Interesting article about Nature editorial endorsing open source software for journal submissions. No mention of climate models, but it certainly seems to play in their insistence for reproducibility . The money quote:
“Reproducibility becomes more difficult when results rely on software. The authors of the editorial argue that, unless research code is open sourced, reproducing results on different software/hardware configurations is impossible. The lack of access to the code also keeps independent researchers from checking minor portions of programs (such as sets of equations) against their own work.”
We certainly saw problems like this when in 2008 GISS released their GISTEMP code.
Written in and old version of FORTRAN, nobody was able to get it to work at first. If I recall correctly, after several weeks of trying, Steven Mosher got most but not all of it to run.
There’s something called the Data Quality Act (DQA) which defines how government agencies must adhere to making data open, accessible and of high quality.
Perhaps we could also do with a Code Quality Act, which would define that any software produced for publicly funded research must be able to be re-run elsewhere for replication of results. Of course for some very large code projects, not everyone has a spare supercomputer lying around to reproduce complex model runs on it. Obviously there are practical limits, but then that also begs the question: who can make sure the coding work done by researchers that have unique supercomputers purchased for specific task is accurate and reproducible?
It’s rather a sticky wicket.
Full story here: