Ron Dean writes in Tips and Notes:
Interesting article about Nature editorial endorsing open source software for journal submissions. No mention of climate models, but it certainly seems to play in their insistence for reproducibility . The money quote:
“Reproducibility becomes more difficult when results rely on software. The authors of the editorial argue that, unless research code is open sourced, reproducing results on different software/hardware configurations is impossible. The lack of access to the code also keeps independent researchers from checking minor portions of programs (such as sets of equations) against their own work.”
We certainly saw problems like this when in 2008 GISS released their GISTEMP code.
Written in and old version of FORTRAN, nobody was able to get it to work at first. If I recall correctly, after several weeks of trying, Steven Mosher got most but not all of it to run.
There’s something called the Data Quality Act (DQA) which defines how government agencies must adhere to making data open, accessible and of high quality.
Perhaps we could also do with a Code Quality Act, which would define that any software produced for publicly funded research must be able to be re-run elsewhere for replication of results. Of course for some very large code projects, not everyone has a spare supercomputer lying around to reproduce complex model runs on it. Obviously there are practical limits, but then that also begs the question: who can make sure the coding work done by researchers that have unique supercomputers purchased for specific task is accurate and reproducible?
It’s rather a sticky wicket.
Full story here:
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Comment by other reader:
“There’s quite a lot of supercomputer simulations which were already proven to be accurate and reliable. For instance today’s airplane motors contain quite a bunch of parts which were calculated on supercomputers. And even weather forecasts, even though they sometimes go completely wrong, can be considered rather reliable. You can’t do either within reasonable timeframe without a supercomputer.”
Rather not true. I’ve been doing complete FEA work on PC’s since 1992 (20 years). The design of turbine blades/aircraft components WERE NEVER DEPENDENT UPON SUPERCOMPUTERS!
Remember the following were accomplished without supercomputers:
1. The P-51 Mustang, 287 days “spec to prototype”, 1940
2. Boeing 707, 737, 747.
3. Apollo Moon Shot
4. Original space shuttle.
As the value of “particle physics” and expensive accelerator is a “societal myth”, so is the value of super computers. 1989 “Finite Element Handbook”, “Supercomputer” FEA of a nuke submarine structure: 3000 elements…
My today’s work, doing some PV (Pressure vessel) support work with 30,000 elements in my model, solving time…1 minute 30 seconds. Toshiba portable.
Forgive me my french but… isn’t this all bullshit?
Small potatoes. I’ve been doing FEA since mid 1980s. Even before that time, FEM-analysis of offshore concrete gravity base structures with several hundred thousand degrees of freedom was not unheard of, using fairly standard VAX-computers.
Today, millions of degrees of freedom on household equipment is “nothing to write home about”.
I think I have yesterday’s supercomputer power in my android phone.. yup sure do!. Heck in 1980, you could by a personal computer with as much power (ram) as the space shuttle it was called the apple2. My point is I seriously doubt that there is any climate modeling that needs a super computer to run today, and even if it requires a lot of today’s power in 2 or 3 years, that will be about normal.
Kasuha said @ur momisugly March 1, 2012 at 6:42 am
Here in SE Australia weather prediction accuracy was 70% out to 7 days several years ago. Accuracy has improved since as has the number of days ahead. Farmers really appreciate this.
Urederra says:
March 1, 2012 at 7:56 am
Bloke down the pub says:
March 1, 2012 at 5:05 am
If it’s not reproduceable, it’s not science.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
“If it’s not reproduceable, it’s not science.” Or engineering or RELIABLE. I spent years doing water hammer analysis starting with simple formulae and a slide rule taking hours, then punch cards on an IBM 360, then on time share terminals; then a PDP 1170, a VAX 730 and on and on, followed by programmable hand held calculators, and finally PC’s. But every project was checked by someone else using similar or different methods … and for large important projects, an outside consultant was used to verify our calculations or vice versa. Reproducibility of results and error checking is pretty basic if you don’t want a pipeline to jump out of the ground when someone actuates a device on the system. The cost of a failure of utility pipelines is miniscule compared to what is being spent on climate research. I wish I had some of that money to build a few water and sewer systems for people who are still chopping holes in the ice or walking miles in Africa to get water. Have been involved in developing systems for both …. such a little amount goes a long way. In climate, it seems the opposite – large amounts of funding just increase the uncertainty. We need to get in touch with reality and get our priorities straight.
Caridad Odens says:
March 1, 2012 at 11:57 am
Forgive me my french but… isn’t this all bullshit?
++++++++++++++++++++++++++++++++++++++++++++++++++++
Merde!!!
Caridad Odens says:
March 1, 2012 at 11:57 am
“Forgive me my french but… isn’t this all bullshit?”
– – – – – – – –
The general line here is “computers, super or not, are very useful in certain areas, and pretty useless in other areas”.
Although there is a shift in the boundary-line, so the “useful area” is widened as the computer capacity increases, the “useless area” for practical purposes still is immensely wide and climate is firmly located in that area and probably will stay there for some long time to come. When you are in the positive mood you might say “the best climate model to date can predict climate for a period of… eh… 16 months” but as climate is defined as a 30 yr (or 60 yr) average this is not even a real climate-model yet at all.
The discussion here is very valid. From it I would conclude: as long as the parameters are well-defined and pretty well known (say a design of a yet-engine) computer simulation is a very helpful tool. When the parameters are not so clear, a simulation or a model at best is a preliminary tool.
This is not difficult actually, or is it?
@AFPhys
1.
Most expensive “(super)computer” models are worthless, for the mere sake that they cost too much. But the principles are not worthless.
Show me a bug free environment, where all code and hardware is free of bugs?
Considering that a million lines of code contains 1-5% of buggy lines of code that you can actually fix as long as you find it, the bugs in the hardware you usually can’t for the mere reason you expect it not to contain any bugs at all. But even if all this would be perfect, enter high frequency noise and degrading hardware… christ just add a “leaking” power cable too close to the serial link cable of a router and imagine if the routers didn’t have error corrections built in, and disregard they still fail, all you need is a tiny little bit of too much of a voltage change that change one little bit, from positive to negative and that’s it.
Why is it that with each supercomputer’s hardware or the software that runs the modeling software or that very software the results gets better and better, and hopefully, more and more accurate, every time with every iteration?
Why would I pay for the first and second version if the third is the rage?
But mostly, why do certain, well paid and groomed, climatologists think their old crappy fortran code, written by non-systems programmers, will function better with faster and more expensive hardware after 30 years of failed results?
2.
Ever heard of GNU and Open Source? It’s like a thing these days, still, after some 30 and 15 years, respectively.
It the climatologist community is comprised of such smart people, why then have they not adopted erlang or at least the SETI mentality, to be able to scale thing aprorietly ro the importance of the case, if it’s so darn important too boot!
Somebody mentioned SETI and friends, there is an existing climate project using BOINC (http://boinc.berkeley.edu/)
http://climateprediction.net/
Garbage in, Gospel Out. It does not matter whether the calculation is done on the back of an envelope with a fountain pen, (in prison) the way Milankovitch did it, or on a super computer. Does the calculated result reproduce and then predict the real physical variables of the system?
The empirical radiative forcing constants used to ‘predict’ surface temperature in the climate models have no basis in physical reality. The calculated ‘equlibrium average surface temperature’ is not a measurable climate variable. There is no flux equilibrium at the Earth’s surface, so the models are invalid before any code is even written down.
There is a mysterious planet called ‘Equlibrium Earth’ that exists only in the realm of computational science fiction that we call climate modeling. It is the home planet of the Computer Climate Trolls. Send money and worship the results. Plant those wind and solar tax farms. You will be saved from computerized global warming hell.
1DandyTroll says:
March 1, 2012 at 4:08 pm
“Show me a bug free environment, where all code and hardware is free of bugs?”
– – – – – –
Maybe that’s it. If you can’t fight them – join them. The programmers have it all wrong in their models. They shouldn’t try to get rid of the bugs, they should allow them, and lots of them. As for climate, the influence of microbes outweighs the human influence by miles. So buggy climate models please, the buggier the better! 😉
Septic Matthew/Matthew R Marler says:
March 1, 2012 at 10:31 am
http://wattsupwiththat.com/2012/03/01/nature-endorses-open-source-software-in-journal-submissions/#comment-909551
“R is open sourced but not (I may be wrong about this) the compilers used.”
If you use R on Linux, it turns out that the whole software stack — operating system, compiler (gcc), libraries — is open-source.
AFPhys says:
March 1, 2012 at 11:32 am
http://wattsupwiththat.com/2012/03/01/nature-endorses-open-source-software-in-journal-submissions/#comment-909603
Modelling vs Reality:
George E.P. Box: “Essentially, all models are wrong, but some are useful.”
John Tukey used to distinguish between “exploratory” data analysis, which looks for “indications” that something interesting might be going on, and “adjudicatory” data analysis, which looks for evidence strong enough to justify spending substantial resources or putting lives at risk. We need to make a similar distinction for modelling. The alarmists are using exploratory-level models as the basis for recommending the dismantling of industrial civilisation.
There is also, in Climate Science, quite a bit of what Richard Feynman called “Cargo Cult Science”:
http://www.lhup.edu/~DSIMANEK/cargocul.htm
Cargo Cult Science tries to mimic the superficial rituals of real science, but without engaging in the relentlessly honest search for, and disclosure of, all possible explanations for the scientist’s results *other* than the explanation the scientist offers.
Open-sourcing the software behind a claim won’t, by itself, satisfy the need for that search and disclosure, but it will be a step in the right direction.
Space Shuttle software development and validation as a pattern to follow:
Even after all that careful software development, including rigorous testing, when they first tried to fly the thing, there was still “The Bug Heard ‘Round The World” that kept it from quite working properly:
http://klabs.org/DEI/Processor/shuttle/garman_bug_81.pdf