Ron Dean writes in Tips and Notes:
Interesting article about Nature editorial endorsing open source software for journal submissions. No mention of climate models, but it certainly seems to play in their insistence for reproducibility . The money quote:
“Reproducibility becomes more difficult when results rely on software. The authors of the editorial argue that, unless research code is open sourced, reproducing results on different software/hardware configurations is impossible. The lack of access to the code also keeps independent researchers from checking minor portions of programs (such as sets of equations) against their own work.”
We certainly saw problems like this when in 2008 GISS released their GISTEMP code.
Written in and old version of FORTRAN, nobody was able to get it to work at first. If I recall correctly, after several weeks of trying, Steven Mosher got most but not all of it to run.
There’s something called the Data Quality Act (DQA) which defines how government agencies must adhere to making data open, accessible and of high quality.
Perhaps we could also do with a Code Quality Act, which would define that any software produced for publicly funded research must be able to be re-run elsewhere for replication of results. Of course for some very large code projects, not everyone has a spare supercomputer lying around to reproduce complex model runs on it. Obviously there are practical limits, but then that also begs the question: who can make sure the coding work done by researchers that have unique supercomputers purchased for specific task is accurate and reproducible?
It’s rather a sticky wicket.
Full story here:
Simple answer: If it “requires” a supercomputer, it’s invalid.
Any result that’s worth changing the world or spending lots of money to implement, is going to appear clearly in the direct results of a carefully designed observation or experiment.
If you need to run a model with thousands of variables, you have ALREADY ADMITTED that you are not objectively analyzing data. You are attempting to generate a desired result.
The CDC6600 was rated at 1 megaflop, 1964
The CRAY-1 was rated at 80 megaflops, 1975
The CRAY-X-MP was rated at 800 Megaflops, 1982
The first multicore CRAY-2 was rated at 1.9 gigaflops, 1985
The garden variety desktop CPU Intel Core I7 920 has been clocked at about 70 gigaflops.
So, essentially, pretty much everyone has access to a super computer. 🙂
SETI and CERN do home number-crunching.
I wouldn’t be surprized if readers of this blog in a similar mechanism could get up to super-computer power.
They are rushing headlong into the 1990s!
Massive computing power is readily available at relatively low prices if you know how to use it properly. Just speak to the good people at Amazon AWS. There is no excuse for not validating software or algorithms that are being used to justify far reaching changes to the world economy.
The Association for Computing Machinery (ACM, the SIGGRAPH people) publishes volumes of ‘Collected Algorithms’ that are validated and available in various languages. Maybe GISS should join and avail themselves of those building blocks.
I would suspect that the other associations, like math or statistical societies, would have their own libraries available, too. Makes no sense for GISS to be reinventing the wheel in addition to the climate.
If it’s not reproduceable, it’s not science.
A few minor corrections, and some more information:
GISTEMP code was released in September 2007, not in 2008. A lot of people made a lot of noise about how convoluted it was, and how impossible it was to run it, but in fact various people, including myself, got the whole thing to run, without much difficulty (a day or two of effort). David Jones and I then rewrote the whole thing in clear, documented Python in the open-source project
http://ccc-gistemp.googlecode.com/. On the way we found a few minor bugs, which had tiny effects on the end result, and which we reported to GISS, and which were swiftly fixed in their original code.
The experience subsequently led to my writing the Science Code Manifesto http://sciencecodemanifesto.org/ last October, which makes basically the same arguments as the Ince/Hatton/JGC paper.
Finally, it’s not an editorial. It’s a Perspective piece: a peer-reviewed article which “may be opinionated but should remain balanced and are intended to stimulate discussion and new experimental approaches.” It doesn’t necessarily reflect Nature’s editorial position (although the NPG has been broadly supportive of moves towards increased code publication).
ToneD says:
March 1, 2012 at 5:03 am
Massive computing power is readily available at relatively low prices if you know how to use it properly. Just speak to the good people at Amazon AWS. There is no excuse for not validating software or algorithms that are being used to justify far reaching changes to the world economy.
————————————————————————————-
Im not giving it to you, you just want to find something wrong with it. 🙂
Sorry for going OT but Svensmark has published a new paper (heavy stuff!) beating CERN and CLOUD to the link between aerosols an CCNs.
http://arxiv.org/pdf/1202.5156.pdf
[Moderator’s Note: When something is OT, the best place for it is in Tips & Notes. Please. -REP]
Unless the complex model actually incorporated real scientific relationships and simply derived equations that simply mimic patterns, I would have little interest in running it. Once the programmers opted for methods that simply reproduced trends, just as any curve can be reproduced by a polynomial, they became effectively worthless as there is no way to realistically enter real world input and values.
Probably half the reason the AGW modelers and founts of wisdom like J Hansen are unwilling to release their software is that, judging from their results, there is little reason to believe that they ever used the software or that it has anything to do with the results. It’s only because the rest of the world thinks they probably needed software to do their work that they have some lying around for window dressing. Of course, the software may not run—it was never meant to or they never attempted to run it, so they did not know it would not run.
[REPLY: Last time we looked, J Hansen had some minor affiliation with GISS and their code was released and replicated. Read the article again and Nick Barnes comment here. -REP]
polistra says:
March 1, 2012 at 4:23 am
Simple answer: If it “requires” a supercomputer, it’s invalid.
______________
There’s quite a lot of supercomputer simulations which were already proven to be accurate and reliable. For instance today’s airplane motors contain quite a bunch of parts which were calculated on supercomputers. And even weather forecasts, even though they sometimes go completely wrong, can be considered rather reliable. You can’t do either within reasonable timeframe without a supercomputer.
The main difference here is, these simulations were already proven to be accurate and reliable.
Science is reproducible … The days of hiding behind a curtain of mystery of computers is gone.
If the data and code are not clear, then there is something devious going on, as we havenow found.
Implying the climate models are simulators is an insult to simulators. It implies the models accurately reflect how the climate works. As you noted, the climate models come nowhere close to this.
Nick Barnes says:
March 1, 2012 at 5:26 am
Nick’s efforts are laudable, and I encourage people to visit his CCC webpage.
However, I couldn’t seem to find any documentation of the GISTEMP algorithm there. The original Hansen paper is confusing and poorly written, and the terse explanation on the GISS webpage is, predictably, inadequate. And this is one of the problems with open source codes – if people are unwilling to adequately document the algorithms, then it takes some effort (maybe a LOT of effort) to ultimately determine what a given piece of code is doing, particularly when you start getting into a climate model which is solving numerous coupled differential equations.
This is just the minimum:
1) The code should be published.
2) The code should be under version control. Ideally, kill two birds with one stone by using something like github, so you can simultaneously publish and version control your code.
3) You should identify the version used for the results. You can “tag” a release in the version control that identifies the exact version used to produce the results in your published paper.
Further than that, LEARN HOW TO CODE. Learn about structure and object-orientation and encapsulation. Unit test your code. If you’re really using a piece of software as a tool in your research, then you should build that code with the same care you’d want in the rest of your equipment.
You beat me to it.
Although in this case I would say “If it is not open source, it is not science”
The fact that Nature has to explain it tells volumes about the state of current science.
We use super-computer simulations for all sorts of things, but I am not aware of anyone promising product on the back of just the simulation results (except climate science.) There is usually an experimental prototype made to make sure the model assumptions were correct before major company-breaking promises are made.
For instance, simulations of candidate drugs are run to make sure the structure doesn’t run afoul of any known biological pathways before it proceeds to the more costly animal testing protocols. It saves a lot of money on testing poor candidates, and prevents some of the more egregious errors of the past. However the company then does the animal and human trial testing before (and as part of) seeking market approval. As more biologic pathways are understood and modeled, they are added to the simulation, but we will always have something else to discover about the way life works so the simulations will never be perfect. The various drug approval agencies will always require the animal and human trials for safety because it is recognized that the simulations will never be perfect.
We have had simulated designs of aircraft engines that looked great in the computer, but had to be reworked in the prototype due to some instabilities the computer didn’t pick up on. This is one of the main reasons that wind tunnels will probably never go out of business. We have had others that worked almost exactly as designed. We still tested the prototype.
Computers have their uses, and they are great tools for finding errors in known processes, but the reliance on computer models in climate theory seems all wrong. The computers should be used to provide outputs based on some assumption, then measure the real world to determine if the assumption is even justified. Climate science seems to go in assuming they have perfect understanding of all variables and then treat the computer output as though it came from the hand of God. The output of the computer is only useful if it tells you what to measure in the real world to confirm or deny your input assumptions!
1DandyTroll says:
March 1, 2012 at 4:38 am
The CDC6600 was rated at 1 megaflop, 1964
The CRAY-1 was rated at 80 megaflops, 1975
The CRAY-X-MP was rated at 800 Megaflops, 1982
The first multicore CRAY-2 was rated at 1.9 gigaflops, 1985
The garden variety desktop CPU Intel Core I7 920 has been clocked at about 70 gigaflops.
So, essentially, pretty much everyone has access to a super computer. 🙂
______________________________
I have a smallish 3.2 TFLOPS machine that anyone could build for less than $1000. Actually, this thing is several years and a far more powerful machine could be built for same.
I am not so arrogant as to assume that I could even get close to writing a program which would mimic anything near the complexities and unfathomable operands of our planet and it’s climate complexities.
However, I’m willing to try. For a mere 1% of federal climate research expenditures (thus freeing up significant monies for support of all the various social justices), I will provide an omnibus overview of the climate threats we face which would answer ( and – ahem- support) all known risks, such as sea-level rise, etc., but would even come up with some new and untried and previously untested scary scenarios and rationalization. I could have the project complete within 365 days.
This is a good step in the right direction, however, it doesn’t go far enough.
1) Release the raw data
2) Release the adjusted data actually used to the model run(s)
3) Document and justify the adjustments
4) Release the source code
5) Release the documentation used to create the source code
6) Release the unit testing code, including the data used to run the unit tests
Everything should be reproducable.
However, this introduces a fundamental problem — Computer model runs are not experiments. If anybody followed these steps above, we would expect everyone to always get the same results. Same input, same code should always produce the same output.*
BTW: Perhaps if others were allowed to review and critic the sloppy code used in these models, the modelers might be inspired or shamed into writing better quality code. Can’t they find some Software Engineering graduate students to write or review the code?
* The discipline of software testing is based on this principle, so please, let’s not get into an OT discussion the extremely rare run-time conditions which might occur so that the results are not always the same.
[Moderator’s Note: When something is OT, the best place for it is in Tips & Notes. Please. -REP]
Thanks for posting that! It ought to be a prominent request somewhere near the top of each article.
Written in and old version of FORTRAN, nobody was able to get it to work at first. If I recall correctly, after several weeks of trying, Steven Mosher got most but not all of it to run.
How much would be required to be “open sourced”? The compiler? The operating system? Some very good statistical software is not “open sourced”, namely SAS and SPSS. R is open sourced but not (I may be wrong about this) the compilers used. I think focusing on the open source is focusing on the wrong problem.
Instead what is required is that the programs be well-commented and as modular as possible (i.e., none of the ancient spaghetti code), that all the variables be declared, and that nifty machine-dependent features be avoided so as to facilitate portability.
More Soylent Green! recommends:
1) Release the raw data
2) Release the adjusted data actually used to the model run(s)
3) Document and justify the adjustments
4) Release the source code
5) Release the documentation used to create the source code
6) Release the unit testing code, including the data used to run the unit tests
If scientists use open source software but do not do Green!’s steps, then the use of open source software accomplishes very little or nothing.
Nick Barnes says:
March 1, 2012 at 5:26 am
Thank you for that.
Luther Wu says:
March 1, 2012 at 8:59 am
“I have a smallish 3.2 TFLOPS machine that anyone could build for less than $1000. Actually, this thing is several years and a far more powerful machine could be built for same.
I am not so arrogant as to assume that I could even get close to writing a program which would mimic anything near the complexities and unfathomable operands of our planet and it’s climate complexities.”
Neither can GISS/Hansen apparently. According to James Hansen the climate models are stripped down versions of weather models–I guess that’s why that don’t want to show ’em in public. :p
Here’s Control Datas 1604 in action, delivered in 1963 to the US Navy.
http://archive.computerhistory.org/resources/text/CDC/CDC.WeatherbyComputer.1963.102641261.pdf
A whooping 48-bit computing capable of in six minutes calculating a 48-hour projection.
So, essentially, since then, apparently, nothing much has happened, other than Climate offices, er, Met offices, around the world wasting tax payer’s hard earned cash on super duper computing power to the same in about the same time, which means the quality of code has degraded to the point of utter parallel chaos, hence chaos computing in a parallel universe. :p
Two comments:
First —
To those who think that (super)computer models are worthless need to educate themselves.
Examples – most Integrated Circuits now are completely computer designed, and in fact could not be designed without the computer modeling of everything from circuit layout to electronic noise. A company would not dare set up a mass production line and distribution based solely on results of their model and simulations, but creation of the designs by hand is not really possible. Likewise, aircraft are now built and “flown” with computer models now, and modified many times, before scale models are are built for wind tunnel tests before full scale prototypes are built before test aircraft is built before commercial production. Modeling is essential, and it can work well. However, it can not be totally relied on, or companies would skip the testing and go directly to mass production.
I have experience with a computer model that worked well compared to the physical item, was changed in a minor manner that “worked” beautifully in the simulation, but when the physical item was modified in that manner (for a mere $10,000,000) the result failed, A large team of people tinkering with the new equipment for three months, and working on the model for over six months, failed to explain the disparity. The machine was returned to its original state, where the model works just fine in explaining many other types of modifications. Obviously, we can’t tinker with the climate or the climate models in the same type of “down to the thousandth of an inch”, “thousandth of a volt”, etc., level. So, the only way to do the modeling is to test it with reality, where it seems to me they clearly lack performance for theoretical reasons and with respect to data, among others.
These are the types of reasons incorporated into why I don’t trust the climate models, and believe it likely they will never be reliable. Nevertheless, to say computer models are worthless in general gives the CAGW believers too easy a target to shoot down, and I strongly discourage that attitude because it is simply wrong.
Second point –
As far as publishing the code so everyone can use it, that is fine. However, I have to caution that there is a danger in everyone simply running the same code. It is important that several separate teams of people, as independently as possible, to do the same type of simulation. Sneaky errors can sneak in and be hidden no matter how many people are looking at the same code (or proofreading the same paragraph, for example). When it comes to code that can be compared to reality at least those errors usually come to light. In something like this climate modeling project where that comparison will be obvious maybe a century after the code is written, that comparison is not useful.
I am reminded of the Space Shuttle computers as a great way to do this – see http://en.wikipedia.org/wiki/Space_Shuttle – do a search on the page for IBM – and pay special attention to how four computers checked each other for hardware failures, and the “Backup Flight System (BFS) was separately developed software running on the fifth computer”.
Given that the majority of the world’s politicians seem to be interested in spending trillions of dollars over the next century, redundant software programs are essential.
Comment by other reader:
“There’s quite a lot of supercomputer simulations which were already proven to be accurate and reliable. For instance today’s airplane motors contain quite a bunch of parts which were calculated on supercomputers. And even weather forecasts, even though they sometimes go completely wrong, can be considered rather reliable. You can’t do either within reasonable timeframe without a supercomputer.”
Rather not true. I’ve been doing complete FEA work on PC’s since 1992 (20 years). The design of turbine blades/aircraft components WERE NEVER DEPENDENT UPON SUPERCOMPUTERS!
Remember the following were accomplished without supercomputers:
1. The P-51 Mustang, 287 days “spec to prototype”, 1940
2. Boeing 707, 737, 747.
3. Apollo Moon Shot
4. Original space shuttle.
As the value of “particle physics” and expensive accelerator is a “societal myth”, so is the value of super computers. 1989 “Finite Element Handbook”, “Supercomputer” FEA of a nuke submarine structure: 3000 elements…
My today’s work, doing some PV (Pressure vessel) support work with 30,000 elements in my model, solving time…1 minute 30 seconds. Toshiba portable.
Forgive me my french but… isn’t this all bullshit?
Small potatoes. I’ve been doing FEA since mid 1980s. Even before that time, FEM-analysis of offshore concrete gravity base structures with several hundred thousand degrees of freedom was not unheard of, using fairly standard VAX-computers.
Today, millions of degrees of freedom on household equipment is “nothing to write home about”.
I think I have yesterday’s supercomputer power in my android phone.. yup sure do!. Heck in 1980, you could by a personal computer with as much power (ram) as the space shuttle it was called the apple2. My point is I seriously doubt that there is any climate modeling that needs a super computer to run today, and even if it requires a lot of today’s power in 2 or 3 years, that will be about normal.
Kasuha said @ March 1, 2012 at 6:42 am
Here in SE Australia weather prediction accuracy was 70% out to 7 days several years ago. Accuracy has improved since as has the number of days ahead. Farmers really appreciate this.
Urederra says:
March 1, 2012 at 7:56 am
Bloke down the pub says:
March 1, 2012 at 5:05 am
If it’s not reproduceable, it’s not science.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
“If it’s not reproduceable, it’s not science.” Or engineering or RELIABLE. I spent years doing water hammer analysis starting with simple formulae and a slide rule taking hours, then punch cards on an IBM 360, then on time share terminals; then a PDP 1170, a VAX 730 and on and on, followed by programmable hand held calculators, and finally PC’s. But every project was checked by someone else using similar or different methods … and for large important projects, an outside consultant was used to verify our calculations or vice versa. Reproducibility of results and error checking is pretty basic if you don’t want a pipeline to jump out of the ground when someone actuates a device on the system. The cost of a failure of utility pipelines is miniscule compared to what is being spent on climate research. I wish I had some of that money to build a few water and sewer systems for people who are still chopping holes in the ice or walking miles in Africa to get water. Have been involved in developing systems for both …. such a little amount goes a long way. In climate, it seems the opposite – large amounts of funding just increase the uncertainty. We need to get in touch with reality and get our priorities straight.
Caridad Odens says:
March 1, 2012 at 11:57 am
Forgive me my french but… isn’t this all bullshit?
++++++++++++++++++++++++++++++++++++++++++++++++++++
Merde!!!
Caridad Odens says:
March 1, 2012 at 11:57 am
“Forgive me my french but… isn’t this all bullshit?”
– – – – – – – –
The general line here is “computers, super or not, are very useful in certain areas, and pretty useless in other areas”.
Although there is a shift in the boundary-line, so the “useful area” is widened as the computer capacity increases, the “useless area” for practical purposes still is immensely wide and climate is firmly located in that area and probably will stay there for some long time to come. When you are in the positive mood you might say “the best climate model to date can predict climate for a period of… eh… 16 months” but as climate is defined as a 30 yr (or 60 yr) average this is not even a real climate-model yet at all.
The discussion here is very valid. From it I would conclude: as long as the parameters are well-defined and pretty well known (say a design of a yet-engine) computer simulation is a very helpful tool. When the parameters are not so clear, a simulation or a model at best is a preliminary tool.
This is not difficult actually, or is it?
@AFPhys
1.
Most expensive “(super)computer” models are worthless, for the mere sake that they cost too much. But the principles are not worthless.
Show me a bug free environment, where all code and hardware is free of bugs?
Considering that a million lines of code contains 1-5% of buggy lines of code that you can actually fix as long as you find it, the bugs in the hardware you usually can’t for the mere reason you expect it not to contain any bugs at all. But even if all this would be perfect, enter high frequency noise and degrading hardware… christ just add a “leaking” power cable too close to the serial link cable of a router and imagine if the routers didn’t have error corrections built in, and disregard they still fail, all you need is a tiny little bit of too much of a voltage change that change one little bit, from positive to negative and that’s it.
Why is it that with each supercomputer’s hardware or the software that runs the modeling software or that very software the results gets better and better, and hopefully, more and more accurate, every time with every iteration?
Why would I pay for the first and second version if the third is the rage?
But mostly, why do certain, well paid and groomed, climatologists think their old crappy fortran code, written by non-systems programmers, will function better with faster and more expensive hardware after 30 years of failed results?
2.
Ever heard of GNU and Open Source? It’s like a thing these days, still, after some 30 and 15 years, respectively.
It the climatologist community is comprised of such smart people, why then have they not adopted erlang or at least the SETI mentality, to be able to scale thing aprorietly ro the importance of the case, if it’s so darn important too boot!
Somebody mentioned SETI and friends, there is an existing climate project using BOINC (http://boinc.berkeley.edu/)
http://climateprediction.net/
Garbage in, Gospel Out. It does not matter whether the calculation is done on the back of an envelope with a fountain pen, (in prison) the way Milankovitch did it, or on a super computer. Does the calculated result reproduce and then predict the real physical variables of the system?
The empirical radiative forcing constants used to ‘predict’ surface temperature in the climate models have no basis in physical reality. The calculated ‘equlibrium average surface temperature’ is not a measurable climate variable. There is no flux equilibrium at the Earth’s surface, so the models are invalid before any code is even written down.
There is a mysterious planet called ‘Equlibrium Earth’ that exists only in the realm of computational science fiction that we call climate modeling. It is the home planet of the Computer Climate Trolls. Send money and worship the results. Plant those wind and solar tax farms. You will be saved from computerized global warming hell.
1DandyTroll says:
March 1, 2012 at 4:08 pm
“Show me a bug free environment, where all code and hardware is free of bugs?”
– – – – – –
Maybe that’s it. If you can’t fight them – join them. The programmers have it all wrong in their models. They shouldn’t try to get rid of the bugs, they should allow them, and lots of them. As for climate, the influence of microbes outweighs the human influence by miles. So buggy climate models please, the buggier the better! 😉
Septic Matthew/Matthew R Marler says:
March 1, 2012 at 10:31 am
http://wattsupwiththat.com/2012/03/01/nature-endorses-open-source-software-in-journal-submissions/#comment-909551
“R is open sourced but not (I may be wrong about this) the compilers used.”
If you use R on Linux, it turns out that the whole software stack — operating system, compiler (gcc), libraries — is open-source.
AFPhys says:
March 1, 2012 at 11:32 am
http://wattsupwiththat.com/2012/03/01/nature-endorses-open-source-software-in-journal-submissions/#comment-909603
Modelling vs Reality:
George E.P. Box: “Essentially, all models are wrong, but some are useful.”
John Tukey used to distinguish between “exploratory” data analysis, which looks for “indications” that something interesting might be going on, and “adjudicatory” data analysis, which looks for evidence strong enough to justify spending substantial resources or putting lives at risk. We need to make a similar distinction for modelling. The alarmists are using exploratory-level models as the basis for recommending the dismantling of industrial civilisation.
There is also, in Climate Science, quite a bit of what Richard Feynman called “Cargo Cult Science”:
http://www.lhup.edu/~DSIMANEK/cargocul.htm
Cargo Cult Science tries to mimic the superficial rituals of real science, but without engaging in the relentlessly honest search for, and disclosure of, all possible explanations for the scientist’s results *other* than the explanation the scientist offers.
Open-sourcing the software behind a claim won’t, by itself, satisfy the need for that search and disclosure, but it will be a step in the right direction.
Space Shuttle software development and validation as a pattern to follow:
Even after all that careful software development, including rigorous testing, when they first tried to fly the thing, there was still “The Bug Heard ‘Round The World” that kept it from quite working properly:
http://klabs.org/DEI/Processor/shuttle/garman_bug_81.pdf