From Gary Strand, software engineer at the National Center for Atmospheric Research (NCAR) commenting on Climate Audit:
As a software engineer, I know that climate model software doesn’t meet the best standards available. We’ve made quite a lot of progress, but we’ve still quite a ways to go.
I’ll say. NASA GISS model E written on some of the worst FORTRAN coding ever seen is a challenge to even get running. NASA GISTEMP is even worse. Yet our government has legislation under consideration significantly based on model output that Jim Hansen started. His 1988 speech to Congress was entirely based on model scenarios.
Do we really want congress to make trillion dollar tax decisions today based on “software [that] doesn’t meet the best standards available.”?
There’s more. Steve McIntyre comments:
Re: Gary Strand (#56),
Gary, if this is what you think, then this should have been reported in IPCC AR4 so that politicians could advise themselves accordingly. I do not recall seeing any such comment in AR4 – nor for that matter in any review comments.
Steve McIntyre July 5th, 2009 at 7:49 pmRe: Gary Strand (#56),
If we can convince funding agencies to better-fund software development, and continued training, then we’ll be on our way. It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.
Boo-hoo. Hundreds of millions of dollars, if not billions of dollars is being spent. PErhaps the money should be budgeted differently but IMO there’s an ample amount of overall funding to have adequate software engineers. Maybe there should be some consolidation in the climate model industry, as in the auto industry. If none of the models have adequate software engineering, then how about voluntarily shutting down one of the models and suggest that the resources be redeployed so that the better models are enhanced?
I’m not making this QOTW to pick on Gary Strand, though I’m sure he’ll see it that way. It is a frank and honest admission by him. I’m making it QOTW because Gary highlights a real problem that we see when we look at code coming from NASA GISS.
But don’t take my word for it, download it yourself and have a look. Take it to a software engineer at your own company and ask them what they think.
GISS Model E global climate model source here
GISTEMP (surface temperature analysis) source here
Sure, this is one of many climate modeling software programs out there, but it happens to be the most influential, since GISS and GISTEMP are the most widely cited outputs in the popular media.
U.S. industry seems to do a better job of software development than government programs, because in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.
In academia, the solution is usually to ask for more grant money.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Looks like it may be more accurate to say that UCAR employees (at least some of them, software engineers may be more conscientious) are underworked and overpaid:
“We also evaluated whether UCAR salaries and wages were properly and ac-curately charged to federal awards. The audit found that UCAR employees 1) were not recording all of their worked hours, 2) charged budgeted rather than actual hours, 3) earned and used unrecorded compensatory time (although UCAR does not officially allow compensatory time), and 4) inaccurately re-corded their time as worked when they were on leave. Furthermore, UCAR did not have detailed written justification to support over 80 percent of the sampled labor costs UCAR transferred between awards. Without a reliable basis of support, UCAR’s $58 million of labor costs charged to NSF and other federal agency awards are at risk of not being accurately allocated. UCAR needs to develop a timekeeping system to accommodate all hours worked by salaried employees, provide its employees specific guidance on timecard completion, and provide more oversight of accounting for leave and transferring of labor costs between awards.”
http://www.nsf.gov/pubs/2008/oigmarch2008/oigmarch2008_2.pdf
Gary Strand stepped forward to participate in discussions at Climate Audit, putting himself in the midst of hard questions and skeptical viewpoints. I for one applaud his courage and professionalism for doing so. However, I suspect he regrets wording his statements that way; but I suspect as well that the code is as he stated.
We’ve all seen old code in our work environments that lead to a lot of head shaking and brow furrowing, so I’m hardly surprised by the admission. Lets hope that Mr. Strand isn’t drawn-and-quartered by his climate science colleagues over this.
why is there no 2009 co2 data?
http://scrippsco2.ucsd.edu/data/mlo.html
does anyone know where it is?
Stephen Skinner (07:59:34) :
Reported in New Scientist 4 July
Boeing has encountered some serious structural sicknesses in the new 787.
Stress tests have revealed that the “wingbox” and 18 areas on the fuselage around the wings require strengthening.
“Data from the test did not match our computer model,” says Boeing vice-president Scott Francher. That highlights the difficulty of predicting the behaviour of advanced CFRP materials being used in very large structures for the first time.”
Boeing has it all wrong. Obviously the test data is wrong and must be adjusted to match the computer model!. Seriously, the interesting thing about this is that the limited number of variables for an airframe model are well defined within a well defined engineering discipline. Imagine the increased complexity in a climate model. Then take one small variable that is incorrectly defined and multiply it out over 100 years. Voila – garbage.
My understanding is that GCM modelers will admit that water vapor and clouds are “not well understood”. How can you build a 100 year climate model when one of your key variables is “not well understood”. Imagine Boeing building a model where the tensile strength of the materials used is “not well understood”. Would you fly in it?
My experience of Fortran coding from many years ago was that to make sense of it and for purposes of verification, somewhere between 20% and 50% of the code should be comments. Without having the time to look at the code, can anybody tell me how much of the code is made up of comments?
I don’t understand why these guys even need software modeling code. The whole principle of AGW rests 99% on more C02, more temperature increase. Of course, you can’t have a linear progression, so just put in a few dips and then back up again.
I think they just need an excel spread sheet, because for all of the little nuances they put into their code – good or bad – they’re just saying that as long as we’re pumping more c02 into the atmosphere, the warmer mother earth becomes.
As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
And why is anyone still coding in Fortran?
Part of me wonders why anyone would care what the model says about 2106 – good code, bad code, erroneous code. Given what has transpired geographically and technologically in the last 100 years, the likelihood of any of the output being relevant is pretty slim.
I get the question about integrity of the code processing, but really…
I would hazard there’s a better than even chance you won’t be able to read that data in 100 years time, especially if its digitally archived. I suspect that we’ll be rooting around in the Smithsonian for printed copies to see what the fuss was (it’s a huge leap of faith that archival institutions like the Smithsonian will still exist in 100 years…). I’d be happy if the models could predict next year with certainty. 🙂
“Rod Smith (09:48:05) :
As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
And why is anyone still coding in Fortran?”
Errrm, income protection. I have to work with people who encrypt “scripts” to interface with Microsoft Systems Manamgement Server…but they are not too bright…
In looking at the subject code, I realized its been nearly 3 decades since I wrote FORTRAN code. I switched to Pascal — which I still use for little fun projects (I’m retired).
Since switching to Pascal decades ago, my goal has always been to make my code readable without need for comments. I did&do it with structure, long variable names, statements written and ordered to facilitate reading (versus clever), and so forth. I made my code read like pseudo-code. It worked, others were able to easily understand my Pascal code — even translate it into their favorite language when they were not knowledgeable about Pascal. Result: My mission critical stuff was really easy to debug for me and others. Consequently, it was in service sooner, cost effective, and easy to update/upgrade.
Readability is what I miss in most code I see in blogs (seems like some take pleasure in getting as much work as possible out of every line and love to abbreviate variable names). The subject FORTRAN is not particularly readable — but neither is much other code I see.
Say what? You must be joking!
First of all, in the case of people like Gavin Schmidt, they are NOT software engineers. One look at their code by any “real” software engineer clearly reveals this fact. They methods are so severely flawed, no respectable software engineer would ever put their name on these projects.
Second, “underpaid”, “overworked” ? … Please, don’t even go there. I have been an active software engineer for almost 30 years. Let me tell you a thing or two about underpaid and overworked. I have spent the bulk of my career working 80-90 hour weeks, without vacations, most of the time without benefits or insurance, all to earn enough to survive. I presently make an “ok” salary with marginal benefits, but I have also been able to only take 2 vacation days in over a year and a half. So don’t even begin to tell me about overworked and underpaid. These idiots (so called “modelers”), operate on budgets many many times larger than anything I have been exposed to. Don’t tell me they don’t have money. They are operating on MY money!
Sorry, that quote just pisses me off…
Sorry, additiaonl, I sometimes run into environments who still use IBM’s REXX. Yes, true, it’s still working…and I actually met Mike Coulishaw (We worked in teh same bulding, while he was a visitor).
At least Boeing had the Guts to admit they were wrong…
Lessons From Chaos Theory
Anyone who ponders the value and accuracy of computer models, especially in simulating immensely complex systems such as our earth’s climate, should revisit the story of Edward Lorenz and his Royal McBee LGP-30, the machine that he used in 1960-61 to attempt to model changing weather patterns and show the potential for using the computer in weather forecasting. His discovery that massive differences in forecast could derive from tiny differences in initial data led to him develop the mathematics behind this phenomena and coining the terms strange attractor and butterfly effect, core concepts in chaos theory.
The story is told in the first chapter of the now well known and superbly written book, “Chaos: Making a New Science” by James Gleick – Published in 1987 by Penguin Books.
From Chapter 1, “The Butterfly Effect”, pages 15, 16 and 17
“With his primitive computer, Lorenz had boiled wither down to the barest skeleton.—-”
“One day in the winter of 1961, wanting to examine one sequence at greater length, Lorenz took a shortcut. Instead of starting the whole run over, he started midway through. To give the machine its initial conditions, he typed the numbers straight from the earlier printout. Then he walked down the hall to get away from the noise and drink a cup of coffee. When he returned an hour later, he saw something unexpected, something that planted a seed for a new science. —–”
“From nearly the same starting point, Edward Lorenz saw his computer weather produce patterns that grew farther and farther apart until all resemblance disappeared.—-”
“Suddenly he realized the truth. There had been no malfunction. The problem lay in the numbers he had typed. In the computer’s memory, six decimal places were stored: .506127. On the printout, to save space, just three appeared: .506. Lorenz had entered the shorter, rounded-off numbers, assuming that the difference–one part in a thousand,–was inconsequential. – – – Yet in Lorenz’s particular system of equations, small errors proved catastrophic.—-”
“But for reasons of mathematical intuition that his colleagues would begin to understand only later, Lorenz felt a jolt: something was philosophically out of joint. The practical import could be staggering. Although his equations were gross parodies of the earth’s weather, he had a faith that they captured the essence of the real atmosphere. That first day, he decided that long-range weather forecasting must be doomed.”
For mankind to put their total faith in computer programs that try to simulate a system as chaotic as the earth’s climate and are so sensitive to precise initial conditions is an act of insanity.
Y’all might be interested in discussions of computer type risks at the “Risks Digest” http://catless.ncl.ac.uk/Risks . Search for key words such as “climate models”, “global warming”, etc. and you will find that many of these issues of simulation software have been discussed for decades. The forum is run by Peter Neumann and was begun in 1984.
As a working software engineer, I must reluctantly agree with you.
There is very little science and too much art to writing software. “Software engineering” is unlike “real” engineering in that anybody can call themselves a “software engineer” while you have to graduate from an approved engineering school and then undertake a rigorous licensing process in order to become a “real” engineer.
Still, I refer to myself as a “software engineer” because I realize there should be a process to developing quality software instead of blindly writing and revising code.
BTW: GISTEMP shows that while anybody can write software, most people should not. The smarter somebody is, the harder it is for them to realize they are not qualified to do something.
Many threads back, someone called this code “a crime against humanity”. I haven’t coded in FORTRAN since the 60’s, and it looks bad even by my low standards. Even so, the code isn’t as much of a problem as the model itself. You have hundreds of variables of unknown sensitivity, and very few proven equations. So you guess.
Someone once said “Assumption is the mother of all …”
(or you can determine the outcome in advance, and adjust the inputs)
Back in the mid-80’s when PC’s were still overgrown toys, one of my engineers decided to model a product of mine using P-Spice, an electronic circuit analysis program. He determined that it had peak output voltages in excess of one gigavolt. As all of us were still alive and we had not seen lightning bolts flying across the lab, I suggested that his model needed some improvement. He dropped the project.
As for the financial problems: very little to do with faulty models. A great deal to do with faulty intervention in the market. Loaning money to people who cannot pay it back doesn’t work. Clobbering the economy to reduce a trace gas that has little effect on climate doesn’t work.
We (at least some) learned the money lesson the hard way. The trace gas lesson could be a whole lot more expensive.
It’s no surprise that climate models are poor.
1) The primary data sources they use are unreliable because of data accuracy problems, paucity of global data collection points and the very crude way that data gets used to provide global estimates.
2) Climate is a chaotic non-linear system and developing a model which can provide really accurate future climate trends is an impossibility with the current data inputs and computer power available. Even something as basic as modelling the effects of clouds is an impossibility, despite these being one of the most important climate regulators. When it comes to modelling ocean currents and solar effects, the situation is even worse as we don’t even have good theoretical models of how they behave and what influence they have on our planet.
3) Being a chaotic system, even the slightest error in the model will have an ever growing effect on the accuracy of the forecast as the years pass by.
I think the people funding the productions of the various GCM’s which are around today are intelligent enough to realise the futility of the task of predicting climate ant the 100+yr level. Provided the results they receive give a ‘good’ fit to past climate and provide them with politically correct forecasts, then why would they invest large sums of money in this area.
An old system no doubt with many, many revisions. All systems like this end up looking fairly bad. They need to redo it, but bad looking code does not necessarily mean bad working code.
Until then, it will continue to be job security for the few who understand it.
Alan, assumptions may be the bane of the pure sciences, but they are the bread and butter of engineers. The main classes in sophmore year supposedly teach basic mathematical concepts, but the main lessons is in reasonable and unreasonable assumptions. If you make unreasonable assumptions, you failed.
There is too much code and not enough time for me to decrypt and understand it, but I think that this would be difficult even in best-practice-compliant Basic (the easiest of all languages to read and written in the most self-descriptive manner possible). While this sort of thing needs to be complicated, I cannot help but feel cheated that I cannot even find the array definitions in many cases, much less where they are set.
One thing that irks me is that they are detailed enough to adjust for the changing weight of dry air in higher CO2, when an increase of CO2 to a full 1% (an incredible amount) is only 0.15 g/mol. This is less than half a percent of the standard estimate (28.8 g/mol). If this affects the model’s outcome at all, then the model is laughably oversensitive, because there is no way that you can get two significant figures of accuracy, much less three, on 90% of the valid inputs. It’s comparable to using a surgeon’s scalpel while blindfolded. A baseball bat is just as accurate.
I asked that question about FORTRAN previously and was roundly spanked by people who informed me that FORTRAN is still used in the engineering field and is also commonly used on supercomputers.
I can only assume Fortran is equally prevalent in science as in engineering. I still question why a more modern, object-oriented language such as C++ (or even Java) isn’t used. Is FORTRAN mult-threaded? I’m also told some versions of FORTRAN have object-oriented extensions, but they aren’t commonly used.
If software engineers are in short supply, that’s a sign that NASA needs to migrate to a more contemporary system. Last I heard, there were a lot of layoffs in the IT field. Surely there are some out-of-work computer science majors who would love to work for GISS?
Does anybody know the hardware and OS GISSTEMP runs on?
First, I’m a software developer with 22 years of post-college professional experience. I used to work for the Jet Propulsion Laboratory where I was responsible for maintaining some of the computer graphics software used by PIO for the Voyager and Galileo “flyby” animations, all written in FORTRAN.
With all due respect to everyone hammering on the quality of the code and the software developers who created this code, the FORTRAN here looks very typical of most of the FORTRAN I encountered 20 years ago at JPL in a professional capacity.
The problem with the code is twofold:
(1) The code is old. You can tell from the varying comment styles and coding styles in various source files that it has been worked on by a large number of people over a long period of time. Typically code like this suffers from the “Lava Flow” anti-pattern: the next person to work on a chunk of code doesn’t know fully how things work, so they delicately add their work, making as little change as possible. Lower layers are “frozen” like old lava, with new code poured on top, and allowed to freeze itself.
(2) The code doesn’t use modern coding techniques such as object oriented coding styles. This is typical of code from the 1980’s: OOP programming (and related techniques, such as using design patterns to keep code maintainable) are innovations that didn’t get wide-spread support in the programming community until the 1990’s. This means the code is hard to read and hard to maintain from current coding standards.
(3) One would hope that there would be a separate document which outlines the equations, formulas and rational used in each section of the code. Unfortunately most programmers take the attitude that “code is self-documenting”, when it is no such thing. When I was at JPL I spent a considerable amount of time documenting the code I inherited so I could keep it straight in my mind–and if I were tasked to maintain this code, that would be the first thing I’d do. Ideally I would try to trace back each of the calculation methods back to the scientific paper or textbook which provided the underlying formula, so there is an understanding why I’m carrying out a particular calculation at a particular point.
This sort of sloppy documentation is inexcusable, of course–but it is an extremely common problem in the software community.
(4) It is a common lament amongst software developers to distrust what they don’t understand–especially any newly inherited code. I had a software developer undertake rewriting large chunks of code I wrote myself–telling me to my face that whoever wrote it was an idiot. (He didn’t know it was me. I let it go.) Thus, no-one working on 20-year-old code will ever say it’s good code, regardless of the quality of the code. And they’re all going to itch to rewrite the code–so they can understand it, not because the quality is necessarily bad.
So I wouldn’t trust the comments of the developers who say the code quality is bad. It looks to me to be typical Fortran code from the 1980’s.
I’d say the real problem here is twofold: (1) The code is not properly documented with references to appropriate scientific texts. And as a consequence (2) we are over-reliant on the predictive model of a chunk of code no-one really understands.
But code quality: having worked as long in the software industry as I have, code quality is a constant lament–and really means “I haven’t figured it out.”
I found this interesting recent news item on models vs reality at http://www.dailyfinance.com/2009/06/24/is-boeing-s-787-safe-to-fly/
Think of the billions Boeing has spent on their models, and they still don’t aren’t perfect!
“Specifically, Boeing found that portions of the airframe — those where the top of the wings join the fuselage — experienced greater strain than computer models had predicted. Boeing could take months to fix the 787 design, run more ground tests and adjust computer models to better reflect reality.”
If realclimatescientists ran Boeing, they would defend the “models”, built the plane, crash dozens, kill thousands, bankrupt Boeing and blame the skeptics!
Yet another software engineer here …
“Keep in mind that the errors could be impacting results both ways, although, given the GISS track record, I doubt it would error on the cool side ;)”
That’s a way bias can creep into any software that produces numbers (eg modelling software) where people have pre-determined expectations for the numbers. Bugs that cause the numbers to differ from the expected result are spotted and fixed. Other errors … well not so much.
John F. Hultquist (08:21:57) :
Good post and useful critique, I think.
But, I first did FORTRAN in 1965 (version II-D) and am quite happy to take your word that these models are not great examples of clarity. I’ve seen all the “do loops” and “if – then” statements I ever want to see.
That being said: Know that I appreciate those of you that are looking at these things and I do trust your judgment.
,,,,,,,,I wrote Economics modeling progrrams in the early 70’s on fortran
S Bridges (08:55:35) :
As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers.
software engineers? They are not engineers. Most have no degree.
I really like this thread however since there are quite a few of us that used fortran. Working for the government and using fortran is kinda 30 years behind the times.
Gary, still using key punch cards?