From Gary Strand, software engineer at the National Center for Atmospheric Research (NCAR) commenting on Climate Audit:
As a software engineer, I know that climate model software doesn’t meet the best standards available. We’ve made quite a lot of progress, but we’ve still quite a ways to go.
I’ll say. NASA GISS model E written on some of the worst FORTRAN coding ever seen is a challenge to even get running. NASA GISTEMP is even worse. Yet our government has legislation under consideration significantly based on model output that Jim Hansen started. His 1988 speech to Congress was entirely based on model scenarios.
Do we really want congress to make trillion dollar tax decisions today based on “software [that] doesn’t meet the best standards available.”?
There’s more. Steve McIntyre comments:
Re: Gary Strand (#56),
Gary, if this is what you think, then this should have been reported in IPCC AR4 so that politicians could advise themselves accordingly. I do not recall seeing any such comment in AR4 – nor for that matter in any review comments.
Steve McIntyre July 5th, 2009 at 7:49 pmRe: Gary Strand (#56),
If we can convince funding agencies to better-fund software development, and continued training, then we’ll be on our way. It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.
Boo-hoo. Hundreds of millions of dollars, if not billions of dollars is being spent. PErhaps the money should be budgeted differently but IMO there’s an ample amount of overall funding to have adequate software engineers. Maybe there should be some consolidation in the climate model industry, as in the auto industry. If none of the models have adequate software engineering, then how about voluntarily shutting down one of the models and suggest that the resources be redeployed so that the better models are enhanced?
I’m not making this QOTW to pick on Gary Strand, though I’m sure he’ll see it that way. It is a frank and honest admission by him. I’m making it QOTW because Gary highlights a real problem that we see when we look at code coming from NASA GISS.
But don’t take my word for it, download it yourself and have a look. Take it to a software engineer at your own company and ask them what they think.
GISS Model E global climate model source here
GISTEMP (surface temperature analysis) source here
Sure, this is one of many climate modeling software programs out there, but it happens to be the most influential, since GISS and GISTEMP are the most widely cited outputs in the popular media.
U.S. industry seems to do a better job of software development than government programs, because in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.
In academia, the solution is usually to ask for more grant money.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
When it comes to GCM’s (general circulation models), I really believe that “software engineers”, along with software developers, is precisely what is necessary if one wants to truly attempt to computer model something as complex as our climate. The task of developing software so complex as this, ultimately demands not just software developers, or coders, but in fact real “software engineers”. Software engineers to determine what technologies employ, determine how those technologies will be employed, determine every aspect of the project design, determine who will play what role in and at what stages, how development processes and methodologies will be utilized when and by whom, all of these things that are required to successfully develop a software product and see it through its SDLC (software development life cycle).
It seems to me that software development project such as this would likely demand new architectural application approaches. Develop and utilize new design patterns, new application modeling techniques, the things that “software engineers” do. To me, a climate model just screams of cutting edge technology approaches such as this. Old-school FORTRAN just doesn’t seem appropriate in this case. Its like trying to build a rocket to Mars using pixy stix. You might be able to do it, but the likelihood of success or desired results, would be rather low.
As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.
I came up through the ranks in the 70s when quite a few coders were also hardware engineers. I was one of them.
I do take exception to the statement that “software engineering” is not real engineering. Ask the aerospace guys (me, me) or the medical eqpt. guys if that is true. Degrees and professional exams are not the measure of all things engineering. Otherwise it would have been a University Professor and not James Watt who brought the steam engine into the modern age.
BTW this hardware/software engineer (non-degreed) had a government inspector look over his code and the inspector said it was the best written code he had seen in years. My secret? The naming of parts. If you get the names of things right the code reads almost like English and even a non-coder should be able to get the gist of it. I did do my work in FORTH which is a great help in making the code readable. But if you are not a disciplined software engineer FORTH makes it much easier to do unreadable code. So it cuts both ways depending on the skill and discipline of the engineer.
We have the example of hardware engineers. You give them a pile of steel and other metals, etc. and they will build you an engine. However, there are no restrictions on how the materials can be used. It is up to the engineer to make the right choices. In “modern” coding languages the effort is to make the compilers and tools cover for (prevent) errors. Such restrictions also prevent you from doing a lot of useful things easily or at all.
All very nice if you are going to employ a bunch of mediocre coders/engineers. But you then wind up with a bunch of mediocre code. In everything. Just think of how long it takes Windoze to load.
So my point – if you don’t trust your software engineers the answer is not training wheels. It is better engineers. There is no substitute for talent and discipline.
BTW PHDs can code well if taught the discipline. If they are apt students it only takes a couple of weeks to train them. If they are not apt they shouldn’t be coding.
@R S Bridges (08:55:35) :
“As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
A more appropriate description for our friends who write programs may be software code developer.”
Hear, hear! Another definition of Engineer, from California’s Business and Professions Code 6701: “Professional engineer,” within the meaning and intent of this act, refers to a person engaged in the professional practice of
rendering service or creative work requiring education, training and
experience in engineering sciences and the application of special
knowledge of the mathematical, physical and engineering sciences in
such professional or creative work as consultation, investigation,
evaluation, planning or design of public or private utilities,
structures, machines, processes, circuits, buildings, equipment or
projects, and supervision of construction for the purpose of securing
compliance with specifications and design for any such work.”
All the FORTRAN code makes the output appear to be ‘scientific.’
Unfortunately most programmers take the attitude that “code is self-documenting”, when it is no such thing.
Well written code IS mostly self documenting. It all depends on choosing your words very carefully.
The government inspector who looked at my code and liked it had no complaints about the lack of commenting (there wasn’t a lot of it) because the code WAS self documenting. That was back in ’79 and I WAS using techniques that did not become widely used in programming for another decade and a half.
All this from a non-degreed (electrical/electronic) aerospace hardware engineer who went into coding out of necessity. So why was my stuff well above average for the time period involved? I cared. I was not just interested in “it works”. I was also interested in elegance and beauty.
BTW I don’t think you can be a good coder/systems engineer without a deep understanding of the underlying hardware. Also a deep understanding of the domain is essential (if only for picking good names for things). Now a days most coders are fairly ignorant of the hardware they are working on and even more so of the domains they work in.
Roger Sowell (13:15:05) :,
You are talking about the LEGAL definition of engineer not the practical definition. A confusion of domains.
Again, just describes my professional role as a “software engineer”.
>> Mac (08:26:46) :
I’ve written my own NASA-like climate model software
10 Rem Global Climate Model
20 Print “The End Is Nigh”
30 Goto 20
40 End (of planet) <<
This is the only example of code in the entire thread. The “End” statement is usually a compiler directive and not an executable statement. However, most compilers will execute a return at this point (and not terminate or stop the machine). A main program will return to the operating system. Statement “30” is an example of an infinite loop–you can never reach statement “40” or exit the loop. It seems appropriate. No matter how much hype we hear about global warming, we will never leave that infinite loop about the coming doom and never reach the “End.”
Jim
Too much value is assigned here to “coding quality”. That may be an issue concerning costs of maintenance and further development of the application. But it is not related to the OUTCOME or calculated result of the system, or quality of the results.
What interests us is – if the calculations are correct. Code quality is irrelevant. You must always check results against some independent benchmark – like say – measured observations. Whether the code styling is nice or not, does not matter. If you cannot test the result in an independent manner, you can never debug the code and make it work correctly.
No amount of “good” software engineering can guarantee that the results are correct. It is impossible to develop and debug code if you have no independent yardstick to evaluate the results by.
A couple of weeks huh? Perhaps back in the 70’s.. it seems you may not have much experience with modern day development processes. It would take a couple of weeks just for them to learn the source control management aspects alone. I would be willing to bet that not one of those individuals would be capable of coding on my current project within a year, perhaps even two.
As a working software engineer/consultant, I can tell you that poor code is the norm. If more programmers knew what they were doing, I would have to find other work.
Software engineers do not have a professional licensing requirement. That means the quality of their work is all over the board. Anybody can claim the title of ‘software engineer.’ In many companies, it’s the job title that replaces ‘programmer/analyst.’
Having a degree in Software Engineering or Computer Science is not a guarantee of the quality of a person’s code. I have met many self-taught programmers who write excellent code and many degreed professionals who just copy and paste and patch their code after it’s put into production.
The difference is in the attitude and aptitude. A person who takes pride in their work strives to write good code. They want to improve and when they don’t know how to do something, they learn it instead of throwing out one bad hack after another.
Take the example of Boeing. They had a nice software model, that certainly helped a whole lot in the design process. Still the result had to be tested in a prototype, where actual stress measurements were made. An error was found that could not have been discovered by “good” software engineering, only by actual measurements. The software will be corrected, and be more useful in the future. Software that cannot be tested cannot be developed to a high degree of reliability. Good coding practices alone are never sufficient.
I also qualify as an engineer (chemical) who was required to learn and then write software in FORTRAN. My field was chemical plants and petroleum refineries, in which we wrote some very complex code. This code was done well, and was robust, and was subject to annual updates / revisions, and was sold as a commercial product throughout the world. Nothing magic in this, we simply understood our field, and FORTRAN. The code had to execute on machines in Japan, Europe, Canada, the US, South Africa, and the Middle East.
After years of writing / debugging / enhancing the code ourselves, we hired a computer science major and had him do the programming. We spent more time explaining things to him than if we had simply done the job ourselves. Management thought we were so progressive, though, in having a professional code-writer. It also enhanced our marketing when we could say the code was professionally written.
Someone earlier asked why do people still code in FORTRAN, and this reminds me of a similar thread earlier on WUWT. Because it is an excellent language for engineering. There are literally millions of lines of excellent FORTRAN operating every day in engineering applications. No one in their right mind would consider re-writing all that perfectly good code.
Someone said: “The model results reflect the assumptions of the authors of the model”.
That would be true if there were no errors (bugs) in the code.
As it is – the models reflect nothing. Their result cannot be treated as anything but random, since we have no independent way of authenticating them, therefore we have no way of debugging the code.
Stefan,
“I recall reading on the web an interview between some climate modelers, and I was struck that they were talking openly and consciously about their models’ flaws.”
The discussion you recall may be this one
http://www.thebulletin.org/web-edition/roundtables/the-uncertainty-climate-modeling#
M. Simon (13:04:11) :
I’m curious Do you think your definition of “software engineer” fits with what the needs are for building a climate model? I think it may, since it requires much more than a coder, which is why the scientists probably decided to do it themselves in the first place anyway.
By your definition, I believe I would be called an engineer. But we will have to disagree on the term software engineer. This company has folks that do everything you mention, and none of them are called software engineer.
Pete
With all due respect to scientists and engineers – leave the coding to the professionals. We set up “sandboxes” where we let the PhD’s play where they wouldn’t hurt anyone.
I was a professional (got paid to do it) programmer during the mid 1980s when a lot of this code was written. I would say ModelE is not up to industry standards of even the 1980s. Our code was much better documented.
This problem reminds me of the Therac-25 software incident:
http://en.wikipedia.org/wiki/Therac-25 that got a lot of play in the IT trade press of the day. A radiation therapy machine somehow dispatched lethal doses of radiation because of a software glitch caused by a certain set of operator instructions.
10 Rem Global Climate Model
20 Print “The End Is Nigh”
30 Goto 20
40 End (of planet) <<
Better program
10 REM Global Climate Model
20 ON ERROR GO TO 40
30 PRINT AT INT (RND * 21),INT (RND*32);”The End is Nigh”
40 GO TO 30
But the RND function is no good since it repeats itself in about 15 times, better take the value of the random SEED generator directly with the aid of a PEEK command.
10 REM Global Climate Model
20 ON ERROR GO TO 40
30 PRINT AT INT (PEEK (16434)*21),INT (PEEK (16434)*32);”The End is Nigh”
40 GO TO 30
Oh my god how long has it been, to long i guess and one thing comes up, that even small and relatively simple program’s do require some debugging, good software takes time, but even then if you feed it with garbage data, it will result in output that is still garbage.
Real engineers
There seems to be a misunderstanding regarding engineers. I use headhunters to hire people and they aren’t confused. Engineering degress are granted by schools of engineering. None of this M.S. in Information systems from church schools. You have an engineering degree from an engineering school or you do not. Microsoft and some other vendors claim to produce them. Nope. Not the real thing. (I cut slack for railroad engineers, they have had that handle and don;’t hold themselves out to have a real engineering degree.) Head hunters also tell me if it is a compromised degree such as B.S. In Engineering Technology. Not as much as a real engineering degree. This excludes the DeVry Universities and some BS computor science degrees. There are real computor engineering Degrees. There are several hundred Engineering School campuses.
Sounds like the next project after SurfaceStations wraps up.
Never! Today USHCN, tomorrow GHCN!
“Just perusing that programming lauguage-it has been more than 30 years since I’ve seen FORTRAN is it me-do I see some sort of deliberate ‘loop’ if you will in the way it it written? I’m no programmer but have downloaded a couple of different wildlife studies
in FORTRAN and got caught in a “garbage in garbage out” -GIGO scenario that dang near got the whole study shut down.-after 3 years of hard field work…
This GISS thing makes me wonder…”
And it feels
Like I been here before
Feels . . .
Like I been here before
And you know
IT [sic] makes me wonder
What’s goin’ on
Under
The ground
The following, from The Future of Everything [originally entitled Apollo’s Arrow: The Science of Prediction and the Future of Everything] by David Orrell, pg. 324), WRT climate models in particular, is apropos relative to global climate models (CGMs) and Gary Strand’s contention at http://www.climateaudit.org/?p=6495 that it’s the science not the software development:
“Einstein’s theory of relativity was accepted not because a committee agreed that it was a very sensible model, but because its predictions, most of which were highly counterintuitive, could be experimentally verified. Modern GCMs have no such objective claim to validity, because they cannot predict the weather over any relevant time scale. Many of their parameters are invented and adjusted to approximate past climate patterns. Even if this is done using mathematical procedures, the process is no less subjective because the goals and assumptions are those of the model builders. Their projections into the future—especially when combined with the output of economic models—are therefore a kind of fiction. The problem with the models is not that they are subjective or objective—there is nothing wrong with a good story, or an informed and honestly argued opinion. It is that they (GCMs) are couched in the language of mathematics and probabilities; subjectivity masquerading as objectivity. Like the Wizard of Oz, they are a bit of a sham.”
The hardest part of understanding the models will be the physics in them rather than whatever programming language they are written in. I suspect therefore primary focus is on that and the programming is secondary.
I miss Gary Strand… He should be commenting here…
Bobn (16:36:30) : I agree the physics is more important – the adjustable params might be more important in a negative way 🙂
But a well written program is easier to understand, easier to test, and easier to modify without causing unintended consequences. The bigger the program, the easier it is to screw it up … well … assuming is was ever in an un-screwed-up state. Apparently the climate models never were.