From Gary Strand, software engineer at the National Center for Atmospheric Research (NCAR) commenting on Climate Audit:
As a software engineer, I know that climate model software doesn’t meet the best standards available. We’ve made quite a lot of progress, but we’ve still quite a ways to go.
I’ll say. NASA GISS model E written on some of the worst FORTRAN coding ever seen is a challenge to even get running. NASA GISTEMP is even worse. Yet our government has legislation under consideration significantly based on model output that Jim Hansen started. His 1988 speech to Congress was entirely based on model scenarios.
Do we really want congress to make trillion dollar tax decisions today based on “software [that] doesn’t meet the best standards available.”?
There’s more. Steve McIntyre comments:
Re: Gary Strand (#56),
Gary, if this is what you think, then this should have been reported in IPCC AR4 so that politicians could advise themselves accordingly. I do not recall seeing any such comment in AR4 – nor for that matter in any review comments.
Steve McIntyre July 5th, 2009 at 7:49 pmRe: Gary Strand (#56),
If we can convince funding agencies to better-fund software development, and continued training, then we’ll be on our way. It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.
Boo-hoo. Hundreds of millions of dollars, if not billions of dollars is being spent. PErhaps the money should be budgeted differently but IMO there’s an ample amount of overall funding to have adequate software engineers. Maybe there should be some consolidation in the climate model industry, as in the auto industry. If none of the models have adequate software engineering, then how about voluntarily shutting down one of the models and suggest that the resources be redeployed so that the better models are enhanced?
I’m not making this QOTW to pick on Gary Strand, though I’m sure he’ll see it that way. It is a frank and honest admission by him. I’m making it QOTW because Gary highlights a real problem that we see when we look at code coming from NASA GISS.
But don’t take my word for it, download it yourself and have a look. Take it to a software engineer at your own company and ask them what they think.
GISS Model E global climate model source here
GISTEMP (surface temperature analysis) source here
Sure, this is one of many climate modeling software programs out there, but it happens to be the most influential, since GISS and GISTEMP are the most widely cited outputs in the popular media.
U.S. industry seems to do a better job of software development than government programs, because in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.
In academia, the solution is usually to ask for more grant money.
AGW has never been about the science. It has been about the social power.
The use of climate science is only incidental. Now that science is ‘settled’, the AGW community could give a fig about the terrible code that was used to sell the politics.
Oh, and like John Galt, I agree completely with RS Bridges that most of today’s “Software Engineers” are really just coders, I’m one of them.
True Software Engineers do exist but they’re developing stuff like flight control systems for aircraft etc, and they’re not using Fortran.
Of course, all this Earth climate model talk is premature – you first have create a model of the Sun that works properly 🙂
Have any of the models managed to ‘predict’ from 1850 to now?
DaveE.
Dr. Robert M. Carter characterizes three climate change realities:
Climate change knows three realities: science reality, which is what working scientists deal with every day; virtual reality, which is the wholly imaginary world inside computer climate models; and public reality, which is the socio-political system within which politicians, business people and the general citizenry work.
————————————————————-
Computer models are not reality. Nature is reality. Political reality is a public display of ineptitude in dealing with reality.
Which of these realities is most conducive to your life, liberty and pursuit of happiness?
According to the google cache here http://74.125.47.132/search?q=cache:8AvVyRofd7UJ:www.stopdodo.com/environment-jobs/applied-climate-scientist–modeler.8197.htm+climate%2Bmodeler%2Bsalary&cd=22&hl=en&ct=clnk&gl=us, this position had a “minimum” $100k salary for a climate modeler…with an advanced degree at least 3 yrs of hands-on experience. That sounds like good money.
Jimmy Haigh (07:45:46) :
“…software engineers when they’re underpaid and overworked.”
“Why not look for another job?”
Probably because it’s nonsense. From 2003:
“Boulder labs: CIRES, NIST, NOAA, NTIA, UCAR, NCAR (JILA, LASP)
Employ 2,566 in Boulder; average salary $64,350
26% higher than overall average $51,220”
http://slochamber.org/Library/3A2.pdf
But I will be one that blames the software engineers for compromised products and predictions that affect everyone’s lives, *especially* when one complains that underpayment and overwork is the problem.
Jeff Id says..
” I’ve learned programming in my spare time – it ain’t rocket science.”
—–
This is the same attitude that the climate modellers have and what leads to these low quality software solutions being created in the first place. Learning a high level language is easy, learning methodology and creating flexible and well documented code is much harder to master.
I can build a deck or a small structure with a set of stairs but I am no carpenter and would never try and build a house that would meet building codes.
As a Software Developer and Networking and IT Consultant I can simply ask those among us in this field to validate the following.
1) Cleaning up an IT Infrastructure maintained by a “learned in my spare time” in-house Network and Systems “Guru” is a waste of time. Start from scratch is faster than fixing it.
2) In house written and maintained software is functionally useless to an outside consultant when developed or maintained by a “this is not rocket science” spare time programmer. Again starting from scratch is cheaper than deciphering and fixing it on a cost basis.
I have made a good living and built a company off these types of situations, so I hope Jeff Id and everyone like him keeps on keeping on because he is the reason my phone rings.
Great book. Recommend it highly. After your jogging of my ancient memory I pulled it out to re-read.
@R S Bridges
http://en.wikipedia.org/wiki/Software_engineer
George Tobin: “The guy who has to translate crappy assumptions…into workable code should not have to field that kind of issue or get caught in the middle.”
I submit that the Montana farm boy who enlists in the Army should not have to get caught in the middle of an ideological war and lose his life. The farm boy, fortunately, disagrees and defends the country according to his own lights.
So it would seem that a programmer, who is aware that software which doesn’t meet the best standards available is assisting the mendacious in destroying the world economy, has an obligation to field any issue and get caught in whatever positions are necessary to alert the world. It probably won’t be fatal to him.
Hubristic “scientists” initiated what has become a fight for the survival of viable economies and scientists must enlist in this fight right now.
R.S. Bridges wrote: “As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.”
It’s a heck of a linguistic promotion from “Code Monkey”.
I take issue with this. I am most definitely a software “engineer”. I don’t believe you have a firm grasp of the roles that some of us of this profession fulfill.
This pretty well sums up my job description, and what I have done for almost 30 years. I am not sure what else you would call my profession other than “software engineer”.
Perhaps, for those of you that have no clue what some of us do.
“…in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.
In academia, the solution is usually to ask for more grant money.”
And in government the solution is to increase an agency’s budget and call for more power.
Perhaps the closet disbelievers who are afraid to speak up on issues of poor computer code or badly created/applied statistical techniques or other issues in climate science could utilized Wikileaks?
http://yro.slashdot.org/story/09/07/06/1321227/WikiLeaks-Daniel-Schmitt-Speaks
Well, some of you may have guessed what the “Code” in my name stands for, and forgive me but my specialty is now being stepped on.
No, coding is not “rocket science”, but coding WELL is.
Anyone involved in this has had to deal with poorly written or poorly laid out or poorly designed code. Anyone can code… yes, but very few can code well.
I previously had a contract to “fix” a project that was 5 years in development using Visual Basic. It took me several months to even figure out what the original author was trying to do, and a few weeks to completely rewrite it properly. They have since hired several other contractors to add function to what I wrote, and none of them have had any problems with it. There is a difference between someone writing code using established methodologies and someone who taught themselves and gets mired in an oversized project.
I also downloaded Model E and was amazed at how poorly written it was. My guess is that it evolved directly from the “proof of concept” phase and was never restarted, reorganized, or reanalyzed, all three of which are essential when your program grows past a certain point.
About models in general, there is nothing wrong with the concept of modeling, as long as you understand that a model is ONLY AS GOOD AS ITS ASSUMPTIONS. For example, the Boeing model was perfectly sufficient for a static design, however one or more of their assumptions was incorrect (material strength? structural load during flight?) As these kinds of problems are encountered and fixed, future models become that much more accurate. If there is EVER a time when you think your model is completely accurate, you will have crossed the line into a fantasy world, since models are only representative, never definitive.
The fundamental, critical flaw in the AGW crowd depending on models is that their models do not, and cannot, effectively model a system as complex as they are attempting to model. We have documented here a few important factors that they apparently didn’t think of. The illusion that a complete climate model is built, or even can be built, is actually very entertaining to me.
I’m definitely NOT a software engineer, and only program’d a bit of PASCAL in the day, but I have a descent grasp of what is meant by “poor coding”. However, it would be nice if someone could give examples of poor coding and show how it affects the outcomes to modeling programs. It would be nice to actually see example of where the lack of “skill” (a Mannian term) affects the outcome of these climate models.
An extension:
While there are a lot of people out there that are “coders”, some of us are not. Some of the things I design and develop include application models, data models and cryptographic systems. Additionally, these systems integrate with other systems using a wide variety of mechanisms, transfer agents and methodologies. Further, I develop source control management processes, procedures and methodologies to manage these projects, from design, coding, distribution, maintenance, auditing, logging, reporting, alerting, change-release, peer-review, integration, systems management, infrastructure integration, the whole spectrum.
Perhaps I am taking issue with the slam on “software engineer” because my background did not stem from an MIS or such degree, but from a Bachelor degree in Computer Science, mathematics minor, physics minor (although I don’t use the math’s as much anymore). Mine is a scientific approach from an engineering perspective of software development.
So, be careful when you round us all up and slam us into a box. True, we are not all engineers, but some of us really are…
I have been a professional computer programmer for over 30 years in the insurance industry. Similar to this situation, we have Actuary’s that write their own software solutions to complex insurance problems. And similar to this situation, it can result in software that no one else can understand at a glance. I have at times been called in to repair/support such actuarial software. It is a challenge to do so, but it is what I do. From these experiences I have a respect for the need of the Actuary’s to experiment with their software in order to learn more about their own models. When a programmer is added to the R&D mix, that programmer must become a subject expert. And in my opinion, not a very high percentage of programmers are capable of PhD level work.
The idea of requiring that every piece of software must be written only by programmers would certainly be the utopia solution. However imposing such a requirement on mankind’s computing needs would greatly harm our productivity and ability to learn by experimentation. In my case for instance, my managers have my workload already laid out for me 2 years in advance. Yes, I already know what projects I’m supposed to be working on in 2011. This is because talented programmers are in fact in short supply when you consider the work that is ahead of us.
And I concur that we are not “engineers”. The state of Oregon has made it illegal to misuse the term (but they never enforce it). Doing so belittles the accomplishments of those that have been certified by the State as being actual engineers.
Pete
Rod Smith (09:48:05) :
As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
And why is anyone still coding in Fortran?
Because it’s still a very good language for M&S in science and engineering.
And because it’s incredibly wonderful for concealing slip shod science.
As I noted in the thread at Climate Audit, my FOIA for the top level software quality documents for GISTEMP and GCM Model E found no responsive documents.
NASA/GISS has not even considered where these projects fit within NASA quality standards.
I recall reading on the web an interview between some climate modelers, and I was struck that they were talking openly and consciously about their models’ flaws. I realised that they are smart enough and professional enough to be perfectly aware of the limitations of their models.
So our disagreement with them is less about quality assurance issues (“yes there is room for improvement” they might say), and more about the attitude towards uncertainty.
For the person in the street, the model results are highly inconvenient. They are a problem for society. But for the people creating the models, the results are very convenient. They raise the status of the field. More than once I’ve read climate scientists refer to their work as “very important”. And isn’t this what any scientist wants to do; make important discoveries?
As climate models appear to have little predictive skill, perhaps we need a new academic field of ecology, one that uses different methodologies, ones which are both more powerful, useful, and which render climate modeling obsolete.
I would certainly agree with you on that point. The percentage of top level programmers and/or software engineers (and especially the latter), is relatively small compared to the total volume of personnel involved in software development as a whole.
We operate similarly where I work. We have specific future goals laid out 2 to 3 years in advance. Because of our limited resources, we must engage in a lot of future planning, specific tasking and engineering.
In my role, I would disagree with this statement. My job roles push me far beyond merely “coding”. Specifically, one of my roles is to “engineer” new application models, scientific research, validation, implementation. These tasks move me far beyond simple software development roles. The developers (we can them “developers”), that work for me, are NOT software engineers as they are not intimately involved in the “engineering” aspects of our software development projects. Whereas, one of my roles is specifically to “engineer” these applications, often times employing cutting edge architectural techniques and unique systems integrations. We have “software developers/coders” here, but we also have “software engineers” here as well, I am one of those engineers.
I think the disconnect here is, most people do not fully understand the world of computer processing and that the subject encompasses. There are times when that world is very vast and requires intimate knowledge and discovery on a vast variety of technical disciplines (ie: engineering).
Pete
Poor coding affects the output because it’s impossible to read the code and follow the logic. In other words, you can’t verify the model was coded correctly.
One aspect of software engineering is unit testing (see http://en.wikipedia.org/wiki/Unit_testing).
Unit testing helps determine the model works as designed but does not help determine whether the model is designed correctly.
The only way to verify the output of these models is to compare the results against the observed climate. Unless you’re grading on the curve, I don’t see that the models hold up really well.
And how do you verify the output of climate models that project the climate for the year 2099? You have to wait until the year 2100 to validate the model; much less time is needed to invalidate.
In modelling when there are unknowns, we are often conservative to cover our asses, but we also have to bear in mind we don’t want to over design or we will cost our client extra $, we have to stay within reasonable bounds but allow for the worst that might be expected. I.e. if designing a reservoir, we have to look at the worst storm possible and assume the catchment is nice and saturated, but we dont design to a storm that is improbable as that would waste money. We have to carefully assess the suitable storm and ensure it isnt unrealistic.
I wander if climate modellors put a limit on how conservative their assumptions are?
I mean, clouds causing warming, 1/3 ocean evaporation, 20kph glaciers, co2 warming the sea etc… etc… when these all come together do they represent a highly unlikely outcome or a realistic outcome?
Personally I like the Idso equation as its the only one empirically derived.