Quote Of The Week #13

Jimmy Haigh (07:45:46) :
“…software engineers when they’re underpaid and overworked.”
Why not look for another job?
I believe the best ones went and worked in the financial modelling sector…..
cheers David

Could anyone hazzard a guess about the Met Offices input/output data in light of the above? No such admissions forthcoming from that quarter or they’ll be out on their ears! It does worry me when the apparent “uncertainties” are explained away as “better understood”, which means nothing & is not the same as saying “we know & undertand how they affect our data output”, particularly when predicting the future!
I still think we are at the limits of computer power & understanding about climate. If you are making assumptions, whether reasonable or not, they are still assumptions, & if wrong, no amount of computing power is going to rectify it & or give you the answer, which seems to be what the MO are implying by getting this new Deep Thought computer, as they seem to want another one 1000’s of times more powerful than the one they have to get the right results! As Prof John Brignell @ Number Watch says, a comuputer model can be right or wrong, but still irrelevant!

No one involved in this mess WANTS functioning, competent software engineers, because they might accidentally do their jobs and reveal the fraud to the world. We Can’t Have That.
When you’re conduction a fraud of this scale, you need to make sure the fraud is perpetuated from top to bottom. Crappy, inscrutable softward code that no one understands but that always gives the “Right” answers is something to be treasured, guarded, and NEVER EVER replaced.

The last piece of code that worked without any unexpected errors was;
10 Print “Hello”;
20 Goto 10
and it still required half an hour of debugging.
The faith that is put in models is terrifying and people always ignore the one underlying problem;
For a model to be correct it has to be close to 100% accurate against the real world situation, if it isn’t it is at worst a waste of time, at best a vague indicator as to whether your original theory was credible but should never be treated as proof of anything.

I’ve written my own NASA-like climate model software
10 Rem Global Climate Model
20 Print “The End Is Nigh”
30 Goto 20
40 End (of planet)

As a longtime professional software developer, I’ve seen millions of lines of code, of varying qualities. But I can make some generalizations. The people who code something up as an assistance to their job, not as a primary focus, in general write lousy code.
So I’m not surprised to see the quality of the code. It pretty much looks like what I’d get if I’m interfacing, say, with a hardware guy who writes a tool to use on the hardware he’s designing. Unreadable, undocumented, unmaintainable code that basically works. The mathematician/statisticians that I’ve worked with are the same way, so I’d guess I’d expect the same thing of climate scientists.
If I had to guess, I’d say that the number of actual professional software engineers they have working on the project is zero, that 100% of this has been coded by the scientists themselves.

“It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.”
UCAR/NCAR stopped public access to wage data in 2008. Apparently how federal funding for this institution is broken down has been hard to find as early as 2006:
http://www.climatesciencewatch.org/index.php/csw/details/greenwire-inhofe/
But if the General Schedule for federal employees is any indication of compensation at NCAR then $80,000+ would be a good starting estimate of “underpaid”.
http://www.bls.gov/oco/cg/cgs041.htm#earnings
Gary, perhaps you should back up your unfounded claims like being “underpaid” and “overworked”, eh?
Or do you just not have the time to post that, being overworked and all, at your office on weekends on your “own time” for instance?

All the software engineers I’ve met were most definitely underpaid. Riiiight….

As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
A more appropriate description for our friends who write programs may be software code developer.

There seems to be a correlation between government involvement and the lack of quality of developed software. In the UK, the Government underwrote the financing of an enormously complex health care system, supposedly to link all the medical records of every family doctor, every hospital into one central database. Although the development was carried out by the private sector, it has been to date nothing less than a complete disaster, and a waste of something like six billion pounds. Now go to the extreme and have the government not only finance the thing but develop it itself and no rational person can expect anything at all out the other end that in any way resembles its stated purpose.

It is perplexing how models are called “experiments” but/and also used for “prediction”.
If the model is understood, it is not experimental.
If the model is experimental, it should not be relied on for prediction.
I don’t really care how much money and time and effort and smarts went into developing the models. There comes a point where we have to decide whether to trust the results. Many projects take up a lot of time and effort but nonetheless, in the end they simply fail.
We need to get honest about whether climate models are successes or failures.

Gary Strand stepped forward to participate in discussions at Climate Audit, putting himself in the midst of hard questions and skeptical viewpoints. I for one applaud his courage and professionalism for doing so. However, I suspect he regrets wording his statements that way; but I suspect as well that the code is as he stated.
We’ve all seen old code in our work environments that lead to a lot of head shaking and brow furrowing, so I’m hardly surprised by the admission. Lets hope that Mr. Strand isn’t drawn-and-quartered by his climate science colleagues over this.

Stephen Skinner (07:59:34) :
Reported in New Scientist 4 July
Boeing has encountered some serious structural sicknesses in the new 787.
Stress tests have revealed that the “wingbox” and 18 areas on the fuselage around the wings require strengthening.
“Data from the test did not match our computer model,” says Boeing vice-president Scott Francher. That highlights the difficulty of predicting the behaviour of advanced CFRP materials being used in very large structures for the first time.”
Boeing has it all wrong. Obviously the test data is wrong and must be adjusted to match the computer model!. Seriously, the interesting thing about this is that the limited number of variables for an airframe model are well defined within a well defined engineering discipline. Imagine the increased complexity in a climate model. Then take one small variable that is incorrectly defined and multiply it out over 100 years. Voila – garbage.
My understanding is that GCM modelers will admit that water vapor and clouds are “not well understood”. How can you build a 100 year climate model when one of your key variables is “not well understood”. Imagine Boeing building a model where the tensile strength of the materials used is “not well understood”. Would you fly in it?

I don’t understand why these guys even need software modeling code. The whole principle of AGW rests 99% on more C02, more temperature increase. Of course, you can’t have a linear progression, so just put in a few dips and then back up again.
I think they just need an excel spread sheet, because for all of the little nuances they put into their code – good or bad – they’re just saying that as long as we’re pumping more c02 into the atmosphere, the warmer mother earth becomes.

Part of me wonders why anyone would care what the model says about 2106 – good code, bad code, erroneous code. Given what has transpired geographically and technologically in the last 100 years, the likelihood of any of the output being relevant is pretty slim.
I get the question about integrity of the code processing, but really…
I would hazard there’s a better than even chance you won’t be able to read that data in 100 years time, especially if its digitally archived. I suspect that we’ll be rooting around in the Smithsonian for printed copies to see what the fuss was (it’s a huge leap of faith that archival institutions like the Smithsonian will still exist in 100 years…). I’d be happy if the models could predict next year with certainty. 🙂

In looking at the subject code, I realized its been nearly 3 decades since I wrote FORTRAN code. I switched to Pascal — which I still use for little fun projects (I’m retired).
Since switching to Pascal decades ago, my goal has always been to make my code readable without need for comments. I did&do it with structure, long variable names, statements written and ordered to facilitate reading (versus clever), and so forth. I made my code read like pseudo-code. It worked, others were able to easily understand my Pascal code — even translate it into their favorite language when they were not knowledgeable about Pascal. Result: My mission critical stuff was really easy to debug for me and others. Consequently, it was in service sooner, cost effective, and easy to update/upgrade.
Readability is what I miss in most code I see in blogs (seems like some take pleasure in getting as much work as possible out of every line and love to abbreviate variable names). The subject FORTRAN is not particularly readable — but neither is much other code I see.

As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
A more appropriate description for our friends who write programs may be software code developer.

As a working software engineer, I must reluctantly agree with you.
There is very little science and too much art to writing software. “Software engineering” is unlike “real” engineering in that anybody can call themselves a “software engineer” while you have to graduate from an approved engineering school and then undertake a rigorous licensing process in order to become a “real” engineer.
Still, I refer to myself as a “software engineer” because I realize there should be a process to developing quality software instead of blindly writing and revising code.
BTW: GISTEMP shows that while anybody can write software, most people should not. The smarter somebody is, the harder it is for them to realize they are not qualified to do something.

It’s no surprise that climate models are poor.
1) The primary data sources they use are unreliable because of data accuracy problems, paucity of global data collection points and the very crude way that data gets used to provide global estimates.
2) Climate is a chaotic non-linear system and developing a model which can provide really accurate future climate trends is an impossibility with the current data inputs and computer power available. Even something as basic as modelling the effects of clouds is an impossibility, despite these being one of the most important climate regulators. When it comes to modelling ocean currents and solar effects, the situation is even worse as we don’t even have good theoretical models of how they behave and what influence they have on our planet.
3) Being a chaotic system, even the slightest error in the model will have an ever growing effect on the accuracy of the forecast as the years pass by.
I think the people funding the productions of the various GCM’s which are around today are intelligent enough to realise the futility of the task of predicting climate ant the 100+yr level. Provided the results they receive give a ‘good’ fit to past climate and provide them with politically correct forecasts, then why would they invest large sums of money in this area.

Alan, assumptions may be the bane of the pure sciences, but they are the bread and butter of engineers. The main classes in sophmore year supposedly teach basic mathematical concepts, but the main lessons is in reasonable and unreasonable assumptions. If you make unreasonable assumptions, you failed.
There is too much code and not enough time for me to decrypt and understand it, but I think that this would be difficult even in best-practice-compliant Basic (the easiest of all languages to read and written in the most self-descriptive manner possible). While this sort of thing needs to be complicated, I cannot help but feel cheated that I cannot even find the array definitions in many cases, much less where they are set.
One thing that irks me is that they are detailed enough to adjust for the changing weight of dry air in higher CO2, when an increase of CO2 to a full 1% (an incredible amount) is only 0.15 g/mol. This is less than half a percent of the standard estimate (28.8 g/mol). If this affects the model’s outcome at all, then the model is laughably oversensitive, because there is no way that you can get two significant figures of accuracy, much less three, on 90% of the valid inputs. It’s comparable to using a surgeon’s scalpel while blindfolded. A baseball bat is just as accurate.

First, I’m a software developer with 22 years of post-college professional experience. I used to work for the Jet Propulsion Laboratory where I was responsible for maintaining some of the computer graphics software used by PIO for the Voyager and Galileo “flyby” animations, all written in FORTRAN.
With all due respect to everyone hammering on the quality of the code and the software developers who created this code, the FORTRAN here looks very typical of most of the FORTRAN I encountered 20 years ago at JPL in a professional capacity.
The problem with the code is twofold:
(1) The code is old. You can tell from the varying comment styles and coding styles in various source files that it has been worked on by a large number of people over a long period of time. Typically code like this suffers from the “Lava Flow” anti-pattern: the next person to work on a chunk of code doesn’t know fully how things work, so they delicately add their work, making as little change as possible. Lower layers are “frozen” like old lava, with new code poured on top, and allowed to freeze itself.
(2) The code doesn’t use modern coding techniques such as object oriented coding styles. This is typical of code from the 1980’s: OOP programming (and related techniques, such as using design patterns to keep code maintainable) are innovations that didn’t get wide-spread support in the programming community until the 1990’s. This means the code is hard to read and hard to maintain from current coding standards.
(3) One would hope that there would be a separate document which outlines the equations, formulas and rational used in each section of the code. Unfortunately most programmers take the attitude that “code is self-documenting”, when it is no such thing. When I was at JPL I spent a considerable amount of time documenting the code I inherited so I could keep it straight in my mind–and if I were tasked to maintain this code, that would be the first thing I’d do. Ideally I would try to trace back each of the calculation methods back to the scientific paper or textbook which provided the underlying formula, so there is an understanding why I’m carrying out a particular calculation at a particular point.
This sort of sloppy documentation is inexcusable, of course–but it is an extremely common problem in the software community.
(4) It is a common lament amongst software developers to distrust what they don’t understand–especially any newly inherited code. I had a software developer undertake rewriting large chunks of code I wrote myself–telling me to my face that whoever wrote it was an idiot. (He didn’t know it was me. I let it go.) Thus, no-one working on 20-year-old code will ever say it’s good code, regardless of the quality of the code. And they’re all going to itch to rewrite the code–so they can understand it, not because the quality is necessarily bad.
So I wouldn’t trust the comments of the developers who say the code quality is bad. It looks to me to be typical Fortran code from the 1980’s.
I’d say the real problem here is twofold: (1) The code is not properly documented with references to appropriate scientific texts. And as a consequence (2) we are over-reliant on the predictive model of a chunk of code no-one really understands.
But code quality: having worked as long in the software industry as I have, code quality is a constant lament–and really means “I haven’t figured it out.”

Yet another software engineer here …
“Keep in mind that the errors could be impacting results both ways, although, given the GISS track record, I doubt it would error on the cool side ;)”
That’s a way bias can creep into any software that produces numbers (eg modelling software) where people have pre-determined expectations for the numbers. Bugs that cause the numbers to differ from the expected result are spotted and fixed. Other errors … well not so much.

AGW has never been about the science. It has been about the social power.
The use of climate science is only incidental. Now that science is ‘settled’, the AGW community could give a fig about the terrible code that was used to sell the politics.

Of course, all this Earth climate model talk is premature – you first have create a model of the Sun that works properly 🙂

Dr. Robert M. Carter characterizes three climate change realities:
Climate change knows three realities: science reality, which is what working scientists deal with every day; virtual reality, which is the wholly imaginary world inside computer climate models; and public reality, which is the socio-political system within which politicians, business people and the general citizenry work.
————————————————————-
Computer models are not reality. Nature is reality. Political reality is a public display of ineptitude in dealing with reality.
Which of these realities is most conducive to your life, liberty and pursuit of happiness?

Jimmy Haigh (07:45:46) :
“…software engineers when they’re underpaid and overworked.”
“Why not look for another job?”
Probably because it’s nonsense. From 2003:
“Boulder labs: CIRES, NIST, NOAA, NTIA, UCAR, NCAR (JILA, LASP)
Employ 2,566 in Boulder; average salary $64,350
26% higher than overall average $51,220”
http://slochamber.org/Library/3A2.pdf
But I will be one that blames the software engineers for compromised products and predictions that affect everyone’s lives, *especially* when one complains that underpayment and overwork is the problem.

Doug Ferguson (10:10:36) :
“Chaos: Making a New Science” by James Gleick – Published in 1987 by Penguin Books.

Great book. Recommend it highly. After your jogging of my ancient memory I pulled it out to re-read.

George Tobin: “The guy who has to translate crappy assumptions…into workable code should not have to field that kind of issue or get caught in the middle.”
I submit that the Montana farm boy who enlists in the Army should not have to get caught in the middle of an ideological war and lose his life. The farm boy, fortunately, disagrees and defends the country according to his own lights.
So it would seem that a programmer, who is aware that software which doesn’t meet the best standards available is assisting the mendacious in destroying the world economy, has an obligation to field any issue and get caught in whatever positions are necessary to alert the world. It probably won’t be fatal to him.
Hubristic “scientists” initiated what has become a fight for the survival of viable economies and scientists must enlist in this fight right now.

R S Bridges (08:55:35) :
As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.

I take issue with this. I am most definitely a software “engineer”. I don’t believe you have a firm grasp of the roles that some of us of this profession fulfill.

From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.”

This pretty well sums up my job description, and what I have done for almost 30 years. I am not sure what else you would call my profession other than “software engineer”.

A more appropriate description for our friends who write programs may be software code developer.

Perhaps, for those of you that have no clue what some of us do.

Perhaps the closet disbelievers who are afraid to speak up on issues of poor computer code or badly created/applied statistical techniques or other issues in climate science could utilized Wikileaks?
http://yro.slashdot.org/story/09/07/06/1321227/WikiLeaks-Daniel-Schmitt-Speaks

I’m definitely NOT a software engineer, and only program’d a bit of PASCAL in the day, but I have a descent grasp of what is meant by “poor coding”. However, it would be nice if someone could give examples of poor coding and show how it affects the outcomes to modeling programs. It would be nice to actually see example of where the lack of “skill” (a Mannian term) affects the outcome of these climate models.

I have been a professional computer programmer for over 30 years in the insurance industry. Similar to this situation, we have Actuary’s that write their own software solutions to complex insurance problems. And similar to this situation, it can result in software that no one else can understand at a glance. I have at times been called in to repair/support such actuarial software. It is a challenge to do so, but it is what I do. From these experiences I have a respect for the need of the Actuary’s to experiment with their software in order to learn more about their own models. When a programmer is added to the R&D mix, that programmer must become a subject expert. And in my opinion, not a very high percentage of programmers are capable of PhD level work.
The idea of requiring that every piece of software must be written only by programmers would certainly be the utopia solution. However imposing such a requirement on mankind’s computing needs would greatly harm our productivity and ability to learn by experimentation. In my case for instance, my managers have my workload already laid out for me 2 years in advance. Yes, I already know what projects I’m supposed to be working on in 2011. This is because talented programmers are in fact in short supply when you consider the work that is ahead of us.
And I concur that we are not “engineers”. The state of Oregon has made it illegal to misuse the term (but they never enforce it). Doing so belittles the accomplishments of those that have been certified by the State as being actual engineers.
Pete

As I noted in the thread at Climate Audit, my FOIA for the top level software quality documents for GISTEMP and GCM Model E found no responsive documents.
NASA/GISS has not even considered where these projects fit within NASA quality standards.

Pete W (11:48:04) :
..
And in my opinion, not a very high percentage of programmers are capable of PhD level work.

I would certainly agree with you on that point. The percentage of top level programmers and/or software engineers (and especially the latter), is relatively small compared to the total volume of personnel involved in software development as a whole.

..
Yes, I already know what projects I’m supposed to be working on in 2011. This is because talented programmers are in fact in short supply when you consider the work that is ahead of us.

We operate similarly where I work. We have specific future goals laid out 2 to 3 years in advance. Because of our limited resources, we must engage in a lot of future planning, specific tasking and engineering.

..
And I concur that we are not “engineers”. The state of Oregon has made it illegal to misuse the term (but they never enforce it). Doing so belittles the accomplishments of those that have been certified by the State as being actual engineers.

In my role, I would disagree with this statement. My job roles push me far beyond merely “coding”. Specifically, one of my roles is to “engineer” new application models, scientific research, validation, implementation. These tasks move me far beyond simple software development roles. The developers (we can them “developers”), that work for me, are NOT software engineers as they are not intimately involved in the “engineering” aspects of our software development projects. Whereas, one of my roles is specifically to “engineer” these applications, often times employing cutting edge architectural techniques and unique systems integrations. We have “software developers/coders” here, but we also have “software engineers” here as well, I am one of those engineers.
I think the disconnect here is, most people do not fully understand the world of computer processing and that the subject encompasses. There are times when that world is very vast and requires intimate knowledge and discovery on a vast variety of technical disciplines (ie: engineering).
Pete

In modelling when there are unknowns, we are often conservative to cover our asses, but we also have to bear in mind we don’t want to over design or we will cost our client extra $, we have to stay within reasonable bounds but allow for the worst that might be expected. I.e. if designing a reservoir, we have to look at the worst storm possible and assume the catchment is nice and saturated, but we dont design to a storm that is improbable as that would waste money. We have to carefully assess the suitable storm and ensure it isnt unrealistic.
I wander if climate modellors put a limit on how conservative their assumptions are?
I mean, clouds causing warming, 1/3 ocean evaporation, 20kph glaciers, co2 warming the sea etc… etc… when these all come together do they represent a highly unlikely outcome or a realistic outcome?
Personally I like the Idso equation as its the only one empirically derived.

As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.
I came up through the ranks in the 70s when quite a few coders were also hardware engineers. I was one of them.
I do take exception to the statement that “software engineering” is not real engineering. Ask the aerospace guys (me, me) or the medical eqpt. guys if that is true. Degrees and professional exams are not the measure of all things engineering. Otherwise it would have been a University Professor and not James Watt who brought the steam engine into the modern age.
BTW this hardware/software engineer (non-degreed) had a government inspector look over his code and the inspector said it was the best written code he had seen in years. My secret? The naming of parts. If you get the names of things right the code reads almost like English and even a non-coder should be able to get the gist of it. I did do my work in FORTH which is a great help in making the code readable. But if you are not a disciplined software engineer FORTH makes it much easier to do unreadable code. So it cuts both ways depending on the skill and discipline of the engineer.
We have the example of hardware engineers. You give them a pile of steel and other metals, etc. and they will build you an engine. However, there are no restrictions on how the materials can be used. It is up to the engineer to make the right choices. In “modern” coding languages the effort is to make the compilers and tools cover for (prevent) errors. Such restrictions also prevent you from doing a lot of useful things easily or at all.
All very nice if you are going to employ a bunch of mediocre coders/engineers. But you then wind up with a bunch of mediocre code. In everything. Just think of how long it takes Windoze to load.
So my point – if you don’t trust your software engineers the answer is not training wheels. It is better engineers. There is no substitute for talent and discipline.
BTW PHDs can code well if taught the discipline. If they are apt students it only takes a couple of weeks to train them. If they are not apt they shouldn’t be coding.

Rod Smith (09:48:05) :
As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
And why is anyone still coding in Fortran?

All the FORTRAN code makes the output appear to be ‘scientific.’

Roger Sowell (13:15:05) :,
You are talking about the LEGAL definition of engineer not the practical definition. A confusion of domains.

M. Simon (13:04:11) :
..
BTW PHDs can code well if taught the discipline. If they are apt students it only takes a couple of weeks to train them. If they are not apt they shouldn’t be coding.

A couple of weeks huh? Perhaps back in the 70’s.. it seems you may not have much experience with modern day development processes. It would take a couple of weeks just for them to learn the source control management aspects alone. I would be willing to bet that not one of those individuals would be capable of coding on my current project within a year, perhaps even two.

Take the example of Boeing. They had a nice software model, that certainly helped a whole lot in the design process. Still the result had to be tested in a prototype, where actual stress measurements were made. An error was found that could not have been discovered by “good” software engineering, only by actual measurements. The software will be corrected, and be more useful in the future. Software that cannot be tested cannot be developed to a high degree of reliability. Good coding practices alone are never sufficient.

Someone said: “The model results reflect the assumptions of the authors of the model”.
That would be true if there were no errors (bugs) in the code.
As it is – the models reflect nothing. Their result cannot be treated as anything but random, since we have no independent way of authenticating them, therefore we have no way of debugging the code.

M. Simon (13:04:11) :
I’m curious Do you think your definition of “software engineer” fits with what the needs are for building a climate model? I think it may, since it requires much more than a coder, which is why the scientists probably decided to do it themselves in the first place anyway.
By your definition, I believe I would be called an engineer. But we will have to disagree on the term software engineer. This company has folks that do everything you mention, and none of them are called software engineer.
Pete

10 Rem Global Climate Model
20 Print “The End Is Nigh”
30 Goto 20
40 End (of planet) <<
Better program
10 REM Global Climate Model
20 ON ERROR GO TO 40
30 PRINT AT INT (RND * 21),INT (RND*32);”The End is Nigh”
40 GO TO 30
But the RND function is no good since it repeats itself in about 15 times, better take the value of the random SEED generator directly with the aid of a PEEK command.
10 REM Global Climate Model
20 ON ERROR GO TO 40
30 PRINT AT INT (PEEK (16434)*21),INT (PEEK (16434)*32);”The End is Nigh”
40 GO TO 30
Oh my god how long has it been, to long i guess and one thing comes up, that even small and relatively simple program’s do require some debugging, good software takes time, but even then if you feed it with garbage data, it will result in output that is still garbage.

The following, from The Future of Everything [originally entitled Apollo’s Arrow: The Science of Prediction and the Future of Everything] by David Orrell, pg. 324), WRT climate models in particular, is apropos relative to global climate models (CGMs) and Gary Strand’s contention at http://www.climateaudit.org/?p=6495 that it’s the science not the software development:
“Einstein’s theory of relativity was accepted not because a committee agreed that it was a very sensible model, but because its predictions, most of which were highly counterintuitive, could be experimentally verified. Modern GCMs have no such objective claim to validity, because they cannot predict the weather over any relevant time scale. Many of their parameters are invented and adjusted to approximate past climate patterns. Even if this is done using mathematical procedures, the process is no less subjective because the goals and assumptions are those of the model builders. Their projections into the future—especially when combined with the output of economic models—are therefore a kind of fiction. The problem with the models is not that they are subjective or objective—there is nothing wrong with a good story, or an informed and honestly argued opinion. It is that they (GCMs) are couched in the language of mathematics and probabilities; subjectivity masquerading as objectivity. Like the Wizard of Oz, they are a bit of a sham.”

I miss Gary Strand… He should be commenting here…

henrychance (15:17:41) :
Thank you for the clarification/education. So there is an actual degree in “software engineering”. Good to know. I’m just ruffled by people that call themselves engineers when they aren’t. M. Simon quite likely is one.
Bobn (16:36:30) :
I agree.
In addition to being complex the climatology models are – and are supposed to be – constantly changing as they learn more of the natural forcings at play. To suggest they should be required to stop now and re-write the code from scratch is… shall we say… not very reasonable.
The models they are working with appear to have been proven fairly accurate over the last 20 years. This video contains some historical perspective of climate models;

Pete

A plumber comments:
It seems that an engineer is in search of reality, while the climate model programmer is in search of continued funding.
Real data in = future warming = continued funding
Real data in = future equivocal = new job opening
Of course I could be wrong…
Mike Bryant

My two Baht’s worth. I’ve worked with computers all my life as a field engineer (gorified mechanic for the most part) and as a code developer who knew the hardware side. My experience has been that the programmer is given specific parameters the code is to work within, and away he/she goes. Personally I have never written a piece of “clean code” on the first try. Just keeping all the “if this then that” routines functioning together is challenging, especially on a tight schedule.
The programmer finally makes the original “assumptions” work as directed, then someone says they actually want something else because of unexpected developments. This is a programmers worst nightmare and can impact thousands of lines of code, not to mention the new coding required for the “unexpected developments”. Computer coding is a dangerous thing if it changes on a regular basis. I once developed a program to track a billion dollar project. Actually the original assignment was easy, even with the tight deadline. As the project developed, management changed their wants and needs and they always wanted it tomorrow. By the time the project ended, I had a bloated program that worked but had become such a nightmare that no one but me would ever be able to follow my logic. Ugly.
I actually feel sorry for the guys who are driven like slaves to accommodate the “climate scientists” whims. It has to be frustrating.
Just my opinion.
Jesse

The climate models do not magically produce results. The problem is not code-related.
They are designed to follow global warming theory on how temperatures increase as GHGs increase. Different modelers will have slightly different assumptions about the temp impact or build in longer lag times or incorporate different assumptions about feedbacks like water vapour or have different assumptions about Aerosols.
But the climate models are just weather models, with a LN (GHGs) module and then some random white noise processes built-in. They all are building toward +3.0C by 2100 give or take and a simple spreadsheet would work just as well.
It is not a question of whether the code is done properly, it is a question of whether the theory is right is the first place. So far, the theory needs lots of little plugs to plug the Watt leakages.

Jesse (17:57:27) : The situation you describe what with the requirements changing as time goes by is normal. That’s what good programming practices are good for – making all that change manageable.

If I were to build a financial model with the same results compared to observations as these climate models, I would be fired. Unless, of course, I was modeling the housing market for a mortgage guarantee pseudo-government corporation. In that case I would get a bonus.
Orwell is laughing.

Frederick,
Convert? Gak! This may be of interest to you if you haven’t seen it:
“The GISS GCM model development process is over 25 years old…”
http://www.giss.nasa.gov/tools/modelE/modelE.html
from Anthony’s link
http://www.giss.nasa.gov/tools/modelE/

Steve in SC, if you’re referring to my comment as “disparaging”, I didn’t mean it that way. I wrote:

There is a difference between someone writing code using established methodologies and someone who taught themselves and gets mired in an oversized project.

Many times I have had to clean up after people who got bogged down in something bigger than they can handle. Part of the “established methodologies” I refer to include stepping back and seeing the big picture, then breaking down or modularizing into sections, then concentrating on building functionality for each module or section, then optimizing.
Most self-taught programmers are very good at one or more of these steps, but few do all of them. Obviously, since I’m writing this, I like to think I am one of those, at least I keep getting called back and others who follow my work are never wading through several feet of muck to get where they need to be.
Either way, I’m trying to summarize what many have also tried:
NO, it’s not all about the code. Sure, it’s about the physics. However, without the code to support the physics you have muck, you can’t verify or confirm accuracy, you can’t test anything, and you have a horrid time making any changes.
The whole point of spreadsheets, for example, was to take programming out of the picture and allow those who worked with numbers to just go ahead and plug in numbers. Yes, programmers were involved, you just didn’t see them. VisiCalc and Lotus 1-2-3 created something of a revolution by letting the numbers and formulas people just work with numbers and formulas, and not have to fight with the computer as well.
Maybe we need to see the development of an independent, publicly accessible, open source GCM, and allow people to share their tweaks and assumptions. Surely then we could come up with a better model.
(yes, I know the current group don’t actually WANT a better model)

“These codes are what they are – the result of 30 years and more effort by dozens of different scientists (note, not professional software engineers), around a dozen different software platforms and a transition from punch-cards of Fortran 66, to Fortran 95 on massively parallel systems.” – Gavin Schmidt, RealClimate.org

John Galt (14:11:13) :
…
Having a degree in Software Engineering or Computer Science is not a guarantee of the quality of a person’s code. I have met many self-taught programmers who write excellent code and many degreed professionals who just copy and paste and patch their code after it’s put into production.
The difference is in the attitude and aptitude. A person who takes pride in their work strives to write good code. They want to improve and when they don’t know how to do something, they learn it instead of throwing out one bad hack after another.

Too true! I have run into this time and time again. I cannot count the number of people I have encountered that have Masters Degrees in Computer Science that are either complete hacks, or simply don’t know their rear end from a hole in the ground. Always makes me laugh. Want to impress me? Don’t show me your papers, that won’t do it…

henrychance (15:17:41) :
Real engineers
There seems to be a misunderstanding regarding engineers. I use headhunters to hire people and they aren’t confused. Engineering degress are granted by schools of engineering. None of this M.S. in Information systems from church schools. You have an engineering degree from an engineering school or you do not. Microsoft and some other vendors claim to produce them. Nope. Not the real thing. (I cut slack for railroad engineers, they have had that handle and don;’t hold themselves out to have a real engineering degree.) Head hunters also tell me if it is a compromised degree such as B.S. In Engineering Technology. Not as much as a real engineering degree. This excludes the DeVry Universities and some BS computor science degrees. There are real computor engineering Degrees. There are several hundred Engineering School campuses.

I would have to agree with this. I personally attended a University, 4 year Bachelor of Science degree, from an accredited engineering and agricultural research University. Many of my courses shared those with EEE students (Electrical and Electronic Engineering), especially Mathematics, Physics, Chemistry, etc… Our “Computer Science” curriculum was very much in line with the Engineering disciplines. I do not consider MIS degrees and all their various permutations, any where near to the class of a real CS degree from an engineering College or University. They simply are not compatible with one another. Perhaps this is where part of this breakdown is when using the term “Software Engineer”. There truly are “Software Engineers” out there (I am one). I would say this is far more the exception than the rule when speaking in general terms of “Software Development”. There is definitely a very sharp contrast between a “Software Developer” and a “Software Engineer”. The latter requires a much larger toolbox. Incidentally, my personal professional title is, and has been for a long time, Sr. Software Engineer. This same title has followed me for many years, even throughout my years working government entities as the DOD, DHS, Pentagon, Navy, and various defense contractors.

We really have been to this movie before.
The Strategic Defense Initiative ‘Star Wars Project’ of Pres. Reagan suffered perhaps its most serious blow when:
“On June 28, 1985, David Lorge Parnas [link] resigned from SDIO’s Panel on Computing in Support of Battle Management, arguing in 8 short papers that the software required by the Strategic Defense Initiative could never be made to be trustworthy and that such a system would inevitably be unreliable and constitute a menace to humanity in its own right.”
see: http://en.wikipedia.org/wiki/Strategic_Defense_Initiative
Remember? We could not show, using computer science protocol already well-established more than 30 years ago that the code-base for SDI could meet a “necessary and sufficient conditions” mathematical analysis. [This is an example of what “software engineering” really is/is about, and is at least roughly analogous to finding the harmonic modes of Galloping Gerdy, before it falls into the Tacoma Narrows).]
see: http://en.wikipedia.org/wiki/Necessary_and_sufficient_condition
The SDI code-base made today’s NASA GISS model E, GISTEMP et al look like a school teacher’s classroom management program. We were capable of using mathematical logic to determine whether the SDI software was actually internally consistent & coherent … and could show rigorously that it was not.
====
There is nothing inherently wrong with Fortran. On the contrary, Fortran work is currently important: it is still the Speed-King of high-level languages, and in plenty of ‘raw-computing’ applications, maximum computation speed is absolutely the bottom line.
There is also nothing inherently wrong with old algorithms/routines. The vast majority of all the algorithms we use are, unavoidably, old. There is actually an advantage to using Grandpa’s code-routine, since it has been hammered/tortured in more ways than some new-fangled method.
The problems with Model E, GISTEMP, etc code-bases aren’t that parts of it are in [guffaw!] Fortran (there is a modern-language conversion-effort underway), or that the routines are [hoot!] 3 decades old, but that these code-bases are not being managed/maintained appropriately.
Why not? Why isn’t this stuff taken care of properly, or subjected to analysis-procedures that can *prove* whether it’s water-tight or not? Because key people don’t care. Because at the levels that count, it’s not important. Remember, Congress cared whether SDI software produced GIGO, or actionable conclusions. Neither Congress nor the White House care if the GISS codes make good sense or not.
Key climate/weather data/analysis is performed by Pentagon & Intelligence departments & agencies. NASA/GISS is a dog & pony show without weight or leverage. Nobody particular cares if their software can even run, much less can be used to say anything verifiably scientific.
The President pays no attention to the NASA/GISS climate work, but he consults with the Pentagon liaison daily. NSA hovers just off his elbow, ’round the clock. Despite Dr. Hansen’s finest theatrics and personal letters to the White House and other Heads of State, Pres. Obama studiously ignores him.
NASA’s climate-work is sub-par, and redundant. Even if their efforts were well-managed, they are woefully antiquated (20 years behind Pentagon/NSA) and under-staffed/funded (by orders of magnitude), compared to the military and intelligence projects that our top leaders actually use to make ‘best-science’ decisions.

CodeTech (20:59:04) :

I was discussing some of these things with one of my software developers today. A major function of our job includes the translation of human processes into computer processes. There are engineering aspects to this, there are also creative aspects to this. These translation processes are not blank and white and can sometime require drawing from a vast array of disciplines and experience. Again, in this arena, I am rarely impressed by someones paperwork. Some of the best software developers and software engineers that I have met, have been your classic academic rebel (I being one of those), and rarely posses impressive credentials. Books are one thing, practice is another.

Ted Clayton (22:58:31) :
…
There is nothing inherently wrong with Fortran. On the contrary, Fortran work is currently important: it is still the Speed-King of high-level languages, and in plenty of ‘raw-computing’ applications, maximum computation speed is absolutely the bottom line.

I would agree, as FORTRAN is just another tool. Is it really the speed king? I would argue with that. C or C++ typically compiles to much more efficient assembler code than does FORTRAN. Further, some versions of Pascal compile to even more efficient assembler code, one of the best being the good old Borland Turbo Pascal, which would compile to about the same compact and efficient assembler code as C/C++, but had the speed advantage of passing parameters through registers rather than pointers to memory heap allocations. Borland Pascal used a very efficient way of utilizing registers and stacks rather than heap memory allocations. I am not saying that FORTRAN is poor, it is not, but I don’t know that I would give it the blue ribbon for efficiency either. I spent several years hand optimizing assembler output from a variety of source compilers and in my experience, C/C++, Pascal and even Modula II typically out performed most all other languages. Metrics from clock cycle testings and audits would confirm this at the time.

Last I checked FORTRAN was still faster on certain numerical calculations, but I agree with you. Who cares on such small differences. The big issue here is maintaining and working with the code. This is classic FORTRAN full of cryptic variable names that are less than 8 characters. It really is old school. How do you even verify that this code is performing correctly. It is over 100,000 lines. No unit tests, no subroutine test drivers. It would take a long time to get your head around all the interactions in the sub-models. One good advantage is that it has formating for line printers. So it’s got that going for it. Which is nice.

Burke (23:00:55) :

Back in the mid 80’s, I was briefly involved in a project with the Chem. department or my University, and one of their research projects. The Chem. dept. is infamous for gobbling processor time on any machine. They were frequently awarded grants from IBM that included the use of newly developed technologies (ie: super computers) of the time. One such computer was the IBM 9000 series Vector Processor machine. My role was to assist in the performance optimization of critical computational components. All of the original code was written in FORTRAN. To achieve our goals, we broke out critical portions and rewrote those components using C which incorporated a lot of hand optimized in-line assembler code. Our results were rather astounding, with an average performance improvement of around 1700-2000%. This was pretty significant. In this example, I would say that our C implementation blew away the computational performance of standard FORTRAN, but to do so wasn’t exactly easy either.

Squidly (23:13:27),
Yes, I swung the praises of Fortran just a bit wide. 😉
To go on & on in Fortran for housekeeping chores isn’t nice. I perused the NASA Model E upgrade project pages about 6 months ago, and saw that a major problem is the use of cryptic, short variable-names. Git out!? Now, that’s a maintainance chore that shoulda got done about a standard career back.
For the most part, Fortran is only going to be ‘in the way’, agreed. Where the aim is to really come to grips with Formula Translations, though, it might have an edge (not that Models, as noted, are doing computationally/numerically-limiting stuff in most of the routines). And, I’m not as current on language-issues (a socially-acceptable 20th C. substitute for religion-arguments? 😉 as I would have been a decade (2?) back…
But I do see on the handiest reference:
“Since Fortran has been in use for more than fifty years, there is a vast body of Fortran in daily use throughout the scientific and engineering communities. It is the primary language for some of the most intensive supercomputing tasks, such as weather and climate modeling, computational fluid dynamics, computational chemistry, computational economics, and computational physics [5 links]. Even today, half a century later, many of the floating-point benchmarks to gauge the performance of new computer processors are still written in Fortran (e.g., CFP2006 [link], the floating-point component of the SPEC CPU2006 [link] benchmarks).”
http://en.wikipedia.org/wiki/Fortran
Like Burke (23:27:06) says, for the right crunch-jobs, it’s evidently still it.
Yes, a climate-modelling team-approach like you describe, or a modern-Celtic OSS approach could rapidly put the computational/modelling portion of climate-investigation (it still needs a huge, ongoing investment of actual, physical science) on a good footing. Dr. Hansen is stuck in a (faulty) loop.

I’ve just been looking at some of the GISS Code and it is schoolboy stuff from a software point of view. I did a quick Lines of code count and it appears that GISS believe that the earth’s climate can be accurately simulated with only 100,000 lines of badly written Fortran. When you consider that windows Vista has 50 Million lines of code it makes you wonder how something that is orders of magnitude more complicated than the climate, could ever be expected to boot up, let alone do anything else.

You should see some of the awful FORTRAN code still used and maintained to do attitude determination, orbital mechanics, guide star occultation, etc. for spacecraft. Complaining that past coding doesn’t meet current standards is just plain silly. As long as it ‘works’ for the primary user, redoing it is just bad engineering (if ain’t broke and all that). Any disagreement with the definition of “works’ has little to do with the ‘quality’ of coding.
Gary was trying to defend what he does for a living. Why, I’m not sure. He calls himself a ‘software engineer’ but is not necessarily any more an ‘engineer’ than a ‘building maintenance engineer’. His ‘admission’ is at about the same level as that of a building maintenance engineer’s admission to the sad state of architectural design. Ho hum.
Should the climate models be used for trillion dollar decisions? Of course not! But for far better reasons than crappy, hard to follow code. Let’s focus on the real issues instead of peripheral ad homs like ‘crappy code’.
Move along people. There’s nothing to see here.

Fortran or WATFIV?

How would one know the difference between true model predictions, and artifacts because of bad coding?
Somewhere in the code I saw that it assumes 360 days in a year, or 12 equal months of 30 days. Not a bad assumption if you are forecasting over a short range of time. But when you are trying to predict global temperatures 100 years into the future – that is a very different story.

I’d thrown together a post a while back on poking a little fun at some of the base assumptions behind catastrophic AGW, and I had a couple paragraphs dedicated to why the models are pretty useless. Of course, I was just looking at the design side, not even getting into the issues of poor coding. Here’s the relevant portion:

…the sheer magnitude and complexity of the system still defies attempts to fully understand all of the interactions, leading to small errors having an ever-growing cascade effect as one tries to predict further into the future.
Put more simply, imagine working on a very complex algebra problem. The issue is that you copied one little part of the problem wrong. In your first pass, simplifying, substituting, and performing other operations, that wrong bit interacts with another element of the problem, the result of which is corrupt. Each subsequent pass through the problem causes the erroneous segment to corrupt more of the total problem as it is combined with other elements, and, by the end, your final answer is utterly wrong. This is very much what inhibits long-term forecasting on even a local scale and short time spans. Taking this up to the global scale allows vastly more chances for error, and projection years into the future gives any errors much more time to cascade to the point of rendering the entire model useless. The calculations may be perfect, but a faulty starting point ruins the entire process. Unfortunately, this isn’t the greatest difficulty. Back to the hypothetical problem, imagine that you don’t even get copies of any of the formulae you need, nor even the base problem. Instead, all you have are the base values to use and thousands, millions of answers, each coming from different base values (you don’t get told what any of them are), from which to figure out the starting equation. It gets worse. Some of those answers are wrong, and you don’t know which ones. It gets worse still. The answers are divided up into hundreds of groups, each group containing answers to certain combinations of formulae and base values, and you aren’t told what formulae or values are used or how they are combined. Nope, not done getting worse, yet. In fact, you are told that none of the groups contain answers to the actual problem you need to solve. Instead, example answers for the overall problem are supposed to be composites, somewhat like the mean, of all of one answer (the tenth, for example) from each group, but you also have to figure out how those subordinate answers get combined into the big one. In the end, you have mountains of data, an unknown amount of which is faulty, from which to determine all of the formulae, how the formulae are used to produce the subordinate answers, and how those are combined to get the main answer before you can even start working on the problem with the given starting values. Guess what? This isn’t even accounting for time lapse yet.
Someone may read that last paragraph and think, “Models on the global scale don’t have to be that precise, though; they can be more general and still get the broad picture.” Well, let’s run with the picture metaphor, shall we? I would roughly equate simplifying components of a model to reducing the number of pixels in an image. Sure, you can do it, but you loose definition. Do it with a complex, intricate picture, and you will very quickly end up with an image people misinterpret, or one from which it is impossible to discern anything at all. Sure, you can dumb down the models by making them less complex with estimations of values instead of solid formulae, but you have to admit that doing so destroys the accuracy of that broad picture. A more direct example comes from graphing a curve to fit plotted points. Reducing complexity could be seen as using fewer data points from which to extrapolate a curve. The problem with this is that, as the data points are made fewer (thereby spreading out on the graph), you may end up missing movement in the actual curve occurring between points and graphing a curve that is completely wrong. When it comes to scientific (quasi-scientific in this case, really, but that’s a topic for another day) analysis of data, generalization is bad, and detail is good.

It appears I have hit a sore spot re: “software engineers.” First of all, I don’t mean to demean the value of the work that they do. They fulfill a very necessary niche in the business world. But at the same time, unless they are engineers in the traditional sense, they must take direction from engineers or scientists to build the models. There are a lot of chemical engineers who develop software for our business (for those in my business, think ASPEN, Simsci, or other chemical engineering software modelling companies). And we would be a hell of a lot less productive without them. I shudder to think of designing a multi-component distillation column by the trial-and-error tray-by-tray method. I remember seeing a thick binder of calculations done by hand. That said, it is still the physical science that is the underlying basis for the models, not the code.
So, Squidly, I salute you, and all of your brethern! Go forth, and prosper!

@ Ted Clayton (00:24:40) :
Ah ha! .. Very thoughtful comments! Nice!
I completely agree with you concerning language construct and organization of FORTRAN and what that brings to numerical formula processing. Your comments are certainly very valid. I would content however, and especially when considering the enormous budgets employed, one could utilize a more modernized language (preferably a fully compiled language that produces binary code for performance optimization, ie: this would exclude Java) and provide even more robust scripting mechanisms that would allow the use of heavy mathematics in robust expressions that could then in turn be pre-compiled into the foundation language, optimized and compiled to binary codes native to the host machine. Such an approach as this will allow for the proper architectural abstractions, allow the software engineers to employ flexible and robust design concepts and patterns, allow software developers to write compartmental coding segments abstracted from adjacent logical modules, and probably most importantly, allow the “scientists” to develop robust mathematical representations of physical processes and complex interactions utilizing language, verbiage and structure that they are used to.
In this scenario, everybody wins, and result could be something that may actually stand a snowballs chance in hell of working. And even if it doesn’t work, you have developed something that can adapt as your learning and understanding culminates and expands.
IMHO…

🙂 hehe webcam for the win Anthony..