Quote Of The Week #13

qotw_cropped

From Gary Strand, software engineer at the National Center for Atmospheric Research (NCAR) commenting on Climate Audit:

As a software engineer, I know that climate model software doesn’t meet the best standards available. We’ve made quite a lot of progress, but we’ve still quite a ways to go.

I’ll say. NASA GISS model E written on some of the worst FORTRAN coding ever seen  is a challenge to even get running. NASA GISTEMP is even worse. Yet our government has legislation under consideration significantly based on model output that Jim Hansen started. His 1988 speech to Congress was entirely based on model scenarios.

Do we really want congress to make trillion dollar tax decisions today based on “software [that] doesn’t meet the best standards available.”?

There’s more. Steve McIntyre comments:

Re: Gary Strand (#56),

Gary, if this is what you think, then this should have been reported in IPCC AR4 so that politicians could advise themselves accordingly. I do not recall seeing any such comment in AR4 – nor for that matter in any review comments.

…and to the second part of the comment:

Re: Gary Strand (#56),

If we can convince funding agencies to better-fund software development, and continued training, then we’ll be on our way. It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.

Boo-hoo. Hundreds of millions of dollars, if not billions of dollars is being spent. PErhaps the money should be budgeted differently but IMO there’s an ample amount of overall funding to have adequate software engineers. Maybe there should be some consolidation in the climate model industry, as in the auto industry. If none of the models have adequate software engineering, then how about voluntarily shutting down one of the models and suggest that the resources be redeployed so that the better models are enhanced?

I’m not making this QOTW to pick on Gary Strand, though I’m sure he’ll see it that way. It is a frank and honest admission by him. I’m making it QOTW because Gary highlights a real problem that we see when we look at code coming from NASA GISS.

But don’t take my word for it, download it yourself and have a look. Take it to a software engineer at your own company and ask them what they think.

download-icon

GISS Model E global climate model source here

GISTEMP (surface temperature analysis) source  here

Sure, this is one of many climate modeling software programs out there, but it happens to be the most influential, since GISS and GISTEMP are the most widely cited outputs in the popular media.

U.S. industry seems to do a better job of software development than government programs, because in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.

In academia, the solution is usually to ask for more grant money.

Advertisements

164 thoughts on “Quote Of The Week #13

  1. “…software engineers when they’re underpaid and overworked.”
    Why not look for another job?

  2. Reported in New Scientist 4 July
    Boeing has encountered some serious structural sicknesses in the new 787.
    Stress tests have revealed that the “wingbox” and 18 areas on the fuselage around the wings require strengthening.
    “Data from the test did not match our computer model,” says Boeing vice-president Scott Francher. That highlights the difficulty of predicting the behaviour of advanced CFRP materials being used in very large structures for the first time.

  3. Jimmy Haigh (07:45:46) :
    “…software engineers when they’re underpaid and overworked.”
    Why not look for another job?
    I believe the best ones went and worked in the financial modelling sector…..
    cheers David

  4. Could anyone hazzard a guess about the Met Offices input/output data in light of the above? No such admissions forthcoming from that quarter or they’ll be out on their ears! It does worry me when the apparent “uncertainties” are explained away as “better understood”, which means nothing & is not the same as saying “we know & undertand how they affect our data output”, particularly when predicting the future!
    I still think we are at the limits of computer power & understanding about climate. If you are making assumptions, whether reasonable or not, they are still assumptions, & if wrong, no amount of computing power is going to rectify it & or give you the answer, which seems to be what the MO are implying by getting this new Deep Thought computer, as they seem to want another one 1000’s of times more powerful than the one they have to get the right results! As Prof John Brignell @ Number Watch says, a comuputer model can be right or wrong, but still irrelevant!

  5. OK , I can see how those in the “climate science” community choose to ignore the holes in the fabric of their theories , but why do our media and our politicians ? (Yes its rhetorical , but still…)

  6. No one involved in this mess WANTS functioning, competent software engineers, because they might accidentally do their jobs and reveal the fraud to the world. We Can’t Have That.
    When you’re conduction a fraud of this scale, you need to make sure the fraud is perpetuated from top to bottom. Crappy, inscrutable softward code that no one understands but that always gives the “Right” answers is something to be treasured, guarded, and NEVER EVER replaced.

  7. Good post and useful critique, I think.
    But, I first did FORTRAN in 1965 (version II-D) and am quite happy to take your word that these models are not great examples of clarity. I’ve seen all the “do loops” and “if – then” statements I ever want to see.
    That being said: Know that I appreciate those of you that are looking at these things and I do trust your judgment.

  8. The last piece of code that worked without any unexpected errors was;
    10 Print “Hello”;
    20 Goto 10
    and it still required half an hour of debugging.
    The faith that is put in models is terrifying and people always ignore the one underlying problem;
    For a model to be correct it has to be close to 100% accurate against the real world situation, if it isn’t it is at worst a waste of time, at best a vague indicator as to whether your original theory was credible but should never be treated as proof of anything.

  9. I browsed through the code a bit. If I take the comments at face value, it looks like you have PhD scientists writing production FORTRAN code – bad idea.
    Industry practice is to have PROFESSIONAL code writers maintain the software. Although I have degrees in Chemical Engineering, I worked as a programmer for 4 years in the mid-1980s on FORTRAN, COBOL, SAS, and several other coding languages. I was the lead programmer for a Fortune 50 company’s in house ChemE simulation program, about 500,000 lines of code. Programmers maintained and updated the code. Our analysts (usually Masters or PhD Chemical Engineers) provided the input and the mathematical modeling, which we then coded. Analysts weren’t allowed to compile or load ANY source code.
    It might be a stretch to call it the worst FORTRAN coding, what little I saw wasn’t very good. I would reserve judgement until I had run some FORTRAN source analyzers and mapping software to see how the code lays out.
    At a minimum they should have a QA/QC and testing program.
    Coding errors wouldn’t be particularly difficult to find. Are the GISS models “peer reviewed”???
    Sounds like the next project after SurfaceStations wraps up.

  10. I’ve written my own NASA-like climate model software
    10 Rem Global Climate Model
    20 Print “The End Is Nigh”
    30 Goto 20
    40 End (of planet)

  11. As a longtime professional software developer, I’ve seen millions of lines of code, of varying qualities. But I can make some generalizations. The people who code something up as an assistance to their job, not as a primary focus, in general write lousy code.
    So I’m not surprised to see the quality of the code. It pretty much looks like what I’d get if I’m interfacing, say, with a hardware guy who writes a tool to use on the hardware he’s designing. Unreadable, undocumented, unmaintainable code that basically works. The mathematician/statisticians that I’ve worked with are the same way, so I’d guess I’d expect the same thing of climate scientists.
    If I had to guess, I’d say that the number of actual professional software engineers they have working on the project is zero, that 100% of this has been coded by the scientists themselves.

  12. Grumbler (08:01:18) :
    I believe the best ones went and worked in the financial modelling sector…..
    Yes I think that they probably did!
    For an overworked software engineer, our Gary still finds lots of time to spend on blogs! When I am working, usually on an oil rig offshore, the 4 hours off, that I manage to salvage from most days, I usually spend sleeping.

  13. “It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.”
    UCAR/NCAR stopped public access to wage data in 2008. Apparently how federal funding for this institution is broken down has been hard to find as early as 2006:
    http://www.climatesciencewatch.org/index.php/csw/details/greenwire-inhofe/
    But if the General Schedule for federal employees is any indication of compensation at NCAR then $80,000+ would be a good starting estimate of “underpaid”.
    http://www.bls.gov/oco/cg/cgs041.htm#earnings
    Gary, perhaps you should back up your unfounded claims like being “underpaid” and “overworked”, eh?
    Or do you just not have the time to post that, being overworked and all, at your office on weekends on your “own time” for instance?

  14. What’s so tough about writing software for climate modeling?
    All you got to do is make sure your model spits out warming.
    Then the media will take care of the rest.

  15. I am a software engineer and I have looked at the GISS software. It is right up there with some of the worst code I have ever seen. The question is not whether it contains errors, the question is just how bad the errors are and how much they impact the results.
    Keep in mind that the errors could be impacting results both ways, although, given the GISS track record, I doubt it would error on the cool side 😉

  16. As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
    A more appropriate description for our friends who write programs may be software code developer.

  17. In The Black Swan (http://www.amazon.com/Black-Swan-Impact-Highly-Improbable/dp/1400063515/ref=sr_1_1?ie=UTF8&s=books&qid=1246893449&sr=1-1)
    Nassim Nicholas Taleb argues it is impossible to successfully model complex systems over periods of time; that the very attempt is foolish. I would be interested to here from anyone involved in complex modeling that thinks otherwise. Anyone?
    Climate modeling software may have problems but it is not the problem. What are the software engineers attempting to code … some model that makes no sense? I heard a congressman ask: did we learn anything from these failed financial models (packaging of mortgages to reduced risk). If you put the slightest confidence in GW models you learned NOTHING. Here is a brief comparison of the two:
    1 Mortgage modelers had a financial incentive to be right. Climate modelers have a financial incentive to spread alarm and generate more government grants. They plan to be dead before the absurdity of their 100-year forecasts becomes clear. (They miscalculated since all 4 IPCC forecasts are not just wrong but fell outside their 90% confidence interval).
    2. Mortgage modelers tested against historical data releasing models only after they completed regression tests using historical data. Climate modelers know the earth cycled through over a dozen ice ages … their models do not, and cannot, predict these cycles. They don’t care. (See #4)
    3. Mortgage modelers attempted to incorporate all known relevant information. Climate modelers are concerned only with “human induced factors” (see #4). Further, they ignore science when it gets in the way of their objectives. For example, every study of CO2’s persistence in the atmosphere (over 30 of them) finds CO2 released into the atmosphere will persist for 4-15 years before being (mostly) consumed by the ocean. I understand GW models assume 100 years because assuming less than 50-years does not produce the desired warming effect. Is this Right? The American Chemical Society considers this a fatal flaw in the GW models.
    4. Mortgage modelers understood the system they were attempting to model. Climate modelers don’t understand much about the climate … nor do they care to. According to the IPCC document, PRINCIPLES GOVERNING IPCC WORK: “The role of the IPCC is to assess on a comprehensive, objective, open and transparent basis the … information relevant to understanding the scientific basis of risk of human-induced climate change.” The “human induced” restriction makes it clear why there is relatively little research into the role of the sun, the earth’s orbit, the formation of clouds (which cool the earth?), cosmic rays (which may seed clouds?), ocean currents or the huge climate changes of the past which moved the earth into and out of over a dozen ice ages; humans have no significant impact on any of these.

  18. Ryan needs to get paid more too, IMHO software developers are often overpaid for the difficulties of their jobs. I’ve learned programming in my spare time – it ain’t rocket science.
    Ryan’s made a big improvement on Steig et al running over 3000 reconstructions of the Antarctic and resolving several outstanding issues. He also directed several comments to the RC guys. Looks like the makings of a bad week for them.
    http://noconsensus.wordpress.com/2009/07/06/tav-to-realclimate-you-can%E2%80%99t-get-there-from-here/

  19. I was unaware that any blame had been directed at GISS software engineers. My understanding was that they were faithfully implementing the bogus sensitivity assumptions and all the rest of that which comprises current climate modeling ideology.
    I am inclined not to read much into this fellow’s candid assessment. And I think Steve M. was a little over the top implying that the programmer should have communicated his misgivings about the models to IPCC, Congress and the media. The guy who has to translate crappy assumptions from higher higher into workable code should not have to field that kind of issue or get caught in the middle.

  20. There seems to be a correlation between government involvement and the lack of quality of developed software. In the UK, the Government underwrote the financing of an enormously complex health care system, supposedly to link all the medical records of every family doctor, every hospital into one central database. Although the development was carried out by the private sector, it has been to date nothing less than a complete disaster, and a waste of something like six billion pounds. Now go to the extreme and have the government not only finance the thing but develop it itself and no rational person can expect anything at all out the other end that in any way resembles its stated purpose.

  21. Just perusing that programming lauguage-it has been more than 30 years since I’ve seen FORTRAN is it me-do I see some sort of deliberate ‘loop’ if you will in the way it it written? I’m no programmer but have downloaded a couple of different wildlife studies
    in FORTRAN and got caught in a “garbage in garbage out” -GIGO scenario that dang near got the whole study shut down.-after 3 years of hard field work…
    This GISS thing makes me wonder…

  22. It is perplexing how models are called “experiments” but/and also used for “prediction”.
    If the model is understood, it is not experimental.
    If the model is experimental, it should not be relied on for prediction.
    I don’t really care how much money and time and effort and smarts went into developing the models. There comes a point where we have to decide whether to trust the results. Many projects take up a lot of time and effort but nonetheless, in the end they simply fail.
    We need to get honest about whether climate models are successes or failures.

  23. Looks like it may be more accurate to say that UCAR employees (at least some of them, software engineers may be more conscientious) are underworked and overpaid:
    “We also evaluated whether UCAR salaries and wages were properly and ac-curately charged to federal awards. The audit found that UCAR employees 1) were not recording all of their worked hours, 2) charged budgeted rather than actual hours, 3) earned and used unrecorded compensatory time (although UCAR does not officially allow compensatory time), and 4) inaccurately re-corded their time as worked when they were on leave. Furthermore, UCAR did not have detailed written justification to support over 80 percent of the sampled labor costs UCAR transferred between awards. Without a reliable basis of support, UCAR’s $58 million of labor costs charged to NSF and other federal agency awards are at risk of not being accurately allocated. UCAR needs to develop a timekeeping system to accommodate all hours worked by salaried employees, provide its employees specific guidance on timecard completion, and provide more oversight of accounting for leave and transferring of labor costs between awards.”
    http://www.nsf.gov/pubs/2008/oigmarch2008/oigmarch2008_2.pdf

  24. Gary Strand stepped forward to participate in discussions at Climate Audit, putting himself in the midst of hard questions and skeptical viewpoints. I for one applaud his courage and professionalism for doing so. However, I suspect he regrets wording his statements that way; but I suspect as well that the code is as he stated.
    We’ve all seen old code in our work environments that lead to a lot of head shaking and brow furrowing, so I’m hardly surprised by the admission. Lets hope that Mr. Strand isn’t drawn-and-quartered by his climate science colleagues over this.

  25. Stephen Skinner (07:59:34) :
    Reported in New Scientist 4 July
    Boeing has encountered some serious structural sicknesses in the new 787.
    Stress tests have revealed that the “wingbox” and 18 areas on the fuselage around the wings require strengthening.
    “Data from the test did not match our computer model,” says Boeing vice-president Scott Francher. That highlights the difficulty of predicting the behaviour of advanced CFRP materials being used in very large structures for the first time.”
    Boeing has it all wrong. Obviously the test data is wrong and must be adjusted to match the computer model!. Seriously, the interesting thing about this is that the limited number of variables for an airframe model are well defined within a well defined engineering discipline. Imagine the increased complexity in a climate model. Then take one small variable that is incorrectly defined and multiply it out over 100 years. Voila – garbage.
    My understanding is that GCM modelers will admit that water vapor and clouds are “not well understood”. How can you build a 100 year climate model when one of your key variables is “not well understood”. Imagine Boeing building a model where the tensile strength of the materials used is “not well understood”. Would you fly in it?

  26. My experience of Fortran coding from many years ago was that to make sense of it and for purposes of verification, somewhere between 20% and 50% of the code should be comments. Without having the time to look at the code, can anybody tell me how much of the code is made up of comments?

  27. I don’t understand why these guys even need software modeling code. The whole principle of AGW rests 99% on more C02, more temperature increase. Of course, you can’t have a linear progression, so just put in a few dips and then back up again.
    I think they just need an excel spread sheet, because for all of the little nuances they put into their code – good or bad – they’re just saying that as long as we’re pumping more c02 into the atmosphere, the warmer mother earth becomes.

  28. As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
    And why is anyone still coding in Fortran?

  29. Part of me wonders why anyone would care what the model says about 2106 – good code, bad code, erroneous code. Given what has transpired geographically and technologically in the last 100 years, the likelihood of any of the output being relevant is pretty slim.
    I get the question about integrity of the code processing, but really…
    I would hazard there’s a better than even chance you won’t be able to read that data in 100 years time, especially if its digitally archived. I suspect that we’ll be rooting around in the Smithsonian for printed copies to see what the fuss was (it’s a huge leap of faith that archival institutions like the Smithsonian will still exist in 100 years…). I’d be happy if the models could predict next year with certainty. 🙂

  30. “Rod Smith (09:48:05) :
    As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
    And why is anyone still coding in Fortran?”
    Errrm, income protection. I have to work with people who encrypt “scripts” to interface with Microsoft Systems Manamgement Server…but they are not too bright…

  31. In looking at the subject code, I realized its been nearly 3 decades since I wrote FORTRAN code. I switched to Pascal — which I still use for little fun projects (I’m retired).
    Since switching to Pascal decades ago, my goal has always been to make my code readable without need for comments. I did&do it with structure, long variable names, statements written and ordered to facilitate reading (versus clever), and so forth. I made my code read like pseudo-code. It worked, others were able to easily understand my Pascal code — even translate it into their favorite language when they were not knowledgeable about Pascal. Result: My mission critical stuff was really easy to debug for me and others. Consequently, it was in service sooner, cost effective, and easy to update/upgrade.
    Readability is what I miss in most code I see in blogs (seems like some take pleasure in getting as much work as possible out of every line and love to abbreviate variable names). The subject FORTRAN is not particularly readable — but neither is much other code I see.

  32. It’s a little harsh, IMHO, to assign blame to software engineers when they’re underpaid and overworked.

    Say what? You must be joking!
    First of all, in the case of people like Gavin Schmidt, they are NOT software engineers. One look at their code by any “real” software engineer clearly reveals this fact. They methods are so severely flawed, no respectable software engineer would ever put their name on these projects.
    Second, “underpaid”, “overworked” ? … Please, don’t even go there. I have been an active software engineer for almost 30 years. Let me tell you a thing or two about underpaid and overworked. I have spent the bulk of my career working 80-90 hour weeks, without vacations, most of the time without benefits or insurance, all to earn enough to survive. I presently make an “ok” salary with marginal benefits, but I have also been able to only take 2 vacation days in over a year and a half. So don’t even begin to tell me about overworked and underpaid. These idiots (so called “modelers”), operate on budgets many many times larger than anything I have been exposed to. Don’t tell me they don’t have money. They are operating on MY money!
    Sorry, that quote just pisses me off…

  33. Sorry, additiaonl, I sometimes run into environments who still use IBM’s REXX. Yes, true, it’s still working…and I actually met Mike Coulishaw (We worked in teh same bulding, while he was a visitor).

  34. Lessons From Chaos Theory
    Anyone who ponders the value and accuracy of computer models, especially in simulating immensely complex systems such as our earth’s climate, should revisit the story of Edward Lorenz and his Royal McBee LGP-30, the machine that he used in 1960-61 to attempt to model changing weather patterns and show the potential for using the computer in weather forecasting. His discovery that massive differences in forecast could derive from tiny differences in initial data led to him develop the mathematics behind this phenomena and coining the terms strange attractor and butterfly effect, core concepts in chaos theory.
    The story is told in the first chapter of the now well known and superbly written book, “Chaos: Making a New Science” by James Gleick – Published in 1987 by Penguin Books.
    From Chapter 1, “The Butterfly Effect”, pages 15, 16 and 17
    “With his primitive computer, Lorenz had boiled wither down to the barest skeleton.—-”
    “One day in the winter of 1961, wanting to examine one sequence at greater length, Lorenz took a shortcut. Instead of starting the whole run over, he started midway through. To give the machine its initial conditions, he typed the numbers straight from the earlier printout. Then he walked down the hall to get away from the noise and drink a cup of coffee. When he returned an hour later, he saw something unexpected, something that planted a seed for a new science. —–”
    “From nearly the same starting point, Edward Lorenz saw his computer weather produce patterns that grew farther and farther apart until all resemblance disappeared.—-”
    “Suddenly he realized the truth. There had been no malfunction. The problem lay in the numbers he had typed. In the computer’s memory, six decimal places were stored: .506127. On the printout, to save space, just three appeared: .506. Lorenz had entered the shorter, rounded-off numbers, assuming that the difference–one part in a thousand,–was inconsequential. – – – Yet in Lorenz’s particular system of equations, small errors proved catastrophic.—-”
    “But for reasons of mathematical intuition that his colleagues would begin to understand only later, Lorenz felt a jolt: something was philosophically out of joint. The practical import could be staggering. Although his equations were gross parodies of the earth’s weather, he had a faith that they captured the essence of the real atmosphere. That first day, he decided that long-range weather forecasting must be doomed.”
    For mankind to put their total faith in computer programs that try to simulate a system as chaotic as the earth’s climate and are so sensitive to precise initial conditions is an act of insanity.

  35. Y’all might be interested in discussions of computer type risks at the “Risks Digest” http://catless.ncl.ac.uk/Risks . Search for key words such as “climate models”, “global warming”, etc. and you will find that many of these issues of simulation software have been discussed for decades. The forum is run by Peter Neumann and was begun in 1984.

  36. As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
    A more appropriate description for our friends who write programs may be software code developer.

    As a working software engineer, I must reluctantly agree with you.
    There is very little science and too much art to writing software. “Software engineering” is unlike “real” engineering in that anybody can call themselves a “software engineer” while you have to graduate from an approved engineering school and then undertake a rigorous licensing process in order to become a “real” engineer.
    Still, I refer to myself as a “software engineer” because I realize there should be a process to developing quality software instead of blindly writing and revising code.
    BTW: GISTEMP shows that while anybody can write software, most people should not. The smarter somebody is, the harder it is for them to realize they are not qualified to do something.

  37. Many threads back, someone called this code “a crime against humanity”. I haven’t coded in FORTRAN since the 60’s, and it looks bad even by my low standards. Even so, the code isn’t as much of a problem as the model itself. You have hundreds of variables of unknown sensitivity, and very few proven equations. So you guess.
    Someone once said “Assumption is the mother of all …”
    (or you can determine the outcome in advance, and adjust the inputs)
    Back in the mid-80’s when PC’s were still overgrown toys, one of my engineers decided to model a product of mine using P-Spice, an electronic circuit analysis program. He determined that it had peak output voltages in excess of one gigavolt. As all of us were still alive and we had not seen lightning bolts flying across the lab, I suggested that his model needed some improvement. He dropped the project.
    As for the financial problems: very little to do with faulty models. A great deal to do with faulty intervention in the market. Loaning money to people who cannot pay it back doesn’t work. Clobbering the economy to reduce a trace gas that has little effect on climate doesn’t work.
    We (at least some) learned the money lesson the hard way. The trace gas lesson could be a whole lot more expensive.

  38. It’s no surprise that climate models are poor.
    1) The primary data sources they use are unreliable because of data accuracy problems, paucity of global data collection points and the very crude way that data gets used to provide global estimates.
    2) Climate is a chaotic non-linear system and developing a model which can provide really accurate future climate trends is an impossibility with the current data inputs and computer power available. Even something as basic as modelling the effects of clouds is an impossibility, despite these being one of the most important climate regulators. When it comes to modelling ocean currents and solar effects, the situation is even worse as we don’t even have good theoretical models of how they behave and what influence they have on our planet.
    3) Being a chaotic system, even the slightest error in the model will have an ever growing effect on the accuracy of the forecast as the years pass by.
    I think the people funding the productions of the various GCM’s which are around today are intelligent enough to realise the futility of the task of predicting climate ant the 100+yr level. Provided the results they receive give a ‘good’ fit to past climate and provide them with politically correct forecasts, then why would they invest large sums of money in this area.

  39. An old system no doubt with many, many revisions. All systems like this end up looking fairly bad. They need to redo it, but bad looking code does not necessarily mean bad working code.
    Until then, it will continue to be job security for the few who understand it.

  40. Alan, assumptions may be the bane of the pure sciences, but they are the bread and butter of engineers. The main classes in sophmore year supposedly teach basic mathematical concepts, but the main lessons is in reasonable and unreasonable assumptions. If you make unreasonable assumptions, you failed.
    There is too much code and not enough time for me to decrypt and understand it, but I think that this would be difficult even in best-practice-compliant Basic (the easiest of all languages to read and written in the most self-descriptive manner possible). While this sort of thing needs to be complicated, I cannot help but feel cheated that I cannot even find the array definitions in many cases, much less where they are set.
    One thing that irks me is that they are detailed enough to adjust for the changing weight of dry air in higher CO2, when an increase of CO2 to a full 1% (an incredible amount) is only 0.15 g/mol. This is less than half a percent of the standard estimate (28.8 g/mol). If this affects the model’s outcome at all, then the model is laughably oversensitive, because there is no way that you can get two significant figures of accuracy, much less three, on 90% of the valid inputs. It’s comparable to using a surgeon’s scalpel while blindfolded. A baseball bat is just as accurate.

  41. Rod Smith (09:48:05) :
    As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
    And why is anyone still coding in Fortran?

    I asked that question about FORTRAN previously and was roundly spanked by people who informed me that FORTRAN is still used in the engineering field and is also commonly used on supercomputers.
    I can only assume Fortran is equally prevalent in science as in engineering. I still question why a more modern, object-oriented language such as C++ (or even Java) isn’t used. Is FORTRAN mult-threaded? I’m also told some versions of FORTRAN have object-oriented extensions, but they aren’t commonly used.
    If software engineers are in short supply, that’s a sign that NASA needs to migrate to a more contemporary system. Last I heard, there were a lot of layoffs in the IT field. Surely there are some out-of-work computer science majors who would love to work for GISS?
    Does anybody know the hardware and OS GISSTEMP runs on?

  42. First, I’m a software developer with 22 years of post-college professional experience. I used to work for the Jet Propulsion Laboratory where I was responsible for maintaining some of the computer graphics software used by PIO for the Voyager and Galileo “flyby” animations, all written in FORTRAN.
    With all due respect to everyone hammering on the quality of the code and the software developers who created this code, the FORTRAN here looks very typical of most of the FORTRAN I encountered 20 years ago at JPL in a professional capacity.
    The problem with the code is twofold:
    (1) The code is old. You can tell from the varying comment styles and coding styles in various source files that it has been worked on by a large number of people over a long period of time. Typically code like this suffers from the “Lava Flow” anti-pattern: the next person to work on a chunk of code doesn’t know fully how things work, so they delicately add their work, making as little change as possible. Lower layers are “frozen” like old lava, with new code poured on top, and allowed to freeze itself.
    (2) The code doesn’t use modern coding techniques such as object oriented coding styles. This is typical of code from the 1980’s: OOP programming (and related techniques, such as using design patterns to keep code maintainable) are innovations that didn’t get wide-spread support in the programming community until the 1990’s. This means the code is hard to read and hard to maintain from current coding standards.
    (3) One would hope that there would be a separate document which outlines the equations, formulas and rational used in each section of the code. Unfortunately most programmers take the attitude that “code is self-documenting”, when it is no such thing. When I was at JPL I spent a considerable amount of time documenting the code I inherited so I could keep it straight in my mind–and if I were tasked to maintain this code, that would be the first thing I’d do. Ideally I would try to trace back each of the calculation methods back to the scientific paper or textbook which provided the underlying formula, so there is an understanding why I’m carrying out a particular calculation at a particular point.
    This sort of sloppy documentation is inexcusable, of course–but it is an extremely common problem in the software community.
    (4) It is a common lament amongst software developers to distrust what they don’t understand–especially any newly inherited code. I had a software developer undertake rewriting large chunks of code I wrote myself–telling me to my face that whoever wrote it was an idiot. (He didn’t know it was me. I let it go.) Thus, no-one working on 20-year-old code will ever say it’s good code, regardless of the quality of the code. And they’re all going to itch to rewrite the code–so they can understand it, not because the quality is necessarily bad.
    So I wouldn’t trust the comments of the developers who say the code quality is bad. It looks to me to be typical Fortran code from the 1980’s.
    I’d say the real problem here is twofold: (1) The code is not properly documented with references to appropriate scientific texts. And as a consequence (2) we are over-reliant on the predictive model of a chunk of code no-one really understands.
    But code quality: having worked as long in the software industry as I have, code quality is a constant lament–and really means “I haven’t figured it out.”

  43. I found this interesting recent news item on models vs reality at http://www.dailyfinance.com/2009/06/24/is-boeing-s-787-safe-to-fly/
    Think of the billions Boeing has spent on their models, and they still don’t aren’t perfect!
    “Specifically, Boeing found that portions of the airframe — those where the top of the wings join the fuselage — experienced greater strain than computer models had predicted. Boeing could take months to fix the 787 design, run more ground tests and adjust computer models to better reflect reality.”
    If realclimatescientists ran Boeing, they would defend the “models”, built the plane, crash dozens, kill thousands, bankrupt Boeing and blame the skeptics!

  44. Yet another software engineer here …
    “Keep in mind that the errors could be impacting results both ways, although, given the GISS track record, I doubt it would error on the cool side ;)”
    That’s a way bias can creep into any software that produces numbers (eg modelling software) where people have pre-determined expectations for the numbers. Bugs that cause the numbers to differ from the expected result are spotted and fixed. Other errors … well not so much.

  45. John F. Hultquist (08:21:57) :
    Good post and useful critique, I think.
    But, I first did FORTRAN in 1965 (version II-D) and am quite happy to take your word that these models are not great examples of clarity. I’ve seen all the “do loops” and “if – then” statements I ever want to see.
    That being said: Know that I appreciate those of you that are looking at these things and I do trust your judgment.
    ,,,,,,,,I wrote Economics modeling progrrams in the early 70’s on fortran
    S Bridges (08:55:35) :
    As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers.
    software engineers? They are not engineers. Most have no degree.
    I really like this thread however since there are quite a few of us that used fortran. Working for the government and using fortran is kinda 30 years behind the times.
    Gary, still using key punch cards?

  46. AGW has never been about the science. It has been about the social power.
    The use of climate science is only incidental. Now that science is ‘settled’, the AGW community could give a fig about the terrible code that was used to sell the politics.

  47. Oh, and like John Galt, I agree completely with RS Bridges that most of today’s “Software Engineers” are really just coders, I’m one of them.
    True Software Engineers do exist but they’re developing stuff like flight control systems for aircraft etc, and they’re not using Fortran.

  48. Of course, all this Earth climate model talk is premature – you first have create a model of the Sun that works properly 🙂

  49. Dr. Robert M. Carter characterizes three climate change realities:
    Climate change knows three realities: science reality, which is what working scientists deal with every day; virtual reality, which is the wholly imaginary world inside computer climate models; and public reality, which is the socio-political system within which politicians, business people and the general citizenry work.
    ————————————————————-
    Computer models are not reality. Nature is reality. Political reality is a public display of ineptitude in dealing with reality.
    Which of these realities is most conducive to your life, liberty and pursuit of happiness?

  50. Jimmy Haigh (07:45:46) :
    “…software engineers when they’re underpaid and overworked.”
    “Why not look for another job?”
    Probably because it’s nonsense. From 2003:
    “Boulder labs: CIRES, NIST, NOAA, NTIA, UCAR, NCAR (JILA, LASP)
    Employ 2,566 in Boulder; average salary $64,350
    26% higher than overall average $51,220”
    http://slochamber.org/Library/3A2.pdf
    But I will be one that blames the software engineers for compromised products and predictions that affect everyone’s lives, *especially* when one complains that underpayment and overwork is the problem.

  51. Jeff Id says..
    ” I’ve learned programming in my spare time – it ain’t rocket science.”
    —–
    This is the same attitude that the climate modellers have and what leads to these low quality software solutions being created in the first place. Learning a high level language is easy, learning methodology and creating flexible and well documented code is much harder to master.
    I can build a deck or a small structure with a set of stairs but I am no carpenter and would never try and build a house that would meet building codes.
    As a Software Developer and Networking and IT Consultant I can simply ask those among us in this field to validate the following.
    1) Cleaning up an IT Infrastructure maintained by a “learned in my spare time” in-house Network and Systems “Guru” is a waste of time. Start from scratch is faster than fixing it.
    2) In house written and maintained software is functionally useless to an outside consultant when developed or maintained by a “this is not rocket science” spare time programmer. Again starting from scratch is cheaper than deciphering and fixing it on a cost basis.
    I have made a good living and built a company off these types of situations, so I hope Jeff Id and everyone like him keeps on keeping on because he is the reason my phone rings.

  52. Doug Ferguson (10:10:36) :
    “Chaos: Making a New Science” by James Gleick – Published in 1987 by Penguin Books.

    Great book. Recommend it highly. After your jogging of my ancient memory I pulled it out to re-read.

  53. George Tobin: “The guy who has to translate crappy assumptions…into workable code should not have to field that kind of issue or get caught in the middle.”
    I submit that the Montana farm boy who enlists in the Army should not have to get caught in the middle of an ideological war and lose his life. The farm boy, fortunately, disagrees and defends the country according to his own lights.
    So it would seem that a programmer, who is aware that software which doesn’t meet the best standards available is assisting the mendacious in destroying the world economy, has an obligation to field any issue and get caught in whatever positions are necessary to alert the world. It probably won’t be fatal to him.
    Hubristic “scientists” initiated what has become a fight for the survival of viable economies and scientists must enlist in this fight right now.

  54. R.S. Bridges wrote: “As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.”
    It’s a heck of a linguistic promotion from “Code Monkey”.

  55. R S Bridges (08:55:35) :
    As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.

    I take issue with this. I am most definitely a software “engineer”. I don’t believe you have a firm grasp of the roles that some of us of this profession fulfill.

    From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.”

    This pretty well sums up my job description, and what I have done for almost 30 years. I am not sure what else you would call my profession other than “software engineer”.

    A more appropriate description for our friends who write programs may be software code developer.

    Perhaps, for those of you that have no clue what some of us do.

  56. “…in business, if something doesn’t work, or doesn’t work well, contracts get lost and/or people get fired. There’s consequences to shoddy work.
    In academia, the solution is usually to ask for more grant money.”

    And in government the solution is to increase an agency’s budget and call for more power.

  57. Well, some of you may have guessed what the “Code” in my name stands for, and forgive me but my specialty is now being stepped on.
    No, coding is not “rocket science”, but coding WELL is.
    Anyone involved in this has had to deal with poorly written or poorly laid out or poorly designed code. Anyone can code… yes, but very few can code well.
    I previously had a contract to “fix” a project that was 5 years in development using Visual Basic. It took me several months to even figure out what the original author was trying to do, and a few weeks to completely rewrite it properly. They have since hired several other contractors to add function to what I wrote, and none of them have had any problems with it. There is a difference between someone writing code using established methodologies and someone who taught themselves and gets mired in an oversized project.
    I also downloaded Model E and was amazed at how poorly written it was. My guess is that it evolved directly from the “proof of concept” phase and was never restarted, reorganized, or reanalyzed, all three of which are essential when your program grows past a certain point.
    About models in general, there is nothing wrong with the concept of modeling, as long as you understand that a model is ONLY AS GOOD AS ITS ASSUMPTIONS. For example, the Boeing model was perfectly sufficient for a static design, however one or more of their assumptions was incorrect (material strength? structural load during flight?) As these kinds of problems are encountered and fixed, future models become that much more accurate. If there is EVER a time when you think your model is completely accurate, you will have crossed the line into a fantasy world, since models are only representative, never definitive.
    The fundamental, critical flaw in the AGW crowd depending on models is that their models do not, and cannot, effectively model a system as complex as they are attempting to model. We have documented here a few important factors that they apparently didn’t think of. The illusion that a complete climate model is built, or even can be built, is actually very entertaining to me.

  58. I’m definitely NOT a software engineer, and only program’d a bit of PASCAL in the day, but I have a descent grasp of what is meant by “poor coding”. However, it would be nice if someone could give examples of poor coding and show how it affects the outcomes to modeling programs. It would be nice to actually see example of where the lack of “skill” (a Mannian term) affects the outcome of these climate models.

  59. An extension:
    While there are a lot of people out there that are “coders”, some of us are not. Some of the things I design and develop include application models, data models and cryptographic systems. Additionally, these systems integrate with other systems using a wide variety of mechanisms, transfer agents and methodologies. Further, I develop source control management processes, procedures and methodologies to manage these projects, from design, coding, distribution, maintenance, auditing, logging, reporting, alerting, change-release, peer-review, integration, systems management, infrastructure integration, the whole spectrum.
    Perhaps I am taking issue with the slam on “software engineer” because my background did not stem from an MIS or such degree, but from a Bachelor degree in Computer Science, mathematics minor, physics minor (although I don’t use the math’s as much anymore). Mine is a scientific approach from an engineering perspective of software development.
    So, be careful when you round us all up and slam us into a box. True, we are not all engineers, but some of us really are…

  60. I have been a professional computer programmer for over 30 years in the insurance industry. Similar to this situation, we have Actuary’s that write their own software solutions to complex insurance problems. And similar to this situation, it can result in software that no one else can understand at a glance. I have at times been called in to repair/support such actuarial software. It is a challenge to do so, but it is what I do. From these experiences I have a respect for the need of the Actuary’s to experiment with their software in order to learn more about their own models. When a programmer is added to the R&D mix, that programmer must become a subject expert. And in my opinion, not a very high percentage of programmers are capable of PhD level work.
    The idea of requiring that every piece of software must be written only by programmers would certainly be the utopia solution. However imposing such a requirement on mankind’s computing needs would greatly harm our productivity and ability to learn by experimentation. In my case for instance, my managers have my workload already laid out for me 2 years in advance. Yes, I already know what projects I’m supposed to be working on in 2011. This is because talented programmers are in fact in short supply when you consider the work that is ahead of us.
    And I concur that we are not “engineers”. The state of Oregon has made it illegal to misuse the term (but they never enforce it). Doing so belittles the accomplishments of those that have been certified by the State as being actual engineers.
    Pete

  61. Rod Smith (09:48:05) :
    As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
    And why is anyone still coding in Fortran?

    Because it’s still a very good language for M&S in science and engineering.
    And because it’s incredibly wonderful for concealing slip shod science.

  62. As I noted in the thread at Climate Audit, my FOIA for the top level software quality documents for GISTEMP and GCM Model E found no responsive documents.
    NASA/GISS has not even considered where these projects fit within NASA quality standards.

  63. I recall reading on the web an interview between some climate modelers, and I was struck that they were talking openly and consciously about their models’ flaws. I realised that they are smart enough and professional enough to be perfectly aware of the limitations of their models.
    So our disagreement with them is less about quality assurance issues (“yes there is room for improvement” they might say), and more about the attitude towards uncertainty.
    For the person in the street, the model results are highly inconvenient. They are a problem for society. But for the people creating the models, the results are very convenient. They raise the status of the field. More than once I’ve read climate scientists refer to their work as “very important”. And isn’t this what any scientist wants to do; make important discoveries?
    As climate models appear to have little predictive skill, perhaps we need a new academic field of ecology, one that uses different methodologies, ones which are both more powerful, useful, and which render climate modeling obsolete.

  64. Pete W (11:48:04) :
    ..
    And in my opinion, not a very high percentage of programmers are capable of PhD level work.

    I would certainly agree with you on that point. The percentage of top level programmers and/or software engineers (and especially the latter), is relatively small compared to the total volume of personnel involved in software development as a whole.

    ..
    Yes, I already know what projects I’m supposed to be working on in 2011. This is because talented programmers are in fact in short supply when you consider the work that is ahead of us.

    We operate similarly where I work. We have specific future goals laid out 2 to 3 years in advance. Because of our limited resources, we must engage in a lot of future planning, specific tasking and engineering.

    ..
    And I concur that we are not “engineers”. The state of Oregon has made it illegal to misuse the term (but they never enforce it). Doing so belittles the accomplishments of those that have been certified by the State as being actual engineers.

    In my role, I would disagree with this statement. My job roles push me far beyond merely “coding”. Specifically, one of my roles is to “engineer” new application models, scientific research, validation, implementation. These tasks move me far beyond simple software development roles. The developers (we can them “developers”), that work for me, are NOT software engineers as they are not intimately involved in the “engineering” aspects of our software development projects. Whereas, one of my roles is specifically to “engineer” these applications, often times employing cutting edge architectural techniques and unique systems integrations. We have “software developers/coders” here, but we also have “software engineers” here as well, I am one of those engineers.
    I think the disconnect here is, most people do not fully understand the world of computer processing and that the subject encompasses. There are times when that world is very vast and requires intimate knowledge and discovery on a vast variety of technical disciplines (ie: engineering).
    Pete

  65. Sonicfrog (11:41:12) :
    I’m definitely NOT a software engineer, and only program’d a bit of PASCAL in the day, but I have a descent grasp of what is meant by “poor coding”. However, it would be nice if someone could give examples of poor coding and show how it affects the outcomes to modeling programs. It would be nice to actually see example of where the lack of “skill” (a Mannian term) affects the outcome of these climate models.

    Poor coding affects the output because it’s impossible to read the code and follow the logic. In other words, you can’t verify the model was coded correctly.
    One aspect of software engineering is unit testing (see http://en.wikipedia.org/wiki/Unit_testing).
    Unit testing helps determine the model works as designed but does not help determine whether the model is designed correctly.
    The only way to verify the output of these models is to compare the results against the observed climate. Unless you’re grading on the curve, I don’t see that the models hold up really well.
    And how do you verify the output of climate models that project the climate for the year 2099? You have to wait until the year 2100 to validate the model; much less time is needed to invalidate.

  66. In modelling when there are unknowns, we are often conservative to cover our asses, but we also have to bear in mind we don’t want to over design or we will cost our client extra $, we have to stay within reasonable bounds but allow for the worst that might be expected. I.e. if designing a reservoir, we have to look at the worst storm possible and assume the catchment is nice and saturated, but we dont design to a storm that is improbable as that would waste money. We have to carefully assess the suitable storm and ensure it isnt unrealistic.
    I wander if climate modellors put a limit on how conservative their assumptions are?
    I mean, clouds causing warming, 1/3 ocean evaporation, 20kph glaciers, co2 warming the sea etc… etc… when these all come together do they represent a highly unlikely outcome or a realistic outcome?
    Personally I like the Idso equation as its the only one empirically derived.

  67. When it comes to GCM’s (general circulation models), I really believe that “software engineers”, along with software developers, is precisely what is necessary if one wants to truly attempt to computer model something as complex as our climate. The task of developing software so complex as this, ultimately demands not just software developers, or coders, but in fact real “software engineers”. Software engineers to determine what technologies employ, determine how those technologies will be employed, determine every aspect of the project design, determine who will play what role in and at what stages, how development processes and methodologies will be utilized when and by whom, all of these things that are required to successfully develop a software product and see it through its SDLC (software development life cycle).
    It seems to me that software development project such as this would likely demand new architectural application approaches. Develop and utilize new design patterns, new application modeling techniques, the things that “software engineers” do. To me, a climate model just screams of cutting edge technology approaches such as this. Old-school FORTRAN just doesn’t seem appropriate in this case. Its like trying to build a rocket to Mars using pixy stix. You might be able to do it, but the likelihood of success or desired results, would be rather low.

  68. As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”.
    I came up through the ranks in the 70s when quite a few coders were also hardware engineers. I was one of them.
    I do take exception to the statement that “software engineering” is not real engineering. Ask the aerospace guys (me, me) or the medical eqpt. guys if that is true. Degrees and professional exams are not the measure of all things engineering. Otherwise it would have been a University Professor and not James Watt who brought the steam engine into the modern age.
    BTW this hardware/software engineer (non-degreed) had a government inspector look over his code and the inspector said it was the best written code he had seen in years. My secret? The naming of parts. If you get the names of things right the code reads almost like English and even a non-coder should be able to get the gist of it. I did do my work in FORTH which is a great help in making the code readable. But if you are not a disciplined software engineer FORTH makes it much easier to do unreadable code. So it cuts both ways depending on the skill and discipline of the engineer.
    We have the example of hardware engineers. You give them a pile of steel and other metals, etc. and they will build you an engine. However, there are no restrictions on how the materials can be used. It is up to the engineer to make the right choices. In “modern” coding languages the effort is to make the compilers and tools cover for (prevent) errors. Such restrictions also prevent you from doing a lot of useful things easily or at all.
    All very nice if you are going to employ a bunch of mediocre coders/engineers. But you then wind up with a bunch of mediocre code. In everything. Just think of how long it takes Windoze to load.
    So my point – if you don’t trust your software engineers the answer is not training wheels. It is better engineers. There is no substitute for talent and discipline.
    BTW PHDs can code well if taught the discipline. If they are apt students it only takes a couple of weeks to train them. If they are not apt they shouldn’t be coding.

  69. @R S Bridges (08:55:35) :
    “As one with a chemical engineering degree, I really despise it when people decribe themselves as a “software engineer”. They are no more an engineer then the guys who pick up my garbage, who are also known as sanitation engineers. From Wikipedia, engineering is defined as “The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.” Last time I looked, “software engineers” do not have to take an engineer-in training (EIT) exam or a professional engineering exam.
    A more appropriate description for our friends who write programs may be software code developer.”

    Hear, hear! Another definition of Engineer, from California’s Business and Professions Code 6701: “Professional engineer,” within the meaning and intent of this act, refers to a person engaged in the professional practice of
    rendering service or creative work requiring education, training and
    experience in engineering sciences and the application of special
    knowledge of the mathematical, physical and engineering sciences in
    such professional or creative work as consultation, investigation,
    evaluation, planning or design of public or private utilities,
    structures, machines, processes, circuits, buildings, equipment or
    projects, and supervision of construction for the purpose of securing
    compliance with specifications and design for any such work.”

  70. Rod Smith (09:48:05) :
    As an old – very old – programmer who has many times been forced by Federal contract requirements to code AND DOCUMENT to Federal Standards, some of which were (to use a polite word) absurd, why is it that Federal Employees don’t have to meet these same standards?
    And why is anyone still coding in Fortran?

    All the FORTRAN code makes the output appear to be ‘scientific.’

  71. Unfortunately most programmers take the attitude that “code is self-documenting”, when it is no such thing.
    Well written code IS mostly self documenting. It all depends on choosing your words very carefully.
    The government inspector who looked at my code and liked it had no complaints about the lack of commenting (there wasn’t a lot of it) because the code WAS self documenting. That was back in ’79 and I WAS using techniques that did not become widely used in programming for another decade and a half.
    All this from a non-degreed (electrical/electronic) aerospace hardware engineer who went into coding out of necessity. So why was my stuff well above average for the time period involved? I cared. I was not just interested in “it works”. I was also interested in elegance and beauty.
    BTW I don’t think you can be a good coder/systems engineer without a deep understanding of the underlying hardware. Also a deep understanding of the domain is essential (if only for picking good names for things). Now a days most coders are fairly ignorant of the hardware they are working on and even more so of the domains they work in.

  72. Roger Sowell (13:15:05) :,
    You are talking about the LEGAL definition of engineer not the practical definition. A confusion of domains.

  73. Roger Sowell (13:15:05) :
    Hear, hear! Another definition of Engineer, from California’s Business and Professions Code 6701: “Professional engineer,” within the meaning and intent of this act, refers to a person engaged in the professional practice of
    rendering service or creative work requiring education, training and
    experience in engineering sciences and the application of special
    knowledge of the mathematical, physical and engineering sciences in
    such professional or creative work as consultation, investigation,
    evaluation, planning or design of public or private utilities,
    structures, machines, processes, circuits, buildings, equipment or
    projects, and supervision of construction for the purpose of securing
    compliance with specifications and design for any such work.”

    Again, just describes my professional role as a “software engineer”.

  74. >> Mac (08:26:46) :
    I’ve written my own NASA-like climate model software
    10 Rem Global Climate Model
    20 Print “The End Is Nigh”
    30 Goto 20
    40 End (of planet) <<
    This is the only example of code in the entire thread. The “End” statement is usually a compiler directive and not an executable statement. However, most compilers will execute a return at this point (and not terminate or stop the machine). A main program will return to the operating system. Statement “30” is an example of an infinite loop–you can never reach statement “40” or exit the loop. It seems appropriate. No matter how much hype we hear about global warming, we will never leave that infinite loop about the coming doom and never reach the “End.”
    Jim

  75. Too much value is assigned here to “coding quality”. That may be an issue concerning costs of maintenance and further development of the application. But it is not related to the OUTCOME or calculated result of the system, or quality of the results.
    What interests us is – if the calculations are correct. Code quality is irrelevant. You must always check results against some independent benchmark – like say – measured observations. Whether the code styling is nice or not, does not matter. If you cannot test the result in an independent manner, you can never debug the code and make it work correctly.
    No amount of “good” software engineering can guarantee that the results are correct. It is impossible to develop and debug code if you have no independent yardstick to evaluate the results by.

  76. M. Simon (13:04:11) :
    ..
    BTW PHDs can code well if taught the discipline. If they are apt students it only takes a couple of weeks to train them. If they are not apt they shouldn’t be coding.

    A couple of weeks huh? Perhaps back in the 70’s.. it seems you may not have much experience with modern day development processes. It would take a couple of weeks just for them to learn the source control management aspects alone. I would be willing to bet that not one of those individuals would be capable of coding on my current project within a year, perhaps even two.

  77. As a working software engineer/consultant, I can tell you that poor code is the norm. If more programmers knew what they were doing, I would have to find other work.
    Software engineers do not have a professional licensing requirement. That means the quality of their work is all over the board. Anybody can claim the title of ‘software engineer.’ In many companies, it’s the job title that replaces ‘programmer/analyst.’
    Having a degree in Software Engineering or Computer Science is not a guarantee of the quality of a person’s code. I have met many self-taught programmers who write excellent code and many degreed professionals who just copy and paste and patch their code after it’s put into production.
    The difference is in the attitude and aptitude. A person who takes pride in their work strives to write good code. They want to improve and when they don’t know how to do something, they learn it instead of throwing out one bad hack after another.

  78. Take the example of Boeing. They had a nice software model, that certainly helped a whole lot in the design process. Still the result had to be tested in a prototype, where actual stress measurements were made. An error was found that could not have been discovered by “good” software engineering, only by actual measurements. The software will be corrected, and be more useful in the future. Software that cannot be tested cannot be developed to a high degree of reliability. Good coding practices alone are never sufficient.

  79. I also qualify as an engineer (chemical) who was required to learn and then write software in FORTRAN. My field was chemical plants and petroleum refineries, in which we wrote some very complex code. This code was done well, and was robust, and was subject to annual updates / revisions, and was sold as a commercial product throughout the world. Nothing magic in this, we simply understood our field, and FORTRAN. The code had to execute on machines in Japan, Europe, Canada, the US, South Africa, and the Middle East.
    After years of writing / debugging / enhancing the code ourselves, we hired a computer science major and had him do the programming. We spent more time explaining things to him than if we had simply done the job ourselves. Management thought we were so progressive, though, in having a professional code-writer. It also enhanced our marketing when we could say the code was professionally written.
    Someone earlier asked why do people still code in FORTRAN, and this reminds me of a similar thread earlier on WUWT. Because it is an excellent language for engineering. There are literally millions of lines of excellent FORTRAN operating every day in engineering applications. No one in their right mind would consider re-writing all that perfectly good code.

  80. Someone said: “The model results reflect the assumptions of the authors of the model”.
    That would be true if there were no errors (bugs) in the code.
    As it is – the models reflect nothing. Their result cannot be treated as anything but random, since we have no independent way of authenticating them, therefore we have no way of debugging the code.

  81. M. Simon (13:04:11) :
    I’m curious Do you think your definition of “software engineer” fits with what the needs are for building a climate model? I think it may, since it requires much more than a coder, which is why the scientists probably decided to do it themselves in the first place anyway.
    By your definition, I believe I would be called an engineer. But we will have to disagree on the term software engineer. This company has folks that do everything you mention, and none of them are called software engineer.
    Pete

  82. With all due respect to scientists and engineers – leave the coding to the professionals. We set up “sandboxes” where we let the PhD’s play where they wouldn’t hurt anyone.
    I was a professional (got paid to do it) programmer during the mid 1980s when a lot of this code was written. I would say ModelE is not up to industry standards of even the 1980s. Our code was much better documented.
    This problem reminds me of the Therac-25 software incident:
    http://en.wikipedia.org/wiki/Therac-25 that got a lot of play in the IT trade press of the day. A radiation therapy machine somehow dispatched lethal doses of radiation because of a software glitch caused by a certain set of operator instructions.

  83. 10 Rem Global Climate Model
    20 Print “The End Is Nigh”
    30 Goto 20
    40 End (of planet) <<

    Better program
    10 REM Global Climate Model
    20 ON ERROR GO TO 40
    30 PRINT AT INT (RND * 21),INT (RND*32);”The End is Nigh”
    40 GO TO 30
    But the RND function is no good since it repeats itself in about 15 times, better take the value of the random SEED generator directly with the aid of a PEEK command.
    10 REM Global Climate Model
    20 ON ERROR GO TO 40
    30 PRINT AT INT (PEEK (16434)*21),INT (PEEK (16434)*32);”The End is Nigh”
    40 GO TO 30
    Oh my god how long has it been, to long i guess and one thing comes up, that even small and relatively simple program’s do require some debugging, good software takes time, but even then if you feed it with garbage data, it will result in output that is still garbage.

  84. Real engineers
    There seems to be a misunderstanding regarding engineers. I use headhunters to hire people and they aren’t confused. Engineering degress are granted by schools of engineering. None of this M.S. in Information systems from church schools. You have an engineering degree from an engineering school or you do not. Microsoft and some other vendors claim to produce them. Nope. Not the real thing. (I cut slack for railroad engineers, they have had that handle and don;’t hold themselves out to have a real engineering degree.) Head hunters also tell me if it is a compromised degree such as B.S. In Engineering Technology. Not as much as a real engineering degree. This excludes the DeVry Universities and some BS computor science degrees. There are real computor engineering Degrees. There are several hundred Engineering School campuses.

  85. Sounds like the next project after SurfaceStations wraps up.
    Never! Today USHCN, tomorrow GHCN!

  86. “Just perusing that programming lauguage-it has been more than 30 years since I’ve seen FORTRAN is it me-do I see some sort of deliberate ‘loop’ if you will in the way it it written? I’m no programmer but have downloaded a couple of different wildlife studies
    in FORTRAN and got caught in a “garbage in garbage out” -GIGO scenario that dang near got the whole study shut down.-after 3 years of hard field work…
    This GISS thing makes me wonder…”
    And it feels
    Like I been here before
    Feels . . .
    Like I been here before
    And you know
    IT [sic] makes me wonder
    What’s goin’ on
    Under
    The ground

  87. The following, from The Future of Everything [originally entitled Apollo’s Arrow: The Science of Prediction and the Future of Everything] by David Orrell, pg. 324), WRT climate models in particular, is apropos relative to global climate models (CGMs) and Gary Strand’s contention at http://www.climateaudit.org/?p=6495 that it’s the science not the software development:
    “Einstein’s theory of relativity was accepted not because a committee agreed that it was a very sensible model, but because its predictions, most of which were highly counterintuitive, could be experimentally verified. Modern GCMs have no such objective claim to validity, because they cannot predict the weather over any relevant time scale. Many of their parameters are invented and adjusted to approximate past climate patterns. Even if this is done using mathematical procedures, the process is no less subjective because the goals and assumptions are those of the model builders. Their projections into the future—especially when combined with the output of economic models—are therefore a kind of fiction. The problem with the models is not that they are subjective or objective—there is nothing wrong with a good story, or an informed and honestly argued opinion. It is that they (GCMs) are couched in the language of mathematics and probabilities; subjectivity masquerading as objectivity. Like the Wizard of Oz, they are a bit of a sham.”

  88. The hardest part of understanding the models will be the physics in them rather than whatever programming language they are written in. I suspect therefore primary focus is on that and the programming is secondary.

  89. Bobn (16:36:30) : I agree the physics is more important – the adjustable params might be more important in a negative way 🙂
    But a well written program is easier to understand, easier to test, and easier to modify without causing unintended consequences. The bigger the program, the easier it is to screw it up … well … assuming is was ever in an un-screwed-up state. Apparently the climate models never were.

  90. henrychance (15:17:41) :
    Thank you for the clarification/education. So there is an actual degree in “software engineering”. Good to know. I’m just ruffled by people that call themselves engineers when they aren’t. M. Simon quite likely is one.
    Bobn (16:36:30) :
    I agree.
    In addition to being complex the climatology models are – and are supposed to be – constantly changing as they learn more of the natural forcings at play. To suggest they should be required to stop now and re-write the code from scratch is… shall we say… not very reasonable.
    The models they are working with appear to have been proven fairly accurate over the last 20 years. This video contains some historical perspective of climate models;

    Pete

  91. I had to take a FORTRAN 90 class while I was studying for my Meteorology degree at Texas A&M. I had to write 3 fairly simple program in FORTRAN 90. It took me most of my study time during the week to write the programs weeks they were due. I was just trying to work out all the bugs in the program. This was the reason I decided not to go into grad school because I did not want to deal with FORTRAN 90 anymore. Once I was finished with the class, I did not write anymore program in FORTRAN 90. I did not like the syntax with that program language. I did write a program for automating the trading for the FOREX market a few weeks and that took two days to complete. A program with same length in FORTRAN 90 took a week to complete because of the debugging.
    I did some modeling as student at Texas A&M, but never dealt with the actual code of FORTRAN. I used the MM5 model to model the weather condition around the Houston area in the summer of 2000. I spent a 2 weeks just reading how to set up the model and run it properly. I then spent about a month just looking at how the model preformed during that time period. It something did not look right or was way off, we made adjustments to the model and reran the model.
    As a Meteoroloist, I learned to look at the model as a tool. If a model did not perform well, I did not use that model or used as a base for me to make my forecast. I learn that skill quite well thanks to many more seasoned Meteorologist who taught me look at the model in that way. I bet that is something none of the people who work on these climate model never learn that skill, looking at the model and learning when the model is not performing at an acceptable level.

  92. A plumber comments:
    It seems that an engineer is in search of reality, while the climate model programmer is in search of continued funding.
    Real data in = future warming = continued funding
    Real data in = future equivocal = new job opening
    Of course I could be wrong…
    Mike Bryant

  93. If “the models” are “fairly accurate” why are they “constantly changing”?
    Is it because of their proven “accuracy” that someone feels the need to “constantly change” them?
    Andrew

  94. My two Baht’s worth. I’ve worked with computers all my life as a field engineer (gorified mechanic for the most part) and as a code developer who knew the hardware side. My experience has been that the programmer is given specific parameters the code is to work within, and away he/she goes. Personally I have never written a piece of “clean code” on the first try. Just keeping all the “if this then that” routines functioning together is challenging, especially on a tight schedule.
    The programmer finally makes the original “assumptions” work as directed, then someone says they actually want something else because of unexpected developments. This is a programmers worst nightmare and can impact thousands of lines of code, not to mention the new coding required for the “unexpected developments”. Computer coding is a dangerous thing if it changes on a regular basis. I once developed a program to track a billion dollar project. Actually the original assignment was easy, even with the tight deadline. As the project developed, management changed their wants and needs and they always wanted it tomorrow. By the time the project ended, I had a bloated program that worked but had become such a nightmare that no one but me would ever be able to follow my logic. Ugly.
    I actually feel sorry for the guys who are driven like slaves to accommodate the “climate scientists” whims. It has to be frustrating.
    Just my opinion.
    Jesse

  95. John Balttutis (15:30:18) :
    John(above) seems to be one of the few commentators here that have focused on the science and not the programming. (Lot’s of touchy programers out there!)That was the whole point of my story on Edward Lorenz above. Good or bad programing(providing it works at all) has more to do with maintaining it and modifiying it(the software model) than it does with whether or not it is a good model. It’s the assumptions on which the model is built is what matters. Lorenz sensed and later showed that chaotic systems like our climate and any other turbulant system are impossible to model for any long range forecasting or predicting because they are not systematic and never really repeat their cycles exactly. Until we can accurately model clouds and cloud behavior, we don’t have a chance of modeling our whole climate with it’s interaction with the earth’s ecosystems.

  96. The climate models do not magically produce results. The problem is not code-related.
    They are designed to follow global warming theory on how temperatures increase as GHGs increase. Different modelers will have slightly different assumptions about the temp impact or build in longer lag times or incorporate different assumptions about feedbacks like water vapour or have different assumptions about Aerosols.
    But the climate models are just weather models, with a LN (GHGs) module and then some random white noise processes built-in. They all are building toward +3.0C by 2100 give or take and a simple spreadsheet would work just as well.
    It is not a question of whether the code is done properly, it is a question of whether the theory is right is the first place. So far, the theory needs lots of little plugs to plug the Watt leakages.

  97. Sorry about the mispelled words in the previous post. I just rushed it out without the engineer’s handy tool, Spellcheck!

  98. ON HOW CLIMATE MODELS OVERSTATE GLOBAL WARMING
    Edited from a previous post:
    Allan M R MacRae (12:54:27) :
    “There are actual measurements by Hoyt and others that show NO trends in atmospheric aerosols, but volcanic events are clearly evident.”
    But increased atmospheric CO2 is NOT a significant driver of global warming – that much is obvious by now.
    What has that to do with aerosols?
    **************************
    The Sensitivity of global temperature to increased atmospheric CO2 is so small as to be inconsequential – much less than 1 degree C for a doubling of atmospheric CO2.
    Climate models assume a much higher Sensitivity, by assuming that CO2 feedbacks are positive, when in fact there is strong evidence that these feedbacks are negative.
    Climate model hindcasting fails unless false aerosol data is used to “cook” the model results.
    Connecting the dots:
    The false aerosol data allows climate model hindcasting to appear credible while assuming a false high Sensitivity of global temperature to atmospheric CO2.
    The false high Sensitivity is then used to forecast catastrophic humanmade global warming (the results of the “cooked” climate models).
    What happens if the false aerosol data is not used?
    No false aerosol data > no credible model hindcasting > no false high climate Sensitivity to CO2 > no model forecasting of catastrophic humanmade global warming.
    Regards, Allan
    Supporting P.S.:
    Earth is cooling, not warming. Pass it on…

  99. I am not a software engineer.
    I am an engineer. I have both mechanical and electrical degrees with many years of dealing with chemists and chemical engineer PHDs in research.
    Software is a necessary evil. Make no mistake about it it is evil.
    That said I have written software for 30+ years. It is evil. I started with Fortran IV in 1967. I make no claim to be an expert but I do have some small degree of competence and a physical understanding of phenomena that most folks do not possess.
    There are coders, developers, software engineers, and programmers behind every bush. Competent ones are very difficult to find. The rest are just code pumps.
    Regarding the disparaging comment about someone being self taught, Newton was self taught.
    A lot of this code is indeed ancient. It does have the lava flow signature and appears to be patched in a random sort of way. (I guess they were afraid to touch it after they got it to run.)
    While it is not undocumented I would have preferred it to be a little more heavily commented.
    While not completely horrible ( what I have looked at), it is fairly painful.
    The long and short of my brief analysis is that given a monster set of initial conditions, this thing will chew on it for a period of time and regurgitate factored data based on their assumptions of the way the world works. Therein lies the rub, as it would seem to me that these guys haven’t figured that part out yet. (really nobody else has either but a lot of their assumptions are faulty.)
    If these people really wanted to have a good or even better than they have model they would throw the flow chart for this beast out there and let everyone have at it. Let everyone pick it apart and accept reasonable suggestions then code from that. I doubt very seriously that a flow chart for this beastie has ever been done. Yeah, I do flow charts because I’m lazy and don’t like doing stuff over N+1 times.
    Just my $0.02 worth.

  100. Jesse (17:57:27) : The situation you describe what with the requirements changing as time goes by is normal. That’s what good programming practices are good for – making all that change manageable.

  101. Allan M R MacRae (18:24:23) : I’m glad you brought up aerosols. Something has been bothering me. The alarmists say aerosols cool, but some clouds warm. But, isn’t a cloud in fact an aerosol? The particle size can vary from the CN to a large hail stone, but most of it is an aerosol, no? If aerosols cool, shouldn’t also clouds that have not progressed to precipitation. (I realize clouds bring a lot more to the table that that, but the aerosol thing has been bothering me.)

  102. Gary Strand was near to providing 2 answers for 2 questions. Did I miss them? Is Gary now staying away? I personally find modeling in fortran is interesting.

  103. jeez (18:58:44) : I thought aerosols acted by virtue of their small size. What’s the significant difference between sulfate aerosols and hydroxyl ones?

    • I’m not qualified to discuss aerosol chemistry. I was addressing your original question.

  104. Mike Bryant (16:54:01) :

    I miss Gary Strand… He should be commenting here…

    Yes, he should be. But Mr. Strand is hiding out. He has been hiding out ever since Caspar and the Jesus Paper was mentioned. Strand put his tail between his legs and ran off yelping. [Can’t blame him; that link completely destroys Strand’s whole argument.]
    Also, thanx to Bob Tisdale for his link to a very interesting back-and-forth with several other commenters: click. I’ve just re-read the entire thread, and it’s as good as it gets in the blogosphere. The putative “authority”, Mr. Strand, gets positively owned. Strand runs ‘n’ hides when Bishop Hill’s well documented article is mentioned — as soon as that article was mentioned, Strand headed for the hills.

  105. If I were to build a financial model with the same results compared to observations as these climate models, I would be fired. Unless, of course, I was modeling the housing market for a mortgage guarantee pseudo-government corporation. In that case I would get a bonus.
    Orwell is laughing.

  106. William Woody (10:32:26) :
    has it just about right.
    I LIVE in FORTRAN — and I’m not underpaid. I was literally up until 2am last night coding an FFT post-processor in FORTRAN. I have a customized version of the old Personal Editor that converts FOR to F90 fast. The LF95 compiler makes FORTRAN the best for serious number crunching.
    This old code doesn’t look that bad to me. I DO NOT have time to fiddle with it this week but, with only one or two small miracles, will have time in a week or two. I’ll try to convert it to something readable.

  107. I wrote my first code in 1963, in a language? called SPS. Then Autocoder, Fortran, RPG, COBOL, Basic. (and now HTML) But I stopped being a programmer because I realized that it was nothing but being a slave to the system! So I became a systems analyst. But I stopped being a systems analyst because I realized that it was nothing but being a slave to the guy holding the money! So I became one of those guys with the money. And I made a lot of money because I remembered what it was like to be a slave to IT and them that hold it. THAT’S WHERE PROBABLY EVERY “PROGRAMMER” OF GLOBAL CLIMATE MODELS IS AT, A SLAVE TO WHATEVER ANSWERS THE GUYS WITH THE MONEY WANT TO SEE. However, I believe what every one of them knows, but would NEVER admit to, is that no digital computer, no matter how perfectly programmed will ever be able to “model” the super complex ANALOG system we know as – global climate. Will there ever be a computer (and language) capable of such? Probably, but it’ll be a “Bionary” (remember you read it here first) unit?, device?, that will be based in nano-biological functionality and be capable of self-correcting its “programming” as it runs processes and finds out-of-bounds anomalies through out the “steps” therein. And these “steps” (functions) will not be limited and set mathematical increments, they will be analog at their core providing an infinite range of conditions and interaction within themselves. Even with such a device it may not be possible to model the global climate or at least not for a long time. Why? Because we do not even know WHAT WE DO NOT KNOW! We don’t know enough about the oceans, clouds, dust from land, dust from space, CO2 dispersions and concentrations, and even the that big bright ball in the sky to fill a small bit bucket, much less feed into a computational device and expect that some lowly programmer has somehow written the code of century that is going to spit out what the temperature at 6ft above the ground is going to be at the corner of 5th and Main in Timbuktu at 3:31am on the 14tn day of July, 2077. And with historical record of any of the above (and more) how could we ever test “to reality” what we created?
    I can program a computer to whistle Dixie while dancing an Irish jig and tell you that in 10 years you’ll be 10 years older – if something doesn’t happen. That doesn’t mean the computer was built in Ireland or programmed at Georgia Tech, or make me a scientist. AND it doesn’t somehow magically put into my hands the data I need to RUN THE MODEL. You can have the most powerful computer physically possible (even a Bionary one) and the most sophisticate language and genius of an analyst and of course a master coder and without the RIGHT data, YOU GOT NOTHING.

  108. “And with historical record of any of the above…” That should be -And without a historical record . . ..

  109. Steve in SC, if you’re referring to my comment as “disparaging”, I didn’t mean it that way. I wrote:

    There is a difference between someone writing code using established methodologies and someone who taught themselves and gets mired in an oversized project.

    Many times I have had to clean up after people who got bogged down in something bigger than they can handle. Part of the “established methodologies” I refer to include stepping back and seeing the big picture, then breaking down or modularizing into sections, then concentrating on building functionality for each module or section, then optimizing.
    Most self-taught programmers are very good at one or more of these steps, but few do all of them. Obviously, since I’m writing this, I like to think I am one of those, at least I keep getting called back and others who follow my work are never wading through several feet of muck to get where they need to be.
    Either way, I’m trying to summarize what many have also tried:
    NO, it’s not all about the code. Sure, it’s about the physics. However, without the code to support the physics you have muck, you can’t verify or confirm accuracy, you can’t test anything, and you have a horrid time making any changes.
    The whole point of spreadsheets, for example, was to take programming out of the picture and allow those who worked with numbers to just go ahead and plug in numbers. Yes, programmers were involved, you just didn’t see them. VisiCalc and Lotus 1-2-3 created something of a revolution by letting the numbers and formulas people just work with numbers and formulas, and not have to fight with the computer as well.
    Maybe we need to see the development of an independent, publicly accessible, open source GCM, and allow people to share their tweaks and assumptions. Surely then we could come up with a better model.
    (yes, I know the current group don’t actually WANT a better model)

  110. Smokey (19:09:54) :
    Mike Bryant (16:54:01) :
    I miss Gary Strand… He should be commenting here…
    “Yes, he should be. But Mr. Strand is hiding out.”
    Gary has been over at climateaudit puffing up and obfuscating.

  111. “These codes are what they are – the result of 30 years and more effort by dozens of different scientists (note, not professional software engineers), around a dozen different software platforms and a transition from punch-cards of Fortran 66, to Fortran 95 on massively parallel systems.” – Gavin Schmidt, RealClimate.org

  112. The difference between a natural scientist or engineer (non-software) and a computer scientist:
    A natural scientist or engineer will get bad code to compile and get some sort of output then declare a scientific conclusion based on this.
    A computer scientist understands there is more to it than that.

  113. John Galt (14:11:13) :

    Having a degree in Software Engineering or Computer Science is not a guarantee of the quality of a person’s code. I have met many self-taught programmers who write excellent code and many degreed professionals who just copy and paste and patch their code after it’s put into production.
    The difference is in the attitude and aptitude. A person who takes pride in their work strives to write good code. They want to improve and when they don’t know how to do something, they learn it instead of throwing out one bad hack after another.

    Too true! I have run into this time and time again. I cannot count the number of people I have encountered that have Masters Degrees in Computer Science that are either complete hacks, or simply don’t know their rear end from a hole in the ground. Always makes me laugh. Want to impress me? Don’t show me your papers, that won’t do it…

  114. Lee (08:22:00) noted: “The last piece of code that worked without any unexpected errors was;
    10 Print “Hello”;
    20 Goto 10

    But I thought it has already been agreed, Lee, that the “10” was a typo as it was meant to have been “Hell”?

  115. henrychance (15:17:41) :
    Real engineers
    There seems to be a misunderstanding regarding engineers. I use headhunters to hire people and they aren’t confused. Engineering degress are granted by schools of engineering. None of this M.S. in Information systems from church schools. You have an engineering degree from an engineering school or you do not. Microsoft and some other vendors claim to produce them. Nope. Not the real thing. (I cut slack for railroad engineers, they have had that handle and don;’t hold themselves out to have a real engineering degree.) Head hunters also tell me if it is a compromised degree such as B.S. In Engineering Technology. Not as much as a real engineering degree. This excludes the DeVry Universities and some BS computor science degrees. There are real computor engineering Degrees. There are several hundred Engineering School campuses.

    I would have to agree with this. I personally attended a University, 4 year Bachelor of Science degree, from an accredited engineering and agricultural research University. Many of my courses shared those with EEE students (Electrical and Electronic Engineering), especially Mathematics, Physics, Chemistry, etc… Our “Computer Science” curriculum was very much in line with the Engineering disciplines. I do not consider MIS degrees and all their various permutations, any where near to the class of a real CS degree from an engineering College or University. They simply are not compatible with one another. Perhaps this is where part of this breakdown is when using the term “Software Engineer”. There truly are “Software Engineers” out there (I am one). I would say this is far more the exception than the rule when speaking in general terms of “Software Development”. There is definitely a very sharp contrast between a “Software Developer” and a “Software Engineer”. The latter requires a much larger toolbox. Incidentally, my personal professional title is, and has been for a long time, Sr. Software Engineer. This same title has followed me for many years, even throughout my years working government entities as the DOD, DHS, Pentagon, Navy, and various defense contractors.

  116. Chris Reed (17:41:37) :
    ….
    As a Meteoroloist, I learned to look at the model as a tool.

    Precisely!!!! You could not have said it better. I cannot count the number of times I have had to remind people that any computer program, computer programming language, etc… IS JUST A TOOL! And like any tool, can be used properly or improperly. The term “use the right tool for the right job” comes to mind…

  117. We really have been to this movie before.
    The Strategic Defense Initiative ‘Star Wars Project’ of Pres. Reagan suffered perhaps its most serious blow when:
    “On June 28, 1985, David Lorge Parnas [link] resigned from SDIO’s Panel on Computing in Support of Battle Management, arguing in 8 short papers that the software required by the Strategic Defense Initiative could never be made to be trustworthy and that such a system would inevitably be unreliable and constitute a menace to humanity in its own right.”
    see: http://en.wikipedia.org/wiki/Strategic_Defense_Initiative
    Remember? We could not show, using computer science protocol already well-established more than 30 years ago that the code-base for SDI could meet a “necessary and sufficient conditions” mathematical analysis. [This is an example of what “software engineering” really is/is about, and is at least roughly analogous to finding the harmonic modes of Galloping Gerdy, before it falls into the Tacoma Narrows).]
    see: http://en.wikipedia.org/wiki/Necessary_and_sufficient_condition
    The SDI code-base made today’s NASA GISS model E, GISTEMP et al look like a school teacher’s classroom management program. We were capable of using mathematical logic to determine whether the SDI software was actually internally consistent & coherent … and could show rigorously that it was not.
    ====
    There is nothing inherently wrong with Fortran. On the contrary, Fortran work is currently important: it is still the Speed-King of high-level languages, and in plenty of ‘raw-computing’ applications, maximum computation speed is absolutely the bottom line.
    There is also nothing inherently wrong with old algorithms/routines. The vast majority of all the algorithms we use are, unavoidably, old. There is actually an advantage to using Grandpa’s code-routine, since it has been hammered/tortured in more ways than some new-fangled method.
    The problems with Model E, GISTEMP, etc code-bases aren’t that parts of it are in [guffaw!] Fortran (there is a modern-language conversion-effort underway), or that the routines are [hoot!] 3 decades old, but that these code-bases are not being managed/maintained appropriately.
    Why not? Why isn’t this stuff taken care of properly, or subjected to analysis-procedures that can *prove* whether it’s water-tight or not? Because key people don’t care. Because at the levels that count, it’s not important. Remember, Congress cared whether SDI software produced GIGO, or actionable conclusions. Neither Congress nor the White House care if the GISS codes make good sense or not.
    Key climate/weather data/analysis is performed by Pentagon & Intelligence departments & agencies. NASA/GISS is a dog & pony show without weight or leverage. Nobody particular cares if their software can even run, much less can be used to say anything verifiably scientific.
    The President pays no attention to the NASA/GISS climate work, but he consults with the Pentagon liaison daily. NSA hovers just off his elbow, ’round the clock. Despite Dr. Hansen’s finest theatrics and personal letters to the White House and other Heads of State, Pres. Obama studiously ignores him.
    NASA’s climate-work is sub-par, and redundant. Even if their efforts were well-managed, they are woefully antiquated (20 years behind Pentagon/NSA) and under-staffed/funded (by orders of magnitude), compared to the military and intelligence projects that our top leaders actually use to make ‘best-science’ decisions.

  118. It is full of RATFOR. I haven’t seen RATFOR since the 80’s. If you are still coding in FORTAN much less RATFOR you probably should look to retire. FORTRAN is still a little faster than C++ on the numerical calculations but given multi-threading, etc. It gets blown away. This thing is junk. It’s full of arcane crap that that ensures job security. If Milton from Office Space were a programmer, this would be his work. I remember a guy who worked maintaining listing for the Yellow Pages. Back when large drives were measured in megs, he built a several gig flat file database. Trouble was he used to drop acid when he coded. No one could figure out how it worked. That said, I think his code was better than this garbage.
    It is typical government coding. I advocate that we spend some of the stimulus money to get them Fisher Price keyboards. That way they can think that they are working, but not cause any real harm.

  119. CodeTech (20:59:04) :

    I was discussing some of these things with one of my software developers today. A major function of our job includes the translation of human processes into computer processes. There are engineering aspects to this, there are also creative aspects to this. These translation processes are not blank and white and can sometime require drawing from a vast array of disciplines and experience. Again, in this arena, I am rarely impressed by someones paperwork. Some of the best software developers and software engineers that I have met, have been your classic academic rebel (I being one of those), and rarely posses impressive credentials. Books are one thing, practice is another.

  120. Agreed.Numerical coding is a bit different from systems coding, however. Without a good background in math or applied mathematics, model coding tends to go astray. It should be noted, however, that 80% of the work in most models is the shuttling, reading and formating of data. Numerical programmers tend to be lousy at that work, as evidenced by this code.
    Did you read any of this stuff? It’s like steam-punk coding. I feel like I should get out a leisure suit to look through it. I realize that it’s government and they can’t afford anyone of quality, but is this some kind of retirement program for failed programmers?

  121. Ted Clayton (22:58:31) :

    There is nothing inherently wrong with Fortran. On the contrary, Fortran work is currently important: it is still the Speed-King of high-level languages, and in plenty of ‘raw-computing’ applications, maximum computation speed is absolutely the bottom line.

    I would agree, as FORTRAN is just another tool. Is it really the speed king? I would argue with that. C or C++ typically compiles to much more efficient assembler code than does FORTRAN. Further, some versions of Pascal compile to even more efficient assembler code, one of the best being the good old Borland Turbo Pascal, which would compile to about the same compact and efficient assembler code as C/C++, but had the speed advantage of passing parameters through registers rather than pointers to memory heap allocations. Borland Pascal used a very efficient way of utilizing registers and stacks rather than heap memory allocations. I am not saying that FORTRAN is poor, it is not, but I don’t know that I would give it the blue ribbon for efficiency either. I spent several years hand optimizing assembler output from a variety of source compilers and in my experience, C/C++, Pascal and even Modula II typically out performed most all other languages. Metrics from clock cycle testings and audits would confirm this at the time.

  122. If I were to build a climate model:
    Personnel ==
    1) Gather up the brightest minds in physics, chemistry, climatology, all the necessary science disciplines required to understand the physical processes involved.
    2) Gather up a good team of “Software Engineers” to design and develop the technologies to be applied and how to apply them. Use the engineers to “engineer” the development processes, application architecture, integrations, etc…
    3) Gather up a strong team of “Software Developers” (ie; coders) capable of witting quality code, implementation, and unit testing individual portions of tasking as designed by the “engineers”.
    4) Gather up a strong team of “translators” or “communicators” capable of communicating between the “scientists” and the “engineers”. Software engineers and physics scientists don’t speak exactly the same language, assistance is needed here to translate one discipline to the other. I would employ a team of people skilled at this practice.
    Processes (SDLC) ==
    1) Engage a peer review coding cycle processes
    2) Engage proper source control management processes
    3) Engage proper QA/QC and validation processes
    4) Engage proper release management processes

  123. Last I checked FORTRAN was still faster on certain numerical calculations, but I agree with you. Who cares on such small differences. The big issue here is maintaining and working with the code. This is classic FORTRAN full of cryptic variable names that are less than 8 characters. It really is old school. How do you even verify that this code is performing correctly. It is over 100,000 lines. No unit tests, no subroutine test drivers. It would take a long time to get your head around all the interactions in the sub-models. One good advantage is that it has formating for line printers. So it’s got that going for it. Which is nice.

  124. Jim (18:57:06) :
    Allan M R MacRae (18:24:23) : I’m glad you brought up aerosols. Something has been bothering me. The alarmists say aerosols cool, but some clouds warm. But, isn’t a cloud in fact an aerosol? The particle size can vary from the CN to a large hail stone, but most of it is an aerosol, no? If aerosols cool, shouldn’t also clouds that have not progressed to precipitation. (I realize clouds bring a lot more to the table that that, but the aerosol thing has been bothering me.)
    ________________
    Good questions Jim, and we need someone better qualified than me to adequately answer them. Maybe Doug Hoyt will step in.
    According to Wiki, I guess you are correct – clouds are aerosols:
    “Technically, an aerosol is a suspension of fine solid particles or liquid droplets in a gas.”
    I have read much about clouds and aerosols, and my impression is that the two are generally considered as different categories for climate model purposes.
    Clouds can form from water vapour, a potent greenhouse gas, and vice versa, so clouds could exhibit more complex behaviour in this regard than other aerosols.
    I have also read that high-level clouds and low-level clouds can have opposite effects wrt warming and cooling. This I have to take with the usual grain of salt, as I haven’t read much that supports these views.
    I guess you are aware of Svensmark’s hypothesis that cosmic rays seed low-level(?) cloud formation, causing cooling. I can accept that low-level clouds cause cooler temperatures than would occur under clear skys during daylight, but it seems to me that at night the low clouds trap heat and reduce the rate of overnight cooling.
    Aerosols from major volcanic eruptions are known to cool the Earth, and these consist of sulfates, particulates, etc.
    All this is rank speculation on my part as I have no time to prepare a proper response. I’ll ask Doug Hoyt if he wants to comment.

  125. Burke (23:00:55) :

    Back in the mid 80’s, I was briefly involved in a project with the Chem. department or my University, and one of their research projects. The Chem. dept. is infamous for gobbling processor time on any machine. They were frequently awarded grants from IBM that included the use of newly developed technologies (ie: super computers) of the time. One such computer was the IBM 9000 series Vector Processor machine. My role was to assist in the performance optimization of critical computational components. All of the original code was written in FORTRAN. To achieve our goals, we broke out critical portions and rewrote those components using C which incorporated a lot of hand optimized in-line assembler code. Our results were rather astounding, with an average performance improvement of around 1700-2000%. This was pretty significant. In this example, I would say that our C implementation blew away the computational performance of standard FORTRAN, but to do so wasn’t exactly easy either.

  126. There are two core problems here. The first is triage. Get this POS cleaned-up. Break the data from the model. Separate data layer from model layer and from the results layer. This work is straight-forward but dull has hell. A few days in you’ll be using the Clockwork Orange eye openers to keep going. It is a necessary first step, however. get it stabilized and documented. I realize that a good deal of documentation exists, but unit tests and regression testing on the model components needs to be in place. There really is nothing quite like a large-scale, non-linear model without regression tests.
    The second stage can follow traditional development techniques like you have outlined above, but you have to get this thing into a malleable form.

  127. Squidly (23:13:27),
    Yes, I swung the praises of Fortran just a bit wide. 😉
    To go on & on in Fortran for housekeeping chores isn’t nice. I perused the NASA Model E upgrade project pages about 6 months ago, and saw that a major problem is the use of cryptic, short variable-names. Git out!? Now, that’s a maintainance chore that shoulda got done about a standard career back.
    For the most part, Fortran is only going to be ‘in the way’, agreed. Where the aim is to really come to grips with Formula Translations, though, it might have an edge (not that Models, as noted, are doing computationally/numerically-limiting stuff in most of the routines). And, I’m not as current on language-issues (a socially-acceptable 20th C. substitute for religion-arguments? 😉 as I would have been a decade (2?) back…
    But I do see on the handiest reference:
    “Since Fortran has been in use for more than fifty years, there is a vast body of Fortran in daily use throughout the scientific and engineering communities. It is the primary language for some of the most intensive supercomputing tasks, such as weather and climate modeling, computational fluid dynamics, computational chemistry, computational economics, and computational physics [5 links]. Even today, half a century later, many of the floating-point benchmarks to gauge the performance of new computer processors are still written in Fortran (e.g., CFP2006 [link], the floating-point component of the SPEC CPU2006 [link] benchmarks).”
    http://en.wikipedia.org/wiki/Fortran
    Like Burke (23:27:06) says, for the right crunch-jobs, it’s evidently still it.
    Yes, a climate-modelling team-approach like you describe, or a modern-Celtic OSS approach could rapidly put the computational/modelling portion of climate-investigation (it still needs a huge, ongoing investment of actual, physical science) on a good footing. Dr. Hansen is stuck in a (faulty) loop.

  128. “In addition to being complex the climatology models are – and are supposed to be – constantly changing as they learn more of the natural forcings at play. To suggest they should be required to stop now and re-write the code from scratch is… shall we say… not very reasonable.
    Pete”
    Wow…. Pete, do you really believe that just stacking more useless wrong data on top of bad code, on a already WRONG Theseus is the correct thing to do? I mean Wow…..
    You say that code will change with more knowledge on natural forcings……… well if the forcings are not known by any person on this planet at the time of the code writing, how do you compensate for that big of error in knowledge of the code?
    eg.. If the earth is said to be flat after extensive scientific study, and then proved wrong, how do you think you might adjust to that realization?….. Hmm, my question…… shall we say… not very reasonable.
    “The models they are working with appear to have been proven fairly accurate over the last 20 years. This video contains some historical perspective of climate models;

    Pete”
    The bottom line Pete, modeling climate is delusional and that video is a crock. You say models have been accurate for 20 years, but not one predicted the cooling that has been going on in the last 10+ years and ZERO mention of a one whole degree of drop in mean temperatures in the last 3 years.
    Really, a heating of .7C in a century and then a drop be 1C in the last few years is a big forcing.
    Sooo, out of control CO2 heating doesn’t seem to exist, but in a computer model.

  129. Phillip Bratby (09:46:34) : My experience of Fortran coding from many years ago was that to make sense of it and for purposes of verification, somewhere between 20% and 50% of the code should be comments. Without having the time to look at the code, can anybody tell me how much of the code is made up of comments?
    Well, nearly zero.
    Taking Step2 as an example, it has about 1319 lines of code (the result of:
    $:~/WeatherModels/GIStemp/STEP2 chiefio$ cat * | wc -l
    1319 ) of which 15 have a leading “C ”
    $:~/WeatherModels/GIStemp/STEP2 chiefio$ !! | wc -l
    grep “^C ” * | wc -l
    15
    And just to show you how helpful those 15 lines are, here they are:
    PApars.f:C *** unit#
    PApars.f:C *** Input files: 31-36 ANN.dTs.GHCN.CL.1 … ANN.dTs.GHCN.CL.6
    PApars.f:C ***
    PApars.f:C *** Output file: 78 list of ID’s of Urban stations with
    PApars.f:C *** homogenization info (text file)
    PApars.f:C *** Header line is added in subsequent step
    PApars.f:C KM = the number of time frames per year
    PApars.f:C 121 LEN(IS)=ILSR(IS)-I1SR(IS)+1 ! (counting gaps also)
    invnt.f:C WRITE(6,*) ITRL(3),INFO(6)+IYR0,INFO(6)+IYRL-1
    text_to_binary.f:C print*, line(1:64)
    text_to_binary.f:C print*, line
    toANNanom.f:C do i=1,ndim
    toANNanom.f:C write(*,*) i,idata(i)
    toANNanom.f:C end do
    toANNanom.f:C if(ndim.gt.0) stop 11
    As you can see, the bulk of them are chunks of commented out code…
    IMHO, having read all of the GIStemp code, it is crap. The comments are at best unhelpful. It is a “hand tool” that expects manual intervention at several points. It is NOT a production product and I would fire the person who came to me with this level of code and said they were ready to ship. (I have managed production of software before and shipped product repeatedly, including the compiler tool chain for GNU aka RedHat…)
    Oh, and to the guy who said software writers are not engineers: Please talk to the the ENGINEERING department that awards ENGINEERING degrees in Computer ENGINEERING at major universities… My Computer Science friends with ENGINEERING degrees from places like U.C. Davis, UCB, UCLA, Stanford, and CalPoly S.L.O. would not agree with you… nor would I, having employed said engineers to write code in the Engineering Department at a couple of name companies that created the computers you use… One of my engineer friends wrote the software that lets the B1B fly (his code controls the tail surfaces). You don’t think that’s engineering? Try flying the B2 without the software… (HINT: You WILL crash. Fast.)
    Back on the “hand tool” style of GIStemp. From STEP4_5 we have:
    $:~/WeatherModels/GIStemp/STEP4_5 chiefio$ grep comment *
    do_comb_step4.sh: echo “or do all compilation first and comment the compilation lines”
    do_comb_step5.sh: echo “or do all compilation first and comment the compilation lines”
    trimSBBX: echo “or do all compilation first and comment the compilation lines”
    zonav: echo “or do all compilation first and comment the compilation lines”
    Notice that this says to do things “by hand” if you like and comment out chunks of code that you don’t like… This is NOT production style. This is what is commonly called a “Kludge”.

  130. George Tobin (09:01:16) : I was unaware that any blame had been directed at GISS software engineers. My understanding was that they were faithfully implementing the bogus sensitivity assumptions and all the rest of that which comprises current climate modeling ideology.
    George, you are laboring under the assumptions that the code was written by a software engineer and that there was a division between the designer and the coder. The code looks to me like it is largely written by folks who learned programming as a sideline or “had a class in it” not by professional programmers. (Yes, anyone can learn to program just like anyone can learn to write poetry… that does not make you a professional programmer nor a good poet…)
    I’ve read GIStemp front to rear. It looks like it was written by folks who had a FORTRAN class once, years ago (I would guess some of it was written by Hansen himself decades ago…) and do not make production product for a living. It also has “layers” that look like kludge added on top of kludge over time. Some blocks are ALL CAPS and look like F77 style (and syntax). Other newer chunks are mixed case and F90 style. In short, “it just growed” is how it looks. There are exceptions. The Python chunk is more professionally done and looks like it was contracted out to a sub.
    But most of it, frankly, looks like someone who learned BASIC and then took a FORTRAN class made a hand tool. Specifically, all the FORTRAN is compiled in line, run, then deleted. It is treated as an interpreted language.
    What few comments there are indicate that the pieces are to be run by hand as you see fit and that if you want to edit the code to change it, well heck, why not! Work files are scattered around in the source “archive” willy nilly and the data files constantly mutate name and function from piece of code to piece of code. Nothing prevents or even encourages preservation of the source code during operation. What part is run in what order is also left as an exercise for the student…
    I am inclined not to read much into this fellow’s candid assessment. And I think Steve M. was a little over the top implying that the programmer should have communicated his misgivings about the models to IPCC, Congress and the media. The guy who has to translate crappy assumptions from higher higher into workable code should not have to field that kind of issue or get caught in the middle.
    While I generally agree with your sentiment, you are assuming that there was a division between the designer and the coder. It looks to me, from inspection of the code, like it was written by the “designer”… in which case one can not expect much in the way of a mea culpa from Hansen…

  131. John Galt (10:30:55) :
    Rod Smith (09:48:05) : And why is anyone still coding in Fortran?
    I asked that question about FORTRAN previously and was roundly spanked by people who informed me that FORTRAN is still used in the engineering field and is also commonly used on supercomputers.
    I can only assume Fortran is equally prevalent in science as in engineering. I still question why a more modern, object-oriented language such as C++ (or even Java) isn’t used. Is FORTRAN mult-threaded?

    Having lived through several “language wars” and written code in more languages than I can remember, I have an opinion about computer languages:
    IT DOES NOT MATTER.
    All that matters is that the person writing the code is competent in the language.
    I’ve worked in BASIC, COBOL, FORTRAN, PL1, Pascal, C, ALGOL, several “non-procedural database languages”, several scripting languages on Unix / Linux machines, programmed several varieties of routers including CISCO in two major variants of their OS, a couple of different assembly languages, and have read Python in GIStemp. Oh, and I once learned APL just to see what it was all about and wrote a few programs in it. It deserves the descriptor “write only language”… And I’ve coded some RPG once too. There are a few others but they don’t float to the top of the memory stack right now… like FORTH that I learned once eons ago…
    At the end of the day, you can write good code in all of them and you can write crap in all of them. Some are a little easier to use for some tasks, but an experienced programmer in any of them can use that tool to run rings around an inexperienced programmer in any language.
    I managed a supercomputer site at one point in my career. FORTRAN was exceptionally optimized on it at that time and C was a new port (mid ’80s). While we wrote a lot of code in C on the Cray, if you had something that really needed to be fast, you used FORTRAN. I doubt if that has changed, simply because there has not been as much money in supercomputers in the 20 years since then (to fund the very large staff needed to make a truly stellar C compiler that would match what had been done with the FORTRAN compiler) as was done in prior years with FORTRAN.
    I’m also told some versions of FORTRAN have object-oriented extensions, but they aren’t commonly used.
    Having managed a software development that lead to 4 software patents and was written in C++, I still have to say that IMHO object-oriented is somewhat over hyped. Yeah, it’s “neat”, but not exactly something that obsoletes everything written before… It’s just another “neat trick” that makes some things easier and some things harder. Good to have if you need it, irrelevant if you don’t need it, and not NECESSARY if you have a non-object oriented language and a programmer skilled in it. Everybody thinks their favorite language or feature is the big deal. It isn’t. It’s what you can do with the hammer that matters, not whos hammer is bigger…
    If software engineers are in short supply, that’s a sign that NASA needs to migrate to a more contemporary system. Last I heard, there were a lot of layoffs in the IT field. Surely there are some out-of-work computer science majors who would love to work for GISS?
    I’d love to get a job re-writing GISS code, but I wouldn’t want to work for Hansen… That said, why would a shortage of engineers argue for a conversion effort that would require hiring more engineers?… It takes fewer engineers to just leave it alone and run it once a month…
    But the problem I see in the code is not just the way it is written. The bigger issue is that it makes bogus assumptions. Things that are demonstrably wrong. One example:
    GIStemp assumes that a station 1000 km away can be used to fill in missing data from another station. I can assure you from a half century of personal experience that not only can you NOT use the San Francisco temperature data to fill in for Lodi, but the two are INVERSELY related in summer. The central valley of California heats up, the updraft pulls ocean cool air over SF and brings the fog in. When Lodi gets hotter, SF gets colder and danker.
    But this “pump” works with a 3 to 4 day cyclicality and with variable time delays between the inland heating and coastal cooling… so you can’t just assume a naive relationship. It changes dynamically (and GIStemp makes no allowance for this).
    Does anybody know the hardware and OS GISSTEMP runs on?
    I don’t think it matters. It needs an F90 compiler. After that the OS and hardware are not very important. The quantity of data and the operations done could be easily handled by a modern PC. It’s pretty simple processing.
    My GUESS would be that it runs on a UNIX workstation, probably a Sun of some sort. I saw no “Crayisms” in the code and nothing tied to a particular platform. It also was blissfully devoid of Micro$oft isms…
    Last time I was at NASA was a while ago, but then they had Crays, SGIs, Suns, and PCs (and I think there where some HP boxes too). Being a government agency they probably have whatever the lowest bidder offered last cycle… I think everything has to either be POSIX or MS Windoze …
    The way it is written ought to be independent of the OS or hardware. There were no “endian” problems AFAIK and they didn’t use much in the way of fancy coding techniques (ie. nothing proprietary). To that extent they did a good job…

  132. Dave Andrews (14:39:07) : The discussion you recall may be this one
    http://www.thebulletin.org/web-edition/roundtables/the-uncertainty-climate-modeling#
    Yes thanks that’s the one. Whilst they are open about limitations, I get the feeling that they just want to continue building models. That’s just what they do. The comment about how they already know with certainty that CO2 causes warming, and is as certain as knowing that a cup falling to the floor will shatter, therefore we must “leap” out of the way, we must act, regardless of what else models may tell us, is an interesting take on things. It suggests that models are just research curiosities. The ones that output significant warming are of course newsworthy, like any techy gadget can be newsworthy, but they are not the original proof or basis for global warming. That, as they say, was already known with certainty.
    Like a shell game, in global warming/climate change we’re still left wondering where the real ball is.

  133. I’ve just been looking at some of the GISS Code and it is schoolboy stuff from a software point of view. I did a quick Lines of code count and it appears that GISS believe that the earth’s climate can be accurately simulated with only 100,000 lines of badly written Fortran. When you consider that windows Vista has 50 Million lines of code it makes you wonder how something that is orders of magnitude more complicated than the climate, could ever be expected to boot up, let alone do anything else.

  134. I don’t think it would matter much in the end if the climate models were completely rewritten from scratch in the best and most modern code available. The end results would still be the same, i.e., the global temperature, CO2, sea level, glaciers, Arctic ice, Antarctic ice etc. would still be rising, rising, rising, melting, melting, melting respectively at even faster rates than they had previously thought (modelled).
    So if it ain’t broke… Everything’s hunky dory. What temperature do you want? What sea level rise do you want? Etc, etc…

  135. You should see some of the awful FORTRAN code still used and maintained to do attitude determination, orbital mechanics, guide star occultation, etc. for spacecraft. Complaining that past coding doesn’t meet current standards is just plain silly. As long as it ‘works’ for the primary user, redoing it is just bad engineering (if ain’t broke and all that). Any disagreement with the definition of “works’ has little to do with the ‘quality’ of coding.
    Gary was trying to defend what he does for a living. Why, I’m not sure. He calls himself a ‘software engineer’ but is not necessarily any more an ‘engineer’ than a ‘building maintenance engineer’. His ‘admission’ is at about the same level as that of a building maintenance engineer’s admission to the sad state of architectural design. Ho hum.
    Should the climate models be used for trillion dollar decisions? Of course not! But for far better reasons than crappy, hard to follow code. Let’s focus on the real issues instead of peripheral ad homs like ‘crappy code’.
    Move along people. There’s nothing to see here.

  136. The trouble with models is that they are pseudo-quantitative. Numbers are based on qualitative judgements.
    Would be better if they used a true qualitative approach such as hazard and operability [hazop] studies where experts and interested parties talked through the issues in a semi structured way. Not dissimilar to a blog really.
    cheers David

  137. >>10 Rem Global Climate Model
    >>20 Print “The End Is Nigh”
    >>30 Goto 20
    >>40 End (of planet)
    This really would be the End of the World – you’ve got a loop there somewhere…. 😉
    .

  138. How would one know the difference between true model predictions, and artifacts because of bad coding?
    Somewhere in the code I saw that it assumes 360 days in a year, or 12 equal months of 30 days. Not a bad assumption if you are forecasting over a short range of time. But when you are trying to predict global temperatures 100 years into the future – that is a very different story.

  139. You need multiple independently coded models to alleviate the problem of coding errors.
    However programs like modelE are implementations of some deep domain knowledge, and the trickiest thing is the application of that domain knowledge rather than the coding itself. Ie problems are more likely to arise because of a physics error than a coding error.
    I have been programming all my life, C++, C#, COBOL too, so this FORTRAN stuff doesn’t look too alien to me. However I can’t make head or tail of the modelE code, not because I can’t follow the flow of the code, but because I don’t understand the domain knowledge (climate physics) behind it.

  140. I’d thrown together a post a while back on poking a little fun at some of the base assumptions behind catastrophic AGW, and I had a couple paragraphs dedicated to why the models are pretty useless. Of course, I was just looking at the design side, not even getting into the issues of poor coding. Here’s the relevant portion:

    …the sheer magnitude and complexity of the system still defies attempts to fully understand all of the interactions, leading to small errors having an ever-growing cascade effect as one tries to predict further into the future.
    Put more simply, imagine working on a very complex algebra problem. The issue is that you copied one little part of the problem wrong. In your first pass, simplifying, substituting, and performing other operations, that wrong bit interacts with another element of the problem, the result of which is corrupt. Each subsequent pass through the problem causes the erroneous segment to corrupt more of the total problem as it is combined with other elements, and, by the end, your final answer is utterly wrong. This is very much what inhibits long-term forecasting on even a local scale and short time spans. Taking this up to the global scale allows vastly more chances for error, and projection years into the future gives any errors much more time to cascade to the point of rendering the entire model useless. The calculations may be perfect, but a faulty starting point ruins the entire process. Unfortunately, this isn’t the greatest difficulty. Back to the hypothetical problem, imagine that you don’t even get copies of any of the formulae you need, nor even the base problem. Instead, all you have are the base values to use and thousands, millions of answers, each coming from different base values (you don’t get told what any of them are), from which to figure out the starting equation. It gets worse. Some of those answers are wrong, and you don’t know which ones. It gets worse still. The answers are divided up into hundreds of groups, each group containing answers to certain combinations of formulae and base values, and you aren’t told what formulae or values are used or how they are combined. Nope, not done getting worse, yet. In fact, you are told that none of the groups contain answers to the actual problem you need to solve. Instead, example answers for the overall problem are supposed to be composites, somewhat like the mean, of all of one answer (the tenth, for example) from each group, but you also have to figure out how those subordinate answers get combined into the big one. In the end, you have mountains of data, an unknown amount of which is faulty, from which to determine all of the formulae, how the formulae are used to produce the subordinate answers, and how those are combined to get the main answer before you can even start working on the problem with the given starting values. Guess what? This isn’t even accounting for time lapse yet.
    Someone may read that last paragraph and think, “Models on the global scale don’t have to be that precise, though; they can be more general and still get the broad picture.” Well, let’s run with the picture metaphor, shall we? I would roughly equate simplifying components of a model to reducing the number of pixels in an image. Sure, you can do it, but you loose definition. Do it with a complex, intricate picture, and you will very quickly end up with an image people misinterpret, or one from which it is impossible to discern anything at all. Sure, you can dumb down the models by making them less complex with estimations of values instead of solid formulae, but you have to admit that doing so destroys the accuracy of that broad picture. A more direct example comes from graphing a curve to fit plotted points. Reducing complexity could be seen as using fewer data points from which to extrapolate a curve. The problem with this is that, as the data points are made fewer (thereby spreading out on the graph), you may end up missing movement in the actual curve occurring between points and graphing a curve that is completely wrong. When it comes to scientific (quasi-scientific in this case, really, but that’s a topic for another day) analysis of data, generalization is bad, and detail is good.

  141. That leads me to a more fundamental question. Are there any logical checks on climate modeling?
    Drawing on my experience as a chemical engineer, because the chemical systems I modeled were non-nuclear, I was bound by the laws of conservation of mass and energy. The models always checked to see it the number of pounds, kilograms, or tons of products equalled the same measure of reactants. If the mass balance did not close, the program printed a big warning.
    Do climate models bother to calculate an energy balance? It would seem to me that if you got a bogus answer of 1 x 10^20 temperature in March 2016, then the answer is “No”.
    If they don’t anergy balance, then how can you have ANY confidence in the results?

  142. It appears I have hit a sore spot re: “software engineers.” First of all, I don’t mean to demean the value of the work that they do. They fulfill a very necessary niche in the business world. But at the same time, unless they are engineers in the traditional sense, they must take direction from engineers or scientists to build the models. There are a lot of chemical engineers who develop software for our business (for those in my business, think ASPEN, Simsci, or other chemical engineering software modelling companies). And we would be a hell of a lot less productive without them. I shudder to think of designing a multi-component distillation column by the trial-and-error tray-by-tray method. I remember seeing a thick binder of calculations done by hand. That said, it is still the physical science that is the underlying basis for the models, not the code.
    So, Squidly, I salute you, and all of your brethern! Go forth, and prosper!

  143. E.M.Smith (01:50:56) :
    George Tobin (09:01:16) : I was unaware that any blame had been directed at GISS software engineers. My understanding was that they were faithfully implementing the bogus sensitivity assumptions and all the rest of that which comprises current climate modeling ideology.
    George, you are laboring under the assumptions that the code was written by a software engineer and that there was a division between the designer and the coder. The code looks to me like it is largely written by folks who learned programming as a sideline or “had a class in it” not by professional programmers. (Yes, anyone can learn to program just like anyone can learn to write poetry… that does not make you a professional programmer nor a good poet…)
    I’ve read GIStemp front to rear. It looks like it was written by folks who had a FORTRAN class once, years ago (I would guess some of it was written by Hansen himself decades ago…) and do not make production product for a living. It also has “layers” that look like kludge added on top of kludge over time. Some blocks are ALL CAPS and look like F77 style (and syntax). Other newer chunks are mixed case and F90 style. In short, “it just growed” is how it looks. There are exceptions.
    ==
    OK. No disagreements with the internals of the code. I will grant your (stated) expertise in the various languages, and the ability of a (good) programmer to write (good) code in virutally language. No problems.
    But … What about the need for configuration control? Is there ANY change control, change testing, testing “debug” notes or corrections?
    Is there any “lost” code that could/would/might produce arbitray trash if a single thing goes wrong? For example, if we put a real value of CO2 and a real value of temperature? Does it produce a valid “radiation in” as a check? Who made the changes? Does it process “real data” if the year or earlier CO2 results and temperatures change? (In other words, if you put in values for 400 ppm CO2 in 1980, does it produce a valid temperaure as it would if the year were 2050 and the CO2 were 400? If CO2 were changed to 200, does it produce real world temperatures for say the year 1800? )
    Who tested each module, if the whole thing has not been tested? how do we know the actual calculations in the repeating module are correct? Who verified its input, and what are its input variables, and what are the “coded in” variables?
    When a change or update was made, who made the change? Who authorized the change? Who tested the result after the change? What was the previous coding, and what was the changed coding? Has it been” hacked” or do the results have any chance of being “hacked”? Were results as printed actually teh results from a test run, or were they merely typed in afterwards?
    If this ran an ATM machine, would you sell it to your bank? Would your congressman put his money into this program if it were a ttax processing software?

  144. @ Ted Clayton (00:24:40) :
    Ah ha! .. Very thoughtful comments! Nice!
    I completely agree with you concerning language construct and organization of FORTRAN and what that brings to numerical formula processing. Your comments are certainly very valid. I would content however, and especially when considering the enormous budgets employed, one could utilize a more modernized language (preferably a fully compiled language that produces binary code for performance optimization, ie: this would exclude Java) and provide even more robust scripting mechanisms that would allow the use of heavy mathematics in robust expressions that could then in turn be pre-compiled into the foundation language, optimized and compiled to binary codes native to the host machine. Such an approach as this will allow for the proper architectural abstractions, allow the software engineers to employ flexible and robust design concepts and patterns, allow software developers to write compartmental coding segments abstracted from adjacent logical modules, and probably most importantly, allow the “scientists” to develop robust mathematical representations of physical processes and complex interactions utilizing language, verbiage and structure that they are used to.
    In this scenario, everybody wins, and result could be something that may actually stand a snowballs chance in hell of working. And even if it doesn’t work, you have developed something that can adapt as your learning and understanding culminates and expands.
    IMHO…

  145. R S Bridges (19:48:32) :

    So, Squidly, I salute you, and all of your brethern! Go forth, and prosper!

    In large part I am in agreement with what you are saying. Conversely the same could be said software engineer to scientist. This, after all, is a cooperative effort. There should be no dictation flowing either direction. I find there is a clear and present contrast between a “programmer” and a “software engineer”, that is the crux of what I have been discussing. To some who have said “a software engineer is not an engineer” or that “there is no such thing as a software engineer”, they are quite naive to the development of software in general. Ultimately, the process of software development is simply the translation of human process to machine process. Of this, there are many facets involved, from the engineering aspects, to the coding aspects, and many branches in between. The trick to developing successful software is proper organization and translation, this is usually not simply a task for a “coder”, as “coders” don’t usually posses the skill set necessary to adequately architect the application structure itself with all components considered.

    “That said, it is still the physical science that is the underlying basis for the models, not the code.”

    I disagree with this. Again, it is a cooperative effort. One relies on the success of the other. All the physical science in the world is simply garbage if not properly constructed and coded. Conversely, the best software design and coding is worthless without proper physical science instruction. So, again, to be successful, one cannot live without the other. They need to be harmonious. These are not easy or trivial tasks, but properly set forth can be accomplished.

  146. @Squidly,
    “All the physical science in the world is simply garbage if not properly constructed and coded,.”
    Not so, sir.
    My engineering career spanned the period before computers were widely used, up to today. I can assure you that sound physics and engineering were performed by competent, real engineers, (in the fields of civil, mechanical, electrical, chemical, and others) without the use of coding into software and running on computers. If you doubt this, then I invite you to have a look at any building, bridge, dam, power plant, chemical plant or refinery with a cornerstone plaque or startup dated earlier than 1970.
    Many times a computer simply adds more time to the task, and useless digits in insignificant places. We had (and still have) perfectly good methods for calculating and designing almost everything — and it was certainly not garbage, and certainly no coding was necessary.
    The legal reality is that almost every large new software system is a failure, and results in lawsuits. While there are many reasons for this, the inability of software *engineers,* systems architects and analysts, code writers and debuggers, to deliver a working product on schedule is a major factor.
    The knowledge that the software *engineers* are coming to deliver a new system causes business owners and others who contract for such systems to cringe. They know the system will fail, and lawsuits will result.
    Just one (of thousands I could mention) is the computerized payroll system for the Los Angeles Unified School District in California, circa 2007. If they (the programmers) cannot do something as simple as add up hours worked, multiply by dollars per hour, perform some equally simple deductions, and print a paycheck, all that for a few thousand employees, how can they be expected to do something a bit more complicated like a climate model?

Comments are closed.