Those of us who have looked at GISS and CRU code have been saying this for months. Now John Graham-Cumming has posted a statement with the UK Parliament about the quality and veracity of CRU code that has been posted, saying “they have not released everything”.
I found this line most interesting:
“I have never been a climate change skeptic and until the release of emails from UEA/CRU I had paid little attention to the science surrounding it.”
Here is his statement as can be seen at:
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc5502.htm
=================================
Memorandum submitted by John Graham-Cumming (CRU 55)
I am writing at this late juncture regarding this matter because I have now seen that two separate pieces of written evidence to your committee mention me (without using my name) and I feel it is appropriate to provide you with some further information. I am a professional computer programmer who started programming almost 30 years ago. I have a BA in Mathematics and Computation from Oxford University and a DPhil in Computer Security also from Oxford. My entire career has been spent in computer software in the UK, US and France.
I am also a frequent blogger on science topics (my blog was recently named by The Times as one of its top 30 science blogs). Shortly after the release of emails from UEA/CRU I looked at them out of curiosity and found that there was a large amount of software along with the messages. Looking at the software itself I was surprised to see that it was of poor quality. This resulted in my appearance on BBC Newsnight criticizing the quality of the UEA/CRU code in early December 2009 (see http://news.bbc.co.uk/1/hi/programmes/newsnight/8395514.stm).
That appearance and subsequent errors I have found in both the data provided by the Met Office and the code used to process that data are referenced in two submissions. I had not previously planned to submit anything to your committee, as I felt that I had nothing relevant to say, but the two submissions which reference me warrant some clarification directly from me, the source.
I have never been a climate change skeptic and until the release of emails from UEA/CRU I had paid little attention to the science surrounding it.
In the written submission by Professor Hans von Storch and Dr. Myles R. Allen there are three paragraphs that concern me:
“3.1 An allegation aired on BBC’s “Newsnight” that software used in the production of this dataset was unreliable. It emerged on investigation that the neither of the two pieces of software produced in support of this allegation was anything to do with the HadCRUT instrumental temperature record. Newsnight have declined to answer the question of whether they were aware of this at the time their allegations were made.
3.2 A problem identified by an amateur computer analyst with estimates of average climate (not climate trends) affecting less than 1% of the HadCRUT data, mostly in Australasia, and some station identifiers being incorrect. These, it appears, were genuine issues with some of the input data (not analysis software) of HadCRUT which have been acknowledged by the Met Office and corrected. They do not affect trends estimated from the data, and hence have no bearing on conclusions regarding the detection and attribution of external influence on climate.
4. It is possible, of course, that further scrutiny will reveal more serious problems, but given the intensity of the scrutiny to date, we do not think this is particularly likely. The close correspondence between the HadCRUT data and the other two internationally recognised surface temperature datasets suggests that key conclusions, such as the unequivocal warming over the past century, are not sensitive to the analysis procedure.”
I am the ‘computer analyst’ mentioned in 3.2 who found the errors mentioned. I am also the person mentioned in 3.1 who looked at the code on Newsnight.
In paragraph 4 the authors write “It is possible, of course, that further scrutiny will reveal more serious problems, but given the intensity of the scrutiny to date, we do not think this is particularly likely.” This has turned out to be incorrect. On February 7, 2010 I emailed the Met Office to tell them that I believed that I had found a wide ranging problem in the data (and by extension the code used to generate the data) concerning error estimates surrounding the global warming trend. On February 24, 2010 the Met Office confirmed via their press office to Newsnight that I had found a genuine problem with the generation of ‘station errors’ (part of the global warming error estimate).
In the written submission by Sir Edward Acton there are two paragraphs that concern the things I have looked at:
“3.4.7 CRU has been accused of the effective, if not deliberate, falsification of findings through deployment of “substandard” computer programs and documentation. But the criticized computer programs were not used to produce CRUTEM3 data, nor were they written for third-party users. They were written for/by researchers who understand their limitations and who inspect intermediate results to identify and solve errors.
3.4.8 The different computer program used to produce the CRUTEM3 dataset has now been released by the MOHC with the support of CRU.”
My points:
1. Although the code I criticized on Newsnight was not the CRUTEM3 code the fact that the other code written at CRU was of low standard is relevant. My point on Newsnight was that it appeared that the organization writing the code did not adhere to standards one might find in professional software engineering. The code had easily identified bugs, no visible test mechanism, was not apparently under version control and was poorly documented. It would not be surprising to find that other code written at the same organization was of similar quality. And given that I subsequently found a bug in the actual CRUTEM3 code only reinforces my opinion.
2. I would urge the committee to look into whether statement 3.4.8 is accurate. The Met Office has released code for calculating CRUTEM3 but they have not released everything (for example, they have not released the code for ‘station errors’ in which I identified a wide-ranging bug, or the code for generating the error range based on the station coverage), and when they released the code they did not indicate that it was the program normally used for CRUTEM3 (as implied by 3.4.8) but stated “[the code] takes the station data files and makes gridded fields in the same way as used in CRUTEM3.” Whether
3.4.8 is accurate or not probably rests on the interpretation of “in the same way as”. My reading is that this implies that the released code is not the actual code used for CRUTEM3. It would be worrying to discover that 3.4.8 is inaccurate, but I believe it should be clarified.
I rest at your disposition for further information, or to appear personally if necessary.
John Graham-Cumming
March 2010
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
To: Larry Geiger 10:14:22
I strongly support your position. Indeed, the funding source is very important, and the funding source is tax money. I am surprised that scientists involved in Climategate don’t acknowledge that and behave in nasty and arrogant way. I have never encountered in my practice and neither do I know any of my colleagues who were ever asked for a code or experimental data. Generally, public is either unaware that they could or isn’t interested. This is despite the fact that billions are spent for LHC, modeling of comet impacts with Jupiter satellites and Earth, string theories, and other science that I would call “entertainment” science suitable mostly for Discovery or History channels. And, arguably and only to my taste, relatively little is spent to support research that are of practical value for public. However, in the climategate case when the international policies making was attempted on the base of publically funded science and these policies will obviously cause tax and unemployment increase and standard of living decrease, the science must be scrutinized and it must satisfy the standards of majority of people. In the mean time all some group of climatologists separated us, the people, into “real scientists”, “real science supporters” and “ignorant trolls” and continue belittling the latter group despite the fact that it seems to be growing majority contributing the most to their well funded (by public) research.
What Eric (10:32:52) : and Mike (09:27:15) : said. It looks like academia code. If your business was dependent on the results or sales of such, you would have code standards in place, version control, business modeling, and test environments. The sell the results and reproducability or accuracy was not their goal. The code just proves what we have always assumed.
I would not be surprised if this were found somewhere in the unreleased code.
x=0;
x=0; //just in case
I bet Pachauri knows what voodoo science is, since he’s seen it before. Might this be voodoo programmimg???
Ecotretas
P Gosselin (08:31:08) :
Somewhat OT, more of the same crap – but a different pile:
http://nofrakkingconsensus.blogspot.com/
Is this Peer Review/Auditor Gate?
Seems like what is lacking according to the code that was released, they lack a decent version tracking system.
ANTHONY, very important
John Graham-Cumming (08:20:24) :
Your headline is slightly inaccurate because I don’t say that the code for CRUTEM3 is ‘below standards’, I was referring to other code at CRU.
The fact of the matter is that if you are developing software for, say, a medical environment, or flight control systems, or engineering design tools, or industrial process control, or any of a thousand other disciplines (even automotive powertrain control) a bug in the code (or any of the code libraries and objects underlying that code) can kill. In thousands of other applications, bugs can result in mechanical failure, excessive operating cost, legal compliance problems, Mars landers missing the planet, etc.
So at an industrial level, there’s a very substantial incentive to get it right.
These scientists are amateurs. For the most part, they plink around trying to produce results, and when they get something that looks like what they want to see they call it finished. There’s no formal test plan, no test data sets to evaluate a full range of possible inputs to the model, no process to ensure that the evaluation is done by properly skilled individuals. Peer review is worthless in this environment because most of the peers are no more structured in their approach to software than the original authors.
Anthony and all,
I haven’t seen it mentioned here, but the full transcript of Jones, and others, testimony, is available at:
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/uc387-i/uc38702.htm
There are also more documents available at
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/contents.htm
with this John Graham-Cumming letter being the last one.
Ecotretas
I don’t know where these FORTRAN DINASOURS are coming from.
Heck, I was raised with crappy old FORTRAN too.
I’ve now processed datafiles with EXCEL which have 10,000,000 elements in them.
Totally TRANSPARENT on the math formuli.
But, I suppose if you are living in 1975…FORTRAN sounds pretty good.
Max
AJStrata (10:20:02) :
“… having architected, designed and managed large software systems…”
OT, but isn’t “architected” and “designed” in the same clause redundant, and infact isn’t “architected” a barbarism? 😉
“Harry” and Dr. Graham-Cumming should get together.
Code, shmode! With all due respect for John Graham-Cumming, (and I’m sure he will agree with this) the fallacy of the whole premise is NOT in the code, good, bad or in-between. It’s in the logic! And NOT the logic behind the code, but the logic (or the illogic if you prefer) supporting the entire SYSTEM! You have AVERAGED, partial (weather!) data, haphazardly recorded, GENERALLY ONCE A DAY, at a miniscule number of points that varied over time within an ANALOG, CUBIC system (global ocean/atmosphere), with no controlled standardization of either equipment or placement, with said equipment never having been design for the intended purpose and virtually no understanding of the system as whole to start with, sub-systems whose effect % we do not understand (not to mention the still to be identified sub-sub-systems or the partial/complete ignoring thereof), no accurate history (data wise) of one time events or again the effect % OR the “effect time frame” of those events…… etc.?!! Even satellite temperature data is at best marginal and so infinitesimally short in length that it’ll be 250+ years before it might have any meaning.
And we’re arguing about code?? How unbelievably arrogant and disingenuous for any “scientist” (or anybody else for that matter) to claim their model/system/code produces any kind of meaningful results, much less accurate projection. Analyze the code till your blue in the face (because of the 40km of ice over your head?) but it’ll mean nothing until you can comprehensively systematize the WHOLE system. YOU CAN NOT GAIN UNDERSTANDING OF THE SYSTEM IN THE COMPUTER LAB! The most elegantly written and documented code is meaningless if it doesn’t accurately model the system OR if the DATA is GARBAGE! And I’m not even going to say it….
Now, go back to work. And if you come into my office again with an incomplete system, based on such illogical analysis, I’ll either fire you or bust you back down to Jr. programmer or maybe even computer operator! Oh, not you John. You’re doing fine work.
mpaul (09:48:33) :
“This statement indicates that Slingo knows almost nothing about software.”
As someone who was for fifteen years a professional systems analyst and programmer (albeit in the commercial sector), this is one of the few things here I can pass an even half-informed comment on.
Running software however often you like isn’t what tests it per se. You analyse, design, code, test and maybe pilot a system before you let it go live, and subsequently, it gets tested by your users, who will be over you like a rash as soon as they pick up on a problem. You damn well better fix it PDQ, especially if it’s mission critical.
All of us here in the UK are the Met’s end users. We’ve been testing their warmed-over soup for a while now, and found it severely wanting. But have they fixed it? Nah. They have this disconnect between modeller’s fantasy land and reality.
I’m sure if they keep on predicting warm summers and mild winters, they’ll get it at least half right the odd year, but I’m not holding my breath. Their shorter-term predictions aren’t too hot (or maybe they are!) either. No one expects them to be perfect, but if even the BBC is contemplating kicking them into touch, things must be really bad.
Slingo’s brass neck and spin is breathtaking.
First, we know and he argues that almost all the weather data is posted all over the world on the Internet so that anyone can get at it.>>
There is a world of difference between “the data I used is in the public domain” and “this is the data I used”. Stating that the data he used is on the internet is meaningless, because it provides no guarantee that what he used was the same, or a subset or a superset or so on. Similarly, publishing the methodology is also meaningless, because a review of the code is required to determine if the methodology is implemented as described. All three are required or the end result is just an opinion.
As for code quality and revision control… There is no doubt that much of the code written for research is buggy and poorly documented. The same is true for a remarkable amount code written within an enterprise for specific purposes. Anyone who dealt with a Y2K mitigation project knows what a nightmare the legacy code was in terms of documentation and revision control and that was for both private enterprise and public enterprise applications. However, code for commercial purposes is an entirely different matter. Companies like Sun, Oracle, Intel, SAP and on and on have rigorous version control, in depth regression testing, and can reproduce the exact code version and the exact dataset tested against at the drop of a dime. The reason is there are substantial financial consequences to them if they cannot do this. The finanancial consequences of the decisions to be derived from this particular combination of code and data demand a standard even higher.
re: wayne (11:00:11)
I am going to add one other aspect to my previous comment.
However, it would be good to have the software used by climate science to be re-written in a very common and forgiving language as C-sharp or Java, freely available, in purely OO design and ignoring any efficiencies, so all programmers can understand what exactly is happening, no matter their expertise. Efficiency could be added by parallel implementations when speed plays in. That would add hugely to the insight and clarity into climate science itself, and in fact, seems the best way to move this science along at its highest speed of development. A common, plug-in type design to, albeit slow at run-time, test basic core logic used to compute these temperatures.
Yes, that would be good!
Mike (09:27:15)
“I worked as a programmer for a physics research group as an undergraduate and in industry before going to grad school in the 1980’s. It never occurred to the physicists to publish or make original code or even the data available. If someone wanted to replicate their work they would do the experiment themselves and write their own code. The papers described the mathematical methods used so that anyone could write a program to do the number crunching. In industry I often had to deal with very poorly documented code. The standards in industry have likely improved, but a lot a valid work was done in the bad old days. ”
If you want the original data for Principia Mathematica it’s available, same for Copernicus, Galileo, Faraday, Maxwell, Einstein, and all the other physicists. Nobody expected to be taken seriously if they couldn’t produce the data and methods by which they came to there results. It explains why the IOP submission was so damning, it’s plainly not science to produce a result that can’t be challenged and tested.
…and in addition to strong version control and regression testing and documentation, may I bring up one more point that IT professionals should be harping on.
Any IT shop should be able to reproduce the exact state of the data on any given day. In this context, “data” means EVERYTHING. Datasets, code bases, e-mails, everything that existed on THAT day. Standard best practices require a weekly full backup and daily incrementals written to tape and stored off site. If an email, or data file, or version of the code existed on a specific day 3 years ago, it should be possible to restore it from backup. The only thing that should escape these best practices is a file etc that was created and destroyed on the same day. Everything else should be there.
John Graham-Cumming (08:20:24) :
Sir: Most of us reading and posting here lean toward being skeptics and not just of the catastrophic effects of climate change. We have learned to treat Anthony’s headlines as we do most everything posted here. And some of us have programming experience – I started with FORTRAN II-D in 1965. Headlines and press releases seem to be a problem and we have learned, with prodding from Anthony, to ask to see the data, the code, the actual article and so. We do not trust climate scientists with peer review and we would rather a person’s work speaks for itself than trust the letters after the name.
While being skeptical, nevertheless I was quite in agreement with your point #1, namely . . .
Although the code I criticized on Newsnight was not the CRUTEM3 code the fact that the other code written at CRU was of low standard is relevant.
Thanks for being public with this. As you now know, not everyone is willing to step forward.
John
Mike — If you want to redo someone’s chemistry experiment should you be able to demand the use of their test tubes?
If this is being used by politicians as the basis to extract an extra 10% of my earnings or make me pay an extra $3 a gallon for fuel, then YES.
Do you not have a concept of what’s at stake here, or why extra diligence might be required for trillion dollar decisions?
lack of basic version control and a unit testing framework reveals extreme incompetence.
Maybe the word is returning its original meaning, which would rather be a compliment than denigration.
Amateur: through French, from the Latin ‘amator’: lover, devoted friend, devotee, enthusiastic pursuer of an objective.
I seem to recall Anthony was called ‘amateur’ several times, too.
He wasn’t just being diplomatic, he was playing high-level gamesmanship. (“My fast to your slow and vice versa.”) Well played.
This bug, does it have a name? I don’t fancy myself an amateur computer analyst, but I do recognize it from many other areas of scientific study, where computer models are replacing lab work. The “bug” works like this:
So is it the Jack Horner bug, by chance?
**”Quasars: Massive or Charged?”
thunderbolts.info
In evolutionary biology, a single organ may serve multiple functions. Conversely, a single function may derive from different organs.
In science, one of the most valuable verification techniques is to address a given hypothesis from different angles: Different data, different experiments, different analytical techniques. Replicating original results by such means inherently renders a hypothesis “robust” in the sense that varying accounts converge.
Generally speaking, “there is no One Best Way.” But there most certainly are invalid, misleading, even self-deceptive ways of doing “science”: Blondlot’s N-rays, transparent junk blithely asserted by such as J.B. Rhine, Immanuel Velikhovsky, Trofim Lysenko, come to mind. Beware Thomas Kuhn’s “Nature of Scientific Revolutions”, for Kuhn fundamentally misreads the enterprise– Einstein did not render classical mechanics obsolete, but stood on Newton’s “giant shoulders” as ’twere.
“Climate science” is a non-empirical discipline, a classification scheme akin to botany that deals in hindsight only. By mathematical and physical principle one can neither extrapolate complex dynamic systems nor assert a “runaway Greenhouse Effect” in terms of a global atmosphere engaging thermodynamic principles of entropy. (Earth’s minimal CO2 effect undergoes cyclical variations, necessarily stable over time.) In sum, though poor coding is not in and of itself a killer, dismissing lazy and malfeasant programming as immaterial invalidates empirical conclusions. If Climate Cultists do not know this, they simply are not scientists at all.
John Graham-Cumming has a BA and DPhil in Maths and Computing from Oxford. Professor Jones read Geography at Lancaster.
Here’s a working hypothesis. Most scientists who advocate AGW didn’t go to the best universities. They’re not all that clever. They got their PhDs by slogging. They are easily swayed by strong personalities. They have second rate minds and do second rate science. It shows.