Those of us who have looked at GISS and CRU code have been saying this for months. Now John Graham-Cumming has posted a statement with the UK Parliament about the quality and veracity of CRU code that has been posted, saying “they have not released everything”.
I found this line most interesting:
“I have never been a climate change skeptic and until the release of emails from UEA/CRU I had paid little attention to the science surrounding it.”
Here is his statement as can be seen at:
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc5502.htm
=================================
Memorandum submitted by John Graham-Cumming (CRU 55)
I am writing at this late juncture regarding this matter because I have now seen that two separate pieces of written evidence to your committee mention me (without using my name) and I feel it is appropriate to provide you with some further information. I am a professional computer programmer who started programming almost 30 years ago. I have a BA in Mathematics and Computation from Oxford University and a DPhil in Computer Security also from Oxford. My entire career has been spent in computer software in the UK, US and France.
I am also a frequent blogger on science topics (my blog was recently named by The Times as one of its top 30 science blogs). Shortly after the release of emails from UEA/CRU I looked at them out of curiosity and found that there was a large amount of software along with the messages. Looking at the software itself I was surprised to see that it was of poor quality. This resulted in my appearance on BBC Newsnight criticizing the quality of the UEA/CRU code in early December 2009 (see http://news.bbc.co.uk/1/hi/programmes/newsnight/8395514.stm).
That appearance and subsequent errors I have found in both the data provided by the Met Office and the code used to process that data are referenced in two submissions. I had not previously planned to submit anything to your committee, as I felt that I had nothing relevant to say, but the two submissions which reference me warrant some clarification directly from me, the source.
I have never been a climate change skeptic and until the release of emails from UEA/CRU I had paid little attention to the science surrounding it.
In the written submission by Professor Hans von Storch and Dr. Myles R. Allen there are three paragraphs that concern me:
“3.1 An allegation aired on BBC’s “Newsnight” that software used in the production of this dataset was unreliable. It emerged on investigation that the neither of the two pieces of software produced in support of this allegation was anything to do with the HadCRUT instrumental temperature record. Newsnight have declined to answer the question of whether they were aware of this at the time their allegations were made.
3.2 A problem identified by an amateur computer analyst with estimates of average climate (not climate trends) affecting less than 1% of the HadCRUT data, mostly in Australasia, and some station identifiers being incorrect. These, it appears, were genuine issues with some of the input data (not analysis software) of HadCRUT which have been acknowledged by the Met Office and corrected. They do not affect trends estimated from the data, and hence have no bearing on conclusions regarding the detection and attribution of external influence on climate.
4. It is possible, of course, that further scrutiny will reveal more serious problems, but given the intensity of the scrutiny to date, we do not think this is particularly likely. The close correspondence between the HadCRUT data and the other two internationally recognised surface temperature datasets suggests that key conclusions, such as the unequivocal warming over the past century, are not sensitive to the analysis procedure.”
I am the ‘computer analyst’ mentioned in 3.2 who found the errors mentioned. I am also the person mentioned in 3.1 who looked at the code on Newsnight.
In paragraph 4 the authors write “It is possible, of course, that further scrutiny will reveal more serious problems, but given the intensity of the scrutiny to date, we do not think this is particularly likely.” This has turned out to be incorrect. On February 7, 2010 I emailed the Met Office to tell them that I believed that I had found a wide ranging problem in the data (and by extension the code used to generate the data) concerning error estimates surrounding the global warming trend. On February 24, 2010 the Met Office confirmed via their press office to Newsnight that I had found a genuine problem with the generation of ‘station errors’ (part of the global warming error estimate).
In the written submission by Sir Edward Acton there are two paragraphs that concern the things I have looked at:
“3.4.7 CRU has been accused of the effective, if not deliberate, falsification of findings through deployment of “substandard” computer programs and documentation. But the criticized computer programs were not used to produce CRUTEM3 data, nor were they written for third-party users. They were written for/by researchers who understand their limitations and who inspect intermediate results to identify and solve errors.
3.4.8 The different computer program used to produce the CRUTEM3 dataset has now been released by the MOHC with the support of CRU.”
My points:
1. Although the code I criticized on Newsnight was not the CRUTEM3 code the fact that the other code written at CRU was of low standard is relevant. My point on Newsnight was that it appeared that the organization writing the code did not adhere to standards one might find in professional software engineering. The code had easily identified bugs, no visible test mechanism, was not apparently under version control and was poorly documented. It would not be surprising to find that other code written at the same organization was of similar quality. And given that I subsequently found a bug in the actual CRUTEM3 code only reinforces my opinion.
2. I would urge the committee to look into whether statement 3.4.8 is accurate. The Met Office has released code for calculating CRUTEM3 but they have not released everything (for example, they have not released the code for ‘station errors’ in which I identified a wide-ranging bug, or the code for generating the error range based on the station coverage), and when they released the code they did not indicate that it was the program normally used for CRUTEM3 (as implied by 3.4.8) but stated “[the code] takes the station data files and makes gridded fields in the same way as used in CRUTEM3.” Whether
3.4.8 is accurate or not probably rests on the interpretation of “in the same way as”. My reading is that this implies that the released code is not the actual code used for CRUTEM3. It would be worrying to discover that 3.4.8 is inaccurate, but I believe it should be clarified.
I rest at your disposition for further information, or to appear personally if necessary.
John Graham-Cumming
March 2010
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
The denigration, minimization, name calling continues with “…amateur computer analyst…” and “…they don not affect the trends…” Quelle surprise!
“3.2 A problem identified by an amateur computer analyst with estimates of average climate (not climate trends) affecting less than 1% of the HadCRUT data, mostly in Australasia, and some station identifiers being incorrect. These, it appears, were genuine issues with some of the input data (not analysis software) of HadCRUT which have been acknowledged by the Met Office and corrected. They do not affect trends estimated from the data, and hence have no bearing on conclusions regarding the detection and attribution of external influence on climate.”
“…don’t…”
Great contribution, but wake me up from this nightmare, please:
All this hype, destroying economies, scaring kids and even killing people by fear, called global warming is based on
– poor weather stations readings, cherry-picked
– enhanced by irreproduceable software, shown here
– compared with unreliable proxies, cherry-picked again
– blaming a trace gas, as only factor, as Schellnhuber’s “linear correlation”
– wrongly mixed with inappropriate statistic methods, as MM have shown
– stonewalled against any checking, as Jones thinks is normal
– arrogantly called this the true and only science by ‘realclimate scientists’
– paid for with a Nobel Peace Price and billions of taxpayers’ money
– brought up as top priority by whatever kind of politicians
You couldn’t invent it.
I am a professional computer programmer who started programming almost 30 years ago. John Graham-Cumming
3.2 A problem identified by an amateur computer analyst
by Professor Hans von Storch and Dr. Myles R. Allen
WUWT?
This must be one of the “sophisticated” climate models I’m always hearing about.
John Graham-Cumming is a smart and fair person.
Since Climate Science (one of the youngest branches of science) is all about projecting the future trends for the Earth’s Climate- computer modeling is their primary tool.
It is quite worrying that there is no standardization for the computer code. Nor is there any review or testing process of the code and data sets used for the results.
Computer science (which is also a young science) has extensive requirements for code and data reliability. Lack of following even the basic standards is negligence, bordering on criminal intent to defraud.
The code is substandard? File this under no [fooling] Sherlock!
Anybody who works in software development knows there are open-source tools available for source control and testing. Some very good databases are also open-source. Surely CRU can afford to use open-source software? In many cases, it’s free!
The climatescience process really is from normalscience, isn’t it!
Conclusions
Results
Method
Apparatus
Thanks to J C-R and others, hopefully soon we will be able to get to the apparatus.
And, I like the idea of Acton being asked by Parliament to verify his statement that the actual CRUTEM3 production code has been released. When you think about it, IF the answer proves to be negative then the consequence would be enormous. In other words, this could turn out to be ‘match point’
So, can anybody comment on the status of the CRUTEM3 code?
I wonder what it takes to be more than “an amateur computer analyst”? If a doctorate and career in the field aren’t enough to qualify one as more than “an amateur computer analyst” then what is required?
While it is a minor point, it does not reflect well on the competence of the authors.
I worked as a programmer for a physics research group as an undergraduate and in industry before going to grad school in the 1980’s. It never occurred to the physicists to publish or make original code or even the data available. If someone wanted to replicate their work they would do the experiment themselves and write their own code. The papers described the mathematical methods used so that anyone could write a program to do the number crunching. In industry I often had to deal with very poorly documented code. The standards in industry have likely improved, but a lot a valid work was done in the bad old days.
Since climatology is such a hot area, I too would like to see greater openness. This will cost more money. Granting agencies will need to provide budget lines for professional programmers (rather than cheap students) and web servers for data archives. But, there is no basis for demonizing researchers who have been following the standards in their field or ignoring past work. If you want to redo someone’s chemistry experiment should you be able to demand the use of their test tubes? Maybe some day chemists will be required to archive all their old test tubes. But, that won’t invalidate all the chemistry that was done before.
Bernie’s Iron Rules of Survival in Modern Organizations
1. Do not lie to auditors.
2. Do not lie about programmers.
3. Never put in an email that which can be said with a wink.
Dr. Graham-Cumming diplomatically leaves out the word ‘amateur’ with which Professor Hans von Storch and Dr. Myles R. Allen denigrated his analysis. ‘Nuff said.
/Mr Lynn
Finding errors is a good thing because errors can be corrected. But implying errors affect conclusions, when they don’t, is not a good thing.
John in L du B (08:30:44) :
So Dr. Graham-Cummings has a BA in Matematics and Computation and a doctorate in Computer Security, has been writing software for 30 years but still only manages, in von Storch and Allen’s opinions, to qualify as an “amateur computer analyst”.
Are they “amature” climate scientists”?
****************************************************
You know the drill. If you don’t have the facts on your side, shoot the messenger. The only problem is that as the bodies accumulate, so do those who look for the facts to verify the claims. That’s the stage we’re in now. For years, the McIntyres and Cummings would get lumped with the crackpot oil-bought “denier’ crowd. There are too many McIntyres and Cummings now for that method to work. As far as I can tell, there is no Plan B, and this whole affair called AGW is working its way toward an ugly end.
For a satirical look at the climategate programming (hiding the decline):
Anthropogenic Global Warming Virus Alert
http://www.thespoof.com/news/spoof.cfm?headline=s5i64103
Should we all weep with laughter or frustration?
I suspect both
“John Galt (09:06:48) :
Anybody who works in software development knows there are open-source tools available for source control and testing. Some very good databases are also open-source. Surely CRU can afford to use open-source software? In many cases, it’s free!”
I am quite sure that CRU do not grasp the concept of open-source.
“Professor Slingo: … We test the code twice a day every day. We also share our code with the academic sector, so the model that we use for our climate prediction work and our weather forecasts, the unified model, is given out to academic institutions around the UK, and increasingly we licence it to several international met services: Australia, South Africa, South Korea and India. So these codes are being tested day in, day out, by a wide variety of users and I consider that to be an extremely important job that we do because that is how we find errors in our codes, and actually it is how we advance the science that goes into our codes as well. So of course, a code that is hundreds of thousands of lines long undoubtedly has a coding error in it somewhere, and we hope that through this process we will discover it. Most of the major testing is very robust.”
This statement indicates that Slingo knows almost nothing about software. The more salient questions are: how must test automation coverage do you have, how do you maintain your test cases, do you do unit testing upon check-in, what’s your procedure for regression testing, do you perform independent code reviews etc.
“We test the code twice a day every day” — this doesn’t make any sense. It sounds like something she just made up on the spur of the moment.
I’d like to recommend this version control software from Perforce Software Inc
http://uk.news.yahoo.com/22/20100303/ttc-uk-china-google-fe50bdd.html
“But, there is no basis for demonizing researchers who have been following the standards in their field or ignoring past work.”
Mike, thats true but by the same token if the standards (of computer coding out of which are born the models) are rather low in this field, should we take their work as good enough to be the basis of policies which could well cause considerable lifestyle changes not to mention rather more taxes?
My submission to the committee appears to have resulted in direct questioning of the people present. If you read the actual transcript you’ll see me mentioned twice.
For a somewhat technical post on this including the answer to my question given by Professor Jones and Professor Slingo see: http://www.jgc.org/blog/2010/02/something-bit-confusing-from-ueacru.html
Perhaps we should set up a project on SourceForge called “CRUTEMP” and help them along a bit! Although funny, I think we should seriously consider it.
We could start with the 160Mb of code and data that we already have (courtesy of our “insider”). We could even put a wiki together – to help the undergrad’s get up to speed.
Although I think we should only allow amateur admin’s. No Dr’s or post grad’s should be allowed access to the delete button. It’s a trust thing.
max (09:19:50) :
“I wonder what it takes to be more than “an amateur computer analyst”? If a doctorate and career in the field aren’t enough to qualify one as more than “an amateur computer analyst” then what is required?”
What is required? That’s very simple: agree with them. That’s all. Agree with AGW and you’re a professional. Disagree and you’re an amateur, denialist, climate criminal, contrarian, etc.
I bet if we asked any of these CRU people what “cyclomatic complexity testing” was done – they would reply that “we don’t test for cyclones in our software”.