NOTE: Part 2 of this story has been posted: see The Smoking Code, part 2
The Proof Behind the CRU Climategate Debacle: Because Computers Do Lie When Humans Tell Them To
From Cube Antics, by Robert Greiner
I’m coming to you today as a scientist and engineer with an agnostic stand on global warming.
If you don’t know anything about “Climategate” (does anyone else hate that name?) Go ahead and read up on it before you check out this post, I’ll wait.
Back? Let’s get started.
First, let’s get this out of the way: Emails prove nothing. Sure, you can look like an unethical asshole who may have committed a felony using government funded money; but all email is, is talk, and talk is cheap.
Now, here is some actual proof that the CRU was deliberately tampering with their data. Unfortunately, for readability’s sake, this code was written in Interactive Data Language (IDL) and is a pain to go through.
NOTE: This is an actual snippet of code from the CRU contained in the source file: briffa_Sep98_d.pro
[sourcecode language=”text”]
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
yearlyadj=interpol(valadj,yrloc,timey)
[/sourcecode]
Mouse over the upper right for source code viewing options – including pop-up window
What does this Mean? A review of the code line-by-line
Starting off Easy
Lines 1-3 are comments
Line 4
yrloc is a 20 element array containing:
1400 and 19 years between 1904 and 1994 in increments of 5 years…
yrloc = [1400, 1904, 1909, 1914, 1919, 1924, 1929, … , 1964, 1969, 1974, 1979, 1984, 1989, 1994]
findgen() creates a floating-point array of the specified dimension. Each element of the array is set to the value of its one-dimensional subscript
F = indgen(6) ;F[0] is 0.0, F[1] is 1.0….. F[6] is 6.0
Pretty straightforward, right?
Line 5
valadj, or, the “fudge factor” array as some arrogant programmer likes to call it is the foundation for the manipulated temperature readings. It contains twenty values of seemingly random numbers. We’ll get back to this later.
Line 6
Just a check to make sure that yrloc and valadj have the same number of attributes in them. This is important for line 8.
Line 8
This is where the magic happens. Remember that array we have of valid temperature readings? And, remember that random array of numbers we have from line two? Well, in line 4, those two arrays are interpolated together.
The interpol() function will take each element in both arrays and “guess” at the points in between them to create a smoothing effect on the data. This technique is often used when dealing with natural data points, just not quite in this manner.
The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.
What the heck does all of this mean?
Well, I’m glad you asked. First, let’s plot the values in the valadj array.
Look familiar? This closely resembles the infamous hockey stick graph that Michael Mann came up with about a decade ago. By the way, did I mention Michael Mann is one of the “scientists” (and I use that word loosely) caught up in this scandal?
Here is Mann’s graph from 1999
As you can see, (potentially) valid temperature station readings were taken and skewed to fabricate the results the “scientists” at the CRU wanted to believe, not what actually occurred.
Where do we go from here?
It’s not as cut-and-try as one might think. First and foremost, this doesn’t necessarily prove anything about global warming as science. It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.
This means that all of those billions of dollars we spent as a global community to combat global warming may have been for nothing.
If news station anchors and politicians were trained as engineers, they would be able to find real proof and not just speculate about the meaning of emails that only made it appear as if something illegal happened.
Conclusion
I tried to write this post in a manner that transcends politics. I really haven’t taken much of an interest in the whole global warming debate and don’t really have a strong opinion on the matter. However, being part of the Science Community (I have a degree in Physics) and having done scientific research myself makes me very worried when arrogant jerks who call themselves “scientists” work outside of ethics and ignore the truth to fit their pre-conceived notions of the world. That is not science, that is religion with math equations.
What do you think?
Now that you have the facts, you can come to your own conclusion!
Be sure to leave me a comment, it gets lonely in here sometimes.
hat tip to WUWT commenter “Disquisitive”
========================
NOTE: While there are some interesting points raised here, it is important to note a couple of caveats. First, the adjustment shown above is applied to the tree ring proxy data (proxy for temperature) not the actual instrumental temperature data. Second, we don’t know the use context of this code. It may be a test procedure of some sort, it may be something that was tried and then discarded, or it may be part of final production output. We simply don’t know. This is why a complete disclosure and open accounting is needed, so that the process can be fully traced and debugged. Hopefully, one of the official investigations will bring the complete collection of code out so that this can be fully examined in the complete context. – Anthony
Sponsored IT training links:
Join today for 646-985 exam prep and get a free newsletter for next 642-072 and 1z0-050 exams.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
bill (09:36:37) :
g hall (08:57:10) :
J. Bob (08:56:09) :
John Galt (08:43:01)
JJ (08:39:45) :
david (08:14:07) etc. etc.
The smoking gun is commented out (see the “;” = comment). Why cannot you see this?? It is obvious.
The code is written by scientists for a 1off use. Why would you clean it up and make it presentable. They are not going to sell it to others.
It’s called professional standards.
Do you take any pride in your work? Are you sloppy when you think nobody is looking? Cut corners when the boss isn’t around?
Any dataset on which this was used would have a output that was divisible by 0.75. You could pick it up using Benfords law.
Audit commander has a free upload and will do a Benfords analysis. All you need are some outputs.
bill (05:58:43) :
Hang on a moment! This code is from 1998.
If this fudge were included in the Briffa etc. documents then there would be no decline.
So all those “hide the decline emails” would be irrelevant.
No, there was a decline and it was hidden in ipcc and other official documents – as described by Steve McIntyre and Jean S..
NOAA also even deleted the post 1960 Briffa data from its archive, as though there was no data to begin with!
Yeah, but …
Can’t we still restructure the entire economy of the US and the West, effectively transferring all manufacturing to China, India, and the Asian Tigers? Along with millions of jobs and trillions in GNP? Just in case?
Technical nit about the description of line 4: The IDL findgen(n) function generates a one-dimensional array of n elements, stepping from 0 to n-1. (So findgen(6) goes from 0.0 to 5.0, and not to 6.0 as listed above.)
Thus, the findgen(19) generates a 19-element array going from 0 to 18 in integer steps, which, after being concatenated to the initial value, makes yrloc a 20-element array as stated (and the ending year for yrloc that you list is correct).
Now that we know that the raw data was discarded by UEA (although personally I believe it is spirited away somewhere – they should confiscate Jones et al’s home computer, etc.) is it possible to refind it by reviewing the “Codes” and removing the fudge factors? A bit of forensic computation is needed here. These things must be considered by the independent commissioner appointed by UAE to ferret out the truth.
WAG (10:00:06) :
Anyone see this?
http://www.nationalpost.com/news/canada/story.html?id=2300282
Apparently Climategate was only one of several organized efforts to break into universities and steal data.
Reply:
Sounds like media spin to make skeptics look like criminal kooks. There is no one to refute his story so it is a very safe spin.
I will believe that AFTER I see a police report dated several weeks ago. At this point I do not believe him and I doubt the media will bother to check if his story is true. Chances are there is a University police department so even checking with the police does not necessarily prove he is telling the truth.
The MSM showed faked film footage of a major riot at Purdue University just after the Kent state riot to get the governor off the hook for the murder of four students. The film was shown nationwide but the riot never happened. I have not trusted the MSM ever since. The Kent State riot was about the town not allowing married adult Vietnam Vets the right to vote if they were students and NOT about protesting the war. However denying voting rights would have drawn a lot more criticism so the story was changed and the riot at Purdue fabricated. (based on first hand evidence by me and my boyfriend we were at the “riots”)
Just a moment, just a moment. I have just picked up a fault in the CRU 35 unit. It will go 100% failure within the next 72 hours.
you guys realized that the variable you are talking about is never even used in the code in that file?
here is the code: http://www.di2.nu/foia/osborn-tree6/briffa_sep98_d.pro
they use this data in the interpolation and get this yearlyadj
and that variable is never used.
i think the CRU data is very incriminating, this is just a very bad example.
WAG (10:00:06) :
Anyone see this?
http://www.nationalpost.com/news/canada/story.html?id=2300282
Apparently Climategate was only one of several organized efforts to break into universities and steal data.
So releasing emails and documents created by the climate researchers themselves would discredit climate science?
I can understand how fake emails and documents could be used to attempt to discredit AGW, but real documents and real emails are even more damning?
I’ll admit, a lot of software is like sausage — you don’t want to know how it’s made or what’s in it — but given the ramifications of AGW, isn’t it necessary for these scientists to actually show their work?
BTW: Aren’t there whistleblower laws that would offer protection to somebody to step forward and shine the light on what is really happening?
George (10:16:34) :
Just a general comment on the code and comments from various links above.
I saw the numbers in the Excel file for grant dollars. If this crap data was in such a state, why the heck did they not use some of the grant money to hire a grad student, even in Computer Sciences, to get them a real database? They could have created a SQL database from all these crap flat files and made life so much easier. Guess I have been in the corporate enterprise world too long and just do not understand acedemia at all. It does not address the raw data issue, but it could have preserved it and made it easier. sigh.
Any database would do, even one of the various open-source databases available. Imagine the analysis that could be done if the data was properly stored in a well-designed database?
“I tried to write this post in a manner that transcends politics. ” – Hahaha!
“I’m coming to you today as a scientist and engineer with an agnostic stand on global warming.” – Double hahaha!!
“It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.” – _ALL_ of the data? This was applied to _ALL_ of the AGW data in 1998???
Bill, Tim,
I doesn’t seem a coincidence that the “correction” corresponds almost exactly to the 2.3 degree temperature fall over 35 years in Briffa’s 1998 paper. This paper was based on 400 trees across the whole N hemisphere; much more meaningful that the later narrow selections. It would have been a great embarrassment to the CRU at the time.
“you guys realized that the variable you are talking about is never even used in the code in that file?”
So, can you say with authority that the scope of the variable is limited to the file? Just curious. I don’t know the answer, but if the variable is never used anywhere in the code… anywhere, than this is much ado about nothing.
Just don’t understand what “hide the decline” means then. Occam’s razor suggests that the answer is probably one of the simpler interpretations of the evidence at hand. I can’t see how “hide the decline” can be use in so many different places to mean the exact thing it needs to mean to avoid being incriminating, even if that thing is different in every “context”
I see though that the toothpaste cleanup crew are out in force to try and force it back into the tube. The truth will out on this code. Objections will be upheld or answered. This is not going to be settled by rhetorical questions.
To all the nerds dissecting the code like some cadaver down to the cellular level get friggin’ real. Tell this to the Inuit’s who for centuries have lived and prospered as an indigenous population as they watch their homes sink into the methane excreting bog that was frozen for milleniums…no snow on Kilamanjaro, glaciers receding around the planet while the desertification belt expands, bird migrations changing and most improtant, something quite easily measureable…duh…the acidification of the friggin’ oceans. Seems the biggest carbon sink of all may sooner than later not be drinking any carbon at all. Rock on number freaks you’ve really proved it now.
Suggestion get out into the real world once in a while…smell the roses tatse the bitters ;(
Wow, Chris that was awesome. You managed a nice round up of debunked AGW theories in a just a few short sentences. Keep drinkin the Kool-Aid! AL Gore is counting on you to keep him well funded. I just wish he would show up and debate someone oneday….
It’s the code stupid.
Thanks Anthony for reminding us all not to get to polarized before we see the entire code warts and all. Context is everything here. I wonder how many commentors actually read your note. All we need are the raw data and the naked code. Surely the FOIA will eventually allow qualified individuals to examine the data in question and all will be made clear. That is my hope. The truth has a way of muscling in and establishing itself. Be patient grasshoppers, strength is gained first through knowledge.
BBC Newsnight programme last night did an item on this code, a expert programmer said that the code was flawed,as it missed date it was supposed to be analysing.
And the person who programme the code was not very good at his job
something the actual code programmer put in to the programes comments
comments like “oops theres my bad programing again”
You might find Newsnight on the BBCiplayer 3/12/2009 starts 22.30hrs
great site Anthony,keep up the good work.
“I can’t comprehend their justification for this obvious blatant fraud. What was Mann thinking when he manipulated this data in this manner?”
He was thinking about all the millions in grant money that he stood to lose as well as the long-term damage to his career, that’s what.
Yup…. It’s true….
I have turned to the dark-side……..
The University site has a form where you can submit your funding research program…..
You know……
Is it just me… or…. do l have fun in a strange way?
Anyhoo………
In the comments section where you can outline what you are after l have proposed the following:
I am interested in funding your Climate Research Unit as long as they come up with the results i’m looking for. Please pass this on to Professor P. Jones as he may be looking for a new job and that means he may be cheaper to afford.
I am also looking for someone to destroy the original raw temperature data after they skew the numbers the way l like them. If you can suggest anyone that would be great.
Also, l can give a job to any and all of the Climate Researchers that lied, manipulated and fudged their way into the “Discredited Scientist” (Seancetist) category. That is, l will pay them to apologise to everyone they have deceived.
Wow, i’m on fire here today! Can you please invite me to Phils’ early “Retirement” party? I will bring the drinks and the credibility (which is sadly lacking there at the CRU)
Brrr… It’s getting cold here….
MUST *giggle BE *giggle GLOBAL *giggle COOLING
Hey, please tell Phil that he may not have proved Global Warming but he really has warmed my heart. The best was when he pretended to be “Weird Al Yankovick” and drank from the toilet bowl. We are still recovering. It was just the best ! *snorting giggle
But we really are greatly interested in his decline. We, here in the Globe that’s Warming, have been looking for his decline everywhere but can not find it. Please ask him if he knows where he last saw it, and we will keep looking here. I hope he has not forgotten where he put his decline.
If he, by mistake, accidentally hid his decline please ask him to keep looking as l would like his decline seen by the whole world.
Thank you… and l hear England is warm and dry at the moment.
Seriously…. It’s called “Global Raining”
Throw it up…. see if it has wings……
Back to me now
If you have any research ideas please contact me as l think this may be a one off thing…..
NickB. (11:00:41) :
You post amounts to nothing more than – “In the absence of data, we are free to make up whatever fits our preconcieved conclusion”
That is Teamspeak, and should not be practiced here.
No one is asking that Anthony take down this post, only that he tone it down so that the conclusions drawn are supported by the data presented.
We do not (yet) have sufficient information to declare this code snippet a ‘smoking gun’ or anything of the sort. We need to keep digging, instead of going off half cocked.
Anthony should tone down the conclusory tone of this post, until the above questions are relaibly answered.
Incidently, for those of you who think ‘It was commented out!’ is some sort of definitive resolution to this issue – It Aint. When we programmers ‘comment out’ code like that, we do it so that we can comment it right back in if we want. If we have no intention of ever running that code again, we dont comment it out. We delete it. That the code was commented out is no proof whatsoever that it was never used.
We need to know what this code does, what the data in it represent, to which data it was applied, what the results were, and how the results were used. Absent that, this is just an interesting lead that needs to be followed. Pretending otherwise is hypocritical, and can backfire in ways that could be used to smokescreen the objects of legitimate criticism that we are seeing in the whistleblowers data.
Why didn’t they use some of those grant dollars to hire one or more people trained in IT? Hire a real programmer, a real DBA, and buy a real database? Well…
(1) They are self-centered narcissists who think they are so smart they don’t need to hire anyone because they are so great they can do anything at all, or
(2) Because it is very CONVENIENT that the original data has been “lost” and that the model code is a rats nest of unintelligible spaghetti.
Or perhaps both.
JJ (14:14:08) :
Admittedly, maybe I misread the following comment:
____________________________________
slow to follow (09:16:30) :
Anthony – please pay attention to the comments above. Leaving this post up with no caveat damages the credibility of your blog. Many will be passing by here without the time/interest to read the comments in detail. Unless this can be shown to produce production output to call it “smoking code” is disingenuous.
____________________________________
That said, this is a repost of the original here: http://cubeantics.com/2009/12/the-proof-behind-the-cru-climategate-debacle-because-computers-do-lie-when-humans-tell-them-to/
As a re-post/guest-post, I’m not sure how exactly Anthony could “tone it down”. It’s my understanding that in this situation the only options are caveat/disclaimer (which he had already done) or removing it entirely
Other than your summary of me that “In the absence of data, we are free to make up whatever fits our preconcieved conclusion”… that’s not what I meant to say and I don’t think that’s what I said.
What I was trying to get to is that these out of context snippets are the only pieces of the puzzle available, so the feeding frenzy should not come as a surprise. This is red meat but without context cannot definitively prove anything. No disagreement on that
Chris,
Yes, we’ve all noticed how all the cold places are getting warmer, the warm places are getting colder, dry places getting wetter, wet places getting drier, calm places windier, windy places calmer, cloudy places sunnier and sunny places cloudier.
Anything you wish to add?
Since there are nearly 200 responses here, and it’s a pain to read through ALL of them, let me repeat this point for those who missed it above. Yes, the use of the array is commented in the exemplar file. However, it is NOT commented, and used twice, in a different source file. ‘briffa_sep98_e.pro’
That said, this does not prove that the offending code was used in any published examples. Only that it could have been.
This is just one more bit of the puzzle to ponder.