Climategate: The Smoking Code

NOTE: Part 2 of this story has been posted: see The Smoking Code, part 2

The Proof Behind the CRU Climategate Debacle: Because Computers Do Lie When Humans Tell Them To

From Cube Antics, by Robert Greiner

I’m coming to you today as a scientist and engineer with an agnostic stand on global warming.

If you don’t know anything about “Climategate” (does anyone else hate that name?) Go ahead and read up on it before you check out this post, I’ll wait.

Back? Let’s get started.

First, let’s get this out of the way: Emails prove nothing. Sure, you can look like an unethical asshole who may have committed a felony using government funded money; but all email is, is talk, and talk is cheap.

Now, here is some actual proof that the CRU was deliberately tampering with their data. Unfortunately, for readability’s sake, this code was written in Interactive Data Language (IDL) and is a pain to go through.

NOTE: This is an actual snippet of code from the CRU contained in the source file: briffa_Sep98_d.pro

[sourcecode language=”text”]

;

; Apply a VERY ARTIFICAL correction for decline!!

;

yrloc=[1400,findgen(19)*5.+1904]

valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75   ; fudge factor

if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’

yearlyadj=interpol(valadj,yrloc,timey)

[/sourcecode]

Mouse over the upper right for source code viewing options – including pop-up window

What does this Mean? A review of the code line-by-line

Starting off Easy

Lines 1-3 are comments

Line 4

yrloc is a 20 element array containing:

1400 and 19 years between 1904 and 1994 in increments of 5 years…

yrloc = [1400, 1904, 1909, 1914, 1919, 1924, 1929, … , 1964, 1969, 1974, 1979, 1984, 1989, 1994]

findgen() creates a floating-point array of the specified dimension. Each element of the array is set to the value of its one-dimensional subscript

F = indgen(6) ;F[0] is 0.0, F[1] is 1.0….. F[6] is 6.0

Pretty straightforward, right?

Line 5

valadj, or, the “fudge factor” array as some arrogant programmer likes to call it is the foundation for the manipulated temperature readings. It contains twenty values of seemingly random numbers. We’ll get back to this later.

Line 6

Just a check to make sure that yrloc and valadj have the same number of attributes in them. This is important for line 8.

Line 8

This is where the magic happens. Remember that array we have of valid temperature readings? And, remember that random array of numbers we have from line two? Well, in line 4, those two arrays are interpolated together.

The interpol() function will take each element in both arrays and “guess” at the points in between them to create a smoothing effect on the data. This technique is often used when dealing with natural data points, just not quite in this manner.

The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.

What the heck does all of this mean?

Well, I’m glad you asked. First, let’s plot the values in the valadj array.

Artificial Hockeystick Graph

Look familiar? This closely resembles the infamous hockey stick graph that Michael Mann came up with about a decade ago. By the way, did I mention Michael Mann is one of the “scientists” (and I use that word loosely) caught up in this scandal?

Here is Mann’s graph from 1999

mann-hockey-stick-graph

As you can see, (potentially) valid temperature station readings were taken and skewed to fabricate the results the “scientists” at the CRU wanted to believe, not what actually occurred.

Where do we go from here?

It’s not as cut-and-try as one might think. First and foremost, this doesn’t necessarily prove anything about global warming as science. It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.

This means that all of those billions of dollars we spent as a global community to combat global warming may have been for nothing.

If news station anchors and politicians were trained as engineers, they would be able to find real proof and not just speculate about the meaning of emails that only made it appear as if something illegal happened.

Conclusion

I tried to write this post in a manner that transcends politics. I really haven’t taken much of an interest in the whole global warming debate and don’t really have a strong opinion on the matter. However, being part of the Science Community (I have a degree in Physics) and having done scientific research myself makes me very worried when arrogant jerks who call themselves “scientists” work outside of ethics and ignore the truth to fit their pre-conceived notions of the world. That is not science, that is religion with math equations.

What do you think?

Now that you have the facts, you can come to your own conclusion!

Be sure to leave me a comment, it gets lonely in here sometimes.

hat tip to WUWT commenter “Disquisitive”

========================

NOTE: While there are some interesting points raised here, it is important to note a couple of caveats. First, the adjustment shown above is applied to the tree ring proxy data (proxy for temperature) not the actual instrumental temperature data. Second, we don’t know the use context of this code. It may be a test procedure of some sort, it may be something that was tried and then discarded, or it may be part of final production output. We simply don’t know. This is why a complete disclosure and open accounting is needed, so that the process can be fully traced and debugged. Hopefully, one of the official investigations will bring the complete collection of code out so that this can be fully examined in the complete context. – Anthony


Sponsored IT training links:

Join today for 646-985 exam prep and get a free newsletter for next 642-072 and 1z0-050 exams.


Advertisements

  Subscribe  
newest oldest most voted
Notify of
INGSOC

Thanks for the great article! Lots to digest. Be prepared for the media blitz that is building however. AGW aint dead yet! I hope we see many more “agnostics” stand up for truth in science. We can beat back the drumbeat of lies by MSM.
Cheers!

Line 8 appears to be missing from the listing.
Excellent article.

Oh, you mean the 8th line since they start with line “00”. Using the files line numbers might make the article more clear.

MangoChutney

i guess Phil Jones leave of absence could be a long one

I can’t comprehend their justification for this obvious blatant fraud.
What was Mann thinking when he manNipulated this data in this manner?

Medic1532

Thanks for the explanation for those of us who aren’t fluent in the various computer languages (I think I still remember how to write a BASIC program lets see 10 Print “Hello World”;20 goto 10 ) I figured the EMail uproar was just the first shot.
Medic1532

Anton

The clear explication provided here highlights a comment I made (as did many others) yesterday. Exclusive focus by the UEA investigation (or the MSM) on the emails would be a perfect way to miss the main point of the released material. Even if the the emails had been entirely innocuous, the computer codes, etc would be more than enough to prove fraud.

Varco

I believe the emails are acting as a distration for the media from a key topic of what what actually done with (to?) the data by way of programming. I found this explanation extremely lucid and hope it inspires passing journalists to review the Harry readme file. I’m not sure you need to be a programmer to understand that ‘Harry’ is not describing best scientific practice let alone programming!
Hopefully work such as this will lead to more detailed investigation of data analysis (manipulation?) techniques being employed by all the major climate data providers?

So the Mann-made hockey stick proxy end was artificially elevated to match the “instrumental record”. The same story as Briffa´s Yamal, where the hockey blade was achieved by cherry-picked series.
The red “instrumental record” is GISS 1890-1999: http://www.woodfortrees.org/plot/gistemp/from:1890/to:1999
Can we use f- word yet?

pwl (05:40:23) :
> Line 8 appears to be missing from the listing.
Anthony needs to put a blank line at the top of the <pre> block to get the line numbers in sync. Currently everything is off by one.
REPLY: the WordPress formatting did that, I’ve switched to a different formatting tool that has a little popup toolbar in the upper right to allow window viewing of long lines of code – Anthony

MangoChutney

just a thought, if the emails and code were obtained by a hacker and not released by a whistle blower, would the evidence be inadmissible in a court of law?

“It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.”
The code is awful. But do we know that the output of the code discussed here was actually published somewhere? I didn’t yet find an explanation of what the significance of this program is, and how it has been used.
(should “cut-and-try” not be “cut-and-dry”?)

SHIRAKAWA Akira

The source code should start from line 01, not 00.

The end!!!
This following short analysis might be easy to understand the reasons why the ’so-called scientists’ would not release the code for scrutiny. If you can’t independently run the code and get the same results as the output the Fraudsters display, then something is wrong. If you can run the code and get the output displayed, then you only need analyze the software code to see what is wrong that produces the FAKED hockey stick. Easy for software engineers, not so easy for lay people — Which is what this is based on, fooling lay people into paying massive new taxes.

bill

Hang on a moment! This code is from 1998.
If this fudge were included in the Briffa etc. documents then there would be no decline.
So all those “hide the decline emails” would be irrelevant.
The only conclusion is that this bodge was done to see what adjustments would have to be made to fit the temp record.
Having done that it was not used for any submitted work

Varco

Off topic, but can we expect an update to the surfacestations work anytime soon? Papers were mentioned at the time of the mid-term census report?
I think more than anything it was the early results from the Surfacestations project that convinced me descriptions of ‘settled science’ were horribly compromized. While Climategate is rightly grabbing attention at the moment perhaps passing journalists would care to check out surfacestations.org for examples of how data collection and reporting should be undertaken. The hard work of the volunteers in collating the damning (IMHO) evidence about the temperature measuring network, and the transparency in which the data has been recorded and presented is commendable and should be given public recognition – we owe these people a debt of thanks. Most importantly it shows there is a way forward for climate science that can be believed, a breath of fresh air in an otherwise fetid atmosphere?

durox

ONE international is sending emails all over the web. once you click the link in the invitation, you sign their petition. you can sign as many times as you want by just clicking, which i find to be in bad taste.
for more info visit http://one.org/international/actnow/copenhagen/index.html?rc=copenhagenconfemail
and pls write about this ongoing unfair effort. thanks

Well we all know that the code was realy the smoking gun. Hopefully the MSM will stop trying to gloss over the issue here and start truly attacking the guilty.

Leon Brozyna

And there we have, in simple form, a recipe for cooking the books.
Give a person weak in science and ethics some information on programming and statistics and they’ll be able to “prove” anything they want.

rbateman

Now there is a thought:
Re-write the IDL into BASIC.
Lots of people understand Visual Basic, and could get it.

Jeremy

Yes I saw this code about a week ago when it first appeared. Your article above is correct. Whether it was used and for what published journals may be difficult to prove, however, the similarity to the trend in the hockey stick plot is a “smoking gun” if I ever saw one.

slow to follow

bitbutter – yes, this should be said loud and clear. And investigated before too much is built on it. Has anyone seen anything establishing the provenance of this code?

Alberto

I’m missing some context here. I would like to know for which series/reconstructions this program was used for, what were the reasons for the adjustments etc.
Without the proper context, it’s difficult to determine the implications.
So a follow up would be nice.

Burch

Sorry, I don’t know IDL. In the initialization of the array valadj, there is a ‘*0.75’ at the end. What does that accomplish? Multiply each initializer by 0.75? If so, then why not just put the proper values into the initializer string? Is it possible that the ‘fudge factor’ comment refers only to the 0.75 constant and not the entire array? Is there anything in the code that explains where the constants in the initializer string came from?
There are two ways to view this. One is they were sincerely trying to correct for something, the other is they were cooking the books to show a trend that did not exist.

Chilled Out

Line 5 – “valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor”
I’d love to see any of the IPCC climate scientist try to explain these numbers – take an apparently arbitrary set of numbers, multiply by 0.75 and bingo we have a “hockey stick” curve.
Note their explantion needs to be backed up by released data, and detailed analysis to explain how each of the factors has been derived and the independent quality assurance that has been applied to the calculations/methodology.
Will we get it – not f**king chance from the CRU, coz they have lost/mislaid/deleted the data 😉

Thomas J. Arnold.

It is precisely what data set scrutiny requires.
A type of public, objectivity and as you put it “uninterested” stance that leads to in depth analysis and results in conclusions which are accurate and that can be believed, Mr. Greiner.
People like yourself, a Physics grad’, familiar in Pure science, with no axe to grind and with other tech’ skills, are needed, fancy a job?…………There will be some going in East Anglia soon…………… I predict.
Trouble is……can you do it for altruistic purposes? After the scandal there will be no money in it, gov’ research funding has taken a big hit.

Tom_R

1. This code section was mentioned in a previous WUWT thread.
2. Note that the valadj array is multiplied by 0.75, so you graph should show 1.95 as the top value, not that it makes much difference in the primary argument.
3. One question I have is whether or not this code segment was actually used to create a published figure. As someone who has written a lot of code, I have often created phony data just to test certain routines. As far as I know, Briffa has not made this claim, so to make it now would be suspicious. If someone can show that this code was used to (say) create a graph for an IPCC report, then that would definitely be a smoking gun.

mikey

Wow everyday more and more is coming out.
slightly off-topic but this new oped on CIF is just shocking:
http://www.guardian.co.uk/commentisfree/libertycentral/2009/dec/04/climate-change-scepticism

wolfwalker

I saw what I think is the same chunk of code analyzed on another blog entry a few days ago. If I could remember which one I’d provide a link, but I don’t so I can’t. Sorry. In any case, one of the commenters there raised a very pertinent question. We can see that the variable ‘yearlyadj’ incorporates the fudged data. Is that variable used anywhere? If it isn’t, then there doesn’t appear to be any way for the fudge to actually infect the program output.

PhilW

It’s just gone mainstream here in UK!

slow to follow

Anthony – do you believe this code is
“actual proof that the CRU was deliberately tampering with their data.”?
If not I think you should caveat this article accordingly.

Douglas DC

“However, being part of the Science Community (I have a degree in Physics) and having done scientific research myself makes me very worried when arrogant jerks who call themselves “scientists” work outside of ethics and ignore the truth to fit their pre-conceived notions of the world. That is not science, that is religion with math equations.”
Don’t sugar coat it Pard, tell us what you think.I agree whole heartedly…

John E.

I don’t see where this code actually obtains any temperature data. yearloc is just an array of years, right? Does interpol() access a database of temp records for the particular year and then apply the skewing fudge factor array to those temps?
I defintely see how the fudge factor array will create a hockey stick. But to reach the conclusion in the post we need to see how this connects to the temperature record that they published.

Hey, thanks for linking to my article 🙂
This website actually was what turned me on to the whole issue.
@Burch, the .75 and “fudge factor” array serve as kind of a guess-and-check hardcoded curve that forces legitimate data to look like whatever the hell the researchers wanted to. Likely the .75 and each individual data value in the fudge factor array were all tweaked little by little until the original data looked like a “hockey stick”, which is what Michael Mann hypothesized a decade ago.
It’s a simple case of falsifying data to fit a pre-conceived feeling.
excellent question.

EdB

It is simple. Put the programmer under oath, get a line by line explaination of what was done and who asked for the code, and why. Then put under oath the people that used the results of the code, and ask them what papers, reports, advice, was based on the coded program.
The truth can come out, if the authorities want it. I am a skeptic though.

hunter

Edb,
Aren’t we all?

John Galt

I think the whole AGW climate doomsday is a travesty of science and has always been a convenient tool for forcing a particular world-view upon us non-believers.
That said, can we verify the authenticity of this code? Perhaps the one thing that might come from this is EUA, GISS, etc., will finally be compelled to release their data and source code.
As some one said, if you can’t replicate the results, it ain’t science.

bill

Robert Greiner you state:
Line 8
This is where the magic happens. Remember that array we have of valid temperature readings? And, remember that random array of numbers we have from line two? Well, in line 4, those two arrays are interpolated together.
The interpol() function will take each element in both arrays and “guess” at the points in between them to create a smoothing effect on the data. This technique is often used when dealing with natural data points, just not quite in this manner.
The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.

Lets look at a bit more of that code:
; Apply a VERY ARTIFICAL correction for decline!!
;yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
yearlyadj=interpol(valadj,yrloc,timey)
;
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20
;
filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=21
yearlyadj=interpol(valadj,yrloc,timey)
Does not this line give a yearly adjustment value interpolated from the 20 year points?
filter_cru,5.,/nan,tsin=yyy,tslow=tslow oplot,timey,tslow,thick=5,color=21
Does not this line plot data derived from yyy
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20

The smoking gun line!!!!
Does not his line plot data derived from yyy+yearlyadj The FUDGED FIGURE
BUT…………
IT’S COMMENTED OUT!!
This is further backed up by the end of file:
plot,[0,1],/nodata,xstyle=4,ystyle=4
;legend,[‘Northern Hemisphere April-September instrumental temperature’,$
; ‘Northern Hemisphere MXD’,$
; ‘Northern Hemisphere MXD corrected for decline’],$
; colors=[22,21,20],thick=[3,3,3],margin=0.6,spacing=1.5
legend,[‘Northern Hemisphere April-September instrumental temperature’,$
‘Northern Hemisphere MXD’],$
colors=[22,21],thick=[3,3],margin=0.6,spacing=1.5

To me this looks as if ‘Northern Hemisphere MXD corrected for decline’ would have been printed in colour 20 – just the same as the smoking gun line. HOWEVER you will note that this section is commented out also.
So in my view this is code left in after a quick look-see.
Remember engineers ans scientist are human and play if bored and do not always tidy up.
have a look at:
http://micro.magnet.fsu.edu/creatures/index.html

Genaddicted

Okay, I hate nitpickers, so I was going to let it go, but someone else asked the question, so here’s the answer:
“cut and try” should not be “cut and dry” either – it’s “cut and dried” – past tense.
Very good post, by the way.

Jack Green

You could digitize the output graph that they have published and get the data if they published one with the raw values. If not you could write some code using this code and back calculate the raw values.
Somebody smarter then me could do it. My training as and expert witness and being around a lot of smart lawyers tells me we could get the raw data from what we have.
I would bet the raw data is somewhere on a HCRU computer just sitting there. They whistle blower as I remember when this all started has more data to release or am I remembering incorrectly.

rw

From the comments about line nos, I gather that most of you folks are not programmers.
Is this code performing Mike’s Nature trick ?

David Schnare

I have queried one of the folks who is familiar with this code and who is a major name in this mess, and he defended the code as follows:
The opening comment: “Apply a VERY ARTIFICAL correction for decline!!” indicates this is not a computer run that would be used in a paper, but is merely one of many runs used to understand how the model operated under different scenarios. He argued that no one could identify a single peer-reviewed paper that relied on this code.
I can believe that one might want to play around with a model to understand how sensitive it is to various scenarios, including presumptions about temperature curves. One would not be looking at the graph, but would be looking at what the forcings would have to be to generate such a graph. This would allow a better understanding of the power of various forcings. It would help tease out which forcings have the greatest significance to model outputs.
There would surely be other ways to reach the same understanding, but one can’t account for how any particular scientist goes about his or her business.
I am forced to reserve judgment until I see how this code was actually used and whether it was used to support a specific academic contribution.

Tim Clark

bill (05:58:43) :
Hang on a moment! This code is from 1998.
If this fudge were included in the Briffa etc. documents then there would be no decline.
So all those “hide the decline emails” would be irrelevant.
The only conclusion is that this bodge was done to see what adjustments would have to be made to fit the temp record.
Having done that it was not used for any submitted work

I don’t think this applies to the Briffa et al work per se:
This is an actual snippet of code from the CRU contained in the source file
IMHO it applies to the CRU temperature reconstruction which was on the same graph as the tree series as a comparison indicating agreement between the two. Since we know that Briffa’s work was bogus, and now that the CRU was also rigged, I guess they are in complete agreement. But nice try Bill, where’s Mary Hinge, Joel S. and RRKampen these days?

imapopulist

So the data output from this program will be the result of an interpolation between (essentially a combination of) the actual temperature data for each period of time and a list of increasing values that have been artificially incorporated into the program. The list of values was intentionally skewed to reflect higher temperatures in the later periods and to conform with periods of increasing CO2 levels.
In other words, this computer program is producing bald faced lies.
This is without any doubt a criminal fraud.
How else can anyone justify such a manipulation of data?

I’d seen this earlier and added it to my page for newcomers explaining “The Decline” and the hiding thereof and theclose-up details of UEA’s latest graph which only digs their hole deeper. At the bottom of that page is more code, with comments, putting the above snippet further in context.

imapopulist

“NOTE: This is an actual snippet of code from the CRU contained in the source file: briffa_Sep98_d.pro”
[snip]

Phil M

There’s a problem with your analysis:
– the ‘yearlyadj’ fudge factor isn’t used in this code!
– if you look a couple of lines further down you can see that the lines which would adjust the data, filter it & then plot it out are actually commented out using ‘;’s

;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20

and code that does the actual plotting doesn’t use the ‘yearlyadj’ fudge factor

filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=21

– so, in this example at least, the fudge-factor isn’t used, even if it looks a little suspicious.
– perhaps it was just some test code, or Briffa changed his mind about using it….
Actual code:
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
yearlyadj=interpol(valadj,yrloc,timey)
;
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20
;
filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=21

Chris

The data array in line 5 is the “hockey stick” and 0.75 multiplier is the “fudge factor”. In other words, the former gives the data a curve effect, and the latter is adjusted to provide the proper steepness (or distortion to the data).
You guys realize don’t you that 2 parameter models (i.e., nothing complicated, very simple) can essentially model most known phenomena to about 10% accuracy.

JonesII

Great post. Small and simple enough to be used by media. Something like this:
“Look how they waterboarded computer programs these pseudo scientists to cheat YOU with the tale of global warming”