NOTE: Part 2 of this story has been posted: see The Smoking Code, part 2
The Proof Behind the CRU Climategate Debacle: Because Computers Do Lie When Humans Tell Them To
From Cube Antics, by Robert Greiner
I’m coming to you today as a scientist and engineer with an agnostic stand on global warming.
If you don’t know anything about “Climategate” (does anyone else hate that name?) Go ahead and read up on it before you check out this post, I’ll wait.
Back? Let’s get started.
First, let’s get this out of the way: Emails prove nothing. Sure, you can look like an unethical asshole who may have committed a felony using government funded money; but all email is, is talk, and talk is cheap.
Now, here is some actual proof that the CRU was deliberately tampering with their data. Unfortunately, for readability’s sake, this code was written in Interactive Data Language (IDL) and is a pain to go through.
NOTE: This is an actual snippet of code from the CRU contained in the source file: briffa_Sep98_d.pro
[sourcecode language=”text”]
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
yearlyadj=interpol(valadj,yrloc,timey)
[/sourcecode]
Mouse over the upper right for source code viewing options – including pop-up window
What does this Mean? A review of the code line-by-line
Starting off Easy
Lines 1-3 are comments
Line 4
yrloc is a 20 element array containing:
1400 and 19 years between 1904 and 1994 in increments of 5 years…
yrloc = [1400, 1904, 1909, 1914, 1919, 1924, 1929, … , 1964, 1969, 1974, 1979, 1984, 1989, 1994]
findgen() creates a floating-point array of the specified dimension. Each element of the array is set to the value of its one-dimensional subscript
F = indgen(6) ;F[0] is 0.0, F[1] is 1.0….. F[6] is 6.0
Pretty straightforward, right?
Line 5
valadj, or, the “fudge factor” array as some arrogant programmer likes to call it is the foundation for the manipulated temperature readings. It contains twenty values of seemingly random numbers. We’ll get back to this later.
Line 6
Just a check to make sure that yrloc and valadj have the same number of attributes in them. This is important for line 8.
Line 8
This is where the magic happens. Remember that array we have of valid temperature readings? And, remember that random array of numbers we have from line two? Well, in line 4, those two arrays are interpolated together.
The interpol() function will take each element in both arrays and “guess” at the points in between them to create a smoothing effect on the data. This technique is often used when dealing with natural data points, just not quite in this manner.
The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.
What the heck does all of this mean?
Well, I’m glad you asked. First, let’s plot the values in the valadj array.
Look familiar? This closely resembles the infamous hockey stick graph that Michael Mann came up with about a decade ago. By the way, did I mention Michael Mann is one of the “scientists” (and I use that word loosely) caught up in this scandal?
Here is Mann’s graph from 1999
As you can see, (potentially) valid temperature station readings were taken and skewed to fabricate the results the “scientists” at the CRU wanted to believe, not what actually occurred.
Where do we go from here?
It’s not as cut-and-try as one might think. First and foremost, this doesn’t necessarily prove anything about global warming as science. It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.
This means that all of those billions of dollars we spent as a global community to combat global warming may have been for nothing.
If news station anchors and politicians were trained as engineers, they would be able to find real proof and not just speculate about the meaning of emails that only made it appear as if something illegal happened.
Conclusion
I tried to write this post in a manner that transcends politics. I really haven’t taken much of an interest in the whole global warming debate and don’t really have a strong opinion on the matter. However, being part of the Science Community (I have a degree in Physics) and having done scientific research myself makes me very worried when arrogant jerks who call themselves “scientists” work outside of ethics and ignore the truth to fit their pre-conceived notions of the world. That is not science, that is religion with math equations.
What do you think?
Now that you have the facts, you can come to your own conclusion!
Be sure to leave me a comment, it gets lonely in here sometimes.
hat tip to WUWT commenter “Disquisitive”
========================
NOTE: While there are some interesting points raised here, it is important to note a couple of caveats. First, the adjustment shown above is applied to the tree ring proxy data (proxy for temperature) not the actual instrumental temperature data. Second, we don’t know the use context of this code. It may be a test procedure of some sort, it may be something that was tried and then discarded, or it may be part of final production output. We simply don’t know. This is why a complete disclosure and open accounting is needed, so that the process can be fully traced and debugged. Hopefully, one of the official investigations will bring the complete collection of code out so that this can be fully examined in the complete context. – Anthony
Sponsored IT training links:
Join today for 646-985 exam prep and get a free newsletter for next 642-072 and 1z0-050 exams.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
For Will Hudson
http://playpolitical.typepad.com/issue_ads/2009/12/the-spectators-fraser-nelson-debates-climate-change-with-an-hysterical-bob-ward-from-the-lse.html
h/t conservativehome.com
For our non-UK viewers : The Spectator is a rather right-wing weekly magazine of immense venerability, but to its credit it does open its doors to a broad range of opinions. It used to be very radical in some of its output ( Mark Steyn was a regular contributor ) but has gone off the boil in the last couple of years after it lost its editor Boris Johnson, who is now Mayor of London.
It has dedicated this week’s issue to the AGW conundrum. http://www.spectator.co.uk
This is a fine example of the level of debate; sceptics trying to be concilitory and just asking for some honesty against shouty, close-eared shrill rudeness and hubris from the warmists. Bob Ward is an intensely tedious presence in blogs’ comments wherever this matter arises.
As to the ‘adjusted’ data not being used anywhere, the following code comment snippets appear to suggest otherwise.
From abdlowfreq2grid.pro:
; HUGREG=Hugershoff regions, ABDREG=age-banded regions, HUGGRID=Hugershoff grid
; The calibrated (uncorrected) versions of all these data sets are used.
; However, the same adjustment is then applied to the corrected version of
; the grid Hugershoff data, so that both uncorrected and corrected versions
; are available with the appropriate low frequency variability. There is some
; ambiguity during the modern period here, however, because the corrected
; version has already been artificially adjusted to reproduce the largest
; scales of observed temperature over recent decades – so a new adjustment
; would be unwelcome. Therefore, the adjustment term is scaled back towards
; zero when being applied to the corrected data set, so that it is linearly
; interpolated from its 1950 value to zero at 1970 and kept at zero thereafter.
From calibrate_correctmxd.pro:
; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered
; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which
; gives a zero mean over 1881-1960) after extending the calibration to boxes
; without temperature data (pl_calibmxd1.pro). We have identified and
; artificially removed (i.e. corrected) the decline in this calibrated
; data set. We now recalibrate this corrected calibrated dataset against
; the unfiltered 1911-1990 temperature data, and apply the same calibration
; to the corrected and uncorrected calibrated MXD data.
and further on in the same file:
; Now verify on a grid-box basis
; No need to verify the correct and uncorrected versions, since these
; should be identical prior to 1920 or 1930 or whenever the decline
; was corrected onwards from.
The BBC take a look – again the question of where and how the code is used doesn’t get asked:
http://news.bbc.co.uk/1/hi/programmes/newsnight/8395514.stm
SunSword (14:14:57) :
Why didn’t they use some of those grant dollars to hire one or more people trained in IT? Hire a real programmer, a real DBA, and buy a real database? Well…
(1) They are self-centered narcissists who think they are so smart they don’t need to hire anyone because they are so great they can do anything at all, or
(2) Because it is very CONVENIENT that the original data has been “lost” and that the model code is a rats nest of unintelligible spaghetti.
Or perhaps both.
———————————————–
There’s a third option. Their approach to data management and code isn’t much different to that of a lot of other research establishments. Mainly writing code as one-offs and they don’t have IT professionals, or won’t use them for the reasons you suggest. In most cases, no one cares whether the analysis in some routine paper is backed by proper IT procedures. The paper just doesn’t matter outside the field and if it’s found to be rubbish, it will simply reflect on the workers and the establishment.
The importance of this work grew, partly through their own political efforts, and they got away with far too much because dubious peer review took the place of verification. It changed from doing one-offs into performing a process and the expectations of QA, and development disciplines are very different. Also, it’s quite reasonable to insist that the quality of work used to inform huge policy decisions is impeccable.
They ended up holding a tiger by the tail. I can’t say I’ve got a lot of sympathy for them because it’s too serious and there’s too much evidence that they’ve behaved badly, but I can see how these things can come about bit by bit until someone finds themselves in the most unbelievable mess.
The dip at point 8 corresponds to 1934, the hottest year on record in the US. By using a negative fudge factor at that point this fact was hidden in the graph, making the temperature rise seem straight line- to match CO2 increases.
The fudge factor inflated the results and the .75 multipler was used to correct this, to bring the inflated tree ring data in line with the instrument data.
chris (13:56:53) :
no snow on Kilamanjaro
glaciers receding around the planet
while the desertification belt expands
the acidification of the friggin’ oceans
Got any more arrows in your quiver?
I find it is the context of the whole package of information in the leaked .zip file that makes a stench so strong it makes me want to hold my nose.
We know the CRU where under pressure to produce the ‘proof’ of CAGW for the IPCC on short lead-times.
We know from the Harry File that the original raw data was in a mess even before it got to him, and can speculate that much of the data it contained may have been interpolated.
We know Jones et al didn’t want anyone outside the cabal to have access to this data.
We know that a program used to produce global average temperature charts had the built in facility to produce ‘fudged’ data for trend charts.
We know the trick to solve the problem of the Briffa tree data work was a done by a data-join to the thermometer set, which was not alluded to when the data was published.
We know many peer reviewed pro-CAGW papers used the trends from the CRU temperature data in their work.
We know the satellite data was calibrated to the thermometer record, so perhaps has also been debased.
The conclusion is that the majority of climate science from the IPCC has been contaminated by poor the poor quality of the original data and subsequent adjustments, and this is the reason the whole group could never let anyone outside of it see the mess.
Oh what a tangled web we weave. When first we practise to deceive!
When first we practise to deceive!
As a “global warming/climate change/whatever they’re calling it this week” skeptic, I’ve been going round and round with Warmers who keep trying to play the classic Wizard of Oz “Pay no attention to that man behind the curtain” strategy.
But my bottom line argument is always this: If the East Anglia data is on the up-and-up, why did the so-called scientists refuse to share it (and threaten to delete it), even in response to an FOIA request? Why did they feel it necessary to ignore, silence and even press for the firing of ideological opponents? And if the original data really wasn’t destroyed, why hasn’t it been publicly released in order to shut up the skeptics once and for all?
Make no mistake: This is bigger than one university, and the Warmers know it. The entire global warming movement hangs in the balance.
As Lord Christopher Monckton pointed out, there are only four major data sets for global temperatures, and this was one of them. But the other three are interrelated, either technically or socially (i.e., the e-mails reveal a close kinship between the scientists). So it really isn’t a case of this being only one of countless databases.
And speaking of interrelated, the East Anglia data underpins the IPCC report, which is what convinces politicians to allocate billions of taxpayer dollars to “saving the planet” and sign treaties such as the Copenhagen Agreement.
That, I believe, was the endgame of the global warming hoax: As the Copenhagen Agreement reveals, the Warmers wanted to extort billions of dollars from industrialized nations and create a transnational agency with power over the countries that signed the treaty.
In other words, like so many scams through history, it all boils down to money and power.
Funny, I don’t recall ever hearing about Albert Einstein refusing to share the data behind his theories. That’s because real scientists have no reason to destroy their data, bully opponents, or — in the case of pseudo-scientist Al Gore — cancel speeches and hide from the public.
P.S. In answer to a previous question … actually, the correct term is “cut-and-DRIED,” not “cut-and-dry.” But perhaps “cut-and-try” was what the author intended.
As for the standard Warmer comeback, “Oh, yeah? So why are the polar ice caps melting?,” I have a simple response: How do you know they are?
Most of us haven’t been there, and the mainstream media are adept at selective reporting. How many of you saw this April 2009 story in the New York Times or any other MSM outlet?
http://www.news.com.au/antarctic-ice-is-growing-not-melting-away/story-0-1225700043191
Ice is expanding in much of Antarctica, contrary to the widespread public belief that global warming is melting the continental ice cap.
The results of ice-core drilling and sea ice monitoring indicate there is no large-scale melting of ice over most of Antarctica, although experts are concerned at ice losses on the continent’s western coast …
[I]ce is melting in parts of west Antarctica. The destabilisation of the Wilkins ice shelf generated international headlines this month.
However, the picture is very different in east Antarctica, which includes the territory claimed by Australia.
East Antarctica is four times the size of west Antarctica and parts of it are cooling. The Scientific Committee on Antarctic Research report prepared for last week’s meeting of Antarctic Treaty nations in Washington noted the South Pole had shown “significant cooling in recent decades”.
___________________________________________
— In the words of the late Paul Harvey, “And now you know … the rest of the story.”
NickB. (14:41:45) :
“As a re-post/guest-post, I’m not sure how exactly Anthony could “tone it down”. It’s my understanding that in this situation the only options are caveat/disclaimer (which he had already done) or removing it entirely”
He can first change the title. He can also emphasize the caveat with reformatting. When you said that he had posted a caveat I did a double take, as I had not seen it. I went up and looked for it, and still missed it. Finally found it … it essentially amounts to fine print buried at the bottom, when the large font head and subhead are sensational. The caveat needs to be BEFORE the poorly reasoned, inflammatory ‘guest post’. He can also make better choices as to what to feature on this blog, frankly. This ‘guest post’ is beneath the quality I expect form this site.
Note the automatically generated ‘Possibly related post’ immediately below the caveat. It links to a blog where someone rips this ‘guest post’ apart and uses it as a broad brush to paint all of us as ‘accusers’ who quote mine and are taking everything found in the leak out of context. This is exactly what I was warning about.
Somebody put their neck on the line to get this stuff out of East Anglia. We owe it to them to not waste it with bullshit ‘red meat feeding frenzies’.
For the code snip to be interesting it has to be demonstrated that the code is actually used in published data or is the basis for such. I’m pretty certain that this is not the whole code, is not the whole data set, and may or may not be something that anything was based on.
That’s why the note at the end of the post is so important. There is no clear context to this code.
As for the hysterics claiming dangerous warming, well, there’s nothing about the current warming conditions (if any, and I believe that we’ve had some) that is unusual. It’s been both warmer and cooler, Inuit’s have sunk into the methane bogs before, and somehow survived. So have Polar Bears. Glaciers have come and gone. Droughts have come and gone. It’s all happened before and will happen again. According to historical records people, plants, and critters do better (overall) when it’s warmer.
Now, if you want dangerous climate change look at the end of the last ice age. I know of at least one flood (Spokane, Wa) which would have wiped about any US city off the map. It happened as the glaciers were receeding and a giant ice dam broke. I’m sure there were others.
“We also don’t know how this code was used. If the results never saw the light of day, then it’s no big deal – just somebody being unsuccessful in trying to pull something (maybe for internal use).”
Why, if this is someone tinkering, was it sitting in a FOIA file?
Sorry, but when I did FOIA requests I trashed anything that was just me jotting notes to myself. Normally they were destroyed as I went along, of course, not at the time of the FOIA because they were never meant to be anything but ephemera. When lawyers do discovery, they present finished documents or at least drafts intended for consideration by others, not every half-arsed revision along the way.
I just don’t credit that this can be anything but an important step along the way for CRU. It might not be the finished article, but I find it difficult to believe it is some random musings.
Indeed, the usage of the manipulated data array is commented out. But why should someone a) even write such a manipulation of data into his code, and b): why would he then just comment it out instead of deleting it?
I think that this does definitively _not_ look like just a programmer playing with code or data. If the manipulation was obvious (if the manipulation would be like [-1.,2.,-3., …] or [-2.,-1.,0., …]) i’d say: ok, someone did play a bit. But to reckognize the influence of the elements of complex data to the results of your algoritm? Even a programming beginner would select data that is obvious in the result, rather than arbitrary values between -0.3 and +2.6. Which — now wonder — perfectly fit the Mann hockeythingy.
Well said, Pamela Gray.
Why no publish the entire piece of code? This is entirely meaningless without seeing how it was intergrated or used. Anyone who’s an experienced programmer knows this is an entirely disingenous post!
“I’m not sure how exactly Anthony could “tone it down”.”
Append a question mark to the current title.
“Oh what a tangled web we weave. When first we practise to deceive!”
“Oh what a tangled web we weave.
When first we practice to believe.”
–Laurence J. Peter (of Peter’s Principle)
I asked about this on Real Climate a couple days ago and Gavin Schmidt was kind enough to reply.
Norman says:
2 December 2009 at 8:41 PM
This is the one that disturbs me. It seems an intentional deception in the program code. Can anyone explain why they did this?
;
; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
; standardised datasets.
; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
; with missing values set appropriately. Uses mxd, and just the
; “all band” timeseries
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
[edit for space]
[Response: This was discussed earlier. It was an artificial correction to check some calibration statistics to see whether they would vary if the divergence was an artifact of some extra anthropogenic impact. It has never been used in a published paper (though something similar was explained in detail in this draft paper by Osborn). It has nothing to do with any reconstruction used in the IPCC reports. – gavin]
JJ:
“The caveat needs to be BEFORE the poorly reasoned, inflammatory ‘guest post’. He can also make better choices as to what to feature on this blog, frankly. This ‘guest post’ is beneath the quality I expect from this site.
“Note the automatically generated ‘Possibly related post’ immediately below the caveat. It links to a blog where someone rips this ‘guest post’ apart and uses it as a broad brush to paint all of us as ‘accusers’ who quote mine and are taking everything found in the leak out of context. This is exactly what I was warning about.”
Correct. I’ve been repeatedly warning that we must not give the other side an opportunity to counterpunch, we must not at this point try for a knockout blow but merely ask for a re-examination, and that we must not focus on the effect of the code and e-mails on the temperature record so much as on their effect on the credibility and trustworthiness of the warmers, of the importance of peer review and consensus, etc.
Incidentally, here’s an neat riposte to “denialist”: insister.
“This is where the magic happens. Remember that array we have of valid temperature readings? ”
There is no reason to believe from what you have published that any array of termperatures is involved here.
“The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.”
As far as I can tell, yrloc contains year numbers (eg., 1940), not temperatures. It is unclear what interpolating such numbers can have, but it sure has nothing to do with temperature, as far as you have shown.
BTW haven’t seen this mentioned anywhere — e-mails have dates so we know when they took place.
Well FWIW the code, data, and other files also have create and last edit dates (see the directory listing) and most of the code (*.pro) files are from 1998 to 2004. Not sure if this will help with the forensic work but seems like it should help create some kind of time sequence to the work.
And the raw data files look like they’re mainly from the early 90s
All the e-mail files have the same date 1.1.2009 but that just seems like a dummy date assigned to all the files to hide the actual dates they were extracted as even e-mails from 2009 have the same date.
Comment on, Comment off.
Why was the code commented out?
There are two ways to misrepresent data:
1. Use the unchanged raw data with a complex formula;
2. Change the data and use a simple formula.
Process 2 has the advantage that if data must eventually be supplied the changes will consistently produce the required result.
So, run the raw data to see what it looks like.
Make modifications by applying successive approximations to variables and constants until the desired effect is achieved.
Toggle the comment and make a final run to check that simple formula produces the expected outcome; if so, leave it as is, no need to go back and change the comment status.
The only potential problem is that the original unchanged data should be compared. How strange that it has been lost.
Maybe it’s a smoking code, maybe not. The point is that the output from this group of researchers is being used right now to invoke a new era of “world governance” in which economies of nations are to be controlled by an organisation that has total contempt for accountability and democratic process, with no avenue of appeal or to opt out.
In the absence of totally transparent data and analysis, is that ok with you?
Thank you for your clean and honest work sir.. Im sure you would be glad to have a unkind eye peer review your work, because it speaks for itself.. Maybe if enough honest scientists come forward the damage can be repaired.. Thanks again for your time and work..
Prosecutor: And where did you find the defendant, Constable Platypus?
Bobby: Be’ind the ware’ouse, standin’ in the shadders by the back gate.
Prosecutor: Was the gate locked when you arrived?
Bobby: Hit was. There was a chain wif a padlock through two o’ the links.
Prosecutor: And what did the defendant have in his hands?
Bobby: ‘E ‘ad a pair o’ bolt cutters, hex’ibit B. (points at evidence table)
Prosecutor: And how did the defendant explain his presence at the back gate of the warehouse, in the dark, with these bolt cutters in his possession?
Bobby: [looking at his notebook] ‘E said, “I were just practisin’, gov’nor. These ‘ere bolt cutters ‘ave been commented out.”
(laughter)
Prosecutor: And had they been commented out?
Bobby: Someone ‘ad put a semicologne on ’em wif a piece o’ chalk, yerse.
Prosecutor: How long would it take for the defendant to remove the semicolon and cut the chain?
Barrister: Objection, M’lud! PC Platypus is not an expert on chalk.
Judge: Over-ruled. You may answer the question.
Bobby: Habout ‘arf a second.
(laughter)
Prosecutor: Thank you, Constable Platypus.
My comments are disappearing.
chris (13:56:53)
If over the past 150 years, or any time frame, the weather or longer range climate did not change, scientists everywhere would be stumped. Climate changes, and to much bigger degrees than we’ve seen the past century. The arctic is not melting in actuality, pay the big bucks and fly up there and find out for yourself. Northerners don’t build on bogs, they drill housing stilts into permafrost. Foundations aren’t usually possible because every intense summer the permafrost shifts, as per normal.
The ocean issue of released methane, and deep ocean issue at that, is one you’d have to ask the Earth about. Does she have unexplored underwater volcanic activity she hasn’t told us about yet? If there’s one thing the past century of what, a +0.6degC, or a more recent – 0.4degC change of AIR temperature cannot do – is warm the massive volumes of oceans on this planet. Only the Sun or the Inner Earth can affect that, ask a physicist or oceanographer. It would be nice if they could step up to the plate and dispel this myth, but I guess it wouldn’t sell too many papers.