NOTE: Part 2 of this story has been posted: see The Smoking Code, part 2
The Proof Behind the CRU Climategate Debacle: Because Computers Do Lie When Humans Tell Them To
From Cube Antics, by Robert Greiner
I’m coming to you today as a scientist and engineer with an agnostic stand on global warming.
If you don’t know anything about “Climategate” (does anyone else hate that name?) Go ahead and read up on it before you check out this post, I’ll wait.
Back? Let’s get started.
First, let’s get this out of the way: Emails prove nothing. Sure, you can look like an unethical asshole who may have committed a felony using government funded money; but all email is, is talk, and talk is cheap.
Now, here is some actual proof that the CRU was deliberately tampering with their data. Unfortunately, for readability’s sake, this code was written in Interactive Data Language (IDL) and is a pain to go through.
NOTE: This is an actual snippet of code from the CRU contained in the source file: briffa_Sep98_d.pro
[sourcecode language=”text”]
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
yearlyadj=interpol(valadj,yrloc,timey)
[/sourcecode]
Mouse over the upper right for source code viewing options – including pop-up window
What does this Mean? A review of the code line-by-line
Starting off Easy
Lines 1-3 are comments
Line 4
yrloc is a 20 element array containing:
1400 and 19 years between 1904 and 1994 in increments of 5 years…
yrloc = [1400, 1904, 1909, 1914, 1919, 1924, 1929, … , 1964, 1969, 1974, 1979, 1984, 1989, 1994]
findgen() creates a floating-point array of the specified dimension. Each element of the array is set to the value of its one-dimensional subscript
F = indgen(6) ;F[0] is 0.0, F[1] is 1.0….. F[6] is 6.0
Pretty straightforward, right?
Line 5
valadj, or, the “fudge factor” array as some arrogant programmer likes to call it is the foundation for the manipulated temperature readings. It contains twenty values of seemingly random numbers. We’ll get back to this later.
Line 6
Just a check to make sure that yrloc and valadj have the same number of attributes in them. This is important for line 8.
Line 8
This is where the magic happens. Remember that array we have of valid temperature readings? And, remember that random array of numbers we have from line two? Well, in line 4, those two arrays are interpolated together.
The interpol() function will take each element in both arrays and “guess” at the points in between them to create a smoothing effect on the data. This technique is often used when dealing with natural data points, just not quite in this manner.
The main thing to realize here, is, that the interpol() function will cause the valid temperature readings (yrloc) to skew towards the valadj values.
What the heck does all of this mean?
Well, I’m glad you asked. First, let’s plot the values in the valadj array.
Look familiar? This closely resembles the infamous hockey stick graph that Michael Mann came up with about a decade ago. By the way, did I mention Michael Mann is one of the “scientists” (and I use that word loosely) caught up in this scandal?
Here is Mann’s graph from 1999
As you can see, (potentially) valid temperature station readings were taken and skewed to fabricate the results the “scientists” at the CRU wanted to believe, not what actually occurred.
Where do we go from here?
It’s not as cut-and-try as one might think. First and foremost, this doesn’t necessarily prove anything about global warming as science. It just shows that all of the data that was the chief result of most of the environmental legislation created over the last decade was a farce.
This means that all of those billions of dollars we spent as a global community to combat global warming may have been for nothing.
If news station anchors and politicians were trained as engineers, they would be able to find real proof and not just speculate about the meaning of emails that only made it appear as if something illegal happened.
Conclusion
I tried to write this post in a manner that transcends politics. I really haven’t taken much of an interest in the whole global warming debate and don’t really have a strong opinion on the matter. However, being part of the Science Community (I have a degree in Physics) and having done scientific research myself makes me very worried when arrogant jerks who call themselves “scientists” work outside of ethics and ignore the truth to fit their pre-conceived notions of the world. That is not science, that is religion with math equations.
What do you think?
Now that you have the facts, you can come to your own conclusion!
Be sure to leave me a comment, it gets lonely in here sometimes.
hat tip to WUWT commenter “Disquisitive”
========================
NOTE: While there are some interesting points raised here, it is important to note a couple of caveats. First, the adjustment shown above is applied to the tree ring proxy data (proxy for temperature) not the actual instrumental temperature data. Second, we don’t know the use context of this code. It may be a test procedure of some sort, it may be something that was tried and then discarded, or it may be part of final production output. We simply don’t know. This is why a complete disclosure and open accounting is needed, so that the process can be fully traced and debugged. Hopefully, one of the official investigations will bring the complete collection of code out so that this can be fully examined in the complete context. – Anthony
Sponsored IT training links:
Join today for 646-985 exam prep and get a free newsletter for next 642-072 and 1z0-050 exams.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
MangoChutney (05:54:06) :
“just a thought, if the emails and code were obtained by a hacker and not released by a whistle blower, would the evidence be inadmissible in a court of law?”
=>The leak of the Pentagon Papers by Daniel Ellsberg set the standard, & the subsequent Supreme Court decision, a 6-3 ruling in favor of The New York Times, which had published the purloined material, also firmly established the First Amendment principle of no prior restraint on publication.
“A stranger’s illegal conduct does not suffice to remove the First Amendment shield from speech about a matter of public concern,” wrote Justice John Paul Stevens.<=
I ran across that piece of information in this article, “Sen. Boxer and ClimateGate: The Terror of Tiny Town,” by Michael Walsh on biggovernment.com
>THE CODE WAS COMMENTED OUT
>IT WAS NOT USED at the point this image was taken
In that particular source file, yes it was commented, but as noted elsewhere, in the file ‘briffa_sep98_e.pro, it IS used, twice. I cannot comment on whether the results of any particular version were used to create published graphs.
As a reminder, the array ‘valadj’ is the “artificially corrected” element.
And remember, it’s really easy to comment or uncomment a line and rerun the code, so we would need to correlate published data to this line of code to know if it was, or was not, used to influence the results.
That said, it is suspicious that it exists at all.
—–
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
————- And later
;
; Restore the Hugershoff NHD1 (see Nature paper 2)
;
xband=x
restore,filename=’../tree5/densadj_MEAN.idlsave’
; gets: x,densadj,n,neff
;
; Extract the post 1600 part
;
kl=where(x ge 1400)
x=x(kl)
densadj=densadj(kl)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densadj=densadj+yearlyadj
I think you are full of BS and what you have set out is crap.
Please ‘hide the decline’ as peer reviewed science demands.
Phil,
So NASA corrected the published data before Steve M brought it up? Do you have a link for that, because that is not how I remember it. I am calling BS here.
Not to mention that no such analysis occurred or can occur on the CRU data set. So the only *reliable* data we have shows that the thirties were warmer than the ’90s, as any history book which deals with the period will corroborate. The claim that the global numbers don’t show this argument carries a *ton* less weight in light of the facts that have come out recently.
The word “unprecedented” is gone, and it ain’t coming back without a lot of extremely careful and transparent work. My guess is that if the reanalysis is done carefully, the thirties will be found to have been warmer, and the only answer you have right now is that I should trust Phil Jones.
I agree with Bill. Greiner’s analysis of this code is fatally flawed if yearlyadj is not actually used in the code. Anthony should write an update.
To elaborate a little further…for this code to mean anything, it needs to be shown how it effects the output/final result. Mr. Greiner has not done that. I would not bother posting this stuff unless you can determine a little more precisely what effect the suspect code has on the output. If the answer is “none”, then there really isn’t necessarily a problem.
I find it interesting how many of the respondents who can’t read and interpret computer code and who know nothing of the author’s veracity, background or motives are willing, without the slightest hesitation, to swallow what the author of this piece says, hook-line-and-sinker. I believe they do so BECAUSE IT TELLS THEM WHAT THEY WISH TO HEAR.
Isn’t that what got “true believing” AGW crowd into the mess they’re now in?
Rein it in a bit, boys and girls!
CH
Interesting AlGorithm.
Been looking at this. I assume that the values in yearlyadj are then applied to the temperature data?
Also this fudge series flattens out in the late 70’s (2.5, 2.6, 2.6, 2.6, 2.6 and 2.6.) what is the effect of that?
REPLY: Not the temperature data, but the tree ring proxy data – Anthony
Love the commenters to this site — even the alarmists like Phil, because most everyone tries to make sense. Looking at this, it’s code for some unknowable purpose, and hopefully the EAU investigation is not a whitewash and looks at the code written over the years at CRU and more importantly why code was written. Was this code a benign ‘test run’ to see if data would react as predicted, so actual model results would be reliable? or was this code part of the 1990s scam to manipulate data? Only a thorough real investigation will reveal that. Based on the Mann code manipulations, the Briffa data nonsense, CRU evading legitmate FOI demands et al, strong suspicion of adjusting data to match model results is clearly warranted. Time will tell, but investigations by government and private grant givers is essential, whistleblowers would be a helpful resource.
The most positive result of these disclosures should be more access to the data and algorithms that are used. It seems to me that looking at source code is a pretty poor way to really determine bias in the algorithms. A much better way would be to simply obtain the model, and run it with constant temperature, thereby graphing the bias directly. Mr. Watts, you seem to be in an excellent position to demand this information. I cannot imagine how this demand could be denied in the present situation. What possible excuse could be invented to oppose such a study?
The most damning thing is the very comment “; fudge factor”. What the heck kind of comment is that to put in code? The point of comments is to explain what the coder is doing and how; it makes the code useful to anyone (including the programmer himself) who comes after to understand the logic and to debug and correct if necessary.
“fudge factor” is, whatever the rationalizations offered by the Mann contingent, NOT a technical term in climatology, the meaning of which would be obvious to anyone using or reading this code; and it’s not a useful comment to anyone reading the code for scientific purposes.
What it sounds like is “This is the adjustment we need to make it come out the way we want; I told you I’d need to apply a ‘fudge factor’ when we discussed the data in person.” So SOMEONE, or some people, in addition to the coder, probably knew what it meant, but no one else in on conversations before the coding would know–and no one else would be meant to know for sure.
I really hope that some time is taken to graphically illustrate these “value added” graphs opposed to the raw data.
The genius of the “hockey stick” is that it is clear and a perfect bookcover illustration. Someone can just point to it and say’ “See, look for yourself.”
Those who question it, need a compelling and equally clear illustration. I am not saying that anyone should, like the hockey team, construct such graph; but this is as much a PR fight as a scientific one.
My hope is that the reason that Climate Audit has not had any recent postings is because this is what is being done. Wouldn’t it be great if it were revealed at Copenhagen.
This post shows a “smoking gun” and it does not matter if the guy with the gun in his hand says he just used it to kill a passing by bird…That will be a matter of investigation. It’s too late now.
Thanks for that information … with it, I have concluded that a lot of phony scientists and Al Gore need to be charged with fraud and prosecuted to full extent of the law.
Governments who have been duped into funding this crap science should be the ones laying the charges. Also, the motion picture academy that gave Gore an Oscar based on his documentary which proved to be a pack of lies and Styrofoam should rescind his Oscar.
The Nobel prize committee has no credibility, so he can keep his phony peace prize. It has the same value as the one given to the man/child Obama and the old terrorist Yassir Arafat. Gore is in good company there.
James, Billy.. we need to go to court and put these programmers in fear of 5 year sentences..
then the truth will come out!
“This code fragment is meaningless unless without the context in which it was used. You need to have evidence that it was used to unjustifyably distort temperature data which was subsequently published. PS. I’m a sceptic, but without context this code is definately NOT a smoking gun.”
Second that.
Look folks, do not squander the opportunity that has been presented to us, by flying off the handle with hastily drawn ‘gotchas’. As currently analysed, this code snippet proves nothing untoward.
Somebody risked a lot to get this data out of East Anglia. With the massive lock down and circling of wagons that is going on, we may not have another chance like this. It would be a travesty if the real smoking guns end up hidden beneath an avalance of water pistols turned up by half assed, Team-like, amateur hour, conjecture driven ‘analyses’.
What do the values in valadj represent? What is the source of the 0.75 coefficient? To which data were the interplations applied? What was the result? Was it ever used? What would that mean?
All of those questions need solid answers before making a big deal out of this. There is plenty of damning material in the whistleblowers gift. No need to exaggerate or make stuff up. Leave that to the Team.
My contribution, with two (de)motivational posters 😀
http://www.flickr.com/photos/45273160@N06/sets/72157622809522485/
For those saying that this isn’t a “smoking gun” because we don’t know how it has been used, you are right as far as that goes, however you are wrong for a more serious reason. No one knows how this model was used. That is the whole point. These scientists would not turn over how the made the model and what their assumptions were based upon. They lost their data, their models were spaghetti, and they increasingly lost control over the process. I don’t care whether this was fraud or incompetency, it sure as heck isn’t science, and it most assuredly should not have been used to justify billions of expenses.
This is a “smoking gun” because even the scientists who MADE this code can explain it and whether or not it was used. As such all their work is now suspect.
Ed Fix (07:23:34) : and others
I agree Ed.
But that has always been the problem; lack of replication to verify results because the original code and data is not published. The release of these documents should in the very least force the release of original data and code (if available) so that people can determine if fudge factors were included. This is what the skeptic community should argue for vehemently. If code is not released, it should be readily apparent to even the most ardent alarmists that there is genuine f***d.
Would this code be accepted for an assignment in an Engineering or Computer Science course? What grade would it get?
I found this comment in calibrate_correctmxd.pro
; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered
; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which
; gives a zero mean over 1881-1960) after extending the calibration to boxes
; without temperature data (pl_calibmxd1.pro). We have identified and
; artificially removed (i.e. corrected) the decline in this calibrated
; data set. We now recalibrate this corrected calibrated dataset against
; the unfiltered 1911-1990 temperature data, and apply the same calibration
; to the corrected and uncorrected calibrated MXD data.
Could it be possible, they used ‘value-added-data’ later on, so there was no need anymore to apply the artificial correction?
The UN IPCC needs to investigate the CRU lies in August about losing their data to avoid FIO:
http://strata-sphere.com/blog/index.php/archives/11715
I agree w/ BB above:
“Unfortunately, the quality of ‘climategate’ discussions is starting to deteriorate and approach that of AGW propaganda at an alarming pace.”
The skeptic side needs to get a grip or run the risk of coming across as tinfoil-hat types.
Here’s some food for thought. Think about this logically. If their intent was to just fudge some numbers and never let anybody see the raw data, why would they need a computer program to do it? Why not just make up a bunch of numbers? Obviously, there is some method to the apparent (to some) madness of the program in question, otherwise why bother? I agree that the AGW side should be completely open with their raw data and with any algorithms for massaging it. Certainly, their secrecy contributes to suspicions. But that doesn’t mean that everything they do is sinister.
The fact that the “fudge factor” is in the code, has to make one wonder.
Yes programmers & analysts can be sloppy at first, but as the project progresses, code is either cleaned up or re-written for traceability or maintenance.
In a number of companies, anyone presenting this code in a design or report review, would be reprimanded or shown the door in quick order.