CRU Emails "may" be open to interpretation, but commented code by the programmer tells the real story

When the CRU emails first made it into news stories, there was immediate reaction from the head of CRU, Dr. Phil Jones over this passage in an email:

From a yahoo.com news story:

In one leaked e-mail, the research center’s director, Phil Jones, writes to colleagues about graphs showing climate statistics over the last millennium. He alludes to a technique used by a fellow scientist to “hide the decline” in recent global temperatures. Some evidence appears to show a halt in a rise of global temperatures from about 1960, but is contradicted by other evidence which appears to show a rise in temperatures is continuing.

Jones wrote that, in compiling new data, he had “just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (i.e., from 1981 onwards) and from 1961 for Keith’s to hide the decline,” according to a leaked e-mail, which the author confirmed was genuine.

Dr. Jones responded.

However, Jones denied manipulating evidence and insisted his comment had been taken out of context. “The word ‘trick’ was used here colloquially, as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward,” he said in a statement Saturday.

Ok fine, but how Dr. Jones, do you explain this?

There’s a file of code also in the collection of emails and documents from CRU. A commenter named Neal on climate audit writes:

People are talking about the emails being smoking guns but I find the remarks in the code and the code more of a smoking gun. The code is so hacked around to give predetermined results that it shows the bias of the coder. In other words make the code ignore inconvenient data to show what I want it to show. The code after a quick scan is quite a mess. Anyone with any pride would be to ashamed of to let it out public viewing. As examples [of] bias take a look at the following remarks from the MANN code files:

Here’s the code with the comments left by the programmer:

function mkp2correlation,indts,depts,remts,t,filter=filter,refperiod=refperiod,$

datathresh=datathresh

;

; THIS WORKS WITH REMTS BEING A 2D ARRAY (nseries,ntime) OF MULTIPLE TIMESERIES

; WHOSE INFLUENCE IS TO BE REMOVED. UNFORTUNATELY THE IDL5.4 p_correlate

; FAILS WITH >1 SERIES TO HOLD CONSTANT, SO I HAVE TO REMOVE THEIR INFLUENCE

; FROM BOTH INDTS AND DEPTS USING MULTIPLE LINEAR REGRESSION AND THEN USE THE

; USUAL correlate FUNCTION ON THE RESIDUALS.

;

pro maps12,yrstart,doinfill=doinfill

;

; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions

; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually

; plot past 1960 because these will be artificially adjusted to look closer to

; the real temperatures.

;

and later the same programming comment again in another routine:

;

; Plots (1 at a time) yearly maps of calibrated (PCR-infilled or not) MXD

; reconstructions

; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually

; plot past 1960 because these will be artificially adjusted to look closer to

; the real temperatures.

 

You can claim an email you wrote years ago isn’t accurate saying it was “taken out of context”,  but a programmer making notes in the code does so that he/she can document what the code is actually doing at that stage, so that anyone who looks at it later can figure out why this function doesn’t plot past 1960. In this case, it is not allowing all of the temperature data to be plotted. Growing season data (summer months when the new tree rings are formed) past 1960 is thrown out because “these will be artificially adjusted to look closer to the real temperatures”, which implies some post processing routine.

Spin that, spin it to the moon if you want. I’ll believe programmer notes over the word of somebody who stands to gain from suggesting there’s nothing “untowards” about it.

Either the data tells the story of nature or it does not. Data that has been “artificially adjusted to look closer to the real temperatures” is false data, yielding a false result.

For more details, see Mike’s Nature Trick

UPDATE: By way of verification….

The source files with the comments that are the topic of this thread are in this folder of the FOI2009.zip file

/documents/osborn-tree6/mann/oldprog

in the files

maps12.pro

maps15.pro

maps24.pro

These first two files are dated 1/18/2000, and the map24 file on 11/10/1999 so it fits timeline-wise with Dr. Jones email where he mentions “Mike’s Nature trick” which is dated 11/16/1999, six days later.

UPDATE2: Commenter Eric at the Climate Audit Mirror site writes:

================

From documents\harris-tree\recon_esper.pro:

; Computes regressions on full, high and low pass Esper et al. (2002) series,

; anomalies against full NH temperatures and other series.

; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N

;

; Specify period over which to compute the regressions (stop in 1960 to avoid

; the decline

;

Note the wording here “avoid the decline” versus “hide the decline” in the famous email.

===============

I’ll give Dr. Jones and CRU  the benefit of the doubt, maybe these are not “untowards” issues, but these things scream for rational explanations. Having transparency and being able to replicate all this years ago would have gone a long way towards either correcting problems and/or assuaging concerns.


Sponsored IT training links:

Need help for EX0-101 exam ? We offer self study 642-436 training program for all your 642-974 exam needs.


0 0 votes
Article Rating

Discover more from Watts Up With That?

Subscribe to get the latest posts sent to your email.

480 Comments
Inline Feedbacks
View all comments
Harold Vance
November 22, 2009 9:02 pm

The really sad thing is that the dendros still have no clue why trees make good thermometers in some years but not others. This is a bigger issue, imho.

Gerald Machnee
November 22, 2009 9:08 pm

RE: Nick Stokes (20:34:52) :
**I may be dense here, but what’s the issue? The red comment says “don’t plot beyond 1960″, because the results are unreliable**
Maybe it means “don’t plot beyond 1960 because the results do not show what we want”?????

Jon Adams
November 22, 2009 9:09 pm

The CRU team and maybe all of these “AGW researchers” are clueless… they are way over their heads and inclined to cheat – wait…its defraud as adults – especially with literally trillions of dollars involved.
How many billions has the US and other countries wasted on this BS…
I want my money back!
One may surmise they did not want any Real Programming Talent Aboard to maybe OUT them? so they tried to fake it themselves…
Anyway… thank God for Anthony… attempting to get some quality to the data sets involved!

rbateman
November 22, 2009 9:09 pm

philincalifornia (20:51:09) :
Yep, saw that, and that’s not the only place he used it. Leaves little doubt that the data was pre-slaughtered, does it not?
Now, who exactly is HARRY?

Jeff Coatney
November 22, 2009 9:13 pm

Unbelievable!!!!!!!!!!!!!!
Or, perhaps, not so unbelievable.
There are ways to game any system. All it takes is a person or persons clever enough to formulate effective methods of cheating.
In the end, what counts more than anything else is the ability to rely on the word of others. To the extent that people are more or less untrustworthy does the potential for dishonesty rise or fall.
Is it just me, or is the societal willingness to indulge in unethical behaviour presently on a significant and massive upswing – Perhaps a little like our friend, the hockey stick curve.

Eric Barnes
November 22, 2009 9:13 pm
Doug in Seattle
November 22, 2009 9:14 pm

P Gosselin (20:47:32) :
WHOA!
I can’t believe it!
One of Germany’s biggest highly respected dailies has it on the front website page.

And in English here – http://translate.google.com/translate?u=http%3A%2F%2Fwww.welt.de%2Fwissenschaft%2Farticle5294872%2FDie-Tricks-der-Forscher-beim-Klimawandel.html%23xmsg_comment&sl=de&tl=en&hl=en&ie=UTF-8

AlexB
November 22, 2009 9:14 pm

The alarmists have gotten so good at misdirection that they get me every time. Their comments are never on the real issue. When I read Dr. Jones explanation of the word ‘trick’ I thought that it was perfectly reasonable so decided to lay that comment to rest. I re-read the e-mail later though and thought ‘hang on!’. Is the word ‘hide’ also commonly used by scientists to mean something other than its common usage? I can’t believe they got me again!

November 22, 2009 9:14 pm

Roger Sowell (20:58:33) :
Stokes: the real issue is that temperatures derived from tree rings are known to not match measured temperatures after 1960.
If tree ring-based temperatures are known to be false compared to actual measurements, then how can they be true in earlier decades or centuries?

That’s not a coding issue. What they say in Briffa 2001 for the Siberian trees is:
““The period after 1960 was not used to avoid bias in the regression coefficients that could be generated by an anomalous decline in tree density measurements over recent decades that is not forced by temperature””
The claim seems to be that they have specific information about a post-1960 divergence, and presumably enough pre-1960 instrumental overlap to satisfactorily calibrate. Now I can’t judge the strength of that, but it’s been discussed in the literature for nearly a decade. The answer won’t be found in a comment in the code. The comment merely reflects the limit stated in the published theory.
REPLY: In other work, Briffa allowed 10 Yamal trees, an unacceptably low sample, to stay in. When he says “anomalous decline in tree density measurements over recent decades” why then would he think 10 trees as a sample upon which to base that the datapoints are OK? As Steve has said many times (and other dendros agree – they say 50 is the minimum sample), Briffa should have truncated that Yamal sample and data. He didn’t. So why is it OK to not truncate a very low sample in one case and not another? Doesn’t make any sense. – A

Editor
November 22, 2009 9:18 pm

Looking in the file HARRY_READ_ME.txt there seems to be some suspicious coder comments as well:
“This still meant an awful lot of encounters with naughty Master stations, when really I suspect nobody else gives a hoot about. So with a somewhat cynical shrug, I added the nuclear option – to match every WMO possible, and turn the rest into new stations (er, CLIMAT excepted). In other words, what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad, but I really don’t think people care enough to fix ’em, and it’s the main reason the project is nearly a year late.”
“You can’t imagine what this has cost me – to actually allow the operator to assign false WMO codes!! But what else is there in such situations? Especially when dealing with a ‘Master’ database of dubious provenance (which, er, they all are and always will be).”

Jon Adams
November 22, 2009 9:19 pm

Everyone… This is Not just about Money … it is about Tyranny… the politicians want to control us till we have no soul…
And we have the Nick Stokes who want to jump on a mention of 1960… temp mismatches… the entire picture is what you need to look at, Nick… study some more and you will begin to see a pattern.

Policyguy
November 22, 2009 9:21 pm

Roger Sowell (20:58:33) :
Stokes: the real issue is that temperatures derived from tree rings are known to not match measured temperatures after 1960.
If tree ring-based temperatures are known to be false compared to actual measurements, then how can they be true in earlier decades or centuries?
Well, perhaps that is a place to start. According to John Dally we know that trees don’t grow on 70% of the earth’s surface (oceans), they don’t grow in deserts or at high elevations. The 15% of the earth’s surface where trees do grow are in those locations where they may also impacted by lack of light (other trees), lack of water (draught), to the extent – that temperature can not be isolated as a cause for growth during any period. It is a false measure.
So lets challengeh tree data as a temperature surrogate altogether. That will take care of Mr. Mann and other tree persons and throw out the hockey stick. Apparently, according to his now published email, even Mr. Revkin now agrees with that.

November 22, 2009 9:23 pm

I agree (as a software developer, especially) comments in code are very rarely any kind of ‘mistake’ as such, although many are out of date, it is true.
Overall, this, the email, and what I suspect to be gleaned from the data, the whole story is a shambles. There is only one possible thing that can be done to salvage the credibility of anyone in the field who supports AGW:
1. All existing data and conclusions should either be examined for veracity or dismissed.
2. There must be a new, non IPCC and non-UN controlled, globally funded task force to re-examine the whole AGW issue. Existing AGW believers and sceptics should be included, especially those who have shown extraordinary effort and dedication in the field to date.
3. The entire investigation should be transparent to the participants, funders (one assumes governments) and also the public. The Internet is a good medium for such a task, as has been proved.
4. All political involvement must be prevented. That cannot be stressed enough.
5. All commercial involvement should be prevented. Oddly enough, I would support ‘big oil’ etc as it seems they are gearing up very swiftly to get ahead of the game in renewable, as is sensible. I suspect that many would cry foul, however.
6. No taxes or political changes should be introduced that rely on the AGW theory being accurate until it is proved that CO2 increases will cause dangerous changes to the climate.
Just my 2c worth…..

Jesse
November 22, 2009 9:24 pm

Once again, you guys are making mountains out of ant hills. This is just normal data processing per the 1960 divergence problem as shown by NUMEROUS sources. This is what happens when a bunch of uninformed amateurs try and “debunk” real scientists. Leave the science to the scientists and go back to your day jobs as custodians, wal-mart employees and laborers.
REPLY: So what do you do down there in Norman? NSSL? U of OK? You might be surprised at the sort of professionals that frequent here. I invite them to sound off. – A

John F. Hultquist
November 22, 2009 9:24 pm

What I interpret this to mean is that the results of their work pre-1960 are as they think they should be. Post-1960, not so much. Post-1960 results have to be adjusted to reflect …. what?
a. a previously non-operating variable, now operating;
b. a previously operating variable that no longer operates;
c. some combination of a & b;
d. a hunch based on years of experience;
e. a revelation from God;
f. other
The code used to ‘artificially adjust’ the results so they will ‘look closer to the real temperatures’ ought to explain which of the above scientific procedures was used. If this code is not in the posted material, I am sure the researchers will happily provide it.
~~~~~~~~~~~~~~~~~~~~
A few months back WUWT had a discussion on programming so, not to go over all that again, I’ll just say many of us (in years gone by) wrote our own code and/or subroutines that were used over and over. I always put a few comments at the top stating a few common things, such as, my name, and the purpose of the code, and the language (FORTRAN 2D, IV, ??). Is there a complete routine in this dump that would identify the programmer?

wes george
November 22, 2009 9:24 pm

Wait and see, the True Believers will soon come out with a “Fake but accurate” defense. Remember, the fate of the whole world hangs in the balance, so what’s a few lies, a bit a fraudulent science and bullying matter as long as the Green agenda of zero-growth, centrally mandated economies and increasing restriction on individual liberties is moved forward.
The end justifies the means in AGW ethics.

Glenn
November 22, 2009 9:26 pm

Richard Sharpe (20:44:52) :
“When people use cut-n-paste coding, they sometime even copy the comments and forget to change them.”
Oh that’s encouraging, the fate of the world hanging on shade tree coders.

WakeUpMaggy
November 22, 2009 9:28 pm

Maybe one of the programmers was the mole.

Dr A Burns
November 22, 2009 9:29 pm

Another strange happening at Hadley … all the hadcrut3 data for this year, except Jan/Feb, has been deleted.
http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt

John F. Hultquist
November 22, 2009 9:32 pm

P Gosselin (20:47:32) : You likely know this but I had to have it translated, except for the two opening words
Die Tricks der Forscher beim Klimawandel
via Google translate
The tricks of the researchers on climate change

Editor
November 22, 2009 9:34 pm

Also, in the Word file jones-foiathoughts.doc it states:
“Options appear to be:
1. Send them the data
2. Send them a subset removing station data from some of the countries who made us pay in the normals papers of Hulme et al. (1990s) and also any number that David can remember. This should also omit some other countries like (Australia, NZ, Canada, Antarctica). Also could extract some of the sources that Anders added in (31-38 source codes in J&M 2003). Also should remove many of the early stations that we coded up in the 1980s.
3. Send them the raw data as is, by reconstructing it from GHCN. How could this be done? Replace all stations where the WMO ID agrees with what is in GHCN. This would be the raw data, but it would annoy them.”
Number 2 seems to indicate the particular station data that they were trying to hide…

November 22, 2009 9:37 pm

rechauffementmediatique (20:57:34) :
I found these comments in osborn-tree6/mann/oldprog in files like maps12.pro and
maps15.pro. These were dated Jan 2000, and the directory name is not encouraging. Seems unlikely that it is currently used code.

Dave Johnson
November 22, 2009 9:44 pm

The Times here in the UK now has the CRU story on one of it’s major comment pages
http://www.timesonline.co.uk/tol/comment/columnists/guest_contributors/article6927598.ece

Neil O'Rourke
November 22, 2009 9:46 pm


rechauffementmediatique (20:57:34) :
Can someone point me what file is this code from?
I have downloaded the package, and found only one file (FOIA/documents/osborn-tree6/mkp2correlation.pro) that includes the above mentioned function. Lines 1 to 7 are identical, but the rest has nothing to do with the screenshot above

documents\osborn-tree6\summer_modes\maps12.pro

philincalifornia
November 22, 2009 9:53 pm

artwest (20:54:26) :
A poster called Asimov has quoted extracts from the “HARRY_READ_ME.txt” file – deeply shocking stuff:
Asimov’s very plausible suggestion is that Harry is a programmer trying, and often failing, to make sense of the garbage data which he’s been lumbered with.
Several posts over several pages:
http://www.tickerforum.org/cgi-ticker/akcs-www?post=118625&page=13
____________________________
I also lifted this from a comment over there:
“Tim Mitchell works at the Climactic Research Unit, UEA, Norwich, and is a member of South Park Evangelical Church.”
South Park ?? You’ve got to be kiddin’ me ??
PS I also tried to check out if he really did spell it “Climactic” but, strangely, the web site appears to be down. Heh heh heh.