CRU Emails "may" be open to interpretation, but commented code by the programmer tells the real story

When the CRU emails first made it into news stories, there was immediate reaction from the head of CRU, Dr. Phil Jones over this passage in an email:

From a yahoo.com news story:

In one leaked e-mail, the research center’s director, Phil Jones, writes to colleagues about graphs showing climate statistics over the last millennium. He alludes to a technique used by a fellow scientist to “hide the decline” in recent global temperatures. Some evidence appears to show a halt in a rise of global temperatures from about 1960, but is contradicted by other evidence which appears to show a rise in temperatures is continuing.

Jones wrote that, in compiling new data, he had “just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (i.e., from 1981 onwards) and from 1961 for Keith’s to hide the decline,” according to a leaked e-mail, which the author confirmed was genuine.

Dr. Jones responded.

However, Jones denied manipulating evidence and insisted his comment had been taken out of context. “The word ‘trick’ was used here colloquially, as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward,” he said in a statement Saturday.

Ok fine, but how Dr. Jones, do you explain this?

There’s a file of code also in the collection of emails and documents from CRU. A commenter named Neal on climate audit writes:

People are talking about the emails being smoking guns but I find the remarks in the code and the code more of a smoking gun. The code is so hacked around to give predetermined results that it shows the bias of the coder. In other words make the code ignore inconvenient data to show what I want it to show. The code after a quick scan is quite a mess. Anyone with any pride would be to ashamed of to let it out public viewing. As examples [of] bias take a look at the following remarks from the MANN code files:

Here’s the code with the comments left by the programmer:

function mkp2correlation,indts,depts,remts,t,filter=filter,refperiod=refperiod,$

datathresh=datathresh

;

; THIS WORKS WITH REMTS BEING A 2D ARRAY (nseries,ntime) OF MULTIPLE TIMESERIES

; WHOSE INFLUENCE IS TO BE REMOVED. UNFORTUNATELY THE IDL5.4 p_correlate

; FAILS WITH >1 SERIES TO HOLD CONSTANT, SO I HAVE TO REMOVE THEIR INFLUENCE

; FROM BOTH INDTS AND DEPTS USING MULTIPLE LINEAR REGRESSION AND THEN USE THE

; USUAL correlate FUNCTION ON THE RESIDUALS.

;

pro maps12,yrstart,doinfill=doinfill

;

; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions

; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually

; plot past 1960 because these will be artificially adjusted to look closer to

; the real temperatures.

;

and later the same programming comment again in another routine:

;

; Plots (1 at a time) yearly maps of calibrated (PCR-infilled or not) MXD

; reconstructions

; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually

; plot past 1960 because these will be artificially adjusted to look closer to

; the real temperatures.

 

You can claim an email you wrote years ago isn’t accurate saying it was “taken out of context”,  but a programmer making notes in the code does so that he/she can document what the code is actually doing at that stage, so that anyone who looks at it later can figure out why this function doesn’t plot past 1960. In this case, it is not allowing all of the temperature data to be plotted. Growing season data (summer months when the new tree rings are formed) past 1960 is thrown out because “these will be artificially adjusted to look closer to the real temperatures”, which implies some post processing routine.

Spin that, spin it to the moon if you want. I’ll believe programmer notes over the word of somebody who stands to gain from suggesting there’s nothing “untowards” about it.

Either the data tells the story of nature or it does not. Data that has been “artificially adjusted to look closer to the real temperatures” is false data, yielding a false result.

For more details, see Mike’s Nature Trick

UPDATE: By way of verification….

The source files with the comments that are the topic of this thread are in this folder of the FOI2009.zip file

/documents/osborn-tree6/mann/oldprog

in the files

maps12.pro

maps15.pro

maps24.pro

These first two files are dated 1/18/2000, and the map24 file on 11/10/1999 so it fits timeline-wise with Dr. Jones email where he mentions “Mike’s Nature trick” which is dated 11/16/1999, six days later.

UPDATE2: Commenter Eric at the Climate Audit Mirror site writes:

================

From documents\harris-tree\recon_esper.pro:

; Computes regressions on full, high and low pass Esper et al. (2002) series,

; anomalies against full NH temperatures and other series.

; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N

;

; Specify period over which to compute the regressions (stop in 1960 to avoid

; the decline

;

Note the wording here “avoid the decline” versus “hide the decline” in the famous email.

===============

I’ll give Dr. Jones and CRU  the benefit of the doubt, maybe these are not “untowards” issues, but these things scream for rational explanations. Having transparency and being able to replicate all this years ago would have gone a long way towards either correcting problems and/or assuaging concerns.


Sponsored IT training links:

Need help for EX0-101 exam ? We offer self study 642-436 training program for all your 642-974 exam needs.


Advertisements

  Subscribe  
newest oldest most voted
Notify of
stevemcintyre

The quote and credit is due to reader “Neal”.

Noelene

Keep it up,I am watching with fascination.You(and the others who love science) must be so angry,they have besmirched the whole field of climate science.Dirty rotten scoundrels.Politicians are carrying on as if this whole issue will go away soon,I hope it doesn’t.

Policyguy

The trillions and gazillions of dollars that are now at risk is the pathetic consequence of a programmer gone berserk. What a travesty. Can it be corrected? Who knows. A call to Science is in order so that reliable individuals can correct this unballance between political/social myth and reality. The challenge will be, how to encourage informed individuals to speak up.

Nick Stokes

I may be dense here, but what’s the issue? The red comment says “don’t plot beyond 1960”, because the results are unreliable. So is there any indication that anyone has plotted beyond 1960? This came up on the Bishop Hill thread, where he drew attention to an email by Tim Osborn where he said that they never plot some treering set beyond 1960 because of a divergence issue. Turns out that that is what Briffa/Osborn say also in Briffa et al 2001. This Briffa/Osborn context may be unrelated, but it seems to me that it may simply just mean what it says. Don’t plot beyond 1960 using this code. And people don’t.

Fred

Its highly unlikely mere facts will be able to stop the AGW religion for advancing from one victory to the next. After all, AGW was never about science, it was about raisng taxes and controlling other people’s lives.

P Gosselin

[snip]
Add that to violating FOIA.
When we’re done, we’ll be able to throw the book at them.

Mann O Mann

But remember – if you question Real Climate Scientists ® then you are a DENIER.
Oooga Boooga!
/sarcasm

Glenn

I’ve been unable to find the definition of “artificial adjustment” in the climatologist’s handbook. It must be called something else. Sure sounds like something else to me. More! More!

Spin that to the moon.
Looks like a duck, quacks like a duck….smells like a duck’s butt.

If that’s the best you can do then you’re whole case is in trouble. Data from independent sources often has to be massaged (the programming terminology is “munged”) in order to make independent series compatible with each other.
The most interesting thing about this is the practice of “copy and paste” coding indicated by the duplicated comments, which ‘real’ programmers don’t like but which are an unfortunate necessity when programming for scientific research.

Alvin

Very telling. I just authored a letter to Senator Graham asking if he is keeping up with this new finding. I also insisted that he provide a public appology to the people of South Carolina and to Senator James Inhofe.

BarryW

[sarcasm on]Don’t you understand? There is reality and then there is TRUTH. If reality doesn’t fit the TRUTH then reality must be adjusted. [sarcasm off]

No wonder they refused to release it even under FOI.

Richard Sharpe

When people use cut-n-paste coding, they sometime even copy the comments and forget to change them.

P Gosselin

WHOA!
I can’t believe it!
One of Germany’s biggest highly respected dailies has it on the front website page.
http://www.welt.de/wissenschaft/article5294872/Die-Tricks-der-Forscher-beim-Klimawandel.html#xmsg_comment

Viktor

Uh, wow. Some fence sitters wanted hard evidence, more than what they perceived to be mere conjecture within the email spool. Well, there you go.
This story has reached another level. Hard evidence of such blatant data manipulation mustn’t be allowed to simply vanish into the news cycle. I hope the few MSM outlets accurately reporting this story pick up on this, because the notes in the code indeed appear to be a smoking gun.

philincalifornia

It may have been posted before so, if so, apologies but I do like this one – to be found in the HARRY_READ_ME.txt file:
“OH F*** THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m
hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform
data integrity, it’s just a catalogue of issues that continues to grow as they’re found.”
(I confess to manipulation of the f-word myself, but only to comply with WUWT policy).
So Copenhagen Comrades, what’s a trillion dollars or so here and there based on “no uniform data integrity” ??

UKIP

No resignations or sackings yet then?

Hank Hancock

I think it is noteworthy that Steve McIntyre comments on Mann and Briffa truncating their MXD data at 1960.
http://www.climateaudit.org/?p=4221

artwest

A poster called Asimov has quoted extracts from the “HARRY_READ_ME.txt” file – deeply shocking stuff:
Asimov’s very plausible suggestion is that Harry is a programmer trying, and often failing, to make sense of the garbage data which he’s been lumbered with.
Several posts over several pages:
http://www.tickerforum.org/cgi-ticker/akcs-www?post=118625&page=13

Can someone point me what file is this code from?
I have downloaded the package, and found only one file (FOIA/documents/osborn-tree6/mkp2correlation.pro) that includes the above mentioned function. Lines 1 to 7 are identical, but the rest has nothing to do with the screenshot above.

Leon Brozyna

Emails may be just chit-chat between various parties, but the coding is the receipe for cooking the books. No wonder AGW leaves such a bad taste — it’s way too overcooked leftovers.

@Nick Stokes: the real issue is that temperatures derived from tree rings are known to not match measured temperatures after 1960.
If tree ring-based temperatures are known to be false compared to actual measurements, then how can they be true in earlier decades or centuries?

Michael Jankowski

Nick,
Exactly who is responsible for “artificially adjusted to look closer to the real temperatures” that the code is advising against plotting, and why would they have done such a thing?
I agree that it’s no problem with the code saying not to plot past 1960, but it is certainly a problem that the code says someone has taken liberty with post-1960 data (or the methodology used to process it) for the purpose of making it “look” more like the instrumental record.

D. King

Wow, the coming week will be most interesting.
Good job Anthony, Steve, and all. Thank you.
Time to look at the sea ice satellite AGC, pointing,
and receiver gain. I think we may find some missing ice.

Harold Vance

The really sad thing is that the dendros still have no clue why trees make good thermometers in some years but not others. This is a bigger issue, imho.

Gerald Machnee

RE: Nick Stokes (20:34:52) :
**I may be dense here, but what’s the issue? The red comment says “don’t plot beyond 1960″, because the results are unreliable**
Maybe it means “don’t plot beyond 1960 because the results do not show what we want”?????

Jon Adams

The CRU team and maybe all of these “AGW researchers” are clueless… they are way over their heads and inclined to cheat – wait…its defraud as adults – especially with literally trillions of dollars involved.
How many billions has the US and other countries wasted on this BS…
I want my money back!
One may surmise they did not want any Real Programming Talent Aboard to maybe OUT them? so they tried to fake it themselves…
Anyway… thank God for Anthony… attempting to get some quality to the data sets involved!

rbateman

philincalifornia (20:51:09) :
Yep, saw that, and that’s not the only place he used it. Leaves little doubt that the data was pre-slaughtered, does it not?
Now, who exactly is HARRY?

Jeff Coatney

Unbelievable!!!!!!!!!!!!!!
Or, perhaps, not so unbelievable.
There are ways to game any system. All it takes is a person or persons clever enough to formulate effective methods of cheating.
In the end, what counts more than anything else is the ability to rely on the word of others. To the extent that people are more or less untrustworthy does the potential for dishonesty rise or fall.
Is it just me, or is the societal willingness to indulge in unethical behaviour presently on a significant and massive upswing – Perhaps a little like our friend, the hockey stick curve.

Eric Barnes
Doug in Seattle

P Gosselin (20:47:32) :
WHOA!
I can’t believe it!
One of Germany’s biggest highly respected dailies has it on the front website page.

And in English here – http://translate.google.com/translate?u=http%3A%2F%2Fwww.welt.de%2Fwissenschaft%2Farticle5294872%2FDie-Tricks-der-Forscher-beim-Klimawandel.html%23xmsg_comment&sl=de&tl=en&hl=en&ie=UTF-8

AlexB

The alarmists have gotten so good at misdirection that they get me every time. Their comments are never on the real issue. When I read Dr. Jones explanation of the word ‘trick’ I thought that it was perfectly reasonable so decided to lay that comment to rest. I re-read the e-mail later though and thought ‘hang on!’. Is the word ‘hide’ also commonly used by scientists to mean something other than its common usage? I can’t believe they got me again!

Nick Stokes

Roger Sowell (20:58:33) :
@Nick Stokes: the real issue is that temperatures derived from tree rings are known to not match measured temperatures after 1960.
If tree ring-based temperatures are known to be false compared to actual measurements, then how can they be true in earlier decades or centuries?

That’s not a coding issue. What they say in Briffa 2001 for the Siberian trees is:
““The period after 1960 was not used to avoid bias in the regression coefficients that could be generated by an anomalous decline in tree density measurements over recent decades that is not forced by temperature””
The claim seems to be that they have specific information about a post-1960 divergence, and presumably enough pre-1960 instrumental overlap to satisfactorily calibrate. Now I can’t judge the strength of that, but it’s been discussed in the literature for nearly a decade. The answer won’t be found in a comment in the code. The comment merely reflects the limit stated in the published theory.
REPLY: In other work, Briffa allowed 10 Yamal trees, an unacceptably low sample, to stay in. When he says “anomalous decline in tree density measurements over recent decades” why then would he think 10 trees as a sample upon which to base that the datapoints are OK? As Steve has said many times (and other dendros agree – they say 50 is the minimum sample), Briffa should have truncated that Yamal sample and data. He didn’t. So why is it OK to not truncate a very low sample in one case and not another? Doesn’t make any sense. – A

Looking in the file HARRY_READ_ME.txt there seems to be some suspicious coder comments as well:
“This still meant an awful lot of encounters with naughty Master stations, when really I suspect nobody else gives a hoot about. So with a somewhat cynical shrug, I added the nuclear option – to match every WMO possible, and turn the rest into new stations (er, CLIMAT excepted). In other words, what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad, but I really don’t think people care enough to fix ’em, and it’s the main reason the project is nearly a year late.”
“You can’t imagine what this has cost me – to actually allow the operator to assign false WMO codes!! But what else is there in such situations? Especially when dealing with a ‘Master’ database of dubious provenance (which, er, they all are and always will be).”

Jon Adams

Everyone… This is Not just about Money … it is about Tyranny… the politicians want to control us till we have no soul…
And we have the Nick Stokes who want to jump on a mention of 1960… temp mismatches… the entire picture is what you need to look at, Nick… study some more and you will begin to see a pattern.

Policyguy

Roger Sowell (20:58:33) :
@Nick Stokes: the real issue is that temperatures derived from tree rings are known to not match measured temperatures after 1960.
If tree ring-based temperatures are known to be false compared to actual measurements, then how can they be true in earlier decades or centuries?
Well, perhaps that is a place to start. According to John Dally we know that trees don’t grow on 70% of the earth’s surface (oceans), they don’t grow in deserts or at high elevations. The 15% of the earth’s surface where trees do grow are in those locations where they may also impacted by lack of light (other trees), lack of water (draught), to the extent – that temperature can not be isolated as a cause for growth during any period. It is a false measure.
So lets challengeh tree data as a temperature surrogate altogether. That will take care of Mr. Mann and other tree persons and throw out the hockey stick. Apparently, according to his now published email, even Mr. Revkin now agrees with that.

I agree (as a software developer, especially) comments in code are very rarely any kind of ‘mistake’ as such, although many are out of date, it is true.
Overall, this, the email, and what I suspect to be gleaned from the data, the whole story is a shambles. There is only one possible thing that can be done to salvage the credibility of anyone in the field who supports AGW:
1. All existing data and conclusions should either be examined for veracity or dismissed.
2. There must be a new, non IPCC and non-UN controlled, globally funded task force to re-examine the whole AGW issue. Existing AGW believers and sceptics should be included, especially those who have shown extraordinary effort and dedication in the field to date.
3. The entire investigation should be transparent to the participants, funders (one assumes governments) and also the public. The Internet is a good medium for such a task, as has been proved.
4. All political involvement must be prevented. That cannot be stressed enough.
5. All commercial involvement should be prevented. Oddly enough, I would support ‘big oil’ etc as it seems they are gearing up very swiftly to get ahead of the game in renewable, as is sensible. I suspect that many would cry foul, however.
6. No taxes or political changes should be introduced that rely on the AGW theory being accurate until it is proved that CO2 increases will cause dangerous changes to the climate.
Just my 2c worth…..

Jesse

Once again, you guys are making mountains out of ant hills. This is just normal data processing per the 1960 divergence problem as shown by NUMEROUS sources. This is what happens when a bunch of uninformed amateurs try and “debunk” real scientists. Leave the science to the scientists and go back to your day jobs as custodians, wal-mart employees and laborers.
REPLY: So what do you do down there in Norman? NSSL? U of OK? You might be surprised at the sort of professionals that frequent here. I invite them to sound off. – A

John F. Hultquist

What I interpret this to mean is that the results of their work pre-1960 are as they think they should be. Post-1960, not so much. Post-1960 results have to be adjusted to reflect …. what?
a. a previously non-operating variable, now operating;
b. a previously operating variable that no longer operates;
c. some combination of a & b;
d. a hunch based on years of experience;
e. a revelation from God;
f. other
The code used to ‘artificially adjust’ the results so they will ‘look closer to the real temperatures’ ought to explain which of the above scientific procedures was used. If this code is not in the posted material, I am sure the researchers will happily provide it.
~~~~~~~~~~~~~~~~~~~~
A few months back WUWT had a discussion on programming so, not to go over all that again, I’ll just say many of us (in years gone by) wrote our own code and/or subroutines that were used over and over. I always put a few comments at the top stating a few common things, such as, my name, and the purpose of the code, and the language (FORTRAN 2D, IV, ??). Is there a complete routine in this dump that would identify the programmer?

wes george

Wait and see, the True Believers will soon come out with a “Fake but accurate” defense. Remember, the fate of the whole world hangs in the balance, so what’s a few lies, a bit a fraudulent science and bullying matter as long as the Green agenda of zero-growth, centrally mandated economies and increasing restriction on individual liberties is moved forward.
The end justifies the means in AGW ethics.

Glenn

Richard Sharpe (20:44:52) :
“When people use cut-n-paste coding, they sometime even copy the comments and forget to change them.”
Oh that’s encouraging, the fate of the world hanging on shade tree coders.

WakeUpMaggy

Maybe one of the programmers was the mole.

Dr A Burns

Another strange happening at Hadley … all the hadcrut3 data for this year, except Jan/Feb, has been deleted.
http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt

John F. Hultquist

P Gosselin (20:47:32) : You likely know this but I had to have it translated, except for the two opening words
Die Tricks der Forscher beim Klimawandel
via Google translate
The tricks of the researchers on climate change

Also, in the Word file jones-foiathoughts.doc it states:
“Options appear to be:
1. Send them the data
2. Send them a subset removing station data from some of the countries who made us pay in the normals papers of Hulme et al. (1990s) and also any number that David can remember. This should also omit some other countries like (Australia, NZ, Canada, Antarctica). Also could extract some of the sources that Anders added in (31-38 source codes in J&M 2003). Also should remove many of the early stations that we coded up in the 1980s.
3. Send them the raw data as is, by reconstructing it from GHCN. How could this be done? Replace all stations where the WMO ID agrees with what is in GHCN. This would be the raw data, but it would annoy them.”
Number 2 seems to indicate the particular station data that they were trying to hide…

Nick Stokes

rechauffementmediatique (20:57:34) :
I found these comments in osborn-tree6/mann/oldprog in files like maps12.pro and
maps15.pro. These were dated Jan 2000, and the directory name is not encouraging. Seems unlikely that it is currently used code.

Dave Johnson

The Times here in the UK now has the CRU story on one of it’s major comment pages
http://www.timesonline.co.uk/tol/comment/columnists/guest_contributors/article6927598.ece

Neil O'Rourke


rechauffementmediatique (20:57:34) :
Can someone point me what file is this code from?
I have downloaded the package, and found only one file (FOIA/documents/osborn-tree6/mkp2correlation.pro) that includes the above mentioned function. Lines 1 to 7 are identical, but the rest has nothing to do with the screenshot above

documents\osborn-tree6\summer_modes\maps12.pro

philincalifornia

artwest (20:54:26) :
A poster called Asimov has quoted extracts from the “HARRY_READ_ME.txt” file – deeply shocking stuff:
Asimov’s very plausible suggestion is that Harry is a programmer trying, and often failing, to make sense of the garbage data which he’s been lumbered with.
Several posts over several pages:
http://www.tickerforum.org/cgi-ticker/akcs-www?post=118625&page=13
____________________________
I also lifted this from a comment over there:
“Tim Mitchell works at the Climactic Research Unit, UEA, Norwich, and is a member of South Park Evangelical Church.”
South Park ?? You’ve got to be kiddin’ me ??
PS I also tried to check out if he really did spell it “Climactic” but, strangely, the web site appears to be down. Heh heh heh.