When the CRU emails first made it into news stories, there was immediate reaction from the head of CRU, Dr. Phil Jones over this passage in an email:
From a yahoo.com news story:
In one leaked e-mail, the research center’s director, Phil Jones, writes to colleagues about graphs showing climate statistics over the last millennium. He alludes to a technique used by a fellow scientist to “hide the decline” in recent global temperatures. Some evidence appears to show a halt in a rise of global temperatures from about 1960, but is contradicted by other evidence which appears to show a rise in temperatures is continuing.
Jones wrote that, in compiling new data, he had “just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (i.e., from 1981 onwards) and from 1961 for Keith’s to hide the decline,” according to a leaked e-mail, which the author confirmed was genuine.
Dr. Jones responded.
However, Jones denied manipulating evidence and insisted his comment had been taken out of context. “The word ‘trick’ was used here colloquially, as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward,” he said in a statement Saturday.
Ok fine, but how Dr. Jones, do you explain this?
There’s a file of code also in the collection of emails and documents from CRU. A commenter named Neal on climate audit writes:
People are talking about the emails being smoking guns but I find the remarks in the code and the code more of a smoking gun. The code is so hacked around to give predetermined results that it shows the bias of the coder. In other words make the code ignore inconvenient data to show what I want it to show. The code after a quick scan is quite a mess. Anyone with any pride would be to ashamed of to let it out public viewing. As examples [of] bias take a look at the following remarks from the MANN code files:
Here’s the code with the comments left by the programmer:
function mkp2correlation,indts,depts,remts,t,filter=filter,refperiod=refperiod,$
datathresh=datathresh
;
; THIS WORKS WITH REMTS BEING A 2D ARRAY (nseries,ntime) OF MULTIPLE TIMESERIES
; WHOSE INFLUENCE IS TO BE REMOVED. UNFORTUNATELY THE IDL5.4 p_correlate
; FAILS WITH >1 SERIES TO HOLD CONSTANT, SO I HAVE TO REMOVE THEIR INFLUENCE
; FROM BOTH INDTS AND DEPTS USING MULTIPLE LINEAR REGRESSION AND THEN USE THE
; USUAL correlate FUNCTION ON THE RESIDUALS.
;
pro maps12,yrstart,doinfill=doinfill
;
; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
;
and later the same programming comment again in another routine:
; ; Plots (1 at a time) yearly maps of calibrated (PCR-infilled or not) MXD ; reconstructions ; of growing season temperatures. Uses “corrected” MXD – but shouldn’t usually ; plot past 1960 because these will be artificially adjusted to look closer to ; the real temperatures.
You can claim an email you wrote years ago isn’t accurate saying it was “taken out of context”, but a programmer making notes in the code does so that he/she can document what the code is actually doing at that stage, so that anyone who looks at it later can figure out why this function doesn’t plot past 1960. In this case, it is not allowing all of the temperature data to be plotted. Growing season data (summer months when the new tree rings are formed) past 1960 is thrown out because “these will be artificially adjusted to look closer to the real temperatures”, which implies some post processing routine.
Spin that, spin it to the moon if you want. I’ll believe programmer notes over the word of somebody who stands to gain from suggesting there’s nothing “untowards” about it.
Either the data tells the story of nature or it does not. Data that has been “artificially adjusted to look closer to the real temperatures” is false data, yielding a false result.
For more details, see Mike’s Nature Trick
UPDATE: By way of verification….
The source files with the comments that are the topic of this thread are in this folder of the FOI2009.zip file
/documents/osborn-tree6/mann/oldprog
in the files
maps12.pro
maps15.pro
maps24.pro
These first two files are dated 1/18/2000, and the map24 file on 11/10/1999 so it fits timeline-wise with Dr. Jones email where he mentions “Mike’s Nature trick” which is dated 11/16/1999, six days later.
UPDATE2: Commenter Eric at the Climate Audit Mirror site writes:
================
From documents\harris-tree\recon_esper.pro:
; Computes regressions on full, high and low pass Esper et al. (2002) series,
; anomalies against full NH temperatures and other series.
; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline
;
Note the wording here “avoid the decline” versus “hide the decline” in the famous email.
===============
I’ll give Dr. Jones and CRU the benefit of the doubt, maybe these are not “untowards” issues, but these things scream for rational explanations. Having transparency and being able to replicate all this years ago would have gone a long way towards either correcting problems and/or assuaging concerns.
Sponsored IT training links:
Need help for EX0-101 exam ? We offer self study 642-436 training program for all your 642-974 exam needs.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
New article from Christopher Brooker in the Daily Mail this morning …
The devastating book which debunks climate change
Front page link too.
Tongue in cheek:
Dr. Mann should have known when he sat out picking cherries that cherry wood is far too brittle to make a hockey stick. Perhaps he should have realized that if he were working with the proper material he might not have made an ash of himself.
Another Daily Mail article by Daniel Martin …
How climate-change scientists ‘dodged the sceptics’
Nice image of Jones (TIP: Place your mouse cursor over it!)
Re Nick Stokes (20:34)
I copied this from “The Air Vent” http://noconsensus.wordpress.com/
Kenneth Fritsch said
November 22, 2009 at 10:30 pm
The scientist is going to concentrate on what is unsual about the proxy, i.e. the proxy does not respond correctly to temperature after 1960. She would not think of plastering something into that time period to deflect attention. She would want to talk about and let you know what she has found – and attempt to explain it.
The advocate and all his defenders, on the other hand, will want to sell their message and when they are questioned will in bewilderment say ” well of course we would not use “bad” proxy data when we have “good” instrumental data and how dare you think we intended to deceive.
Drudge gets +20 million hits a day, has this in right column :
Hostility among foes…
…Britain’s Climate Research Unit of the University of East Anglia reveal an intellectual circle that appears to feel very much under attack, and eager to punish its enemies….”Kevin and I will keep them out somehow — even if we have to redefine what the peer-review literature is!”…”I will be emailing the journal to tell them I’m having nothing more to do with it until they rid themselves of this troublesome editor,”…
http://www.drudgereport.com/
http://www.washingtonpost.com/wp-dyn/content/article/2009/11/21/AR2009112102186_pf.html
I started looking at some of the non-Email and quickly found myself in the midst of documents/HARRY_READ_ME.txt . It’s a dairy of a three year saga to bash a lot of code to run on a new platform and his dealings with data files in as bad shape as the code.
This post didn’t exist at the time, plus the file would be more interesting to the denizens of Chiefio, so I posted http://chiefio.wordpress.com/2009/11/21/hadley-hack-and-cru-crud/#comment-1664 over there.
I’ve written similar diaries, but not for a project this long or with code this fouled up. Programmers will appreciate it the most (especially those with long weekend nights under their belt).
Anthony,
REPLY: In other work, Briffa allowed 10 Yamal trees…
I didn’t want to get into dendro issues here. The thread is about the meaning of a comment in the code, and all I’ve been saying is that the comment is consistent with what is done in the 2001 Briffa et al paper – namely the use of pre-1960 data. The code also dates from the time the 2001 paper would have been being written.
However, I can’t see the relevance of the Yamal issue. Apart from anything else, the 10 tree period there was 1990 and beyond, while here data is not taken past 1960. And it’s a NH paper – Yamal would be a very small part.
REPLY: My point is about exclusions based on sample size, the strategy of exclusion is inconsistent. – A
And how about this admission from Ed Cook, a dendro: http://whatcatastrophe.com/drupal/node/47
No single piece of this jigsaw is determinative and it will take some time for all the relevant material from the leaked/hacked files to be collated.
We have all read some of the emails containing suspicious wording but it should not be overlooked that many of them are capable of innocent interpretation. That is why the “big picture” is so important. Notes in coding also only give a snapshot of the thinking of the coder (is that the correct term?) at the time. Isolated soundbites do not make a case, a coherent body of evidence is needed and that can result only from detailed examination of everything in its proper context.
Of particular importance are the following questions: (a) Does a pattern of behaviour appear? (b) Do private thoughts recorded in emails and notes correspond with the writer’s public pronouncements? (c) When Team Member A suggested something ostensibly underhand or improper to Team Member B, did Team Member B expressly agree, expressly dissent or remain silent? (d) Were the financial interests of the writer or his employer taken into account when deciding how to present scientific conclusions?
If we concentrate on those questions (although many more are also relevant) we will be able to see clearly the extent of any jiggery-pokery.
My first impression is that the documents disclosed fall a long way short of being a complete package. Perhaps the leaker/hacker is planning a second instalment to rebut detailed explanations and excuses from members of the Team. Time will tell.
So far, so good, but don’t get overexcited by individual items.
Yes, Asimov also had quotes from the code up about how they had lost all the cloud data before 199X (sorry on the X, cant remember). That code is a very interesting piece of work.
The HARRY_READ_ME.txt is a MUST READ to understand the utter chaos of the CRU TS. Here are a few more snippets:
BEGIN FILE =========
So.. should I really go to town (again) and allow the Master database to be ‘fixed’ by this program? Quite honestly I don’t have time – but it just shows the state our data holdings have drifted into. Who added those two series together? When? Why? Untraceable, except anecdotally.
It’s the same story for many other Russian stations, unfortunately – meaning that (probably) there was a full Russian update that did no data integrity checking at all. I just hope it’s restricted to Russia!!
There are, of course, metadata issues too. Take:
……….
..knowing how long it takes to debug this suite – the experiment
endeth here. The option (like all the anomdtb options) is totally
undocumented so we’ll never know what we lost.
22. Right, time to stop pussyfooting around the niceties of Tim’s labyrinthine software suites – let’s have a go at producing CRU TS 3.0! since failing to do that will be the definitive failure of the entire project..
……….
Tried running anomdtb.f90.. failed because it couldn’t find the .dts file! No matter that it doesn’t need it – argh!
Examined existing .dts files.. not sure what they’re for. Headers are identical to the .dtb file, all missing values are retained, all other values are replaced with one of several code numbers, no idea what they mean.
END FILE ==========
It goes on and on and on like this. I’ve never seen so much confusion in any coding project (and I’ve worked on more than a few). From what I’ve seen, I wouldn’t trust them to code a toy app for an iPhone.
Jeff Id (20:40:17) :
….smells like a duck’s butt.
I’ll just take your word for that.
😉
As someone who spent 25 years for the most part working on one complex custom software system for a certain agency of the Federal Government (that shall remain nameless), all I can say is I fully agree with above comment by Jeff:
Like most programmers on complex software systems with a couple 100 thousand lines of code, we tried hard to accurately and fully describe the functions of each module in the code comments. Failure to do so leads to chaos; and any programmer worth his or her salt will do the same.
If it looks, walks, flys, swims, and quacks like a duck, theoretically it still COULD be something else. But lacking overwhelming evidence to the contrary, odds are REAL good that it’s a duck. The duck quacked in this case, and their goose is cooked.
I’m a statistical programmer for “BIG PHARMA” . For every new drug application, the FDA requires that we give them: raw data, analysis datasets (which are a merging of raw datasets and algorithms applied to raw data), a description of the algorithms and statistical methods, AND all our code. Then, the FDA reviewers try to come up with our results. This is done for every drug or medical device before approval.
The societal impact of global warming er.. climate change is greater than any one drug. If we go through all this independent review for a drug, we should demand a similar review process be applied to AGW claims.
The FDA, acting as a public protector, has to assume we are trying to “cheat” (and that is a reasonable approach). We never throw out data. Granted, our clinical trials are more controlled , but this ‘give the reviewers everything’ approach should be applied as much as possible in climate research.
But then again, we are the evil, capitalist, profit-seeking, “BIG PHARMA” and the people need to be protected from us.
Jesse (21:24:11) :
You’re right Jesse, there’s probably nothing here for you. Maybe you can spend your time at RealClimate. They’re more your type. We’re too far below you.
Say! Did you happen to see the post by Roy Spencer about elitism? Just wondering.
Keep looking folks; at the end of the day this another yawner.
There has to be something beyond more egg on the face for the usual suspects to for this to go anywhere. Either problems in the HadCRU data (due to errors or malfeasance), or implication of someone else.
You can only dump on Mann, Jones, Briffa et al. for so long. It’s lots of fun, but it’s a distraction from the main event: scientific evidence that the climate just isn’t that sensitive to CO2 levels. That’s what Spencer is after, and that’s what matters. Not what happened in some tree ring in 1962.
Off Topic…
Mainstream press overnight (US time) is now more confident and pushing harder…
UK Daily Mail pushing the FOI avoidance:
http://www.dailymail.co.uk/news/article-1230122/How-climate-change-scientists-dodged-sceptics.html
Canada’s Edmonton Journal also highlights the directions to delete stuff:
http://www.edmontonjournal.com/technology/Good+climate+news+alarmists/2252439/story.html
Wall Street Jounal – Also more strident today is – even highlights a link for readers to get the original files!
http://online.wsj.com/article/SB125883405294859215.html?mod=WSJ_hpp_sections_news
Meanwhile….
BBC says:
‘Hopes for the Copenhagen climate summit in December have been boosted after it emerged that more than 60 presidents and prime ministers plan to attend.’
http://news.bbc.co.uk/2/hi/europe/8373551.stm
Will they…NOW??
and poor old Mother Jones (I used to love that mag) is left explaining the meaning of ‘trick’
http://www.motherjones.com/kevin-drum/2009/11/tricked-out
Jesse (21:24:11) :
. . . bunch of uninformed amateurs try and “debunk” real scientists. Leave the science to the scientists and go back to your day jobs as custodians, wal-mart employees and laborers.
REPLY: So what do you do down there in Norman? NSSL? U of OK? You might be surprised at the sort of professionals that frequent here. I invite them to sound off. – A
Hi Jesse
Let me take a break from from floor waxing to say that I did grad school down in Norman. The professors all said they were trying to build a grad school their football team could be proud of. Well, gotta get back to buffing Wal-mart’s wide aisles.
Hi everyone else
I think the real meat here will be coming from the data and code, not in the email fluff. Here’s hoping M&M find the foi data they were looking for.
wes george (21:24:44) :
Wait and see, the True Believers will soon come out with a “Fake but accurate” defense.
They feel they can’t go down. The Titanic couldn’t sink too.
Once again, you guys are making mountains out of ant hills. This is just normal data processing per the 1960 divergence problem as shown by NUMEROUS sources. This is what happens when a bunch of uninformed amateurs try and “debunk” real scientists. Leave the science to the scientists and go back to your day jobs as custodians, wal-mart employees and laborers.
Nothing as convincing as some ad hom attacks.
Lots of folks posting here have actual academic qualifications. (I have an MSc in Clinical Research and 20 years of experience working with clinical data sets FWIW.)
The work from The Team is a joke compared to the standards that the FDA and EMEA require. That’s the problem with “appeal to authority” as a debating technique. When your “authority” turns out to be a tiny cabal of shoddy scientists, things kind of fall apart)
….meanwhile the earth is cooling…. and the general population is slowly finding out that there was collusion among the top circle of global warming scientists….
Glenn (20:38:14) :
I’ve been unable to find the definition of “artificial adjustment” in the climatologist’s handbook. It must be called something else.
Look under ‘cha-ching’
Doug (21:56:11) :
Dr. Mann should have known when he sat out picking cherries that cherry wood is far too brittle to make a hockey stick.
Here’s a case of being able to tell a lie about a cherry tree.
In SOAP-D15-intro-gkss.doc, it states:
“Osborn and Briffa, together with other co-authors (Rutherford et al., 2005), examined the sensitivity of temperature reconstructions to the use of different target variable (annual or seasonal temperature), target domain (hemispheric or extratropical) and reconstruction method. They found that when the differences in target variable and domain are accounted for, the differences in several reconstructions are small and that all reconstructions robustly indicate anomalous warm conditions in the 20th century, with respect to the past millennium.”
Since they are subbing real temp data after 1960, isn’t this fraud?
I mean, I think this is a pretty bald face lie to me.
Jesse (21:24:11) :
Once again, you guys are making mountains out of ant hills. This is just normal data processing per the 1960 divergence problem as shown by NUMEROUS sources. This is what happens when a bunch of uninformed amateurs try and “debunk” real scientists. Leave the science to the scientists and go back to your day jobs as custodians, wal-mart employees and laborers.
IF THERE EVER WAS AN APOLOGY FOR THE CURRENT ELITISM (and it isn’t even a good one) THIS QUOTE…IS IT.
Contrary to Ayn Rand, who was an “elitist” of sorts, but saw the inherent value in anyone with a heart and soul, no matter what their “blue-collar” “job”, this individual pulls the curtain back and exposes himself for who he really is:
The “we scientists” part is extremely pungent given the fact that, empirically speaking, some of the first individuals to cave in the Third Reich….were the scientists.
Explain them apples.
Not sure, but it might have to do that scientific intelligence is not the end-all, be-all.
Yes, yes….science SHOULD rule (am in total agreement there).
Its just when you turn it to these nasty, nasty, ad hominem comments about custodians, and Wal-mart greeters [ugh i hate Wal-mart but don’t hate the people who need a job…so they work there], and others that YOU, JESSE, so quickly deride.
But beyond that, Jesse, this is not “making mountains of of ant hills.”
It is standing up for the truth, whatever it may be.
And…in passing…those FIRE ANT mounds that have taken over in the southeast US thanks to the positive AMO, are, to us, mountains….and if it is a big deal to you, their opportunistic, swarming, life-choking habits will hopefully come to a halt when the AMO turns negative.
So this whole CRU-gate saga is…it IS….a big deal.
It most certainly is a big deal. Precisely why you are your opportunistic mates are so up at arms.
So I say hear hear HAIL HAIL let’s end this cowardistic hi-jack of more than a few opportunistic ideologues who are trying to control the world and lets get on with….
SCIENCE BUSINESS AS USUAL!
AGW is dead. (Thank bl**dy g*d!)
Chris
Norfolk, VA, USA