The people -vs- the CRU: Freedom of information, my okole…

I found this particularly interesting
http://www.eastangliaemails.com/emails.php?eid=304&filename=1048799107.txt
especially this part
8.0 Articles
A) How women matter in decreasing world population
B) The energy we need
C) Mining the impacts
D) Symbiotical relationship of religion and global life-support systems
E) Celebration of Life Day
F) The hidden agenda: China
G) Earth Government now a priority
H) The splitting of America into separate independent states living at peace for the
good of all
I) The war industry: the modern evil at work in the Middle East
J) Earth security
K) Earth governance
L) The Earth Court of Justice holds the people of the U.S.A. and Britain as criminals
M) Foundation for the new world order, Earth Government

This made me smile:
“Ben,
Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. “…
The rest makes me weep….

Hat tip for the use of some local Hawaiian language 🙂

As I said, the issue is not Trenberth or scientists talking smack.

Translation for those not from the US: As I said, the issue is not Trenberth or scientists badmouthing people or being nasty.

Outstanding !!
Can you show a flow chart based on The AGW Method??

Dean of the College of Earth and Mineral Sciences , W. E. Easterling told me by email this afternoon that Penn State would be releasing a statement later today. Has anyone seen it?

Anthony,
It’s obvious why Phil Jones didn’t supply the data. You foolishly asked for the “raw” data. You should have asked for the “artificially adjusted” data, as that seems to be what they use.

Pseudscience has the first step as “Belief”, or “assumption based on a consensus” and not observation. No one has observed AGW occurring. This is a subtle but crucial difference, and one that also occurs in astronomy, archaealogy and geology. AGW is hence deductive science minus the initial empirical basis and climate science spends all of its efforts proving the initial assumption, not realising that no scientific theory can ever be proven, but they can be falsified. Hence AGW is the misuse of the scientific method for political purposes.
Except the AGW crowd can’t seem to understand this, hence the animosity towards the sceptics.

Interesting to see the interplay regarding FOI. It really helps to have your side as well. It’s sad that there are people who claim to be scientists who will function as apologists for this behavior – it’s wrong and they know it.
Unless they all want to cop a “by reason of insanity” defence.

OK…this is just…deeply…deeply disturbing.
I especially love the comparison between McIntyre and McCarthy (!)…McIntyre is requesting that publicly funded data and research methods be made available as is consistent with the scientific method. And their response is that he’s being a totalitarian and a schoolyard bully?
Fabulous. Just fabulous.

So I wonder. Will Warwick Hughes now get the data and code he was after?

This goes rather neatly with my posting earlier today on PJM.

I can now see what realclimate was talking about when they said that these emails were made available without the proper context. Giving the full context is much clearer – these people’s behavior is despicable and no excuse can now be found for it.

Wow. Wonder what comes from the emails when you look at whose papers were released, when they were released, who is cited in the paper, and what the paper covers.

Reads like Exhibit #1 in Sen. James M. Inhofe (R-Okla.) investigation.

Part of the difficulty with climate science is that, unlike all other physical sciences, it does not study things — instead it studies averages.
This is because climate by definition is the average of weather over a suitably long period of time (typically taken as a minimum of 30 years).
As a result, much of the study that goes on, and the papers that are written, deal almost exclusively with mathematics and statistics. This is the reason that access to the computer codes is so critical.
It’s simple in the physical sciences to describe an experiment, e.g. “I took three grams of carbon and subjected them to a pressure of 50,000KPa and a temperature of 500C. Unfortunately, the experiment did not succeed, I could not replace the diamond I had lost from my wife’s wedding ring.” Anyone can reproduce that experiment (and get the same results).
But when you say “I took the raw temperature data, variance-adjusted it, averaged it, gridded it, area-adjusted it, extrapolated results to data-free areas within 250 km, and made a global temperature record”, that’s far from enough information. In order to determine what was done, we need far more detailed information in climate science, because in general we are describing intricate mathematical operations. These are often very hard to describe clearly in spoken or written language.
And even a crystal-clear description is not enough. Despite what he says he has done, if the scientist has inadvertently used an improper procedure (e.g. the uncentered principal components analysis used in Mann’s Hockeystick), we’ll never be able to determine that the answer is demonstrably wrong unless we have the actual code that he used. Otherwise, we could spend years trying to guess where he went wrong, but we would never be able to show that he went wrong as science demands.
This is why the insistence of scientists that their computer codes are sacrosanct private secret documents best kept under Hermetic seal in a clandestine vault is lethal to good science. Without the codes, we can’t tell if what has been done is correct and free from hidden mathematical error. Of course, this may be unconnected with the reason that Jones et. al are hiding their codes …
Given that climate science is not the study of things but of the averages of things, and that as a result math and statistics are central to climate science, the findings of the Wegman Report are now seen to be even more insightful, trenchant, and valid. They said:

It is important to note the isolation of the paleoclimate community; even though they rely heavily on statistical methods they do not seem to be interacting with the statistical community. Additionally, we judge that the sharing of research materials, data and results was haphazardly and grudgingly done. In this case we judge that there was too much reliance on peer review, which was not necessarily independent. Moreover, the work has been sufficiently politicized that this community can hardly reassess their public positions without losing credibility. Overall, our committee believes that Mann’s assessments that the decade of the 1990s was the hottest decade of the millennium and that 1998 was the hottest year of the millennium cannot be supported by his analysis.

And presciently, that was written thee years ago, well before we got the CRU emails …

Can’t argue with Popper, Popper wins of Aristotle any day of the week. Indeed it was Aristotle’s views and methods that helped make the dark ages so dark; except there is no evidence that the dark ages were dark.
PC analysis has shown that using paving-stone wear patterns as proxies for intellectual development clearly indicate that the people in the dark ages had a profound understanding of the natural world, the scientific method and the dangers of too much atmospheric CO2. They deliberately didn’t burn fossil fuels to save future generations.

They are elitists. They are not to be questioned by YOU. And they enlist their elitist friends in helping them. Since they see themselves as THE elite scientists they believe themselves to be above the petty requirements of the FOI act, these acts don’t apply to THEM. We see this all the time in politics, in journalism, throughout academia, nothing special in it, it is as common as dirt. The really interesting part is that these idiots are frequently socialists, they really want socialism to take over the world. And once it does it is these same elitists that quickly end up in the gulags as slaves or dead. It is quite logical actually, these elitists are insufferable to be around so once the Marxist thugs take over they quickly get rid of them. It is true, look at the old Soviet Union, Cuba, China, anywhere socialism took over, the elitists help them to power then get purged. They are not really that smart, they do this over and over again, history means nothing to them. Their ego tells them they are above history. If they are lucky they live somewhere that never goes socialist and never have their wish fulfilled, and they live a full life honored by their equally elitist friends, writing Marxist drivel, living in moderately well off surroundings; bitter, clinging to their ideology and their tenure.
Keep pushing, these are brittle people, they’ll crack before they’ll bend.

This is interesting:
From the progrm file FOI2009/FOIA/documents/harris-treebriffa_sep98_e.pro
—–
;
; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
; standardised datasets.
; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
; with missing values set appropriately. Uses mxd, and just the
; “all band” timeseries
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
; Now normalise w.r.t. 1881-1960
;
mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj
;
; Now plot them
;
—–
‘Oooops!’ indeed. I wonder what the ‘plot’ was?
Oh well…, back to the code.

Willis, couldn’t agree more. This is what I said in Tips and Notes earlier:
David (12:31:24) :
…..In the HARRY_READ_ME.txt, a search for ’synthetic cloud grids’ brings up this particular piece of information ‘This means that the cloud data prior
to 1996 are static.’ If the climate models can reproduce past climate with static cloud figures, then they are quite obviously wrong.
It is past time for the climate models to become open source, so that everyone can replicate the results of these models to their hearts content. It is also imperative that the data used to model climate become open to the public as well, as the disturbing trend of silencing critics of the AGW theory revealed in the CRU leak is simply a reminder that these scientists are also human and subject to the same internal faults as those in the financial community regularly excoriated for their crimes against investors (which, by the way, led to the practice of auditing financial statements as we know it today.)
—————
If I understand HARRY_READ_ME correctly, isn’t that the GCM they are talking about having cloud data prior to 96 as static? That seems like a huge mistake if that is the case.

Willis Eschenbach (16:31:46)
If they can’t identify the stations they’re collecting data from, then presumably they don’t know what it refers to. Its akin to going to the bank and depositing cheques, and the cashier putting them into random acounts. Just imagine 100 people sitting exams and the examiner allocating results based on alphabetical order..
It looks like a rather clever “trick” (to use Jones language) to circumvent FOI

Willis,
I believe that Charles was referring to first comment at 16:07:03 by Steve.
Thank you for all your work on this. I’ve long thought that the Climate Emperor’s wardrobe was a bit sparse. Now it appears to be much more hole than cloth.

More than anything, the series of emails from those people really reflect their personalities, and it is clear that they don’t have nice personalities or respect for others that don’t think as they do… shame on them.

I think this will be the end of what was known as “the peer-review process” as we know it. I like the idea of putting things up for all to see and comment on. What we will need are websites that will compile and manage all that information… a wiki type of management of science… Wikijournals maybe?

Sorry, Willis, I was in a dead hurry because I was due to do a podcast interview, didn’t mean to be obscure. And thanks for recalling Wegman’s conclusions, because I think that’s exactly the process through which CRU got itself into this mess.

Dear Mr. Willis Eschenbach,
If a fiction writer made up this story, no one would believe him. Truth is stranger than fiction. I admire your patience and persistence.
What someone should now do is file an FOIA request for all communications (e-mails sent and received, letters sent and received, internal memos, etc.) regarding the avoidance of providing information requested in a actual or anticipated FOIA request. Specific examples should be given as to the dates, subject line, addressee, sender, that were in the “hacked file”, and ask for these and any others like them. The request should be for both those items which still exist as well as any items which were deleted. Any communications that involve someone outside of UEA as a sender or receiver could then begin another FOIA request to another organization. The replies (if any) could be interesting.

Andrew (17:44:58), thanks for pointing this out. You say:

This is interesting:
From the progrm file FOI2009/FOIA/documents/harris-treebriffa_sep98_e.pro
…
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
; Now normalise w.r.t. 1881-1960
;
mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x)
densall=densall+yearlyadj

That is more than interesting. It is a perfect example of why the code needs to be released.
The “oops” is just an error message if the number of years in their value adjusting data doesn’t match the number of years in the record. That’s not an issue, I do the same. It’s just an error test, that’s good programming.
But look at what they are actually doing. They are fudging the numbers by an arbitrary amount to get the result they want. You probably understand this code, but let me see if I can explain it in English for those who don’t.
First, they put together a list of numbers that will be used to calculate the yearly adjustment. The numbers start at zero, get a bit smaller, and then gradually increase. This list is called “valadjust”.
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,2.6,2.6,2.6]
At the end of their calculations, they use valadj to make a yearly adjustment string, yearadj, by interpolating (taking intermediate values) from the values in valadj. This gives a value for each year, by which each year’s data will be adjusted up or down.
densall=densall+yearlyadj
But of course, they’re fudging things, so it won’t come out right the first time. To control the process, they put in a “fudge factor”, a single number that they can use to change the size of all the data adjustments. This is the “fudge factor” referred to in the code, which at the moment is set to 0.75. This is the 0.75 at the end of the line for setting up the valadj values:
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
They are using arbitrary values plus a fudge factor to make the result just what they want … YIKES! Or as Hal said in 2001, I DON’T THINK I CAN DO THAT, DAVE!
Upon such solid well tested “simple physics” calculations is the lofty edifice built … this is definitely why we need access to the code. This is far worse than I had feared, and I feared that it would be bad. Somebody’s got some serious ‘splaining to do.
w.

Reading this email trail gave me a distinct sense of deja vu. In another life I had been party to meetings concerning the deregulation of the electricity industry. The subject was rife with political agendas that played out in these meetings while some of us old goats around the table were concentrating on keeping the lights on (we failed in August 2003 for much of north east America). As the discussions wallowed in the political mire I used to play with my car key fob that still has this quip imprinted upon it: “I’m trying to see things from your point of view, but I can’t get my head that far up my ass.” Good manners and a lack of cojones prevented me from reading it out loud at the appropriate moment. However, I feel the folks at CRU and elsewhere who are playing fast and loose with science for political gain should take note because I think I see their August 2003 on the horizon.

Ken Roberts (18:00:02) :
“Warming increases the risk of civil war in Africa”
They need an excuse?

re: Letter from Ben Santer to [numerous] (not dated above, but immediately before letter to Tom Quigley of Dec 12, 2007)
The letter ends:
I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be
subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.
With best personal wishes,
Ben
***********
In my experience one would expect a letter like that to be a defense exhibit. I’ve even seen people write things like that for that express purpose.
Mr. Santer seems like a savvy guy.

Willis:
This is an extraordinary tale – elegantly retold. As someone above mentioned you have provided the context necessary to pull together one of the primary dimensions of this entire drama: The FOI theme.
A number of things strike me. First, this story is clearly worth some major investigative reporting. It is a shame that so many players appear to be in the bag. It looks like Fox may be the only outlet that would pursue this story and apart from Stoessel, Reilly, Beck et al have the attention span of gnats. Will Andy Revkin want to do the right thing? Will he ask you to prepare a piece for dot.earth?
Second, one theme that emerges clearly in the FOI issue, is that this group has been infected with a belief that you and Steve and others are part of a conspiracy to disrupt the progress of climate science. They simply do not believe that you, Steve and others are a bunch of essentially puzzle freaks who have discovered an area where they can apply their considerable data analysis skills and who have strong feelings about the integrity of science. They appear to have taken whatever happened in tobacco research issues and defined you in the role of the tobacco companies. This belief is so strong that it becomes self-fulfilling. It is pretty close to paranoia – though I think the holes in their research makes the defensiveness somewhat self-serving.
The third point is that they grossly over-estimate their own cleverness and capabilities and even more obviously under-estimate the smarts and capabilities of those requesting information and analyzing the research papers.
Again it is a tremendous piece – a tragedy in the making?

And if anyone’s wondering what the “decline” looks like, here’s your answer:
http://akwag.blogspot.com/2009/11/hiding-decline-what-that-really-looks.html

RE, my

yonason (19:35:55) :
The problem with the CRU is that they believe in making it up first, then “Verification and Validation through Replublication” (the lie oft repeated, and all that)

Which is why they had to control the peer review process.
So,
(1) – controlling the data was necessary to provide support for their hypothesis.
(2) – controlling peer review was to shield their lies from exposure.
Maybe Ellis or someone else, based on the email and other revelations, could come up with a flow chart showing what they actually did, as opposed to what they should have done? It might go a long way to convincing those who can’t understand how they engineered the “consensus,” by showing how they did it.

I also like the statement “In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science.” Didn’t Joe McCarthy expose communists in the u.s.!
HHMMM. I wonder?

This is simply jaw droping stuff Willis. I never actually thought that there was anything conspiratorial going on with the AGW cabal. I thought that the only glue that guided their actions and direction was a commonality of agenda. Now it is only too clear that efforts to fool the public, avoid scientific review, and avoid their legal obligations were in fact directly and explicitly coordinated between them. And despite their protestations, there was far more to their work than just some scientist trying to do their science. You can tell from their references that they were looking forward to having their work converted to public political policy, and that “hide the decline” was most likely the kind of work that they did in many other areas.

“Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists then attack the claim by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand.”
Exactly. And so, rather than give a couple of ‘buddies’ his work for peer review, Willis decided to show his work to the entire world (Steel Greenhouse). Everybody took a swing at him, but he remains standing. That is how science is supposed to work. Great job Willis.

WAG (20:02:38) :
A carefully laid trap, and perfectly baited. Obtain the digital data, doctor it, replace. Lather, rinse, repeat. Anyone now attempting to examine the ‘decline’ will find that the present data supports the cause in a rather weakened state, but warmer that reality nonetheless.
This was done in at least 2 distinct steps.
Witness the twice moved 1900-1999 US temps graph.
HARRY_READ_ME.txt indicates he was already working on version #1 of doctored US data. Data for stations beyond the range of present day “in place” data. Present day data with decades missing, in places necessary to support the agenda hypothesis.
Cooked books.
See, this is exactly the type of things Steven McIntyre was asking for. Where’s the data?
Where are the 2 previous versions of the data that made 1900-1999 US Temps graphs?
Ah, but there’s a problem. That would mean outing cooked books and doctored data.
Busted by HARRY_READ_ME.txt.

WAG (20:01:50) : “time to move on”
Do you mean as in the EPA helping to reduce the temperature by 16 thousandths of a degree C by 2100?
Let’s just sit tight for about 10 years while pouring a few billion dollars into the skeptical bag of tricks and then compare notes.

yonason (20:15:18) :
Sallie Baliunas is a wonderful person. She is careful with her science. She has not deserved the persecution she has gotten from the AGW wolves.

WAG, that’s a ludicrous cherry pick. Start in any year other than 2000. Or show us the actual timespan Jones was faking.

WAG:
“Case in point: it is clear that “hide the decline” has nothing to do with doctoring evidence, as Willis admits here. Yet the skeptic community continues to trumpet it around as if it mattered.”
You seem to be misinformed on this WAG. “Hide the decline” has everything to do with doctoring the evidence. It is not an issue of plotting surface data and proxy data on the same chart. Mike’s trick was to take surface data and add it to proxy data and to call the entire thing proxy data. Then he claimed that the proxy data agreed with the surface data. This is clearly a fraudulent thing to do. Your site tries to make some kind of explanation about there not being a decline using surface data. This shows that you have completely missed the point. No one has tried to indicate that there was a decline in the surface data. There was a decline in the proxy data, and that decline was doctored to hide it. Your other explanation about the divergence being a known issue is also off base. The divergence of the proxy data is something that has only recently been admitted to by the AGW cabal. The claim that it is only a later 20th century phemomena is completely without merit. No one has given an explanation for why such a divergence should only happen in the later part of the twentieth century. Such divergences could well have happened all through history – basically making dendro data useless. For example, how do we know that the same divergence didn’t happen during the medival warming period?
“McIntyre found an error, the error was corrected, and the fundamental shape of the graph was no different.”
No, you are wrong again. The shape of the graph was different. There was still a kind of hockey stick, but the severity of the blade was diminished and the MWP and LIA were slightly larger. Only someone with an agenda would say that the correction made no difference. But the story didn’t end with Mann’s errors. McIntyre also found that Mann’s primary data source was cherry picked. Mann got his blade from Graybill’s data, and Graybill got his blade by cherry picking strip bark bristlecone pines. When new and more extensive bristlecone samples from the same areas were taken for Lenah Ababneh’s doctoral dissertation, as well as samples taken in Colorado my McIntyre himself, there was no blade in modern times. In other words, the dendro data was unable to reflect the warming that we are suppose to have had in the later part of the 20th century. For a long time the AGW cabal avoided using contemporary dendro data in order to cover up this split.
So you have it backwards. The twisting and interpretation was not done by the skeptics, but by the supposed peer reviewed climate scientists themselves. The solution is to get all the data out there to the public. Let the light of day decide who is twisting and who is telling the truth. Obviously the climate cabal is afraid of the light of day. If someone like McIntyre “twists” their data, they can always refute him and show him for a fool. But their real fear is that they will themselves be shown to be frauds.

Glenn (21:49:35) :
http://www.telegraph.co.uk/earth/environment/globalwarming/6637006/Climate-change-scientists-face-calls-for-public-inquiry-over-data-manipulation-claims.html
“Dave Britton, a spokesman for the Met Office which works with the UEA on climate monitoring, said: “We are utterly confident that there was no collusion or manipulation. All the data used was peer reviewed and we are certain it is fully reliable.”
That’s curious. Data is peer-reviewed? And how have they accomplished that, if all the excuses they have given for why data isn’t provided under FOI are true?
Great point. Britton should be provided with a small list of the excuses, including the one that says they don’t even have a list of the sites currently being used, and that even if they had it, it would not be useful because they “would not be able to link the sites with sources of data.” Wow.

“Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!”
Gloating about hiding their funding from Parliament because the requesting Tory MP didn’t know that they are listed under UEA and not CRU… How quaint!
I’m sure Parliament will be happy to know that!

… an explanation for “some of” the divergence.

Gene Nemetz (21:46:58) :
yonason (20:15:18) :
Sallie Baliunas is a wonderful person. She is careful with her science. She has not deserved the persecution she has gotten from the AGW wolves.

I know. That’s why I posted the video of her talking about the old witch-hints. They are so reminiscent of the modern ones, only now they call us “climate criminals” instead of “witches.” Same small minded wolves as in the past, same craving of power and control based on the “religion” of “scientific consensus” they cooked up themselves.

Well, the internet ate another post. The CO2 and NOx do not explain the divergence, they explain some of the rise before the drop.
Time for bed.

Now this is how to put together a meaningful story. Unimpeachable documentation, easy to follow, rich yet economical. Massive thumbs up.
And as Henry Kissinger would say, with the added advantage of being true.
WILLIS ESCHENBACH FOR UEA PRESIDENT….NOW!!!!!!

As usual, the “Nature trick” and “hiding the decline” is explained best at ClimateAudit, in a clear post by a statistician named “Jean S.”
w.

Very nice description.
Clearly the following admission kills CRU’s whole argument
I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.
However it seems to me the HARRY_READ_ME.txt is also critical. Look at the first line :
READ ME for Harry’s work on the CRU TS2.1/3.0 datasets, 2006-2009!
Now other comments in the file make it clear that “HARRY” was trying to recreate CRU’s published results FOR THREE YEARS AND FAILED. Do you see the significance of this?
It is absolutely fatal for CRU and anything they have produced.
What we have here is a documented THREE year effort by a CRU programmer, who has access to all the data, access to all the code, access to all the people who developed the code he could still NOT duplicate CRU’s OWN results. If he can’t it simply means the CRU’s methods can not be duplicated by ANYONE and so anything produced by CRU can not be reproduced.
http://www.devilskitchen.me.uk/2009/11/data-horribilis-harryreadmetxt-file.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheDevilsKitchen+%28The+Devil%27s+Kitchen%29&utm_content=Netvibes

This is the best WUWT post, indeed the best blog post of any kind, that I have ever read. Excellent expose!

Thank you Willis. You tie it all together. I am sure this will help people who would not otherwise understand the context of the emails.
@ ” chasrmartin (16:47:13) :
Okay, Steve, you may have found the first shill email.”
There was a “Phil the Shill” character played by Phil Collins
(of Genisis) in an episode of Miami Vice in the 1980s. A very dodgy character.
I don’t know if the word “shill” existed before then.

THNX Willis (and several others with illuminating comments).
This is the first true “smoking gun” proving intent to deceive and withhold information under FOIA.
SPREAD THE NEWS FAR AND WIDE, FOLKS!
Willis is indeed correct to note that transparency (BLOCKED) and the need to permit repeatability is the key to sound science. Your demonstrative exposition, dear Willis, is the deepest and most important to appear so far, IMHO. Words fail me: HATS OFF!!

Whilst this is indeed a ‘hunt the fox’ jaunt of unparalleled complexity and dexterity in terms of evading tactics, delays, retrenchment and stand-offs, one key question does come into my mind.
‘How does a scientist in this day and age not simply hand over the fruits of 25 years hard labour to competitors who will jump on the bandwagon and trump them with the grant awarding bodies?’
I am minded to think of Bill Gates’ solution to programmes to develop HIV vaccines using his Foundation money. There, he saw that with a winner-takes-all attitude, all approaches would be reluctant to drop their trial hence trial data was likely to be distorted. Gates stated that all selected participants would benefit equally as all had as a goal the worthy aim of developing an HIV vaccine.
It does seem to me that primary credit via senior authorship on ‘seminal’ papers is likely to be a key block to progress here. Ditto membership of key scientific advisory boards.
What further appears clear is that, in all likelihood, CRU were early movers in this field, at a time when the research was relatively low-key and not intrinsic to enormous economic decisions of politicians. Under such circumstances, retaining data and methods in house is OK, if not optimal.
Once, however, CRU has become a global reference centre for data and predictions which lead to Kyoto and Copenhagen, the individual careers of the scientists, whilst important, are irrelevant in the wake of the mind-numbing changes being proposed globally. The sole thing which is important to justify spending trillions of dollars over the next 30 years is that the science is right. And if Jones et al can’t see that, I think they need to stand down.
Where people are coming from is a worry that the whole global warming story is an artefact of how the raw data is manipulated and how the overall constellation of raw data inputs evolves over time. I’m not saying it is or it isn’t, but that key objection MUST be overcome if this argument is to be resolved.
Releasing this data is not like releasing how to make a hydrogen bomb. Now that WOULD be amenable to FOI blocks! But given the studies carried out by McIntyre on previous statistical anomalies and given the examination of the maintenance of weather stations by Watts, it would appear that justifiable worry exists that the rigour of preparing, selecting and maintaining data reference points may (but may not) be insufficiently good for the conclusions drawn to be reliable or justifiable. Those are the reasons for FOI requests and they are justifiable ones.
Politicians must, however, realise that scientists are rather like bulls locking horns to win the cow’s mating rights. Laissez faire isn’t the way ahead here. There needs to be firm leadership and executive direction on this one and it must not only be free of factional interests, it must be SEEN to be free of them too. Only then, will all protagonists be prepared to accept what comes next.
Because what comes next is an impartial arbiter CO-ORDINATING the examination of the datasets and the data manipulations by multiple groups of independent groups. A bit like a public servant holding a public enquiry and soliciting evidence before weighing it up and coming to conclusions.
I do think, very strongly, that unless this takes place then climate science will continue to suffer from this wrangling in public, that the science, reliable or not, will continue to lose favour with the non-specialist public and that, in the case that the warming scientists are right, that the correct actions will not be taken.
I repeat, the egos of CRU scientists are irrelevant in this matter. This is a matter of the highest international stakes on the planet and the decisions which are to be taken must be taken knowing that the scientific community has examined the assumptions made in reaching conclusions and declared them to be sound.
Not to get their next grants or to curry favour with journal editors.
But because on the basis of the best scientific judgement of the day, that conclusion is the right one to be reached.
Now if Copenhagen were to initiate THAT sort of programme, then the carbon footprint left by all the scientists, journos and politicos in their jet planes etc etc etc will have been worth it.
I wonder what odds I would get at the bookmakers of THAT happening, eh?

Whoops,
I meant to say that you can’t compute a statistic from intensive variables that are physically meaningful.

Just so. These people may be the “Black Sox” of science and all I can get across to several people so far is something like peeking at the next card in solitaire.

Today’s Wall Street Journal runs with the story.
An excerpt:
Yet all of these nonresponses manage to underscore what may be the most revealing truth: That these scientists feel the public doesn’t have a right to know the basis for their climate-change predictions, even as their governments prepare staggeringly expensive legislation in response to them.
http://online.wsj.com/article/SB10001424052748704888404574547730924988354.html?mod=WSJ_hp_mostpop_read
As the kids say — “Shweeet”…

A great read, absolutely compelling evidence and the kind of forensic work the press *should* be doing, but are not. Which is scary in itself.
I’m bigging this article up all over the palce, hope you don’t mind…

Here’s the best way to deal with those computer models which haven’t been working out….

Just posted this at realclimate
“Hi Gavin,
Seeing as lots of people complain about the emails being “taken out of context” I would like to point you guys to one of the players in the email drama and his requests for information under FOI Act. His name is Willis Eschenbach. He lays out his emails and requests for information from CRU and their replies in light of some of the leaked emails. Very illuminating about the scientific method.
Warning: this is not flattering for CRU at all.
http://omniclimate.wordpress.com/2009/11/24/willis-vs-the-cru-a-history-of-foi-evasion/
I hope this post gets through as it adds something to the debate and there is nothing offensive here.”

Tilo Reber,
Thanks for your post about Mann’s hockey stick. I had assumed that the hockey stick was debunked by M&M and Wegman, so I was quite suprised, to say the least, when WAG said it was still the same after the corrections. Now I understand that there were 2 points. 1) After the correction to the algorithms there was a somewhat hockey stick shape, but this was due to 2) cherry picked data. After a wider dataset was used, the hockey stick disappeared altogether.
This is what I love about this site – you always learn something new. It shows how desperate the warmists are to try and resurrect it. The question is why are they doing this?

Don’t count the Information Commissioner out just yet. I doubt he was appraised of the full facts of the situation when his advice was sought and he might actually be pretty pissed off he has been used as a smoke-screen.

My post to the BBC RIchard Black Bog is below ( they refused to allow it to be posted – it took almost a week for them to reply but I’ll give them credit they actually replied ):
Richard,
The internet is on fire with news of corruption at CRU. Hackers have found evidence of collusion by Phil Jones and other scientists to;
1) manipulate data to “hide decline”
2) control/subvert the scientific journal peer review process
3) cover up illicit activities through deleting of emails correspondence in order to evade Freedom of Information Act
At the very least one must expect the resignation of Phil Jones but surely questions must be asked of Government Ministers as to how Government Employees are involved in such a massive fraud.
Why is the BBC not covering this? Telegraph, Guardian and many others have written about the fraudulent science and unethical collusion by top “man-made” global warming scientists?
By chance, have you received instructions from the UK Government that the BBC are not allowed to cover this breaking story? (I see that all posts mentioning the content in the hacked emails have been deleted – in an effort to protect the people at CRU – in short it looks like a massive cover up is underway.)
Is the UK Government (which planned to spend 6 Million pounds on advertising that CO2 is a dangerous monster that will drown children’s pets) complicit in this whole endeavor to fabricate propaganda and control “accepted” science to the effect that CO2 is causing dire global warming (something which is patently false)?

Thanks Willis for this. Did a full read of this last night. I’ve been waiting for segments like this that put a chronological context to the story.
I’m struck by the behavior of the Team that most parents will recognize. They spend an immense amount of time, effort and emotion avoiding doing what they’re supposed to do when compliance would be much simpler and quicker. Really childish stuff.

GISS are HadCRUT use the same raw data and different methods to clean up the raw data. I am not an expert on those methods used, but they aren’t identical. So in the sense that the GISS team and the HadCRUT team made use of independently derived methods to deal with some of the uncertainties and problems with the raw station data, they are indeed independent. However, they use the same raw data (there is only the raw data obtained from each nation in the WMO…it’s not like these countries keep different sets of data to release to different groups), and because James Hansen and the HadCRUT team suffer from samethink, there’s a good probability their methods are at least SIMILAR.

Related to just adjusting the record.
I found a chart of the new HadSST3 sea surface temperature record (which is still in draft form).
This is the one that Tom Wigley proposed to Phil Jones just be adjusted by 0.15C (to hide the decline or hide the increase which ever they decided to do).
Well, it looks like that was already in the works.
In the top chart here – the Green Line is the new draft HadSST3 while the Red Line is the HadSST2 (which is currently used in the record).
http://img524.imageshack.us/img524/1205/hadsst3.png
This is outlined in this draft paper from the WMO.
https://abstracts.congrex.com/scripts/jmevent/abstracts/FCXNL-09A02a-1662927-1-Rayneretal_OceanObs09_draft4.pdf
So, we have no idea what the warm years were or the cold years were. We just have some charts produced by GISS, NCDC, CRU and the Met Office.

My post got through at
http://www.realclimate.org/?comments_popup=2019#comment-144607
I wonder what they will now say about these emails being taken out (assuming they are all genuine and untampered with).

Here is the UEA Academic Code of Conduct
“Beyond honest errors and errors caused through negligence are a third category of errors: those that involve deception. Making up data or results (fabrication), changing or misreporting data or results (falsification), and using the ideas or words of another person without giving appropriate credit (plagiarism)—all strike at the heart of the values on which science is based.
“These acts of scientific misconduct not only undermine progress but the entire set of values on which the scientific enterprise rests. Anyone who engages in any of these practices is putting his or her scientific career at risk…”

going through that harry_read_me file it shows that errors in climatology, like in radiative physics are the maximum that error will allow. Like in radiative physics, they make up the equations as they go along, without any regard for actual measurement and this has been going on for over 100 years. It is this historical antecedent that allows them to continue it with impunity, probably unaware of how derived numbers relate to the physical process, or how temperatures, both absorbed and emitted in fact, relate to molecular composition, density and absorbtion wavelengths than a constant.. This methodology, permits some rather arbitrary extrapolations that can be applied to derive earth temperatures from the past as is seen desirable. If so much theoretical physics is in the wrong, and left unchecked, then likewise, so much climatology will be in the wrong for the same reasons

Vincent:
“After a wider dataset was used, the hockey stick disappeared altogether.”
Actually, Vincent, it’s a three part drama. After the Bristlcone pine cherry picking and divergence problem was resolved Mann’s original hockey stick was destroyed. But the assertion that Mann wanted to make with it was still alive. The dendro alarmists claimed that nothing had changed because the hockey stick could still be produced even when you took out the Graybill bristlecone data. They were right. But what basically happened was that the blade that was provided by that series was replaced by the blade that was created by the Briffa Yamal series. This is why McIntyre struggled so long to see why Briffa was able to get results that didn’t seem to be present in other series.
When Briffa was finally forced to release his data several things became clear. First, the sample set that Briffa used was pitifully inadequate for making statements about trends. Everything depended on a handful of trees. Second, the Russians that took the Yamal samples did not find a hockey stick. Third – and Briffa admitted this himself – the series that were used for reconstruction were based upon their agreement with the surface temperature record. So while individual trees were not cherry picked, series were cherry picked. The fourth problem had to do with the growth characteristics of the Larches that were used in the Yamal studies. Their growth tends to increase as they get older. Due to the way that the trees are selected and cored, this means that the rings representing the later part of the twenthieth century all came from old tree rings, while the rings that came from the earlier history were from a mixture of young and old tree rings. And last, there were other, more complete, more comprehensive series available in the same area that actually showed cooling.
When one looked at the various hockey sticks that were created by the hockey team, virtually all of them used either the Graybill series, or the Yamal series, or both. So now, not only is Mann’s hockey stick broken, but virtually all of the hockey sticks that were created by various memebers of the hockey team are broken – particularly those that make up the IPCC hockey stick spagetti graph. Proxies other than tree rings tell a very mixed story, depending on the particular proxy and the type of proxy. What is clear, however, is that the statement that the hockey team seeks to make – that the second half of the twentieth century is unique in the climate record of the past 2000 years – is not even close to accurate.

Just picked up this from an article by Dr. Roy Spencer on IcecapUS
“If all of this sounds incompatible with the process of scientific investigation, it shouldn�t. One of the biggest misconceptions the public has about science is that research is a straightforward process of making measurements, and then seeing whether the data support hypothesis A or B. The truth is that the interpretation of data is seldom that simple.
There are all kinds of subjective decisions that must be made along the way, and the scientist must remain vigilant that he or she is not making those decisions based upon preconceived notions. Data are almost always dirty, with errors of various kinds. Which data will be ignored? Which data will be emphasized? How will the data be processed to tease out the signal we think we see?”
Well, count me amongst those naiively harbouring such misconceptions, and I would probably include most MPs and reporters in that, particularly where such a basic function (from a layman’s point of view) as taking temperatures is concerned.
How can that data be dirty and be subject to various kinds of error? Why on earth would would you ignore some data and emphasise some other? There can only be one way to process data and that is to process it straightforwardly to produce a result and definitely not in such a way as to tease out a signal that we think we see.
The reading is what it is. Modifying and massaging it for whatever reason is dishonest, especially so when that teases out such a footling anomaly as 0.7C in a century, for which we are expected to stand civilisation on it’s head.

Excellent review, and it hit a nerve on something I have been saying for quite some time – when it comes to AGW it is time for the PhDs to grow up and deal in the real world.
http://strata-sphere.com/blog/index.php/archives/11509
BTW, I am not trying to disparage all PhDs – most our very fine, outstanding and admirable people. It is this type you find huddled in the alarmist camps that really give all us propeller heads and geeks a bad rap.
Cheers, AJStrata

I just found this in a post at the Daily Mail and it made me laugh so I thought I’d pass it on.
“I think we can safely rename the University of East Anglia, to that of the University that’s Easily Angled.”
Brilliant!

Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like
Dr. Judith Curry, GA. Tech Climatologist, is doing the same thing at CA. It’s a pitiful attempt, but there you are. She’s actually very easily outed and rebutted as to this tactic, and its irrelevance to the real issue. The question is whether she’ll actually do anything to restore real Science. I’m not holding my breath.

I wonder if, as a government organisation, the MET office is required to use data produced to an International Standard (although it seems the UN isn’t too interested in standards either given their use of Wikipedia!)? Obviously, the CRU temperature data relied on by the MET wasn’t produced to any kind of standard whatsoever. FOI request?

I have a somewhat regular critic of my blog who is also a AGW true believer. He only now just got around to commenting on my blog about this. Despite, taking so much time his reply only focused on the personal emails were illegally obtained, Climate Audit and others are pseudo/non-science blogs and Professor Mandia (who commented earlier) was correct.
This would only be hilarious If all this did not affect trillions of dollars of economic activity and the way of life of millions in several countries.

Just for completeness, John in NZ:
shill.

J. Peden (09:16:15) :

Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like

Dr. Judith Curry, GA. Tech Climatologist, is doing the same thing at CA. It’s a pitiful attempt, but there you are. She’s actually very easily outed and rebutted as to this tactic, and its irrelevance to the real issue. The question is whether she’ll actually do anything to restore real Science. I’m not holding my breath.

J. Peden, what you say is not true at all, and is an attack on on an honest scientist. Dr. Curry is one of the few establishment climate scientists to defend her work at ClimateAudit. She is open and sharing about her work and her data. She openly states her reservations about others work. In my interactions with her, she has been consistently pleasant and open about her beliefs and concerns about the state of climate science.
More to the point, however, I posted on the Climate Audit thread you reference, inviting her to read my account. She did, and posted the following upstream:

Judith Curry (03:46:31) :
Willis, thank you very much for this lucid and comprehensive summary. While I’ve avoided censuring individual scientists and others over this whole episode until the context is better understood, you have now provided such a context.
I’ve also noticed that people are questioning your motives etc (i think i saw this over at Pielke Jr’s site. I would like to make this public statement. In all the time that I have spent over at climateaudit, i have found Willis Eschenbach to fair, quite knowledgeable about statistics and data analysis, and unfailingly polite and respectful to others.

So, quite contrary to your claim, she is one of the few true scientists in the field, who is willing to publicly change her position when new facts emerge. Thank you, Dr. Curry, I can only wish that all climate scientists were as honest and as gracious as you are.
w.

Robert Burns (18:53:18) :
Dear Mr. Willis Eschenbach,
If a fiction writer made up this story, no one would believe him. Truth is stranger than fiction.
It certainly is! As so brilliantly demonstrated by Michael Crighton in his 2005 novel State of Fear 😉
http://www.heartland.org/policybot/results/16260/Michael_Crichton_Is_Right.html

@Willis Eschenbach (10:00:22)
Another respected scientist speaks out:
http://pielkeclimatesci.wordpress.com/2009/11/24/beware-saviors-by-demetris-koutsoyiannis/

But see “Scientists aren’t science – and science isn’t a method”:
http://vulgarmorality.wordpress.com/2009/11/25/scientsts-arent-science-and-science-isnt-a-method/

Well, someone sent me a quote from Professor Trevor Davies, Pro-Vice Chancellor for Research and Knowledge Transfer of the EAU. He said:

The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.

Ooooh, I am ashamed to confess it, but I said bad words and forbidden epithets, it angrified my blood immensely. Now they’ve gone and done it. I expect crooks to be crooks, that’s no surprise, I’m not angry at Jones et. al. As long as there is a system, people will try to game it.
But I also expect the people in charge to do something when they notice the crooks, not just spout platitudes.
So I’ve just sent the following email:

Dear Professor Davies:
I see that you are quoted regarding the hacked emails as saying

“The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.”

If you believe this, I encourage you to read my account of my experience with my FOIA requests to CRU below.
I would greatly appreciate a response, particularly if (as I hope) you are actually taking action to insure that the FOI requests are handled properly in future.
Many thanks,
w.

I attached a copy of my story of my battle with the CRU, which is posted at
<http://wattsupwiththat.com/2009/11/24/the-people-vs-the-cru-freedom-of-information-my-okole…/
I sent copies to the following individuals:
To: ,
Information Commissioner’s Office (responsible for FOI)
“Palmer Dave Mr (LIB)” FOI Officer, UEA
“Inglis Kitty Ms (LIB)” FOI Appeals, UEA
Justice Department, UK (responsible for the Information Commissioner)
CC: Editor, Times of London
Editor, New York Times
Editor, Daily Mail (UK)
Editor, Chicago Tribune
Editor, LA Times
Mark Morano, Climate Depot
Plus individual copies to the following journalists:
George Stephanopoulos, ABC
Nightline, ABC
Ted Koppel, ABC
Jennifer Loven, AP
Kathleen Carroll (Executive Editor), AP
Larry Margasak, AP
Michael Silverman (Managing Editor), AP
Michelle DeArmound, AP
Nedra Pickler, AP
Ron Fournier, AP
Sandra Johnson (Washington Bureau Chief), AP
Seth Sutel, AP
Terry Hunt, AP
48 Hours, CBS
60 Minutes, CBS
60 Minutes II, CBS
Bill Plante, CBS
CBS Evening News with Dan Rather, CBS
Dick Meyer (Editorial Director), CBS
Early Show, CBS
Erin Moriarty, CBS
Face The Nation, CBS
Mark Knoller, CBS
Peter Maer, CBS
Sunday Morning, CBS
American Morning, CNN
Anderson Cooper 360, CNN
Bill Schneider, CNN
Candy Crowley, CNN
Howard Kurtz, CNN
In The Money, CNN
Jeanne Meserve, CNN
Jim Walton (President of CNN News Group), CNN
Judy Woodruff, CNN
Kyra Phillips, CNN
Lou Dobbs, CNN
Lou Dobbs, CNN
Rick Davis (Executive Vice President – CNN News Standards and Practices), CNN
At Large with Geraldo Rivera, Fox
Brian Wilson, Fox
Bulls & Bears, Fox
Cashin’In, Fox
Cavuto on Business, Fox
Forbes on FOX, Fox
FOX & Friends, Fox
FOX News Live, Fox
FOX News Sunday, Fox
FOX News Watch, Fox
FOX Report with Shepard Smith, Fox
Hannity & Colmes (Alan Colmes), Fox
Hannity & Colmes (Sean Hannity), Fox
Heartland w/ John Kasich, Fox
James Rosen, Fox
Jim Angle, Fox
Major Garrett, Fox
Molly Henneberg, Fox
On the Record with Greta, Fox
On the Record with Greta, Fox
Special Report with Brit Hume, Fox
Studio B with Shepard Smith, Fox
The Beltway Boys, Fox
The O’Reilly Factor, Fox
Viewer Services, Fox
War Stories, Fox
Wendell Goler, Fox
Your World with Neil Cavuto, Fox
Alison Stewart, MSNBC
Amy Robach, MSNBC
Bob Kur, MSNBC
Chris Jansing, MSNBC
Chris Matthews, MSNBC
Contessa Brewer, MSNBC
Countdown with Keith Olbermann, MSNBC
Dan Abrams, MSNBC
David Schuster, MSNBC
Deborah Norville, MSNBC
Deborah Norville, MSNBC
Dennis Sullivan (Executive Editor – Campaign Coverage), MSNBC
Don Imus, MSNBC
Don Imus, MSNBC
Hardball with Chris Matthews, MSNBC
Imus in the Morning, MSNBC
Joe Scarborough, MSNBC
Joe Trippi, MSNBC
Keith Olbermann, MSNBC
Lester Holt, MSNBC
Lester Holt Live, MSNBC
MSNBC Investigates, MSNBC
Randy Meier, MSNBC
Richard Kaplan (President), MSNBC
Ron Reagan, MSNBC
Scarborough Country, MSNBC
Dateline, NBC
Meet The Press (Tim Russert), NBC
Nightly News with Tom Brokaw, NBC
Today Show, NBC
and 75 other overseas journalists.
Professor Palmer has angrified the blood of the wrong man …

Smokey, the problem with those lists are they are links to web-based forms. They don’t let you attach a file, and I’d have to do them one by one.
What I need are email addresses … sheesh, you’d think they were trying to keep us from getting in touch with them … oh, wait …

Excellent, truly excellent.
It seems to me this story is taking a twist besides climate. And that is the conduct of FOI organization. First, there are many many journalist who are sympathetic to this angle and have had their own severe frustrations with FOI and obtaining information that should rightly be public. Their instinctive reaction is that all information should be public (so they can get a story out of it). So, I think the FOI angle has legs. Official complaints can be made, calls for investigation, noting the similarities between this struggle and some similar problems that journalists have had.
PS Dont forget the National Post in Canada. Its business section, The Financial Post, is a long time climate sceptic.

Charles, my bad, delete the first one if you’d be so kind.

Tim Russert is dead, Dobbins left CNN, Colmes left Hannity, Dan Rather retired, Tom Brokaw retired.
Hopefully enough live bodies are caught in the shotgun blast that this gets through and on the air somewhere.
Frankly, I don’t watch tv, haven’t for years now. A person who actually cares about any issue will keep up with it on the internet.

Despite what he says he has done, if the scientist has inadvertently used an improper procedure (e.g. the uncentered principal components analysis used in Mann’s Hockeystick)

Inadvertently?? You’re being much more magnanimous than I could have been.