U-CRU

From Kate at Small Dead Animals: No U-turns allowed

Flashback to April 18th…

Dear Tom,

I find it hard to believe that the British Antarctic Survey would permit the deletion of relevant files for two recent publications or that there aren’t any backups for the deleted data on institutional servers. Would you mind inquiring for me? In the mean time, would you please send me the PP format files that you refer to here for the monthly sea ice data for the 20th century models discussed in your GRL article and the 21st century models referred to in your JGR article.

Regards, Steve McIntyre

Then in July… “Unprecedented” Data Purge At CRU

On Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory ftp.cru.uea.ac.uk/. The next day, on July 28, Phil Jones deleted data from his public file – see screenshot with timestemp in post here, leaving online a variety of files from the 1990s as shown in the following screenshot taken on July 28, 2009.

The Telegraph, todayClimategate: University of East Anglia U-turn in climate change row ….. Leading British scientists at the University of East Anglia, who were accused of manipulating climate change data – dubbed Climategate – have agreed to publish their figures in full….

Now, here comes the other shoe! Hide the Decline!

SCIENTISTS at the University of East Anglia (UEA) have admitted throwing away much of the raw temperature data on which their predictions of global warming are based.

It means that other academics are not able to check basic calculations said to show a long-term rise in temperature over the past 150 years.

The UEA’s Climatic Research Unit (CRU) was forced to reveal the loss following requests for the data under Freedom of Information legislation.

The data were gathered from weather stations around the world and then adjusted to take account of variables in the way they were collected. The revised figures were kept, but the originals — stored on paper and magnetic tape — were dumped to save space when the CRU moved to a new building.”

[…]

In a statement on its website, the CRU said: “We do not hold the original raw data but only the value-added (quality controlled and homogenised) data.”

The CRU is the world’s leading centre for reconstructing past climate and temperatures. Climate change sceptics have long been keen to examine exactly how its data were compiled. That is now impossible.

Advertisements

  Subscribe  
newest oldest most voted
Notify of
Stephen Brown
Lee

The more you uncover Anthony the more it stinks…….
Keep up the very good work you all. You are doing work of International importance and i mean that sincerely….

kuhnkat

TRUST ME!! I know what I am doing and that is why I got rid of the raw data!!!
Now give the UN control of your country so we can fix everything else for you too!!
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

galileonardo

Have no fear. This sounds like a job for Mr. Jones. In his 1255298593.txt email to Rick Piltz he says:
“The original raw data are not lost either. I could reconstruct what we had from some DoE reports we published in the mid-1980s. I would start with the GHCN data. I know that the effort would be a complete wate of time though. I may get around to it some time. As you’ve said, the documentation of what we’ve done is all in the literature.”
I say it’s time we use FOI to make him waste some time.

SABR Matt

And the third shoe is…
The investigation will not find fault in CRU for doing this…and we’ll never know if they threw out data that could prove them wrong.

I wonder how much of this really relates to the AGW proponents. Could this have more to do with Jones?

Michael D Smith

Nobody is that stupid. “Value-added (quality controlled and homogenised)” is not data. It is a result.

Ben M

And they want us to hand over $6 trillion, on the basis that their quality control and homogenisation was perfectly done?
Alchemy, anyone?

Roger Knights

The warm has turned.

Gabriel

If the derived data is no longer reproducible, it is not trustworthy. It is no longer useful for science but must itself be thrown out as there is now no way to test or verify it.

Glenn

“Jones was not in charge of the CRU when the data were thrown away in the 1980s, a time when climate change was seen as a less pressing issue. The lost material was used to build the databases that have been his life’s work, showing how the world has warmed by 0.8C over the past 157 years.”
When in the 80s was global warming “not an issue”? Certainly an issue in 1988 when Hansen testified before Congress. Why keep the “value-added” data in that case.
My guess is that whoever made the decision to throw the paper and tapes away didn’t do so because of a location move but because they thought the value-added data was good enough for government work. Incompetence? CRU has certainly demonstrated incompetence lately, and perhaps since 1995 or before.
Maybe Jones can be persuaded to write some code to extract the raw data from the “value-added”, before the door hits him in the arse while he’s on the way out. He may not be responsible for the purge, but he knew.

bil

This I just can’t believe. A lot of this information would be recorded in journals of national historical import, notwithstanding their scientific value.
This is truly a scandal of epic proportions.
The man must resign. The results of all his work are not worth the paper (oh, hang on, he threw all that away) they were printed on.

Mikey

This idea of “value added data” is a new one for me. What’s the scientific history behind that one? Did Galileo, Newton, Faraday, Rutherford, Eratosthenes, etc. also deal in value added data?

Well, he did say he would delete the files rather than giving them to Steve, Phil is evidently a man of his word…

David Hoyle

The Science is scuttled!!!

Ian Lee

One would have to say that the adherence to accepted scietific practice isn’t much to the fore at The Climatic Research at the University of East Anglia. I am aware that data do get misplaced and mislaid but to trash original raw data which are valuable in their own right and particularly in the current climate (no pun intended) could and should be available just as a check that all is kosher gives me a very uneasy feeling about the whole AGW scenario. GISS seems always to modify data upwards, the way “hide the decline” was perpetrated is hardly best practice and refusing to release data is just arrogant. That Professor Jones would rather destroy than release data is on record and it seems as if his sentiments are comonly held at the University of East Anglia. The whole AGW edifice is so shonky that it beggars belief that so many accept without question the mantras that “the science is settled” and “we must cut carbon pollution”. Surely even the most scientifically illiterate politicians must be starting to have misgivings about the rush to emissions trading schemes. Mustn’t they?

Michael

My question is how do you “homogenise” temperature data?
Cheers
Michael

Ian Lee

Sorry typo. That should read Climatic Research Unit at the University of ast Anglia

Vg

Just moved over to Bing much better news coverage on climategate. Just shortcut save the main page of bing news on your desktop real easy transition LOL. Oh Im sure google will reconsider we hope.

Glenn

http://www.cru.uea.ac.uk/cru/about/history/
Tom Wigley was “in charge” of CRU from 1978 to 1993.
“In 1979, CRU hosted a remarkable, international, interdisciplinary conference (Climate and History), a turning point for the future work on historical climatology and the influence of climate on human societies. This type of work still has an important place in CRU’s research portfolio to the present day, although it has broadened to include the development and analysis of early instrumental records and the extension of important climate indicators and datasets as far back in time as possible.”
“As it became clearer, in the 1980s, that the world was warming, a question that was asked with increasing frequency was how much, if any, of the warming was a consequence of human activity? CRU had made an important contribution to the posing of that question, so was in an excellent position to attract some more research funding to address it. The UK Government became a strong supporter of climate research in the mid-1980s, following a meeting between Prime Minister Mrs Thatcher and a small number of climate researchers, which included Tom Wigley, the CRU director at the time.”
Good stuff. Hope it doesn’t inadvertently get deleted.

KimW

These issues were long suspected by many who frequented Climate audit and this site but now is being publicised for the whole blogosphere. I went to the Canadian Broadcasting Commission site and read Bob McDonalds article. I note that he actually stated, ” …. climate science stands on a firm footing of rigorous research and appropriately skeptical peer review”. Rigorous research seems to consist of denial of data and then its deletion and “Peer review ” looks like becoming a popular punchline for a bad joke.
I am reminded of the story that in 1945 the captured German Atomic scientists were secretly recorded discussing the news of the Hiroshima bomb and their speculations were so bizarre that the Los Alamos personnel listening to the conversations were driven to comment, ” Who are these clowns ? “.

WOW
So how do we take the article statement:
‘Jones was not in charge of the CRU when the data were thrown away in the 1980s’ That is, piles of papter records chucked out when they moved office in the 1980s
And then this press release:
CRU climate data already ‘over 95%’ available (28 November)
Over 95% of the CRU climate data set concerning land surface temperatures has been accessible to climate researchers, sceptics and the public for several years the University of East Anglia has confirmed.
“It is well known within the scientific community and particularly those who are sceptical of climate change that over 95% of the raw station data has been accessible through the Global Historical Climatology Network for several years. We are quite clearly not hiding information which seems to be the speculation on some blogs and by some media commentators,” commented the University’s Pro-Vice-Chancellor, Research Enterprise and Engagement Professor Trevor Davies.
http://www.uea.ac.uk/mac/comm/media/press/2009/nov/homepagenews/CRUupdate
curious and curiouser…

Ross Berteig

Its going to be fun watching them reconcile the many positions they’ve taken over the past few years. Their recent promise to finish getting releases from all the national weather services and to release all the data contradicts the idea that they only hold “value added” data.
Personally, I’m reasonably sure that without access to the actual raw data, as well as access to their complete process of “adding” “value” it is impossible to resuscitate their global temperature product.
The problem is that they weakened their own authority by publicly refusing review. Now, their clear actions with respect to FOI combined with their intimidation of critics, journals, and editors punctures any remaining gravitas they had fatally.
The UEA ought to be looking out for its future reputation. Being the home of a scientific scandal is not good for a university, regardless of how it comes out in the end.

martin brumby

Nothing to hide, of course. And who would want ‘raw’ data when they could have it expertly cooked by Jones, Briffa and the team?
A data Chow Mein, anyone? Just the stuff on which to base major policy decisions.
The special recipe used is secret, however, so you can’t do this at home, folks.

mack520

Not to worry, Gavin has now collected and released all the data and code skeptics should want in a directory on RC. While he makes the point that all the data or code any skeptic should want has always been publicly available, he has now gone to the trouble of compiling this directory. If something you want isn’t there- it will be because you don’t need or should not want it. This is a huge change.

Dave

http://www.timesonline.co.uk/tol/news/environment/article6936328.ece
They say they dumped much of the raw data when they moved due to not having enough data storage capacity back then.
Okay, remember HARRY_READ_ME.txt
http://www.devilskitchen.me.uk/2009/11/data-horribilis-harryreadmetxt-file.html
From the HARRY_READ_ME.txt file:
http://www.anenglishmanscastle.com/HARRY_READ_ME.txt
The first line:
“READ ME for Harry’s work on the CRU TS2.1/3.0 datasets, 2006-2009!”
Then later in the file:
“You can’t imagine what this has cost me – to actually allow the operator to assign false
WMO codes!! But what else is there in such situations? Especially when dealing with a ‘Master’
database of dubious provenance (which, er, they all are and always will be).”
and then from near the end of the file:
“OH FUCK THIS. It’s Sunday evening, I’ve worked all weekend, and just when I thought it was done I’m
hitting yet another problem that’s based on the hopeless state of our databases. There is no uniform
data integrity, it’s just a catalogue of issues that continues to grow as they’re found.”
But CRU’s press release says that v3 has fixed all the issues. Okaaay, then.

Pingo

If you can’t repeat it, it’s not science.
This reminds me of the last scene in the Lord of the Flies. The adults have just turned up late to the party.

Raven

The post is misleading because no raw data was deleted by the BAS. It was simply converted to a format before use in the papers and deleting those files created more work for SteveMc. It may be annoying but it does not go against the broad principal of data retention.
I think it is very important to distinguish between this case and the CRU where it appears the original data was deleted making it impossible to anyone to check the HadCRUT series.

John Trigge

The science of prestidigitation – now you see it, now you don’t.

JustPassing

Roll up, roll up, ladies and gentlement, find the data.
Is it under this cup ……… no
Is it under the other cup …….. no
Lets start again. Place your bets.
(shuffle, shuffle)

Did UEA incinerate the paper data?
…In an old oil drum in the parkinglot??
….how many of Al Gore’s polar bears died because of that???
……I hope old Al is choking down his fraudulant $100M words…
Watts Up With That… Thank you for making data and stories available to average, non-scientific persons.
Any recent news of how Alan Carlin has faired at the EPA, since the SS enforced the Obama Climate Agenda?

Dr A Burns

I’d posted this previously but I’m not sure if I had a response. Does anyone know why the past 8 months of hadcrut3, hadsst2 and crutem3 data have been deleted ?
http://www.cru.uea.ac.uk/cru/data/temperature/hadcrut3gl.txt

Phil Jones WILL lose his job over all this, and the pro-AGW lot will want to do what is now the buzz thing to do – wait for it… ‘Time to move on’. That’s what is said here in the UK now whenever a scandal surfaces. Those involved say, ‘Well, yes we made some mistakes and people have lost their jobs. Now let’s move on’. It’s meant to say that enough has been done to rectify the situation. Of course, in reality no such thing has happened and the problem surfaces again in the future! We’ve had this with social services getting involved in child cruelty cases. We’ve even had judges and social care managers saying, ‘This must never be allowed to happen again’. Of course, it does happen again because not enough is done in the first place to stop it happening again!

Lee

Barry Foster (01:42:17) :
As a fellow Brit i can only agree with you…..what shit state they’ve gotten us into.

Charles. U. Farley

MSM still head in the sand.
Complaints sent to 3 broadcasters and two letters to newspapers regarding A) Bias and B) Non reporting of an unfolding trainwreck.
Do likewise id suggest.

Paul Vaughan

“value-added”
The spin comes with a good laugh.

Donald (Australia)

As Bill (above) said, this is a scandal of epic proportions.
Surely it is time to have these fellows arrested.

Climategate Quiz on Youtube:
The news site that made the original quiz liked Jeffrey LaPorte’s video enough to post it on their website. Check it out here:
http://us.asiancorrespondent.com/gavin-atkins-shadowlands/quiz-now-on-youtube.htm
Jeffrey LaPorte’s website: http://www.JeffreyLaPorte.com
Twitter: http://tinyurl.com/lrxrnk

Stephen Shorland

Nothing about it on radio4 this morning (uk). Just Milliband saying great things about Copenhagen.I find it hard to believe but it wouldn’t surprise me if ‘they’ did do a legally binding deal.There’s so much cash and Government tied up in this thing that it’s still going to take a fight like the Russians in Stalingrad to kill the monster. I like the NIWA scandal best – cranking down 6/7 stations temperatures right from the start of the dataset in each case by AT LEAST 0.5C. That’s a real clear cut frad IMO. The fact that NIWA won’t address it and keep coming back to subsequent station relocations is a sign they know the game’s up,I think.

The combination of “science” and politics is a deadly one, as we have seen in the last century.

BradH

“Don’t you see that the whole aim of Newspeak is to narrow the range of thought?… Has it ever occurred to your, Winston, that by the year 2050, at the very latest, not a single human being will be alive who could understand such a conversation as we are having now?…The whole climate of thought will be different. In fact, there will be no thought, as we understand it now. Orthodoxy means not thinking—not needing to think. Orthodoxy is unconsciousness.”….
“In Oceania at the present day, Science, in the old sense, has almost ceased to exist. In Newspeak there is no word for ‘Science’. The empirical method of thought, on which all the scientific achievements of the past were founded, is opposed to the most fundamental principles of Ingsoc. And even technological progress only happens when its products can in some way be used for the diminution of human liberty.”….
“And if all others accepted the lie which the Party imposed—if all records told the same tale—then the lie passed into history and became truth. ‘Who controls the past’ ran the Party slogan, ‘controls the future: who controls the present controls the past.'”

rbateman

Great damage appears to have been done to both the current records and the GHCN. Where are all the Observers Reports that are handwritten?
How much of them were destroyed?
The GHCN does not jive with the stations mentions in HARRY_READ_ME as how far back in time the stations are supposed to go.
I have found some early station data in the US from the AMS records, which were taken from early observers, Signal Corps, etc. They are in monthly mean or monthy max/monthly min format.
Sacramento, CA from AMS jounal of 01-01-1884
YEAR(S) JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC ANN
1853 —- —- 59.8 61.0 68.0 77.0 75.0 71.0 76.0 78.0 53.0 —- —-
1854 43.0 52.0 53.6 60.0 62.0 67.0 80.6 69.5 65.0 60.0 55.0 48.0 59.5
1855 43.7 52.5 54.8 51.8 60.2 71.1 72.5 73.0 68.0 63.0 50.6 47.9 59.5
1856 48.0 52.6 57.0 58.8 63.9 71.1 75.1 69.6 70.9 58.0 52.2 46.9 60.1
1857 48.5 50.2 56.4 63.3 65.5 71.9 71.4 71.3 67.9 61.5 53.2 43.9 60.7
1858 45.0 52.2 53.7 59.8 65.2 69.4 70.8 70.6 68.9 59.5 54.2 47.7 59.5
1859 44.9 50.5 51.5 57.1 63.0 74.8 69.1 67.2 65.9 63.3 54.0 44.5 58.7
1860 46.2 49.8 53.3 57.8 58.5 65.6 73.2 73.5 67.6 59.8 53.5 43.5 59.0
1861 47.1 52.2 55.0 60.6 63.7 66.2 73.6 69.7 67.8 59.9 53.6 49.3 60.1
1862 46.4 47.5 53.6 58.0 61.2 69.3 73.2 75.0 70.4 67.6 53.1 50.9 60.2
1863 46.9 48.0 57.6 59.5 67.1 69.1 75.6 70.7 69.0 62.8 52.7 46.4 60.3
1864 49.2 53.6 56.1 62.1 68.5 71.1 74.8 74.7 69.8 64.5 53.5 46.5 62.8
1865 47.4 49.0 53.6 59.3 70.2 73.5 74.0 71.7 68.8 63.1 50.9 50.2 61.0
1866 46.5 53.5 54.2 61.9 63.1 72.2 76.2 76.0 72.2 65.2 53.8 44.1 62.1
1867 48.2 47.8 50.7 59.7 64.4 70.3 73.2 71.7 68.8 62.7 54.8 50.2 59.9
1868 47.0 50.5 55.0 60.1 64.2 69.5 73.8 71.2 68.3 62.0 53.0 46.8 60.1
1869 47.6 49.9 53.6 59.0 64.2 70.8 74.4 71.3 69.9 63.1 54.0 47.0 61.4
1870 48.6 51.1 53.0 57.0 61.0 69.3 71.8 72.6 68.0 63.6 53.4 46.4 59.6
1871 48.3 49.4 56.0 59.2 61.5 71.1 70.2 72.0 67.4 62.2 50.2 45.5 59.6
1872 48.5 53.3 56.8 57.6 67.0 69.2 71.4 73.1 68.8 58.9 51.2 48.7 60.4
1873 52.7 48.2 56.8 60.0 67.9 71.7 73.2 66.3 69.9 61.4 57.5 49.0 60.7
1874 45.7 49.3 52.9 59.5 64.7 70.2 72.8 70.9 70.7 61.7 63.5 47.7 59.8
1875 46.9 52.7 53.7 63.0 68.1 70.6 73.3 72.5 55.7 69.9 56.7 45.0 62.5
1876 48.8 50.2 54.6 59.5 65.7 76.9 74.0 72.8 70.1 63.5 53.3 48.0 61.7
1877 49.1 55.0 59.0 60.2 64.5 72.5 75.0 72.9 72.5 62.9 54.7 45.5 61.2
1878 49.7 51.3 56.7 59.4 65.5 71.8 73.4 73.4 69.0 62.9 55.5 48.6 61.3
1879 45.5 55.0 57.4 60.3 60.2 72.1 71.8 74.7 70.5 61.5 50.9 47.2 60.3
1880 43.5 46.0 48.8 54.6 61.6 66.6 70.9 69.7 68.0 62.1 49.7 44.0 57.5
1881 49.2 53.5 55.5 60.8 64.5 66.1 71.1 68.2 67.8 56.8 50.8 50.3 59.2
1882 45.1 46.3 53.0 55.8 64.0 68.1 73.4 71.9 68.4 58.4 49.5 46.2 59.4

So have they actually deleted the original data or simply put the original data in electronic form and then deleted the paper copies?

RexAlan

Well I’ve just canned google. Bling Bling for me!

RexAlan

No more Adwords $ for you boys.

KeithGuy

So the CRU have lost the original temperature data and replaced it with their homoginized, quality controlled, constructed data. Oh dear!
But all is not lost because they can still troll through some DOE documents and reconstruct the original data. Hoorah!
These guys are so clever! I would never have worked out such a brilliant way of getting back to where I started.
Is there such a word as de-homoginized?

rbateman

Correction: The above was the January 1883 Montly Weather Review from AMS.

EW

Michael (00:50:41) :
My question is how do you “homogenise” temperature data?

Homogenization of data in meteorology and climatology is nothing nefarious per se.
It is simply a procedure to join data from various scribal versions or grafting a modern station record to the record of a nearby discontinued old station or correcting a long record for a UHI with the help of the neighboring rural stations.
Of course, it is best done with the actual knowledge of the issues with the homogenized dataset and the stations involved, i.e., when it’s done by local scientists from the original records and not by an algorithm just averaging points on grids.

Robert Wood of Canada

Perhaps, a punishment for these crimatologists, while doing their time in prison, would be to manually reconstruct and recover the original data 🙂