Met Office and CRU bow to public pressure: publish data subset and code

Just over a month after Climategate started, we have breaking news from Climate Audit

Steve McIntyre writes:

The UK Met Office has released a large tranche of station data, together with code.

Only last summer, the Met Office had turned down my FOI request for station data, saying that the provision of station data to me would threaten the course of UK international relations. Apparently, these excuses have somehow ceased to apply.

Last summer the Met Office stated:

The Met Office received the data information from Professor Jones at the University of East Anglia on the strict understanding by the data providers that this station data must not be publicly released. If any of this information were released, scientists could be reluctant to share information and participate in scientific projects with the public sector organisations based in the UK in future. It would also damage the trust that scientists have in those scientists who happen to be employed in the public sector and could show the Met Office ignored the confidentiality in which the data information was provided.

However, the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence. If the United Kingdom does not respect such confidences, its ability to protect and promote United Kingdom interests through international relations may be hampered…

The Met Office are not party to information which would allow us to determine which countries and stations data can or cannot be released as records were not kept, or given to the Met Office, therefore we cannot release data where we have no authority to do so…

Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept. The Met Office received the data from Professor Jones on the proviso that it would not be released to any other source and to release it without authority would seriously affect the relationship between the United Kingdom and other Countries and Institutions.

The Met Office announced the release of “station records were produced by the Climatic Research Unit, University of East Anglia, in collaboration with the Met Office Hadley Centre.”

The station data zipfile here is described as a “subset of the full HadCRUT3 record of global temperatures” consisting of:

a network of individual land stations that has been designated by the World Meteorological Organization for use in climate monitoring. The data show monthly average temperature values for over 1,500 land stations…

The stations that we have released are those in the CRUTEM3 database that are also either in the WMO Regional Basic Climatological Network (RBCN) and so freely available without restrictions on re-use; or those for which we have received permission from the national met. service which owns the underlying station data.

I haven’t parsed the data set yet to see what countries are not included in the subset and/or what stations are not included in the subset.

The release was previously reported by Bishop Hill and John Graham-Cumming, who’s already done a preliminary run of the source code made available at the new webpage.

We’ve reported on a previous incident where the Met Office had made untrue statements in order to thwart an FOI request. Is this change of heart an admission of error in at their FOI refusal last summer or has there been a relevant change in their legal situation (as distinct from bad publicity)?

Advertisements

  Subscribe  
newest oldest most voted
Notify of

Congratulations, Steve!
That’s a great step in the right direction.
Best wishes for the Holidays,
Oliver K. Manuel

docattheautopsy

Nice. I’m wondering if the data had been “homogenized” before release to minimize the CRU shennanigans.

docattheautopsy – I wonder the same thing. There’s been enough devious , underhand stuff happen to make ruling this out somewhat difficult.

Michael R

According to the met website FAQ’s
(blockquote)The data that we are providing is the database used to produce the global temperature series. Some of these data are the original underlying observations and some are observations adjusted to account for non climatic influences, for example changes in observations methods or site location.
The database consists of the “value added” product that has been quality controlled and adjusted to account for identified non-climatic influences. It is the station subset of this value-added product that we have released. Adjustments were only applied to a subset of the stations so in many cases the data provided are the underlying data minus any obviously erroneous values removed by quality control. The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.(/blockquote)
and
(blockquote)The data set of temperatures, which are provided as a gridded product back to 1850 was largely compiled in the 1980s when it was technically difficult and expensive to keep multiple copies of the database.
For IT infrastructure of the time this was an exceedingly large database and multiple copies could not be kept at a reasonable cost. There is no question that anything untoward or unacceptable in terms of best practices at the time occurred.(/blockquote)
Meaning there is a high chance this data set is still largely the “value added” data making verification of the temperatures impossible.

boballab

couple of things
1. If you read the FAQ page at the Met office you find out this isn’t the Raw data that CRU used. This is the adjusted data that makes up CRUTEM3 and goes into the HadCRUT Global temp data. John Graham-Cumming in his latest blog post has it plotted against CRUTEM3 and to my eye they look almost identical (very slight differences).
http://bishophill.squarespace.com/blog/2009/12/22/met-office-code.html
2. John Graham-Cumming once he looked at the code released believes this is not the CRU code but code the Met office whipped up just for this release of the subset because of the bug he found in it.
http://www.jgc.org/blog/

Peter of Sydney

Comments by other blog sites say the data is not raw but “value add”. ALso, it’s not all there. Too early to tell yet. In any case, I for one have a lot of suspicion so I hope the analysis involved checking the data to see if it makes sense. One way to do this is to compare it with raw data obtained from other sources.

savethesharks

The measured….methodical….patient approach of M&M strikes again.
Something to be said for the Canadian Scots.
Get ’em boys!
Chris
Norfolk, VA, USA

George Turner

However, the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence.

My head is still spinning.
I can write everything from engineering equations to Shakespeare, but the doublespeak behind that second sentence has me floored.
It’s almost like “I’ll tell you, but then I’ll have to kill you.”
We’re not talking about the formula for Coca-Cola, Kentucky Fried Chicken, or W-87 Minuteman III warhead designs. We’re talking about once-a-day outdoor thermometer readings.

Rereke Whakaaro

Come on guys – don’t be so skeptical – they have given you a Christmas present.
I was probably made by an “emerging economy”, it will undoubtedly break the first time you use it, it may well poison you if you ingest it, but it was still a nice thought …
Have a happy and safe holiday, and thanks for all the fun.

jorgekafkazar

savethesharks (22:10:25) : “The measured… methodical… patient approach of M&M strikes again. Something to be said for the Canadian Scots.”
Och, aye, eh?

Dave F

Peter of Sydney (22:07:55) :
Well, they’ve lost the raw data. Not like it is a set of car keys, but they managed.

Michael R

Ok i tried standard quote marks, then i tried quote marks suggested by another reader and i still cant make quotes, can someone clarify what tags i need to add to make them?
REPLY:

Like this are block quotes

Use the word blockquote and /blockquote inside of left and right arrows, which I can’t display here, but are the ones above the comma and period on your keyboard – Anthony
<i> gives Italics </i> and you can do the same with <b> for bold </b> or <blockquote> for inset block quotes as noted by Anthony. Use the <strike> tag for strikeout font. -ems

Bulldust

Merry Christmas M&M! I sense a busy holiday season for you two 🙂

Olle

Well??
Im thinking about Proffessor Wibjörn Karlen here in Sweden.Doesnt this give him the opportunity to check his national and nordic original “raw data” with the “massaged” CRU s ? And isnt ALL historical rawdata still availeble in most cuntries? If this is the case,… shit can really hit the fan.Am i wrong??The real audit is now possible?

Jay

I agree with the previous posts. Unless the FULL RAW data gets published this if only to give the appearance of cooperation. I am not a conspiracy theorist but I still am very suspicious of these people. What I know of them from their emails makes me think, How long does it take to look through the data, throw together a nice mixture of numbers that reflect a warming trend and enough outliers to appear genuine, and then some code that does not do much but reflect what you want it to. I would not be suprised to hear that the code is not the original and the data is manipulated to reflect what they want. Now they are going to go out and say how the data was released in full along with the code and we are still not happy. This is a smart bunch. They have everything to lose at this point.
Someone smarter than me please catch them in this new lie! It is a battle of David and goliath for modern times. I live for the day when those with vested interests will be exposed. I watch NBC with their GE commercials about “Renewing America”.
Follow the money. It is all Psychology. If someone did not benefit in some way, they would not do it. Green used to be as uncool as Captain Planet. Only when it became profitable did everyone start to care so
much.
My final point is a quote from my father after I go over all of the peer-reviewed skeptical arguments, the emails, the UHI effect, and all the other things I have learned since actually questioning what I was hearing.
“I see what your saying, but I still believe in global warming.”
How do you argue with that?

Phillip Bratby

I’d like to see the QA procedures they use for control and use of the data and code. Ten to one there is no QA trail for the code released and that used prior to release. We need to see every version of the code since it was first used years ago, the reasons for all the changes and the effects of the changes.

Bill Parsons

Steve, Anthony:
Congratulations to you two for your tenacity, and to all you others who’ve hung in there. And, good luck to all those who dig in to this material to see what’s left.
Here’s hoping that “raw” means “raw”. And that it’s complete.

Michael

For those who missed the conversation on a previous thread.
Oh how deep the rabbit hole goes.
Agenda 21 For Dummies

George Turner

Anthony,
I lost the slash in front of my closing blockquote above, where I said my head was still spinning. The comment still reads okay, but you do have to flip your laptop upside-down to get a feel for the intended indentation. I’m confident that WUWT readers won’t have a problem with it since they’ve already learned to look at Mann’s sediment samples upside-down.

Bulldust

Ah I see Darwin Airport is there… I sense a follow up article in the Darwin series…

boballab

@Olle (22:40:02) :
Yes and no.
No because what the Met office released is a subset not the full set of data and yes he can check to see what is in the subset against the raw back in his home country. However he still will have no idea why any adjustements were made, just that they were.

Dan Martin

My first reaction to this was “this is great”. After thinking about it though I have to be suspicious of any data that was being held from the public like this. It’s temperature records for goodness sakes, what is so secretive about that? Unless the providers of this data are worried about the quality of their sensor sites and they have fudged the data. Mr. Watts has shown what kind of quality we can expect from US stations, who’s to say that other countries don’t have the same problem.
Hopefully all of you who are much more learned than I am can sort through this mess and show everyone what is really happening.

Ray

I would not trust the data they released since these individuals could not be trusted. How can we make sure the raw data have not been modified to fit their agenda? Maybe the only way would be to get the original sheets of paper from individual stations and compare them with what CRU and MET gave up.

Michael

The more rocks you turn over, the more secretes you find that they have been hiding from you. Don’t look so surprised. They’ve been at it for decades. Who are they? Your owners. They see you as useless eaters and the unwashed masses. You are nothing to them. What I just said doesn’t sound so crazy anymore does it, not after what you have seen and read on WUWT.

MarkR

Off-topic:
It looks like Build-a-Bear has pulled the global warming animated propaganda kiddie videos from their website (though still available on YouTube http://biggovernment.com/author/mflynn/). Here’s the email I just got from the Build-a-Bear CEO.
“Thank you for your email and candid comments regarding our animated holiday webisodes. We are sorry we disappointed you.
Our goal is to entertain and engage the imagination of children with our stuffed animals, our store environment, and online. Our intention with the “Under the North Star ” webisodes was to tell a story through the voices of our animal characters of how kids can make a difference in their own individual ways. We did not intend to politicize the topic of global climate change or offend anyone in any way. The webisodes concluded this week with Santa successfully leaving on his journey to deliver gifts around the world. The webisodes are no longer available on the site.
I started Build-A-Bear Workshop as a place for families and children to come for a …
Sincerely,
Maxine Clark
Maxine Clark
Founder and Chief Executive Bear
Build-A-Bear Workshop®
1954 Innerbelt Business Center Drive
St. Louis, MO 63114-5760
maxine@buildabear.com

Malaga View

As we get closer to the CRU kitchen [where they prepare the raw data ingredients] the smell of cooking is becoming overpowering….
Global Warming Goulash
First take your Darwin Zero raw station data and add in some step changes to spice up this otherwise bland dish:
http://wattsupwiththat.files.wordpress.com/2009/12/eschenbach_before-after1.jpg
Then add some additional spice by boiling down the number of station locations since 1850 with some step changes identified by John Graham-Cumming
http://www.jgc.org/blog/uploaded_images/a-709767.png
Please note that you should coordinate your step changes if you wish to get this mixture to rise properly in the oven.
The next step is to combine these ingredients in a computer so that they can be blended to perfection by removing any unwanted lumps and bumps.
The final steps in seasoning this dish are really for the connoisseur as detailed by Musings from the Chiefio at http://chiefio.wordpress.com/gistemp/
Moving your raw ingredients to lower altitudes (preferably a beach or an airport) will enable you to add some BBQ flavour and heat… if you find that the mix is not spicy enough then please move your raw ingredients to higher latitudes where they can be heated and roasted by the sun….
The finished dish should look something like this:
http://www.appinsys.com/GlobalWarming/GW_Part2_GlobalTempMeasure_files/image007.jpg
Global Warming Goulash is Mann made by good cooks everywhere.

This is not the raw data. This is a bone thrown to the dog to make him quiet.

I have compared the CRU data for some of the UK sites to the Met Office Station data and there are differences, I have asked the Met Office to check my assumptions. You can see the difference for Oxford at my website, generally it’s small but suddenly after 1979 there are large differences.
http://www.akk.me.uk/Climate_Change.htm

MartinGAtkins

Michael R (22:23:26) :
Ok i tried standard quote marks, then i tried quote marks suggested by another reader and i still cant make quotes, can someone clarify what tags i need to add to make them?
Like this. (I think)
<blockquote>

Turns blockquote on.

</blockquote>
Turns blockquote off.

Chris Schoneveld

Jay (22:40:56) :
“I see what your saying, but I still believe in global warming.”
As an atheist I am used to that convincing line of argument.

I have already compared the temperatues from Met Office and ECA for my hometown, Lisbon, Portugal, and they show some interesting difference between them. Please check out the difference graph at http://ecotretas.blogspot.com/2009/12/receita-de-tratamento-de-temperaturas.html
Ecotretas

Ed Zuiderwijk

Before you do anything with this data, do a Benson analysis on it.

Michael

I think Nike and Apple should re-think their actions of pulling out of the Chamber of Commerce because of their beliefs in AGW. I know I’ll be shunning those companies.

So, the value added station data is released but the explanations for the values which have been added have not.
Ho Hummm. Oh well, it gives us a bone to chew over the holiday break I guess.

I’ve used this new UK Met Office data source to create linear graphs for 24 BoM monitoring sites in Western Australia which compare it against historic records from the BoM, high quality data from the BoM and GISS data. Where available, I refer to it as HadCRUT3 data but I must admit I’m not too sure what to call it. UKMO09?
Albany
Bridgetown
Broome
Busselton
Cape Leeuwin
Cape Naturaliste
Carnarvon
Derby
Donnybrook
Esperance
Eucla
Eyre
Geraldton
Halls Creek
Kalgoorlie
Katanning
Kellerberrin
Marble Bar
Merredin
Perth
Rottnest Island
Southern Cross
Wandering
York

I wonder if the Met Office has been put under legal pressure here. From the outset, the original claim that Met Office/CRU couldn’t hand over data because of various legal restrictions sounded very thin. First, did Jones ever do more than assert that that was the case – did he prove in any way his claim that he couldn’t share data? Part of the claim was that the data had commercial value. Help out an ignoramus like me: in what way are, say, Singapore temperature readings 1930-1935, commercially valuable. Second, of the 193 countries in the world, strange that they all took the same line, that you can’t share this stuff with anyone because its ours and its jolly valuable. Strange that there weren’t one or two more generous spirited countries, at least. So maybe this is how we’ve got to here: someone has finally said, OK, show us the agreements where it says you can’t. And do you know, no-one could find any such agreements.

johnh

This is classic UK Govt spin, please remember I paid for this and its not worth the paper its written on, please use it for the only use left for recycled paper.

Steven Douglas

Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept.”
That has GOT to be the most weasled and weak explanation ever. There is a finite number of countries – NO recollection of those with whom agreements/”strict understandings” were made? No records kept? It’s THAT SLOPPY? Could be anyone? Can’t state or name anyone specific with any degree of confidence?
Worse yet — just how difficult would it have been — even over this past week — to make contact, or send out less than two hundred pieces of correspondence even, to get everyone on record as agreeing or objecting to the release of the data they provided…and then publicly name them as such?
The stench of coverup is still very thick in the air. The information they released? Virtually worthless. They can’t even distinguished between raw and adjusted data, and don’t even know what adjustments were made to what sets, or why. Good luck sorting that out. Meanwhile, good luck to the credibility of ANYONE who cites the CRU datasets in their work.

Charles. U. Farley

I knew this is what theyd do.
I sent them a request for the RAW, unadjusted, unmolested data weeks ago….”we’ll get back to you” they said, they never did.
What-a-surprise, more “adjusted” stuff.

Perry

Jay (22:40:56) :
You wrote about your father, “I see what your saying, but I still believe in global warming.”
How do you argue with that?
You say (with tongue in cheek) “Ah, elderly and respected parent, please step outside the igloo for a short time”. http://www.straightdope.com/columns/read/2160/did-eskimos-put-their-elderly-on-ice-floes-to-die
Check please! I’ll get my bearskin.

Max

I see the only data set used in Northern Ireland is Aldergrove and guess what it is an international airport… Cherry picking or what… We have an Observatory here in Armagh that has been diigently collecting data since the 1840’s……
It has an accurate forecast that shows no significant warming, even after its data has been homogenised by it own climate scientists.
Nothing to see here, move along

stephen richards

Michael R (22:00:19)
Meaning this is ‘probably’ data manipulated by the CRU ‘Harry’ software. In other words ‘Crap’.
It is not the raw data, which I believe they must have, but the ‘value added’ data.
And what is it with the software they have provided?

Dev

The is the equivalent of the Washington DC Friday “Data Dump”.
Anyone familiar with Washington knows that bad news, bad economic numbers, possibly troublesome policy decisions, etc, are always released publicly on Friday afternoon. Nobody reads the weekend papers or watches the news broadcasts. By the time Monday comes around, the administration can claim that it is “old news” and ‘nothing to see here’.
It’s the day before Christmas Eve, and most people are busy with the holidays and family until after the First of the year. Smart timing by the MET, but something tells me CRU isn’t going to get away with saying ‘old news, nothing to see here’ over this data…
My browser will be autorefreshing for all the latest from Anthony, Steve, Roger, William, Lucia, Jeff, Bishop, John, and all the other blogs over the holidays. Should be fun reading!

Malaga View

Global Warming Goulash – Notes for Wikipedia and Chefs
The original recipe for Global Warming Goulash is Top Secret and is believed to have originated in East Anglia although others claim it comes from Pennsylvania… perhaps the truth with never be really know…. However, it is now generally acknowledged that this dish is very smelly and is best swallowed with a large pinch of salt… However, dedicated individuals have managed to reverse engineer the main ingredients of this toxic concoction based upon random sampling, analysis and hard work. But please note that this recipe should never be released into the public domain without prior approval and peer review by the Magic Circle of AGW.
Buying your raw ingredients from the same manufacturing is most likely to produce the best results. However, some Chefs do experience problems with the ingredients and techniques used to produce the perfect Global Warming Goulash. If you should find that your mixture fails to rise, or is half-baked, then please do not release your dish for public consumption. Please place your failures in the Hide The Decline bin for safe disposal.

DavePrime

Now the domers can say, ” We have given them the [whisper](cut/recut,adjusted/re-adjusted,value added/value reduced,location added/location reduced)[/whisper] data. We are sure *smerk* that it will show everything we said it did….”
It is time to take a VERY close accounting of this crud they have just handed out and nt let up until either they release the true RAW DATA, of admit OUT LOUD and IN PUBLIC that they no longer have it. (Thus invalidating their ENTIRE premise…)

DavePrime

That should have read “Doomers”.

DavePrime

…and *smirk*.
*sigh*

P Wilson

Yet a year previously the they published such information openly. Then it was cross referenced against satellite data, which was more reliable and accurate, and put into doubt due to the enormous and growing divergence with GISS satellite data, (and then quickly withdrawn from the Met’s information pages).
it would be interesting to know why this precept isn’t applied equally across all fields of information they make available – such as willingness to tell us their projected temperatures, erroneous or otherwise, all around the world for the next 100 years, the weather for the next season, or indeed tomorrow’s weather forecast. During the week they wrote this international crisis that would come of telling us the temperatures, the daily forecast changes enormously, for example. At the beginning of that week it was going to be “hot and dry by Thursday” then at mid week it was changed to the weekend, and on Friday evening they apologised and said it would be later – hot and dry this week instead.
So its unsure that the Hadley centre and CRU do anything of serious value, especially in view of the erroneous (mistaken) forecasts for the last 8 consecutive seasons, which are quite entertaining to read in their archives. On their web pages they almost boast about not only uncertainty about their own models, but uncertainty about the science behind them. They make quite an issue of it in fact, and then, on that basis, proceed to make very certain predictions up to 100 years ahead.
on their info page they say they take different computer models that all produce different results, mash them all together (like custard with the steak tartare) to produce what is probably a thoroughly confused set of predictions about even the very near future – eg the next few days..

P Wilson

Above addressed to George Turner (22:17:52) :