Just over a month after Climategate started, we have breaking news from Climate Audit
Steve McIntyre writes:
The UK Met Office has released a large tranche of station data, together with code.
Only last summer, the Met Office had turned down my FOI request for station data, saying that the provision of station data to me would threaten the course of UK international relations. Apparently, these excuses have somehow ceased to apply.
Last summer the Met Office stated:
The Met Office received the data information from Professor Jones at the University of East Anglia on the strict understanding by the data providers that this station data must not be publicly released. If any of this information were released, scientists could be reluctant to share information and participate in scientific projects with the public sector organisations based in the UK in future. It would also damage the trust that scientists have in those scientists who happen to be employed in the public sector and could show the Met Office ignored the confidentiality in which the data information was provided.
However, the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence. If the United Kingdom does not respect such confidences, its ability to protect and promote United Kingdom interests through international relations may be hampered…
The Met Office are not party to information which would allow us to determine which countries and stations data can or cannot be released as records were not kept, or given to the Met Office, therefore we cannot release data where we have no authority to do so…
Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept. The Met Office received the data from Professor Jones on the proviso that it would not be released to any other source and to release it without authority would seriously affect the relationship between the United Kingdom and other Countries and Institutions.
The Met Office announced the release of “station records were produced by the Climatic Research Unit, University of East Anglia, in collaboration with the Met Office Hadley Centre.”
The station data zipfile here is described as a “subset of the full HadCRUT3 record of global temperatures” consisting of:
a network of individual land stations that has been designated by the World Meteorological Organization for use in climate monitoring. The data show monthly average temperature values for over 1,500 land stations…
The stations that we have released are those in the CRUTEM3 database that are also either in the WMO Regional Basic Climatological Network (RBCN) and so freely available without restrictions on re-use; or those for which we have received permission from the national met. service which owns the underlying station data.
I haven’t parsed the data set yet to see what countries are not included in the subset and/or what stations are not included in the subset.
The release was previously reported by Bishop Hill and John Graham-Cumming, who’s already done a preliminary run of the source code made available at the new webpage.
We’ve reported on a previous incident where the Met Office had made untrue statements in order to thwart an FOI request. Is this change of heart an admission of error in at their FOI refusal last summer or has there been a relevant change in their legal situation (as distinct from bad publicity)?
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
From Free Software Magazine, Gary Richmond on: Open Science and climategate: The IPCC/CRU needs to take a leaf out of CERN’s Book.
“The data set of temperatures, which are provided as a gridded product back to 1850 was largely compiled in the 1980s when it was technically difficult and expensive to keep multiple copies of the database.”
Nonsense, I began my civilian career in the mid-70s on Wall St doing maintenance on the computers in their data centers. We always had back up to mag tape. Not expensive and not a big problem. Their dataset was certainly was a lot smaller then the banks back then.
email 1112622624 may help resolve some of these issues:
From: Phil Jones
To: “Brohan, Philip”
Subject: Re: HADCRUT various
Date: Mon Apr 4 09:50:24 2005
Cc: Peter Thorne
Philip,
I’m not unhappy at all. If I am it is more about HadCRUT2 and 3.
I read through the report to DEFRA and will be sending some comments later today. I also commented on what Harry has written as a report for you. I’ve left those comments with him as he’s away this week and I’m off April 6-15.
It is a bit odd with HadCRUT2 that the problem has surfaced now and my old mask hasn’t made any difference.
Cheers
Phil
At 15:33 01/04/2005, Brohan, Philip wrote:
Phil.
I’ve just had a chat with Peter Thorne about HadCRUT2 and 3, and I get the impression that you are concerned, so we thought I should clarify what is going on. In particular I want to assure you that we are not trying to change the system without your approval.
To make things quite clear, we have two HadCRUT systems here:
1) Peter is running HadCRUT2. This is our operational system which produces the new data every month that we send to you and everyone. This is a fixed system, it does exactly what you agreed with Peter a couple of years ago. We don’t plan to change it at all.
We did, unfortunately, make a mistake while running the system; we think a land-mask file was changed. This is what Peter’s recent messages have been about. We’re still not quite sure how this happened, but whatever fix we apply will be to restore the system to the original, agreed state.
2) I am coordinating HadCRUT3. This currently encompasses Harry’s work on the data, Simon’s work on blending, John Kennedy’s work on variance correction, and my work on errors and gridding. Some combination of this work will become the new dataset.
I have a clear picture of what I think should form the new dataset. However, we won’t produce HadCRUT3 unless you (and all the other contributors) agree. If I can’t persuade you of the value of a change, it won’t happen. In particular, I see the land station data as entirely under your control, both now and in the future.
If I (or Peter) misread the vibes and you were not worrying about any of this, please don’t start. There are not serious problems with either system.
Have fun,
Philip.
Prof. Phil Jones
Climatic Research Unit Telephone +44 (0) 1603 592090
School of Environmental Sciences Fax +44 (0) 1603 507784
University of East Anglia
Norwich Email p.jones@uea.ac.uk
NR4 7TJ
UK
Sounds like averaged data, not day-to-day records. Useless, most likely.
From Met Office Q&A:
“The database consists of the “value added” product that has been quality controlled and adjusted to account for identified non-climatic influences. ”
“The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.”
My take on it, this data is useless.
David Shepherd (05:03:14) :
The data was not confidential it was commercial as Jones said about a year ago. You cannot redistribute commercial data – try putting commercial DVD data on youtube and see the result.
The data is not raw data – That is available from the originating met offices if no available from CRU.
Jones said the data would be released when all MOs (who considered their data to be commercially valuable) had given approval. He also said that gt 90% of the data was already available as GISS sites.
The MO have now released the gt 90% of data as a file. I.e they have only released data that has no commercial value.
This is a good example of what Jones had as raw data
http://climate.arm.ac.uk/scans/1878/08/187808a1.jpg
other locations would have been more difficult.
To store such data in the 80s would have been difficult – 1000 stations, 150kB per picture per month per am/pm=1000*150*2= 300MB/month
To store the text hand converted would be much less but then you loose the raw data.
Armargh noted in a post above has a discontinuity of 1 deg C in september 1878 which raises all preceeding months by 1 deg C. Overall the hockey stick shape is preserved. when this is taken into account:
http://img51.imageshack.us/img51/3635/armargh1878discontinuit.png
Note that the met office has had on line for some time unadjusted temps of a few uk stations
Fixed it:
Boudu (01:43:45) :
On the first day of Christmas
HADCRut sent to me
A portion of
rawvalue added dataAn a partidge in a Yamal tree
And on the second day of Christmas
Hadley sent to me
Two hashed up PERL scripts
A portion of
rawvalue added dataAnd a partridge in a Yamal tree
1) There is still no raw data.
2) In response to a question about FOI requests they said:
“We take our responsibilities under the Freedom of Information Act very seriously and have, in all cases, handled and responded to requests in accordance with its obligations under the legislation.”
That doesn’t seem to square well with the leaked e-mail messages.
Interesting aside. CRU in frech means uncooked. Seems something was lost in the translation !
“I see what your saying, but I still believe in global warming.”
“How do you argue with that?”
Don’t. The crux of the matter isn’t whether the globe is warming, which it is, but what is the cause. Dr. Akasofu argues that it’s the rebound from the LIA plus the warm phase of the PDO, in: “Two Natural Components of Recent Climate Change,” (as a 50-Mb PDF):
http://people.iarc.uaf.edu/~sakasofu/little_ice_age.php
“I see what your saying, but I still believe in global warming.” and how do you argue with that? Well given the importance and relevance of the UK Mets flip-flopping and total buffoonery now, you have to ask yourself which national leader will crack first and say ‘enough is enough’ and we need to stop this juggernaut right now while we all cool off and take a calm, calculated look behind the walls of current climatology. Nothing like a decent commission of enquiry to deflect the usual political approbrium and the temptation must be growing. They must be watching their scientists and experts at the commanding heights covering their butts and heading for the exits and getting increasingly nervous. Once one cracks the rest will follow with alarming speed I’ll warrant. Look how quickly the Berlin Wall fell once a certain momentum had gathered and then a tipping point was reached.
The Met office says:
“For IT infrastructure of the time this was an exceedingly large database and multiple copies could not be kept at a reasonable cost. There is no question that anything untoward or unacceptable in terms of best practices at the time occurred”.
OK, they didn’t keep multiple copies. But the question is, did they keep *any* copies – or even just the original tapes (or possibly disk packs)? Note that a a “copy” of something is not the “original” something.
Wait. I thought ALL the data was already released and publicly available. Hasn’t that been the defense for the last month?
”a network of individual land stations that has been designated by the World Meteorological Organization for use in climate monitoring”
World Meteorological Organization?? – What’s that? Not another UN body,I hope? Do some of ‘The 42’ work there?
Jay (22:40:56) :
“My final point is a quote from my father after I go over all of the peer-reviewed skeptical arguments, the emails, the UHI effect, and all the other things I have learned since actually questioning what I was hearing.
‘I see what your saying, but I still believe in global warming.’
How do you argue with that?”
I’ve had to cope with the same problem convincing friends, acquaintances and even my two adult sons.
One problem is that belief is almost impossible to defeat with facts alone, as it usually based on perception rather than reality. To find out what’s happening with your father try some inquiry using some planned questions and a technique which I call the ‘Depth Gauge’ – remember just listen during the questioning phase and don’t refute anything he says, no matter how daft it may sound to you! Oh, and no ‘closed’ questions.
Build the question deck you as questioner are going to ask the respondent in this sequence:-
What, how, when, why, where –
1. Understanding of facts. (e.g. How is global average temperature calculated?)
2. Context of facts in relation to hypothesis. (e.g. How does this impact CAGW?)
3. Importance of topic to respondent. (e.g. What does this mean to you?)
4. Feelings of respondent to topic (e.g. How do you feel about it?)
At the end of this process, you should have a good idea of the level of your fathers factual knowledge, the context of its importance to him and his feelings about it. Now you can give him the correct information a bit at a time and hopefully change his perception of reality.
I figured why the MET is able to release the data now when they previously could not do so due to confidentiality requirements. The confidentiality agreement was that they could not share the original data. But this data has been massaged and homogenized, such that it in no way resembles the original data, and thus the MET can release the data and stay in compliance with their confidentiality agreements!
amortiser:
“As the data is further picked apart, the reputations of politicians who have stood by these charlatans will also be destroyed.”
I’d love to believe that… In the UK ALL the major political parties’ leaders are fully, indeed rampantly, behind the AGW cause. I call it a cause, we’d call it a scam, of course.
Personally, UKIP will get my vote. Only credible people at least standing up to all this EU/UN/AGW/NWO BS.
Then again, they’re politicians as well and once you start giving them power then, well, you know what power does, don’t you?
Just have to look to the biggest political organisation in the world to see the global height of corruption. Not the pinnacle of corruption, for that place is occupied by the people who set up and funded the UN from the beginning: the money men.
Again , anyone had to conclude that the data and emails ‘ obtained ‘ from the University of East Anglia’s Hadley Climate Research Unit (CRU) were in fact leaked and not hacked .
On October 12, 2009 , Paul Hudson , a BBC weatherman was ‘ forwarded ‘ the ‘ ClimateGate ‘ files from a ‘ anonymous person ‘ or ‘ mole ‘ a month before the internal breach of security was discovered and the ClimateGate files went public . I seems that the question of whether the ClimateGate files were leaked or hacked should had already been determined .
I stated a month ago that :
” To verify this one would have to compare the ClimateGate file that Paul Hudson from the BBC was forwarded on October 12 , 2009 . again , the word ‘ forwarded ‘ implies to the files being ‘ Copied ‘ , not Taken as Jones had stated to Invesigate Magazine .”
” This means that the ‘ anonymous person ‘ must be a ‘ insider ‘ within the Climate Research Unit , as he had opportunity to ‘ update ‘ the FOIA2009 folder from within until November 12th , 30 days after Hudson at the BBC possibly had an altermatum to either expose ClimateGate to the public OR the ‘ insider ‘ would leak the data him/her/them selfs .”
I wondered if Paul Hudson had been contacted or interviewed to get his side of this story until I found that most BBC forecasters are not directly employed by the BBC, but by the Ministry of Defence (MOD) ‘s Meteorological Office ? Yet since 2007 Paul Hudson is reported to be a full-time member of the BBC staff , and not the Meteorological Office ?
September 11 , 2007. ( or 911 ? ) a new Integrated Climate Programme (ICP) was for the first time combined for the needs of the Department for Environment, Food and Rural Affairs and the Ministry of Defence, to gather information on climate change.
The Met Office Hadley Centre is the UK’s official centre for climate change research. Partly funded by Defra (the Department for Environment, Food and Rural Affairs) and the Ministry of Defence. The MOD has signed a three-year deal with the Met Office Hadley Centre worth £12m while Defra has signed a five-year deal worth £74m.
While it is stated that Paul Hudson is no longer directly associated with the Meteorological Office , you can be sure that personal and professional ties still exist .
With this information , it is more clear to me now that on the day that Phil Jones was informed of the ‘ Breach of Security ‘ (internel?) by the Administrators at RealClimate , that same afternoon it was reported that Jones was interviewed by ‘ police ‘ about the scandal , AND that Two Plain Clothes Officers arrived in an unmarked car and took Jones to Norfolk Police’s headquarters in nearby Wymondham to give a statement ?
Two Plain Clothes Officers in a Unmarked Car ? Doesn’t that seem more like a Ministry of Defence visit rather then a conventional local police matter ? While it is fact that the Ministry of Defence had signed that three-year deal with the Met Office Hadley Centre worth £12m , this could be the reason why the Ministry of Defence was involved , but if not ? ? ?
Like I’ve Always Said . . . FOLLOW THE MONEY .
From Aaron Shenck, Deputy Director, (PA)Senate Education Committee:
“Penn State has informed Senator Piccola that university policy requires an internal inquiry process to occur first before a full investigation can be undertaken. In most cases the internal inquiry process may take no more than 60 days by university policy, and if this process determines a full investigation is warranted, that investigation must be completed no later than 120 days after it is initiated. The internal inquiry process in Dr. Mann’s case started on November 30. The Senate Education Committee will monitor this process and will review the outcomes of it. “
Actually,publishing the data late is no excuse for not prosecuting them. Delaying the publishing is illegal; I just hope it is punishable with something painful enough.
What is the purpose of criminalising something if there is no punishment?
If the subsets are based on the TS2.1 dataset they are homogenized. CRU claims that the later TS3.0 is not homogenized but Harry_Read_Me seems to make it clear that TS3.0 is derived from TS2.1. That’s if the subsets are from the TS data products. They could be from some other data product or scraped together source data discovered in the Recycle bin (Trash bin for Mac users).
I’ve got FOIAs in process submitted a couple weeks ago to see if CRU complied with their regulations regarding confidential data.
Plus the code just displays the numbers as best as I can see. So they havent released the code or the data. Just some code and some data. When will they learn that we won’t stop.
3×2 (05:24:26) :
GWG (MO-A)
Although we have every confidence that the food served at our restaurant in Norwich has never contained Cockroaches or Rat poison, we feel it is our duty to investigate.
The latest information indicates that the Norwich restaurant will be closed for sometime because cockroaches and rats still have free access to the building. Additionally, the latest environmental health inspection of the Norwich premises has identified that the Chefs have actually been using genetically modified ingredients in the kitchen. This severely undermines their claims that the Global Warming Goulash only contains 100% pure organic ingredients.
I wouldn’t know TS-1 from a nine iron, but if the data release has been homgenized, I think the popular case (probably legal as well) may be building to demand meta-data showing exactly what was changed, when, and how.
Can the stewards of individual stations legally turn over raw data to the public? (either past or future?)
A T50 (50 MB) disk could be backed up on one (1) 9-track Pertec tape – a T200 (200 MB) disk took three tapes I think. And a 990 minicomputer usually had one T50 (system disk) and several (2 or 3) T200 ‘user’ disks. Incremental backups took place each night, and full backups took place once a week; there were sufficient ‘tapes’ on a storage rack that were cycled through to accomodate this practice without reusing the prior tape made the day or week before.
A mini-datacenter at a site (we had three sites at least in the Dallas area) had about five (5) 990 mincomputer systems each (not counting any specific ‘project’-owned 990 minis) run by the group known as ‘DCS’ (Distributed Computer Services) within the company (TI). I say mini-datacenter because this does not count the BIG IBM iron that was at a couple different ‘CIC’ (Corporate Information Center) sites (South Bldg and Lewisville at the time) and to which the 990’s had been networked via applications on the 990’s written to IBM’s SNA arcitecture and protocols (RJE, BDT, TSO).
And this was at a defense contractor in the 80’s (before the VAX’s began to sweep through and replace the 990 minis) …
.
.