While I’ve been getting lots of attention for “take down” of a single file that infringed on my copyright, another, much more broad and serious event is unfolding in the UK at the Hadley Climate Centre.
It appears that the “mole” has caused a Centre-wide panic and they are purging publicly available climate data.
If their climate science is so solid, so unassailable, why would they need to do this? Why hide the climate data gathered from public domain sources worldwide such as NOAA and NCDC? Steve McIntyre tells the story and wonders also. I’m sure all of those who complained in my case will do the same about Hadley, since it will affect the climate community worldwide. I suppose we now have a new term: “Climate Data Deniers” – Anthony
“Unprecedented” Data Purge At CRU
On July 31, 2009, the purge of public data at CRU reached levels “unprecedented” in its recorded history. Climate Audit reader Super-Grover said that the data purge was “worse” than we expected.
On Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory ftp.cru.uea.ac.uk/.
The next day, on July 28, Phil Jones deleted data from his public file – see screenshot with timestemp in post here, leaving online a variety of files from the 1990s as shown in the following screenshot taken on July 28, 2009.
The following day, the following listing of station data available since 1996 (discussed in my post CRU Then and Now) was deleted from public access: ftp.cru.uea.ac.uk/projects/advance10k/cruwlda2.zip, though other data in the file remained.
This morning, everything in Dr Phil’s directory had been removed.
This is part of a broader lockdown at CRU. Ian Harris, Dave Lister, Kate Willett, Tim Osborn, Dimitrios, Clive Wilkinson and Colin Harpham all altered their FTP directories this morning. Only one directory (Tim Osborn -see below) has added material.
Revisiting the Advance 10K webpage this morning, all Advance 10K data was deleted from their FTP site. None of the Advance 10K data links at www.cru.uea.ac.uk/advance10k/climdata.htm work any more.
If you go to the directory page ftp.cru.uea.ac.uk/projects which formerly hosted ftp.cru.uea.ac.uk/projects/advance10k directory, it now contains only two directories between Sept 1999 and the present, both dated 8/1/2008, but containing data from 2001.
On July 31, 2009 at 10:41 am, Tim Osborn published a webpage entitled “controversy.htm”. It is located in a folder entitled www.cru.uea.ac.uk/people/timosborn/censored/ and the webpage www.cru.uea.ac.uk/people/timosborn/censored/controversy.htm itself is of course censored.
I presume that the data has not been totally destroyed, only that, after many years of public availability, it has been put under lock and key. It’s as though CRU is having a collective temper tantrum.
bill (08:06:46) : “Was it paid for? data was ‘obtained’ from institutions who would nomally charge. Should this be given freely over the internet when you do not own it?”
Ooooh, I see. Steve McIntyre wasted his time filing FOI requests for data he could have just paid for.
So where do we go to buy it?
[cue chirping crickets]
I’d give Christopher Booker’s stuff a miss BTW – don’t waste your time on such nonsense. He has – what we call on the East side of The Pond – “Previous”! Life’s too short to spend it reading ill-opinionated bull!
But I still think data should be freely and internationally available. It would give a common talking-shop, and once and for all it would stop people wasting their time with conspiracy-theories, be it in climate or whatever.
As I said earlier, this started under the Thatcher years, when some smart alec figured that if data can be obtained via public funds, it can be sold on to generate more public funds. “Added value” I think it was called! Several other countries realised that this might be a “good thing”. It gets more complicated past that moment – but you can see where it ends up!
All the best – John
“I’d give Christopher Booker’s stuff a miss BTW – don’t waste your time on such nonsense. He has – what we call on the East side of The Pond – “Previous”! Life’s too short to spend it reading ill-opinionated bull!”
I, however, wouldn’t give him a miss. Indeed the random ad. hom. really should be supported by some of link unless, of course, you feel your opinion alone should be followed.
bill (06:10:41) :
Its all about when the FTP site was created/laziness/students.
This ftp site was used by many people students/researchers/staff/CRU
It is indeed about laziness on the hand of the administrators of those servers who never bothered to restrict access to this server (like so many other that are still open to the public).
But there is a flaw in your thinking, if this was al about cleaning up and restricting access to the Data then why not notify all those who are involved, namely students/researchers/staff/CRU that you can acces the server with a password and username starting at date X time Y.
In case of leak like this the first thing you should do is close the door and then clean up the mess. Not the other way around.
All in all i guess that the Mole was indeed the server itself (The IP-adress deep within), the only conspiracy i see is that they try cover up their blunder for have restricted data open for all to see. In the panic/stupidity they started to delete the offending data instead of closing public access to the server.
with “the only conspiracy” i mean regarding to the deletions, not the content of those files because that is another story.
Just like Germany. BEFORE the war!
Jim (09:33:00) :
Have you tried getting the data for EMC/EMP testing od a 747 jumbo?
Millions of lives depend on this for safety during thunderstorms. If the data is from reliable sources then the data can be assumed valid.
pyromancer76 (09:33:39) :
If necessary, pay for the data.
The dat even if paid for by CRU cannot be sent to others. If you want the data YOU pay for it! (if you buy software you cannot give it to friends)
Robert van der Veeke (11:19:49)
closing access to the server is no different to removing data. The whole point is, thanks to McIntyre the UEA IT police have been alerted to improper storage of data on the system. This data is not for the public – and is probably of no use to the public. It can and should therefore be removed.
There is no conspiracy.
Watts suggests that he will update his Surface Stations data base when he publishes. But leaving a database online which is 1.5years out of data means someone can read this and think it is current (the old data warning is off screen).
How about a new conspiracy.
Watts is not updating his database because he wants to ensure that the temperature profiles of high grade stations all go in the direction that disproves global warming.
There’s the conspiracy theory now prove it is false. Watts could of course update the DB now, but he says he will not. Watt’s he hiding!!!?
bill (11:49:07) :
It’s not a conspiracy, it’s a game of ‘you made me mad’. Grown, educated men acting like 2 year olds. Read McIntyre’s correspondence, and you will see what I mean. “Why should I give you the data, when you are just trying to prove it wrong?” If there is nothing wrong with it, why not?
bil, I think you are getting a little hysterical.
This is somewhat off topic but it does bear at least tangentially on the issues.
I have a post up linking to Microsoft’s freely available Feynman Physics Lectures. The first video discusses the nature of physical law and how they are discovered. It is not only amusing it is educational.
I also link to the book version of the lectures. In order to avoid the spam filters I’m posting this with just one link.
http://powerandcontrol.blogspot.com/2009/08/feynman-physics-lectures-video.html
And Anthony and Gang – thank you so much for what you do. I love reading the comments where intelligent AGW proponents have their say. They make me think. And as any one knows who has tried it thinking often hurts. But like any good medicine it may do some good.
Yep, AGW climate “science” begins with obfuscation of data, continues with questionable and specious manipulation of that data, concludes with erroneous and fudging analysis of the data, and ends with complete revocation of the data.
Oh yeah, how “science” should be done.
From the Feynman lecture linked above:
This is the key of modern science, and is the beginning of the true understanding of nature: this idea to look at the thing, to record the details, and to hope that in the information thus obtained may lie a clue to one or another of a possible theoretical interpretation.
It is about 13 minutes into the video.
This is why the data as RECORDED is so critical.
Note to Anthony. – Any post about data esp. but also possibly methods should have that quote from Feynman at the top.
And if I got the quote slightly wrong – no matter. There is a text reprise of the video words that scrolls along with the video. You can check my work and fix it.
MikeU (09:47:07) : “None of their links pointed into those directories.”
Actually, the primary data directory has a link provided on the main CRU web page. The other directories are accessible by clicking a provide “higher directory” link on the data ftp site. So all CRU had to do was protect the higher directory link. They still have done nothing to block access to the people directory, even though the readers at CA and WUWT have been accessing the files since at least July 29.
I actually think that CRU has been monitoring which folder people have been accessing and then removing those. There were a couple of files I had looked in, and was curious about, but did not have time to investigate. When I went back the next day, the folders were removed.
Well guys like me who have an interest in reading produced data,considers this behavior one of skulduggery,because of some of the participants of the purge caper,have past history of questionable behavior in producing data,to support actual published science papers.Dr. Jones has resisted releasing even TWENTY year old temperature data!.a pathetic situation we have to put up with.
It is the lack of openness that hurts the pursuit of science research,the secretiveness,the childish refusals to release data to people,who are doing science research.It will only implode on themselves because research that depends on good data,will fail.Their irrational reputation exposes them as a backwater institution in science.A place to avoid and for people to go elsewhere for credible data.
I will add CRU to my list of science bodies to ignore,considering them hostile to science.A place where bogus science are being produced,because we no longer can know that what they claim are in fact verifiable.
Why not drop ALL ground based temperature data and go with Satellite temperature data instead?,since we know that both RSS and UAH are far more open with their data and how they calculate the raw data into a final form.
If ALL scientists and interested observers of climate phenomenon boycott these tainted data centers,and go to Satellite data,we will all be better for it.Putting pressure on those to respect the method of science research,the sharing of data to further science research.
Re: Sandy 11:05,
previous Noun. A criminal record. Usually police and criminal vernacular. E.g.”Be careful if you decide to employ him, he has some previous.”
This is a very serious development. Panic appears to be setting in at the leadership level, which is usually where an organization needs restrained-cool to prevail.
This is how science is done in the postmodern era. There is no truth, it’s the narrative that is important. Data means nothing except when it supports the narrative.
Along that line of thought, science using unverifiable data and obscure and questionable analytic methods is ripe for mischief of the Sokal’s hoax variety.
http://www.museumofhoaxes.com/hoax/archive/permalink/the_sokal_hoax/
*******************************
bill (11:49:07) :
Jim (09:33:00) :
Have you tried getting the data for EMC/EMP testing od a 747 jumbo?
Millions of lives depend on this for safety during thunderstorms. If the data is from reliable sources then the data can be assumed valid.
********************************
This is yet another red herring. If a family member dies, you can bet my lawyer will get the data if relevant. But that isn’t the point. This data isn’t used in science. It is more technical data than anything else. I don’t have a problem with it kept secret. There is plenty of QC and testing of planes. In a similar manner, a work produced by a scientist should be verifiable by other scientists. That means EVERYTHING used in the work should be public so anyone can verify it. How can you possibly defend the use of secret data for the purposes of science?
Chris (14:02:30) “There is no truth, it’s the narrative that is important. Data means nothing except when it supports the narrative.”
Wise words.
Can anybody answer my question as to whether running the model backwards in time or running from 1600 AD is a valid method of testing it’s accuracy?”
In theory what you propose is perfectly valid. A digital computer program can be run in either direction (although there is no commercial demand for running programs backwards and hence no products that do this).
As others have pointed out the quality of the data from that long ago makes the initial conditions (data) too uncertain.
But then I would argue that data from most of the 20th century is too uncertain for use in the climate models and hence their ‘precision’ is spurious.
Philip_B (00:26:03) :
Huh? A person or company in the private sector is looking to make money off their product or information. How is that the same basis as public funded non-profit, government work? Once again, WHY, in specific, should such info be withheld?
You appear to be confusing,
Whether data is the property of some entity.
Whether data should be the property of some entity.
Many public bodies make money from selling their data, which requires that it be proprietary, and I believe the UK’s Met office is one of them.
Very reasonable except for one small point.
It’s all retrospective.
DaveE.
I have to believe that these lost records have been already downloaded by quite a few people. I gather we will be blocked out from the new stuff. I recall a forensic analysis on this blog some months ago ferreted out the hidden data tweaking formula of Steig who refused to provide it voluntarily. There must be a similar possibility to unearth the secrets of Hadley Centre and CRU. Probably starting with the IPCC projections to determine the fudge factors necessary to manipulate the world data to fit. For the future, it is clear that we can’t trust the agencies that we are paying for to keep things straight and will need a competing network of data – probably requiring volunteers. It would be gratifying if the initiative of the surface stations evaluation in the US by Watts et al was taken up by like-minded concerned people in other countries. Lets contact likely candidates in Australia, South America, Europe, Japan, India and China. The latter two are not pushing the AGW agenda.
I’m sure they are just running out of storage space. A hard drive costs about $0.20/Gigabyte. Budget cuts, you know.
“Gary Pearse (19:11:25) :
I have to believe that these lost records have been already downloaded by quite a few people. I gather we will be blocked out from the new stuff. I recall a forensic analysis on this blog some months ago ferreted out the hidden data tweaking formula of Steig who refused to provide it voluntarily. There must be a similar possibility to unearth the secrets of Hadley Centre and CRU. Probably starting with the IPCC projections to determine the fudge factors necessary to manipulate the world data to fit. For the future, it is clear that we can’t trust the agencies that we are paying for to keep things straight and will need a competing network of data – probably requiring volunteers. It would be gratifying if the initiative of the surface stations evaluation in the US by Watts et al was taken up by like-minded concerned people in other countries. Lets contact likely candidates in Australia, South America, Europe, Japan, India and China. The latter two are not pushing the AGW agenda.”
I am quite happy to volunteer here in Sydney, Australia.
This whole thing stinks of knee-jerk panic action. There is only one reason why data that was freely available, for years, is now “protected” and unavailable to the public. I gaurantee you it’s not because someone just “noticed” it was available either. This is a yet another top-down directive to remove yet more tools/information from public access.