While I’ve been getting lots of attention for “take down” of a single file that infringed on my copyright, another, much more broad and serious event is unfolding in the UK at the Hadley Climate Centre.
It appears that the “mole” has caused a Centre-wide panic and they are purging publicly available climate data.
If their climate science is so solid, so unassailable, why would they need to do this? Why hide the climate data gathered from public domain sources worldwide such as NOAA and NCDC? Steve McIntyre tells the story and wonders also. I’m sure all of those who complained in my case will do the same about Hadley, since it will affect the climate community worldwide. I suppose we now have a new term: “Climate Data Deniers” – Anthony
“Unprecedented” Data Purge At CRU
On July 31, 2009, the purge of public data at CRU reached levels “unprecedented” in its recorded history. Climate Audit reader Super-Grover said that the data purge was “worse” than we expected.
On Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory ftp.cru.uea.ac.uk/.
The next day, on July 28, Phil Jones deleted data from his public file – see screenshot with timestemp in post here, leaving online a variety of files from the 1990s as shown in the following screenshot taken on July 28, 2009.
The following day, the following listing of station data available since 1996 (discussed in my post CRU Then and Now) was deleted from public access: ftp.cru.uea.ac.uk/projects/advance10k/cruwlda2.zip, though other data in the file remained.
This morning, everything in Dr Phil’s directory had been removed.
This is part of a broader lockdown at CRU. Ian Harris, Dave Lister, Kate Willett, Tim Osborn, Dimitrios, Clive Wilkinson and Colin Harpham all altered their FTP directories this morning. Only one directory (Tim Osborn -see below) has added material.
Revisiting the Advance 10K webpage this morning, all Advance 10K data was deleted from their FTP site. None of the Advance 10K data links at www.cru.uea.ac.uk/advance10k/climdata.htm work any more.
If you go to the directory page ftp.cru.uea.ac.uk/projects which formerly hosted ftp.cru.uea.ac.uk/projects/advance10k directory, it now contains only two directories between Sept 1999 and the present, both dated 8/1/2008, but containing data from 2001.
On July 31, 2009 at 10:41 am, Tim Osborn published a webpage entitled “controversy.htm”. It is located in a folder entitled www.cru.uea.ac.uk/people/timosborn/censored/ and the webpage www.cru.uea.ac.uk/people/timosborn/censored/controversy.htm itself is of course censored.
I presume that the data has not been totally destroyed, only that, after many years of public availability, it has been put under lock and key. It’s as though CRU is having a collective temper tantrum.
Now that is an abuse of power. Plain and simple. Does climate data threaten the landed gentry?
I think it is a little pre-mature to assume this is because “they” have ulterior motives. I am constantly adding and deleting files at my website for various reasons. Does this mean I have something to hide? Who is to say this is not part of a general re-organization of website?
I think of this like the right to remain silent and right to not allow search without a warrant. Some may say that if you have nothing to hide then you should allow your property to be freely searched by law enforcement. The US government was founded on preventing such broad assumptions.
“Site is temporarily down for maintenance. ”
The SurfaceStations database is down for “maintenance”. Any idea of how long temporarily is likely to be?
That’s pretty weird stuff.
I agree with Wade and others who think this cleanup is being over-hyped.
These are files on an ftp server of a university computer network. Without doubt there are institutional rules governing to files that may be put there along with retention policies. Equally certain, periodically people ignore, forget or simply don’t know the rules – or they get changed.
When a very public security event like Steve’s “mole” occurs, various people cop it in the neck and computer administrators audit and cleanup in accordance with institutional policies. On an optional server like this one the simplest process may even be to archive everything and see what gets put back. The rest likely shouldn’t have been there anyway.
Lol. Why should they make their data available to people might find something wrong with it?
I realise the UK is a long way from California, but you have teh intertubes even there right? If so, you might want to note that the Hadley Centre is a government funded research lab in Exeter, Devon (SW England) which is part of the UK Met Office, while CRU is a centre at the University of East Anglia, in Norwich (SE England). Two different groups, with different employees and different websites.
I agree with Wade that we should hold out hope that this is a reorganization, rather than just assume the worst. However, should the worst turn out to be the case, the justification for paying their salaries fades away.
If their intentions are to cover up the data, this behavior will backfire on them. It is important to keep the heat on them until the media (perhaps the Times) and politicians pay attention. This has the potential to be as much of a scandal as Michael Mann’s behavior regarding the “hockey stick” data. Perhaps this will be another crack in the foundation of AGW orthodoxy.
It is, I think, time for a volunteer cooperative of scientists and and interested volunteers to be formed to produce another land based temperature index. Perhaps a US centric collection where the records from the best stations with the longest records could be regathered, and published into the public domain. Such a site would not only make available the station temps in .txt files, but could also host PDF files of the scanned images of the old original paper records recorded by station keepers from years ago.
Furthermore, any processing of such data would be done by opensource code stored in an open source code repository and hosted at an open source site such as sourceforge.net. The language would be modern and easy to use. Something like Ruby perhaps — can be used either functionally or in an OOP fashion, and with plenty of useful math and graphics libraries.
Seems to me our Sun has just pulled off one of these data purging tricks. Mr. Sun and Mr. Jones are following the same playbook just to confuse mere mortals.
Seems pretty clear to me that the data access removal is related to the presence of the mole.
Could be coincidence, as some have said, but the “mole” time line speaks volumes.
Sad day for climate science.
So …what are they hiding?
Yes. Two different groups one Hadley funded directly by HMG and the other funded by grants from HMG. Different departments, same central government.
Although I will write a letter to the Times it will do no good, under its new editor it has become a slavish political instrument of the Green brigade.
There are however ways and means of getting at what is going on and I will do some digging but with August, holiday season, not much will start happening until late September.
I am used to doing this kind of thing but it does take time.
The only advantage is that with the Labour administration due to fall civil servants are neither willing to do much for Ministers nor hide their master’s indiscretions provided they can blame them on their policies.
So useful to be whiter than white and in tune with the incoming new government’s policies you know.
So it may be possible to get somewhere. Time will tell.
Kindest Regards.
Somewhat OT but related to data shenanigans, certain persons at wikipedia are again providing an “altered” perspective on the Satellite measurements. As it happens, they had not updated their image from here: satellite temperature measurements since 2005 until I provided a reference to a RSS and UAH graphic updated as of November 2008. They (Kim Peterson) quickly reverted my image reference indicating that trendlines were not okay (although there are linear trend lines in the current image).
So fine. I accessed the up to data public domain UAH and RSS data, loaded it into an Excel workbook, generated a chart, and uploaded it to wiki commons making it public domain. I posted a thumb link to it to the page. I highly doubt it will last as — surprise — suddenly the existing image has been updated as of July 31st 2009. However the RSS and UAH data lines DO NOT look at all like the RSS and UAH data lines posted elsewhere. In fact they mysteriously track with GISS and, the linear trends have been extended.
Interesting, yes?
one of the rare occasions, where the question about motives is productive, because incompetence doesn’t explain this.
I detect the hand of lawyers here. Who will be asking questions like,
If this data is copyright, why are you making it publicly available without the relevant copyright notices?
Otherwise I see the apologists got in early to this thread.
Wade, this may surprise you, but the US Constitution has no legal standing in the UK.
Hummm— A possible UK purgegate in the making?
What is lacking in the discussion, but is occasionally brought up in various threads, is the logical basis for withholding information gained by non-profit, government paid individuals or organizations. If tax dollars paid for the info to be gathered, then on what basis is it withheld? The reasons given in other threads and stories posted have been laughable and very ambiguous. What gives?
I see a move like this doing nothing but fueling public distrust of AGW science. Too bad you don’t hear of this type of stuff in the media.
If tax dollars paid for the info to be gathered, then on what basis is it withheld?
The same basis as info gathered by another entity. The fact tax money paid for it is irrelevant to who owns it or whether it is owned at all.
Americans have a distorted perspective on this issue because the US government makes all tax payer funded data, public domain. The US government is unique in this respect (AFAIAA).
Re: Tim McHenry (19:19:16) :
Well, that isn’t really the problem either. It doesn’t matter why it is being withheld from McIntyre if it is also withheld from everyone else that asks. But it isn’t. If one has a track record of supporting AGW in their research, they apparently don’t have a problem getting the data. If one has a record of questioning AGW, the data is off limits. The bottom line is in a statement made by Jones where he asked why he should give it to someone who wants to find “something wrong with it”. Which defeats the entire purpose archiving data and methods … so that others might “find something wrong” with or validate the conclusion as the case may be.
David (16:50:21)
Not much connection to the gentry, landed or otherwise, surely? I think you give too much credit to HRH’s influence. This is more like the work of apparatchiks.
Have you asked them why?
You know this is pretty tiresome behaviour, if you want to be taken seriously you need to at least portray a semblance of objectivity.
If you want to be part of scientific debate you need to pay some respect to Occam’s razor.
Joe Black (16:57:07) :
āSite is temporarily down for maintenance. ā
The SurfaceStations database is down for āmaintenanceā. Any idea of how long temporarily is likely to be?
Bump
Also http://mi3.ncdc.noaa.gov/mi3qry/login.cfm is down. Is this typical on weekends?
Perhaps the files were taken offline because they show massive alteration when compared to earlier versions. Wouldn’t want incriminating files lying about. Hard to tell what the thinking is. Most science sites I know of put up a “Temporarily down due to maintenace” note. They don’t just start deleting files under full view.
Wade (16:52:02) :…. “I think of this like the right to remain silent and right to not allow search without a warrant. Some may say that if you have nothing to hide then you should allow your property to be freely searched by law enforcement. The US government was founded on preventing such broad assumptions.”
————————————————————–
You miss the point Wade…. These people are publicly funded. That data is public data, it belongs to the the UK taxpayer.
Therefore it is not CRU’s property to hide…… These people are getting above themselves.
I speculate that even the AGW Scammers might be getting weary of The Scam (I sure am) at this point (because nobody intelligent falls for it), and the inexplicable behavior belongs in the classic ‘Criminal Who Wants To Get Caught’ scenario. š
Andrew
Mac (19:49:54) :….. But Mac, it is CRU that won’t share data an who is stopping other people from examining their findings and methodology. CRU is abusing the FOI act on publically funded research data….. Why shouldn’t Anthony kick up a stink at every possible excuse?
Plus they don’t answer his “why’s”.
I don’t know where you get the idea that Anthony’s behaviour is “tiresome”…. CRU’s behaviour is beyond childish.
…. Anyway, we’ll wait and see what this tree shaking exercise does. See what nuts fall out;-)
Wade (16:52:02) :
“I think of this like the right to remain silent and right to not allow search without a warrant. Some may say that if you have nothing to hide then you should allow your property to be freely searched by law enforcement. The US government was founded on preventing such broad assumptions.”
In asserting this may be nothing, Wade is correct.
But the legal rights he cites here are specifically designed to protect individual citizens from the overweening power of the state. They are not designed to protect the overweening power of the state from individual citizens.
Nor to hide public data from the tyrannical citizenry which, in fact, owns it.
Surely, the mole has already backed up all of this data.
This data was visible to the public, whether it was intended to be visible or not. If any of these files have been published, such as through links from their web site, then their deletion is odd.
Of course, they should be keeping copies as required by their policies.
Perhaps they are responding to the FOI requests- gathering all data that has been asked for by Steve and others. After collation the entire data set with code and intermediary steps will be forwarded with best wishes.
So,what if a researcher,say AlGore, wants the data,is it kept at the CRU HQ, under ground in a bomb proof vault on say, a series of linked 60’s vintage mainframe IBM computers,all accessible with the appropriate knowlege of FORTRAN and use of punchcards?
There are subtleties here in the UK which our US friends might not appreciate.
I cannot go further at this stage until I have found out why the data has been removed, which is the first step.
It might be wholly innocent you know, servers down, it’s August and we are revamping the site or whatever.
But once I have answers to that, credible or otherwise, I can start to dig into the procedures and protocols by asking individual ministries, departments and government agencies to provide answers.
That is because this information is not the sole province of any one of them: so each must answer as to its interest in and usage of the data.
And there are more of them than you might think: and more Acts of Parliament that define their powers than you might imagine. And strange ways and means apart from an FOI that you have probably never heard of.
Nor is the money they spend unaccountable either. Far from it. We can ask about that too: and track it.
This is far from filing a simple FOI and hoping for an answer, it is a slow and cumbersome process but eventually like the mills of God it grinds exceeding small.
It is true that civil servants working for a new and robust administration can be very obstructive, especially when the government has the Meeja behind it. Then you really do have to flex some muscle to get anywhere.
But this is a dying administration which many civil servants at any level, especially the middle and upper echelons, have no cause to love, and in any event they know it must necessarily fall by the middle of next year.
So I don’t imagine they will put up too robust a defence.
We shall see.
Kindest Regards.
a jones,
Ploy Away!
and heartfelt thanks in advance….
It appears that the āmoleā has caused a Centre-wide panic and they are purging publicly available climate data.
Things aren’t always what they appear to be. This might be true, but if it isn’t, you better believe they’re going to use this to their advantage to discredit this blog and Climate Audit as over-reactionary.
Just erring on the side of caution for now, as others have stated. There could be another explanation. However, if it turns out to be true, this has 8 more legs than “CarbonGate” ever did.
Call me a denier for the next few days until this one shakes out.
REPLY: Well the timing is suspect. Some of these these files have been there for years. – Anthony
Two things; first, a jones, it will be most interesting to hear your updates. Thank you.
Second, SunSword brought up a matter that concerns me. I am not mathematically or technically trained, as are the commenters and our gracious host, so I am dependent upon clear expository writing and honest data and graphs to even begin to keep up with you. The idea of controversialists jacking with online sources such as Wikipedia so as to falsely promote their position makes me more angry than I care to abuse the moderators’ patience by expressing. This is exactly the sort of (snip) I expect from them — they can’t support their position with science so they resort to dirty tricks to fool people like me who can’t catch them without help. Whatever it takes to win. There’s money at stake here!
Infuriating.
Re: Philip_B (19:34:57) :
“If tax dollars paid for the info to be gathered, then on what basis is it withheld?
The same basis as info gathered by another entity.”
Huh? A person or company in the private sector is looking to make money off their product or information. How is that the same basis as public funded non-profit, government work? Once again, WHY, in specific, should such info be withheld?
crosspatch (19:36:34) – good point on fairness, though that is a different topic than what I was addressing
Wade sez:
“I think of this like the right to remain silent and right to not allow search without a warrant…”
Dude, in science the hypothesis is always assumed guilty and does not have the right to remain silent or not have it’s data or methods searched.
Huh? A person or company in the private sector is looking to make money off their product or information. How is that the same basis as public funded non-profit, government work? Once again, WHY, in specific, should such info be withheld?
You appear to be confusing,
Whether data is the property of some entity.
Whether data should be the property of some entity.
Many public bodies make money from selling their data, which requires that it be proprietary, and I believe the UK’s Met office is one of them.
regards
SunSword (18:31:41) :
I was going to give CRU the benefit of the doubt until I found this on the UEA web site. It looks Phil Jones wants to ‘adjust’ the MSU temperature record now. A self funded research project entitled ‘Can we create a better Microwave Sounding Unit climate record through the use of high-quality in-situ data?’ with Professor Phil Jones as one of the supervisors can be found here:
http://biobis.bio.uea.ac.uk/Resproject/show.aspx?ID=23
‘However, the [MSU] measurements have been made with forecast input in mind leading to inevitable and insidious non-climatic influences permeating the record’.
‘Taken together these data should be sufficient to allow a fundamentally different approach to be undertaken to MSU dataset development and hence to re-evaluate currently available MSU-based series’.
Self funding probably means funding by some environmental group. The outcome of the research is ‘inevitable’. No doubt the ‘insidious non-climatic influences’ will be removed and the warming will reappear.
Anthony, this might be worth a separate post. I believe the cops call this MO.
I bet the order came down to clean up all the folders that can be acessed by the pubnlic. Jones has always maintained that no data can be made public, and there it was, on a publicly accessible ftp server.
I have submitted a FOI request to the Met Office seeking an explanation fo the denial of this data.
Previuosly, I had asked whether their model could be run backwards in time. Their response was :-
> > Date: Mon, 5 Jan 2009 11:35:35 +0000> > From: enquiries@metoffice.gov.uk> > Subject: re[2]: FWD: Climate prediction model> > To: john_edmondson@hotmail.co.uk> >> > Dear John,> >> > We do not run climate models with time going backwards> > in a literal sense as climate models are complex numerical tools> that> > are definitely not designed for this purpose (an analogy being to> try to> > operate a car by injecting exhaust gases through the tail pipe and> > expecting it to drive itself backwards and for petrol to accumulate> in> > the tank – clearly nonsensical).> >> > Running a model “prediction” with time going forwards but climate> > forcing (such as greenhouse gases and aerosols) changing in a way> that> > reverses what has happened in the past is possible, but we have no> > particular reason to do this since it does not help much in> answering> > relevant scientific or modelling questions.> >> > We do, however, run climate models regularly (in a forward sense) as> a> > means of evaluating their ability to reproduce past climate changes> (of> > the last 150 years, or of more ancient periods when the Earth’s> orbit> > around the Sun presented a significantly different pattern of solar> > insolation through the year at given latitudes).> >> > I hope this helps.> >> > Regards,> >> > Suzanne.> > (On behalf of an expert)> >> > Weather Desk Advisor> > Met Office, FitzRoy Road, Exeter, Devon, EX1 3PB, United Kingdom.> > Tel: 0870 900 0100 Fax: 0870 900 5050 Email:> enquiries@metoffice.gov.uk http://www.metoffice.gov.uk
I later asked if the model could be run from 1600 AD , but all I got back was some scientific paper sidestepping the question.
Can anybody answer my question as to whether running the model backwards in time or running from 1600 AD is a valid method of testing it’s accuracy?
Hadley and the CRU are intertwined, hence ‘HadCRUt’. The giveaway is in the name.
It may of course be possible that:
1. The “mole” was actually the ftp server itself.
2. That would account for the public announcements from CA and WUWT indicating that the source was valid and the IP address had been confirmed as within CRU.
3. It would also account for why CA et al. were happy to discuss it openly (despite many posters being concerned regarding the “mole’s” future employment.)
4. It would also account for why the ftp server is now being “purged”.
So, my theory (for what it’s worth) is that data was discovered on a publicly accessible ftp server in folders which the owners thought were private.
Just my theory :o)
It may of course be possible that:
1. The “mole” was actually the ftp server itself.
2. That would account for the public announcements from CA and WUWT indicating that the source was valid and the IP address had been confirmed as within CRU.
3. It would also account for why CA et al. were happy to discuss it openly (despite many posters being concerned regarding the “mole’s” future employment.)
4. It would also account for why the ftp server is now being “purged”.
So, my theory (for what it’s worth) is that data was discovered on a publicly accessible ftp server in folders which the owners thought were private.
Just my theory :o)
P.S. – Sorry, forgot to tell you great post!
J.Hansford (20:17:26) :
Wade (16:52:02) :ā¦. āI think of this like the right to remain silent and right to not allow search without a warrant. Some may say that if you have nothing to hide then you should allow your property to be freely searched by law enforcement. The US government was founded on preventing such broad assumptions.ā
āāāāāāāāāāāāāāāāāāāāā
You miss the point Wadeā¦. These people are publicly funded. That data is public data, it belongs to the the UK taxpayer.
Therefore it is not CRUās property to hideā¦ā¦ These people are getting above themselves.
————————————————————————————
There are plenty of things funded by the UK taxpayer that are not public-domain. There may be commercial reasons (data can be worth money) or other sensitivities – remember that MI5/MI6 are publically funded – would you send them an FOI request about the latest Al-Q plots?
So to talk about this being public-funded as a major objection is a red herring.
My taxes fund the British Geological Survey but if I want mundane geological datasets from them I have to pay a lot for the privilege – and that’s the stuff that CAN be accessed. It started getting tight like this during the Thatcher years and really needs looking at.
The CRU data as I understand involve a lot of stuff obtained via various international agreements.
Bye for now – John
Anthony, you and SteveM’s got some good coverage on this week’s podcast by Jim Puplava. The link to the audio is here:
http://www.financialsense.com/fsn/main.html
Click on 3rd Hour with Jim & John – part 2. The climate part starts 34 mins from the start. The first part is about the health care bill which is even more scary than the climate change scam. I always find these podcasts great to listen at the wekend.
the story finally made the press today
http://www.telegraph.co.uk/comment/columnists/christopherbooker/5955955/Weather-records-are-a-state-secret.html
Philip_B (00:26:03) :
“Many public bodies make money from selling their data, which requires that it be proprietary, and I believe the UKās Met office is one of them.”
I am a British Tax payer. That makes me with all of the other British Tax payers proprietors of the Met Office (and the data). I can’t see any national security implications on this data being in the public domain, so whats the problem
Cleaning up ftp-servers every once and a while is standard practice in every company and institute in the world.
The only thing this cleaning is good for is to demonstrate that climate skeptics indeed are not so much skeptical towards the science, but rather have a paranoid vision towards the world around them.
Which of course has been known for ages
I know that. But this is an American website and they are being accused by Americans for actions which they have no proof, just coincidences. They should know better. It was also to show how our rights are slowly being justified away. “Oh … it looks bad, therefore they are GUILTY!” Prove it! We ask the global warming believers to offer proof, but when we have to do it ourselves, we justify it away. Skeptics are always called apologists. Our rights should be upheld. The right to be innocent until PROVEN guilty is a fundamental human right. Don’t let the eco-Communists win by taking it away.
It is an easy trap to fall in to.
Found something interesting on CRU’s FTP server
First Dimitrios…
http://img269.imageshack.us/img269/5906/dimitrios.png
…who was working for a guy called Le Chiffre…
http://img510.imageshack.us/img510/8427/007dimitrios.jpg
…who was working for a secret organisation called Quantum which was run by an environmental group called Greene Planet which belonged to the Al Gore-like Dominic Greene
http://img197.imageshack.us/img197/5314/greene.jpg
Then another intriguing find is project SPECTRE…
http://img9.imageshack.us/img9/7429/spectre.png
…which has been known to stand for SPecial Executive for Counter-intelligence, Terrorism, Revenge and Extortion, which is what Greenpeace gets up to quite often. SPECTRE used to be led by this animal loving, anti-development, anti-industry tinpot…
http://img6.imageshack.us/img6/3357/blofeld.jpg
So there you have it folks!
Re: Philip_B (00:26:03
Certainly, I was not arguing the present state of things but what they SHOULD be doing. All I hear is “confidentiality agreements” (though they are never explained – what agreements??) or that the government or institutions may indeed WANT to make money off of this data. In the latter case I am arguing that they SHOULD NOT be publicly funded!
Oh yes, Fredo, Hadley makes a huge fuss about refusing FOI requests, refusing to hand over public data to anyone who might “just try to find something wrong with it”, then find out that some of the “top secret” (ie, public) data has sneaked out anyways and – OF COURSE! it’s just a little cleaning, standard practice in every company in the world!
The people doing this really aren’t worth having their boots licked, Fredo. Think about that.
“Dude, in science the hypothesis is always assumed guilty and does not have the right to remain silent or not have itās data or methods searched.”
LOL!! Wes George: can I use your line in all my intro physics classes? I don’t think I’ve ever seen such a succinct, and humorous statement of the “scientific method”.
Its all about when the FTP site was created/laziness/students.
This ftp site was used by many people students/researchers/staff/CRU
It was used to transfer data from person to person to home etc
It was used to receive/transmit data from/to other external sources
The structure and directory names are standard on many setups and are not particularly meaningful.
When it was set up the internet was a much more technical (and friendly) place where there were not many hackers trying to destroy your system. Security was not uppermost in peoples minds.
Students directories would be created when they arrive and should be archived and removed when they leave. Students are students and will store garbage in silly directory names if they are allowed!
The University of East Anglia of which CRU is a part is like all british universities – underfunded.
IT in the early days would havebeen by some student who knew about “these things” not by a specialised department.
For nearly 30 years the ftp setup was adequate.
McIntyre with a bunch of followers then comes along and starts roaming the inadequately protected servers.
The UEA IT dept gets alerted by the activity of McIntyre and his cohorts and asks that what should be used for temporary files gets cleaned, and then only used for its designated purpose.
Unecessary files get deleted – the current situation.
I would imagine McIntyre and gang has scraped the data from each and every available directory, So there is no need for CRU to retain it!!
as fredo (03:13:37) says to think directory cleaning is to hide data is paranoid.
In the 1980s it is quite possible that verbal gentlemen agreements were made that prevented passing on of commercial data:
Conditions of a Verbal Agreement
Under law there are two basic terms that constitute a binding agreement. The verbal agreement will be binding if there was an agreement on the services to be performed and an agreement was reached on remuneration for this service. This agreement can be reached by a verbal exchange in person, via telephone or via an email.
http://www.contractsandagreements.co.uk/law-and-verbal-agreements.html
Remember there was no internet as we know it. there was no pdf files and operating system/data/and office applications all fitted on one floppy disk. Agreements if not verbal would have been paper – not as easy to search for as on a hard disk!
The CRU has been around since the 70’s under different leadership:
“Hubert Lamb retired as Director in 1978. He was succeeded by Tom Wigley (to 1993), Trevor Davies (1993-1998), Jean Palutikof and Phil Jones (jointly from 1998 to 2004) and Phil Jones (to the present).”
http://www.cru.uea.ac.uk/cru/about/history/
The Ftp site was created before security was an issue – few password protected directories
Laziness – users have nor removed files transfered to them, after they have reteived them.
Students are notoriously bad at housekeeping, security, and invalid use of storage.
Why make a conspiracy out of it?
Wade (16:52:02) :
“I think it is a little pre-mature to assume this is because ātheyā have ulterior motives.”
“I think of this like the right to remain silent and right to not allow search without a warrant. Some may say that if you have nothing to hide then you should allow your property to be freely searched by law enforcement. The US government was founded on preventing such broad assumptions.”
Nothing premature whatsoever. By definition their motives are ulterior as they have not stated nor revealed them.
As far as freedom from unreasonable search and seizure goes there are two problems.
One is it applies to individuals not government entities. There is an accountability issue here. Governments tend to hide science to achieve their political ambitions.
Second, we are not talking about law enforcement here. We are talking about a public wanting of supporting information for the science that they are going to be taxed over and their freedoms limited.
fredo:
Periodic cleaning up ftp-servers, etc. is indeed often a standard practice, but it is not standard to start such clean-ups when your data is under a data request. In fact, Arthur Anderson was convicted of a felony when it followed its pre-established data-retention policy. While I do not believe that Jones and Hadley are subject to any investigation that would make its clean-ups a potential crime, still there is no evidence or suggestion that it is following a standard practice.
Indeed the paranoid behavior belongs not to the skeptics but to many leading Global Warming Pessimists who vehemently object to release of data, algorithms, and documentation that would support their conclusions.
I used to be a Global Warming Pessimist, but the refusal for proponents of CO2-based GW to be open about their studies was one of the reasons that I have departed from their camp.
Anthony, you and Steve McIntyre are performing a great service to all science and scientists worthy of the name. All those apologists for the CRU purge of public data (with no explanation as to when it will be restored) will be quite chagrined when they find themselves impoverished by taxes and ever expanding fees from “cap-and-trade-and-sequestering” of the relatively puny human contribution to CO2 in the atmosphere. (Google Large Igneous Provinces and see their massive spouting of a variety of atmospheric gases.)
Raw data files gathered by tax-payer money must remain uncompromised and accessible to those who paid for it. All data necessary for scientific endeavors must be available to all scientists.
Christopher Booker has it about right: The 1990 head of the Met Office, Sir John Houghton set up the Hadley Center in Exeter; it was linked to the Climate Research Unit (CRU) at U of East Anglia; the result would be “a record of global temperatures based on surface weather stations across the world — a data set known as HadCrut”; Sir John and his CRU ally Professor Phil Jones programed the HacCrut computer models to conclude ANTHROPOGENIC CO2 GLOBAL WARMING; “Sir John played a central role in running the IPCC, selecting many of the contributors to its reports”; he and Prof Jones were prominent champions of the IPCC’s notorious “hockey stick” graph. The stench arising from these malicious-to-science activities is overwhelming.
It is time to through the blaggards out! Or, “Arrr, keelhaul the blaggards!” All “scientists” who are practicing the black arts of denying access to data, deleting data, or cooking the data should begin to wake up to their dark future. I hope Great Britain’s elections next year will do just that to prevaricating politicians. While we are at it, U.S. voters should end the far leftist experiment that began here in January of this year. All leftist are authoritarians — there will be no science worthy of the name under their regime.
“Tim McHenry (05:47:50) :Certainly, I was not arguing the present state of things but what they SHOULD be doing. All I hear is āconfidentiality agreementsā (though they are never explained ā what agreements??) or that the government or institutions may indeed WANT to make money off of this data. In the latter case I am arguing that they SHOULD NOT be publicly funded!”
Could not agree more.
Furthermore…I don’t care if its the UK..USA…wherever…it is a really bizarre, unacceptable curiosity in today’s world…namely:
Taxpayer-funded “public servants” who have unmerited and ill-gotten “private-sector egos”!!
Matters of national security withstanding, f you are funded by the public, then you belong to the public, period!
To put it another way…[and listen to this all you big-egos in public places….I wont mention any names JAMES HANSEN]….to put it another way:
If the public underwrites your research, or cuts your paycheck, then you are, in essence, the public’s bitch!
CHRIS
Norfolk, VA, USA
Bill, I’m not claiming conspiracy or anything at all.
It’s simply amusing to watch HADCRU cover their arses in a hurry.
pyromancer76 (07:22:10) :Raw data files gathered by tax-payer money must remain uncompromised and accessible to those who paid for it.
Was it paid for? data was “obtained” from institutions who would nomally charge. Should this be given freely over the internet when you do not own it?
The whole thing smells of tape erasing/file shredding.
I am not a crook.
Fret not, the smaller fish will get thrown under the bus.
The smarter agencies who realized they were being used distanced themselves.
Next up: Who knew what/when and internal fingerpointing as things go very badly for the hucksters.
When natural phenomena turn hard againt the agenda, the handwriting is on the wall as to the investigation.
Come on fredo, you have to admit the timing on this file deletion thing smells a little, I don’t know, fishy?
SMMOOOOOCHH!! I know it was you HADCRU, you broke my heart. You broke my heart…
bill (06:10:41) wrote: “Why make a conspiracy out of it?”
Answer: The timing.
John Edmondson (01:29:15) :
You wrote…
“I later asked if the model could be run from 1600 AD , but all I got back was some scientific paper sidestepping the question.
Can anybody answer my question as to whether running the model backwards in time or running from 1600 AD is a valid method of testing itās accuracy?”
It appears that no one has answered your inquiry, so I shall try to do so.
First the issue of running forward from 1600AD: Running from 1600AD might be a valid method of testing the accuracy (reliability might be a better term) of the simulation program except that one possibly needs much better information regarding initial conditions circa 1600AD than one really has. In other words, if the modeling program results do not match the reality of the past 409 years, is the departure due to algoritm problems, or due to incorrect initial conditions?
The question about running in reverse is more complex: I presume you mean run in reverse and duplicate climate that we have already observed, and in doing so validate the algorithm or code in the same manner as running forward from 1600AD? In the simplest sort of idea one might think to just input a negative Delta time step with current conditions and let the program run. I doubt this could even work in practice because the program was never intended to run thusly, and so quantities that in reverse should have a negative increment, may have a positive one just the same.
Moreover, even if the program could run in such a manner, the idea is just plainly wong in theory. If any portion of the simulation involves dissipation, then information is lost in the forward time step, and to try to simply reverse things will lead to completely wrong results. Turbulence, mixing, diffusion, and so forth are all dissipative processes that destroy information in the forward direction. Once information is irreversibly gone, to try to reverse the process and try to run backward in time generally results in oscillation. To suppress the oscillation one reduces resolution through smoothing because the oscillation appears first at small spatial scales. Eventually smoothing eliminates all resolution completely.
In many disciplines in geophysics we try to go backward against irreversible processes as a matter of data analysis, (analytic continuation of potential fields in space, time reversing of acoustic fields in seismology, time reverse heat conduction to retrieve past climate, which was really a worthless endeavor, and so forth). I have been associated with all these ideas in one way or another. The results? Resolution is often poor, results depend very greatly on noise in the data, and are suspect as a result, results are ambiguous, and so forth.
*******************
bill (08:06:46) :
pyromancer76 (07:22:10) :Raw data files gathered by tax-payer money must remain uncompromised and accessible to those who paid for it.
Was it paid for? data was āobtainedā from institutions who would nomally charge. Should this be given freely over the internet when you do not own it?
**************************
I think the question of who owns the data and computer code is a red herring. The real question here is should such data be used for scientific works. I believe the answer to that is a resounding NO!! No public scientific journal should publish works that use secret data and methods, digital or analog. Anyone should be able to access data and methods so that it can be verified or refuted. So OK, let it remain a secret, but don’t allow it to be used for scientific endeavors or allow those works to be published as science. It isn’t science if it can’t be verified.
Bill, this data is for a public record of global temperatures which will be (is being) used to make public policy. No secrecy, ever. If necessary, pay for the data. And I can imagine many freeby quid-pro-quos that might be arranged in order to acquire data. Or, you scratch my back and I’ll scratch yours. Public data. No deals.
Bill, this is about science, not “ownership”.
I am just thinking out loud here: If the raw data was subjected to a proprietary manipulation/adjustment/fudge factor and then tabulated or listed, it is no longer raw data and thus WOULD have copyright protection. I have noticed that each private/public group that gives us graphed temp data has their own version of adjustments. That process would result in copyrighted data files. What would then be done under normal published peer reviewed practice is to write an article describing the manipulation/model/algorithm so that others can critique and discuss its merits and application on the raw data.
To that end, in order to free up space for scientific discussion instead of all this feather ruffling, the publicly funded entity that collects temperature data from the sensors should have this raw, unadjusted temperature data available to the public in two forms, raw actual and raw anomoly. Then each of us can have a go at it with our own algorithm adjustment as we see fit and post it as copyrighted material.
To that end, I think we should be knocking at the door of the supplier of the raw data, not an end user of it.
as much as i’d like to see some of these guys paraded down the street with dunce caps like in the cultural revolution, I think this is part of a routine clean up that may be highly commendable. distributing undocumented half baked data is probably not a good idea. they should publish all their data from public sources, together with documentation about where the data came from and how it was derived.
Im my opinion, when someone or some institution wants to change our life, our fiscal taxation and many more impositions to us, and uses that data to impose that changes, should give us all data. Freely and openly to confirm their studys or their propaganda. Otherwise its ditadorial imposition, as any communist system.
We dont need to have to pay for that data because its used to impose us their mantra. And we need to check their claims. Otherwise is a dictadorship system.
They dont have the right to impose us their policies, using that data and hiding that data to us, to check the reliability of their claims.
Its not a question of copyrights. Its a question of Democracy, Freedom and basic Human Rigths.
Im my opinion, were suffering the abuses of a new modern totalitarism. Were living in Orwells Times. This is the new comunism or new nacional-socialism. This is the western civilization in decadence.
This is a new brave world. The world where the State has the power, the means and the desire to control us as they want, to enslave us, as they conspirate to treat us like animals.
Sorry by my bad english.
The most likely explanation for this is simply that they were negligent with their data, never intending for it to be public. None of their links pointed into those directories. They did have reason to want to keep it private, since Phil Jones apparently makes everyone he gives the data to that they won’t give it to anyone else. I agree that’s bogus in science that’s publicly funded and has no compelling reason (such as classified research) to keep it private. However, I those are their current ground rules.
Once it became clear that this gargantuan “hole” had been found, they undoubtedly panicked, moving the data somewhere else until they can restrict access to it the way they want to. Without such steps, people like Phil Jones would stop providing them with data (or worse, demand that the data they already had be deleted, not just moved to a more secure area).
Long-term, the proper way to deal with this is of course to change the rules, so that such data cannot be hidden from fellow scientists and academics. Do that, and the gate-keepers of such data will either relent and abide by the rules, or else become irrelevant over time, replaced by researchers who will.
CRU has just shot themselves in the foot. By making their data so secretive and unverifiable, they’ll also be taking themselves out of the loop when it comes to credible research. Researchers (even those with confidential agreements providing access to CRU data) will be forced to use other data sources because no sane person in the scientific community should believe research conclusions based on data that is unverifiable. Consequently, if I was a climate researcher, I wouldn’t waste my time using this data. I’d use other sources that my peers can corroborate.
bill (08:06:46) : “Was it paid for? data was ‘obtained’ from institutions who would nomally charge. Should this be given freely over the internet when you do not own it?”
Ooooh, I see. Steve McIntyre wasted his time filing FOI requests for data he could have just paid for.
So where do we go to buy it?
[cue chirping crickets]
I’d give Christopher Booker’s stuff a miss BTW – don’t waste your time on such nonsense. He has – what we call on the East side of The Pond – “Previous”! Life’s too short to spend it reading ill-opinionated bull!
But I still think data should be freely and internationally available. It would give a common talking-shop, and once and for all it would stop people wasting their time with conspiracy-theories, be it in climate or whatever.
As I said earlier, this started under the Thatcher years, when some smart alec figured that if data can be obtained via public funds, it can be sold on to generate more public funds. “Added value” I think it was called! Several other countries realised that this might be a “good thing”. It gets more complicated past that moment – but you can see where it ends up!
All the best – John
“Iād give Christopher Bookerās stuff a miss BTW ā donāt waste your time on such nonsense. He has ā what we call on the East side of The Pond ā āPreviousā! Lifeās too short to spend it reading ill-opinionated bull!”
I, however, wouldn’t give him a miss. Indeed the random ad. hom. really should be supported by some of link unless, of course, you feel your opinion alone should be followed.
bill (06:10:41) :
Its all about when the FTP site was created/laziness/students.
This ftp site was used by many people students/researchers/staff/CRU
It is indeed about laziness on the hand of the administrators of those servers who never bothered to restrict access to this server (like so many other that are still open to the public).
But there is a flaw in your thinking, if this was al about cleaning up and restricting access to the Data then why not notify all those who are involved, namely students/researchers/staff/CRU that you can acces the server with a password and username starting at date X time Y.
In case of leak like this the first thing you should do is close the door and then clean up the mess. Not the other way around.
All in all i guess that the Mole was indeed the server itself (The IP-adress deep within), the only conspiracy i see is that they try cover up their blunder for have restricted data open for all to see. In the panic/stupidity they started to delete the offending data instead of closing public access to the server.
with “the only conspiracy” i mean regarding to the deletions, not the content of those files because that is another story.
Just like Germany. BEFORE the war!
Jim (09:33:00) :
Have you tried getting the data for EMC/EMP testing od a 747 jumbo?
Millions of lives depend on this for safety during thunderstorms. If the data is from reliable sources then the data can be assumed valid.
pyromancer76 (09:33:39) :
If necessary, pay for the data.
The dat even if paid for by CRU cannot be sent to others. If you want the data YOU pay for it! (if you buy software you cannot give it to friends)
Robert van der Veeke (11:19:49)
closing access to the server is no different to removing data. The whole point is, thanks to McIntyre the UEA IT police have been alerted to improper storage of data on the system. This data is not for the public – and is probably of no use to the public. It can and should therefore be removed.
There is no conspiracy.
Watts suggests that he will update his Surface Stations data base when he publishes. But leaving a database online which is 1.5years out of data means someone can read this and think it is current (the old data warning is off screen).
How about a new conspiracy.
Watts is not updating his database because he wants to ensure that the temperature profiles of high grade stations all go in the direction that disproves global warming.
There’s the conspiracy theory now prove it is false. Watts could of course update the DB now, but he says he will not. Watt’s he hiding!!!?
bill (11:49:07) :
It’s not a conspiracy, it’s a game of ‘you made me mad’. Grown, educated men acting like 2 year olds. Read McIntyre’s correspondence, and you will see what I mean. “Why should I give you the data, when you are just trying to prove it wrong?” If there is nothing wrong with it, why not?
bil, I think you are getting a little hysterical.
This is somewhat off topic but it does bear at least tangentially on the issues.
I have a post up linking to Microsoft’s freely available Feynman Physics Lectures. The first video discusses the nature of physical law and how they are discovered. It is not only amusing it is educational.
I also link to the book version of the lectures. In order to avoid the spam filters I’m posting this with just one link.
http://powerandcontrol.blogspot.com/2009/08/feynman-physics-lectures-video.html
And Anthony and Gang – thank you so much for what you do. I love reading the comments where intelligent AGW proponents have their say. They make me think. And as any one knows who has tried it thinking often hurts. But like any good medicine it may do some good.
Yep, AGW climate “science” begins with obfuscation of data, continues with questionable and specious manipulation of that data, concludes with erroneous and fudging analysis of the data, and ends with complete revocation of the data.
Oh yeah, how “science” should be done.
From the Feynman lecture linked above:
This is the key of modern science, and is the beginning of the true understanding of nature: this idea to look at the thing, to record the details, and to hope that in the information thus obtained may lie a clue to one or another of a possible theoretical interpretation.
It is about 13 minutes into the video.
This is why the data as RECORDED is so critical.
Note to Anthony. – Any post about data esp. but also possibly methods should have that quote from Feynman at the top.
And if I got the quote slightly wrong – no matter. There is a text reprise of the video words that scrolls along with the video. You can check my work and fix it.
MikeU (09:47:07) : “None of their links pointed into those directories.”
Actually, the primary data directory has a link provided on the main CRU web page. The other directories are accessible by clicking a provide “higher directory” link on the data ftp site. So all CRU had to do was protect the higher directory link. They still have done nothing to block access to the people directory, even though the readers at CA and WUWT have been accessing the files since at least July 29.
I actually think that CRU has been monitoring which folder people have been accessing and then removing those. There were a couple of files I had looked in, and was curious about, but did not have time to investigate. When I went back the next day, the folders were removed.
Well guys like me who have an interest in reading produced data,considers this behavior one of skulduggery,because of some of the participants of the purge caper,have past history of questionable behavior in producing data,to support actual published science papers.Dr. Jones has resisted releasing even TWENTY year old temperature data!.a pathetic situation we have to put up with.
It is the lack of openness that hurts the pursuit of science research,the secretiveness,the childish refusals to release data to people,who are doing science research.It will only implode on themselves because research that depends on good data,will fail.Their irrational reputation exposes them as a backwater institution in science.A place to avoid and for people to go elsewhere for credible data.
I will add CRU to my list of science bodies to ignore,considering them hostile to science.A place where bogus science are being produced,because we no longer can know that what they claim are in fact verifiable.
Why not drop ALL ground based temperature data and go with Satellite temperature data instead?,since we know that both RSS and UAH are far more open with their data and how they calculate the raw data into a final form.
If ALL scientists and interested observers of climate phenomenon boycott these tainted data centers,and go to Satellite data,we will all be better for it.Putting pressure on those to respect the method of science research,the sharing of data to further science research.
Re: Sandy 11:05,
previous Noun. A criminal record. Usually police and criminal vernacular. E.g.”Be careful if you decide to employ him, he has some previous.”
This is a very serious development. Panic appears to be setting in at the leadership level, which is usually where an organization needs restrained-cool to prevail.
This is how science is done in the postmodern era. There is no truth, it’s the narrative that is important. Data means nothing except when it supports the narrative.
Along that line of thought, science using unverifiable data and obscure and questionable analytic methods is ripe for mischief of the Sokal’s hoax variety.
http://www.museumofhoaxes.com/hoax/archive/permalink/the_sokal_hoax/
*******************************
bill (11:49:07) :
Jim (09:33:00) :
Have you tried getting the data for EMC/EMP testing od a 747 jumbo?
Millions of lives depend on this for safety during thunderstorms. If the data is from reliable sources then the data can be assumed valid.
********************************
This is yet another red herring. If a family member dies, you can bet my lawyer will get the data if relevant. But that isn’t the point. This data isn’t used in science. It is more technical data than anything else. I don’t have a problem with it kept secret. There is plenty of QC and testing of planes. In a similar manner, a work produced by a scientist should be verifiable by other scientists. That means EVERYTHING used in the work should be public so anyone can verify it. How can you possibly defend the use of secret data for the purposes of science?
Chris (14:02:30) “There is no truth, itās the narrative that is important. Data means nothing except when it supports the narrative.”
Wise words.
Can anybody answer my question as to whether running the model backwards in time or running from 1600 AD is a valid method of testing itās accuracy?ā
In theory what you propose is perfectly valid. A digital computer program can be run in either direction (although there is no commercial demand for running programs backwards and hence no products that do this).
As others have pointed out the quality of the data from that long ago makes the initial conditions (data) too uncertain.
But then I would argue that data from most of the 20th century is too uncertain for use in the climate models and hence their ‘precision’ is spurious.
Philip_B (00:26:03) :
Huh? A person or company in the private sector is looking to make money off their product or information. How is that the same basis as public funded non-profit, government work? Once again, WHY, in specific, should such info be withheld?
You appear to be confusing,
Whether data is the property of some entity.
Whether data should be the property of some entity.
Many public bodies make money from selling their data, which requires that it be proprietary, and I believe the UKās Met office is one of them.
Very reasonable except for one small point.
It’s all retrospective.
DaveE.
I have to believe that these lost records have been already downloaded by quite a few people. I gather we will be blocked out from the new stuff. I recall a forensic analysis on this blog some months ago ferreted out the hidden data tweaking formula of Steig who refused to provide it voluntarily. There must be a similar possibility to unearth the secrets of Hadley Centre and CRU. Probably starting with the IPCC projections to determine the fudge factors necessary to manipulate the world data to fit. For the future, it is clear that we can’t trust the agencies that we are paying for to keep things straight and will need a competing network of data – probably requiring volunteers. It would be gratifying if the initiative of the surface stations evaluation in the US by Watts et al was taken up by like-minded concerned people in other countries. Lets contact likely candidates in Australia, South America, Europe, Japan, India and China. The latter two are not pushing the AGW agenda.
I’m sure they are just running out of storage space. A hard drive costs about $0.20/Gigabyte. Budget cuts, you know.
“Gary Pearse (19:11:25) :
I have to believe that these lost records have been already downloaded by quite a few people. I gather we will be blocked out from the new stuff. I recall a forensic analysis on this blog some months ago ferreted out the hidden data tweaking formula of Steig who refused to provide it voluntarily. There must be a similar possibility to unearth the secrets of Hadley Centre and CRU. Probably starting with the IPCC projections to determine the fudge factors necessary to manipulate the world data to fit. For the future, it is clear that we canāt trust the agencies that we are paying for to keep things straight and will need a competing network of data ā probably requiring volunteers. It would be gratifying if the initiative of the surface stations evaluation in the US by Watts et al was taken up by like-minded concerned people in other countries. Lets contact likely candidates in Australia, South America, Europe, Japan, India and China. The latter two are not pushing the AGW agenda.”
I am quite happy to volunteer here in Sydney, Australia.
This whole thing stinks of knee-jerk panic action. There is only one reason why data that was freely available, for years, is now “protected” and unavailable to the public. I gaurantee you it’s not because someone just “noticed” it was available either. This is a yet another top-down directive to remove yet more tools/information from public access.
bill (11:49:07) :
Bill, I think you need a rest.
How much is the data? Do they take MasterCard?
What I don’t get is how come HadleyCRU don’t simply explain to all of us what it is that they are doing. Simple stuff, and there may well be a good reason.
They clearly need to enage a PR firm!
rcrejects, have you asked them?
rcrejects (02:14:44) :
‘What I donāt get is how come HadleyCRU donāt simply explain to all of us what it is that they are doing. Simple stuff, and there may well be a good reason.’
Let me recap the issue. The IPCC relies on research from a lot of scientists. A lot of researchers relay of ātemperature dataā from Jones and Hansen to project their computer generated āpet theoriesā models which have been āpeer reviewedā. Steve at Climate Audit broke Mannās āHockey Stickā (which took a US Senate investigation) brings up the question. If past temperature used by IPCC is wrong, how reliable is modern temperature data?
According to IPCC regulations, all data used by the IPCC shall be made available (See Climate Audit). The IPCC told Steve (Climate Audit) it not their job to provide the āDataā (If the IPCC wanted to, they could force Jones to release the ādataā but they choose not to.)
HadCrut purging temp data is kind of like the time when Rosemary Woods “purged” 18 minutes of Nixon’s phone records.
Purging is easy: Rx.for bored climatologists: Take three spoons/day of castor oil.
These guys possibly ignore there is an international law priciple called “habeas data”, which obliges to render all data to the public.
Aw, it’s just a scientific law.