Over at Climate Audit there is renewed interest in data availability with McIntyre asking whether journals that don’t guarantee data archiving (The Holocene in this case) should be cited in IPCC reports.
It happened that yesterday Phil Jones of CRU gave a talk at KNMI in De Bilt, The Netherlands, where he also talked about availability of data, in this case the data behind the recently published Crutem4 and Hadcrut4 graphs. The talk itself was pretty neutral, just explaining what had been done to produce these two datasets. However at the end Jones made a statement that is relevant to the long lasting discussions about data availability:
For raw temperature data you have to contact the NMS’s.
NMS’s stands for National Meteorological Services, like the Met Office in the UK or KNMI in The Netherlands. The good news is, as we can also read in his latest Crutem4 paper, that CRU will make all data available. However the bad news is that these data have already been homogenized by the NMS’s and the original data are not available at the Crutem4 webpage.
Jones explained that he and Moberg concluded already in 2003 that the homogenisation of temperature data is best done at the NMS’s. So for Crutem4 whenever possible they used these homogenized data of the NMS’s directly, as can be seen at their webpage.
Now in itself this is a fair approach. If the NMS’s cannot figure out what happened with their stations in the past and how best to control for station moves and instrument changes who else can?
…
We know that in some countries adjustments to raw data determine a large part of the trend. In New Zealand sceptics fight (see also here) with NIWA (the NMS of New Zealand) over the adjustments made to the raw data. The temperature trend in the raw data is only 0,3 degrees per century while the adjusted data show a trend of 1 degree per century. Jones uses the adjusted NIWA data in Crutem4. Later this year the High Court in New Zealand will consider this case.
Jones seemed satisfied with the new situation. Anyone asking him for the raw data in the future will be referred to the NMS’s.
================================================================
Anthony: This is particularly worrisome, because as we’ve seen, metadata for GHCN global stations is very poor to virtually non-existent, and from what we know, GHCN and CRU takes very little metadata into account. While the NMS’s may have a better handle on metadata, given the disparity of quality of met services globally, this pretty much ensures that no individual researcher is going to get their hands on a complete set of all data. Phil Jones is essentially blowing off the issue saying “let them figure it out, not our responsibility”. Whatta guy!
================================================================
Crok continues:
In his future answers to sceptics asking for data he can almost copy this paragraph of Joelle Gergis blowing off McIntyre when he requested some tree ring data from her:
This list allows any researcher who wants to access non publically available records to follow the appropriate protocol of contacting the original authors to obtain the necessary permission to use the record, take the time needed to process the data into a format suitable for data analysis etc, just as we have done. This is commonly referred to as ‘research’.
In the case of Crutem4 the raw data in many cases is also not publicly available and anyone interested has to contact each of the NMS’s trying to get these data. There is no guarantee at all that they will release the data.
Now although Jones was very obstructive to data requests from sceptics in the past, I don’t say that Jones is to blame for the current situation. At least he tried to get permission from the NMS’s to release the data as he promised in this Nature article in 2009, which was also covered in several Climate Audit posts (see here for example). In the Nature article Jones said:
“We’re trying to make them all available,” says Jones. “We’re consulting with all the meteorological services — about 150 members [of the World Meteorological Organization] — and will ask them if they are happy to release the data.” A spokesperson for the Met Office confirmed this, saying “we are happy for CRU to take the lead on this, as they are their data”.
But getting the all-clear from other nations won’t be without its challenges, says Jones, who estimates that it could take several months. In addition, some nations may object if they make money by selling their wind, sunshine and precipitation data.
In his new paper on Crutem4 he reports back on this attempt:
In November 2009, the UK Met Office wrote on our behalf to all NMSs to determine if we could release the versions of their monthly temperature series that we held. Of the about 180 letters, we received 62 positive replies, 5 negative replies, and the remainder did not reply.
These results are worrisome in itself. Almost two-thirds of the NMS’s didn’t even bother to answer to a request concerning one of the most important climate graphs in the world. For these countries Crutem4 uses the GHCN data.
===============================================================
The need for a journal that demands all data, (used and excluded) up front, along with methodology, code, and supplementary material to ensure the work is reproducible, before even considering a paper for review is becoming clearly obvious. – Anthony
h/t to Dr. Roger Pielke Sr.
Nick Stokes;
Well, you can. And individuals have done it. You can get raw data from GHCN, the various SST sources and the NMS’s. But CRU doesn’t use it. It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?>>>>
1. They had to compile it for their own purposes, so suggesting there is ANY additional work to be done on that score is silly; and
2. Yes. Already being compiled, all theny need do is release it.
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says.
D. J. Hawkins says: @ur momisugly June 1, 2012 at 5:07 pm
….That would have been quite a feat for thermometers graduated in whole degree increments. Drop a decimal point and I’ll agree.
_______________________________
And that is why this whole “adjustment” of historical data is so much nonsense. The numbers are good to +/- 1 degree at best and these idiots are trying to squeeze a 0.07°C/decade warming out of the data.
They all need remedial math classes in my opinion. Taken while enjoying the hospitality of the government.
See: http://joannenova.com.au/2012/03/australian-temperature-records-shoddy-inaccurate-unreliable-surprise/
http://strata-sphere.com/blog/index.php/archives/11420
Richard Lawson says: June 1, 2012 at 5:35 pm
‘Nick Stokes says: June 1, 2012 at 4:10 pm
“I can’t see what the criticism of Jones is here”’
“Nick Stokes – defender of the indefensible.”
In fact I’m not defending anything. The reason is that people can’t specify the accusation. What is this post saying that Jones has done wrong? What should he have done?
davidmhoffer says: June 1, 2012 at 6:24 pm
“They had to compile it for their own purposes”
How do you know that? It seems quite contrary to what the post says. CRU obtained homogenized data direct from the NMS’s. They did not use raw data.
LazyTeenager says: @ur momisugly June 1, 2012 at 5:27 pm
You have filled out your tax records in a way that is entirely ethical and honest…..
_________________________________________
YOU have never dealt with an IRS audit have you?
What you do is hand over your information and smile sweetly as you do so. Then you call your tax account and and if necessary your lawyer.
I have 20 years of tax data for my business including every single receipt stored in such a way that I can hand them the correct boxes within minutes of being asked for it. EVERYTHING is stored by line item per the tax code and a copy of the tax code for that year is included in the boxes with the receipts, journals and my copy of my tax filings.
So in other words it would take me less than 10 minutes to hand over the tax information. In the case of large businesses the companies provide offices for the IRS tax auditors in their office buildings. That is from a friend who was CFO for a large corporation.
Sorry Lazy, That dog don’t hunt either.
I find it sad that so many scientists have such an unhealthy attitude to errors.
All humans make mistakes. If you publish something, you can be fairly sure it contains errors. Sometimes the errors are significant, sometimes they are not — but they are always there.
If you find a mistake in my work, that isn’t a failure. It’s a success. Every error found and fixed improves the quality of the work.
With this in mind, why would you make it hard for people to reproduce your work? You should make it as easy as possible. If more people check the work, more errors will be found and fixed.
LazyTeenager;
You have filled out your tax records in a way that is entirely ethical and honest.
But you had a personal disagreement with a tax inspector who demonstrated complete ignorance of how your business operates. And is likely incompetent in any case.
>>>>>>>>>>>>
The tax inspector doesn’t NEED any understanding of how your business operates. What he NEEDS is your financial information provided to him based on GAAP (Generaly Accepted Accounting Practices). If you did that, there’s no problem.
LazyTeenager;
That tax inspector has indicated up front he’s gonna nail you.
>>>>>>>>>>>>
You should be very excited if this is the case because as a government official announcing such a thing he has basically convicted himself of abuse of official powers and the judiciary will take a very dim view of that when your lawyer hauls the issue into court.
LazyTeenager;
And then he initiates a complete tax audit of your business that is going to take 3 months to complete and which is going to keep you at his beck and call for that time.
How cooperative are you going to be. Are you going to tell him to get f@&$d?
>>>>>>>>>>>>>>>>
I’m going to cooperate as is required of me by law. If the tax inspector abuses his authority, he will pay the price. In your example however, the shoe is on the other foot. An audit has been requested of Jones and the CRU, that audit has a basis in law as well as accepted scientific principles, complying with the audit is not onerous as the data has already been compiled and exists in electronic format which can be shared with but a few mouse clicks, and yet Jones and the CRU continue to hide behind legal processes in defiance of the law of the land. Why?
The only logical conclusion is that their data was not assembled in an ethical and honest manner. Defend away buddy, all they have to do to exonerate themselves is release the data.
Phil. says:
June 1, 2012 at 6:26 pm
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says;
>>>>>>>>>>
Complere with sources and metadata?
If so, I stand corrected.
Manfred says:
How do these folk sleep at night?
Soundly. Whatever happens, their asses are covered. Even if the entire gig is busted a decade from now, Phil can simply point the finger at the NMS’s and their “bad data”. His analysis was, of course, perfectly sound.
Gail Combs says:
BINGO!
The problem has metatasized to most fields in science. Science is getting a black eye and it is fully justified.
It’s true. But part of the problem lies outside of the science and the scientists. A lot has to do with the manner in which science is funded. Increasingly, government as well as non-governmental agencies are only willing to fund research which supports preordained conclusions. As we’ve seen with climate work in particular, scientists are under enormous pressure to ensure their studies are fully compliant with the party line. It should not be a shock that such funding policies drive an increase in corruption and fraud.
I am not excusing the behavior for a second. But we should be clear that the solution lies less with science/scientists and more with the politicians and others in public administration.
[b]The need for a journal that demands all data, (used and excluded) up front, along with methodology, code, and supplementary material to ensure the work is reproducible, before even considering a paper for review is becoming clearly obvious.[/b] – [i]Anthony[/i]
NEEDs Repeating for Mann, Hansen, Gore, Clarke, Jones, Briffa, et.al.
David, there’s a link to the data with description in the original post, so yes you stand corrected.
Tom in Worcester: June 1, 2012 at 2:22 pm
“Does anyone know the status of the FOI request from U of Penn/Mann? Here’s to hoping that they reveal some really horrifying behavior on the parts of “Team”. It seems to me that these guys are always “weasling” when it comes to the release of all data. A comuppance would seem in order.”
Well, it would hardly be very effective to submit a FOI to U of Penn regarding Michael Mann because, for everyone who does not know the difference (BUT SHOULD), U of Penn is a private, Ivy-League School of exceptional repute located in Philadelphia, PA that has NO association whatsoever with Michael Mann of PENN STATE infamy (Penn State being a pseudo-public PA state institution located in University Park, PA, about 200 miles northwest of Philadelphia, PA.). Please do not disparage an institute of outstanding academic merit and integrity by falsely associating it with the likes of Michael Mann!
Phil. says:
June 1, 2012 at 7:32 pm
David, there’s a link to the data with description in the original post, so yes you stand corrected.
>>>>>>>>>>>>>>>
Well not so fast….
I followed the link.
1. It lists a whole bunch of sources, about 1/4 of which have no link. No link = you have to request it yourself from that source, and you have no way of verifying if what they sent you and what they sent Jones are the same thing.
2. Of the sources that do have links, so what? We’re trying to VERIFY work here. Unless we see the actual grid data on a grid by grid basis, we don’t know that the numbers Jones used match those in the link to the source.
3. Some of the source descriptions are plain silly. Like “Data from various NMSs received in 2010”. Yeah, I can track that down in a snap….NOT!
4. At the bottom of the list of sources is a link to the a UK MET Office formatted version of the station data. I followed that too. First of all, it is for HadCrut3, not HAdCrut4. Second, the first line in the description says “subset” so it isn’t even complete and hence useless.
To be fair, there is also a link to HadCrut4 station data in a zip file. I didn’t bother opening it, but will take it at face value that this is the actual data they used. My arguments above still stand. We have no way to verify if the numbers they used match a considerable amount of the data they claim they were sent, we don’t know what they discarded and what they didn’t, we don’t know if they screwed up when they transposed the data they were sent into a format that would match the rest of their data…. In brief, if I missed something obvious, please correct me (yet again) but I just do not see how this amounts to disclosing their data. Itz just a bunch of numbers that they claim to be accurate with no way on earth to check the accuracy.
LazyTeenager says:
June 1, 2012 at 5:27 pm
Chuck Nolan on June 1, 2012 at 1:36 pm said:
Why do they fear FOI?
—————-
Let’s consider this scenario.
You have filled out your tax records in a way that is entirely ethical and honest.
But you had a personal disagreement with a tax inspector who demonstrated complete ignorance of how your business operates. And is likely incompetent in any case.
That tax inspector has indicated up front he’s gonna nail you. He has an agenda.
And then he initiates a complete tax audit of your business that is going to take 3 months to complete and which is going to keep you at his beck and call for that time.
How cooperative are you going to be. Are you going to tell him to get f@&$d?
======================================================================
Sooo …. what you’re saying is, “Only “The Team” has the secret recipe for making Hockey Sticks so leave them alone!”?
Phil. says:
June 1, 2012 at 6:26 pm
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says.
=========================================================
“For raw temperature data you have to contact the NMS’s.”
Sooo … He’s making available the data he has by telling people to get it from someplace else. Got it!
Just two thoughts for everyone:
Simplicity and replication.
Ask anyone who has taught anyone something. If you can not dumb a subject down and teach an advanced subject to a dumb person, how can you possibly think you are smart enough to control science which is about simplicity and replication?
Make your work easy to understand and make the information necessary to replicate it easy to find and easy to understand. If you can’t do that for climate sceptics and the rest of us who are advanced in the understanding of this field, how can you possibly think that the average person can likewise follow what you are doing?
I just want to know how I can obtain the same information P. Jones got when its likely that even if I could somehow magically obtain all of his data from over 100 sources all over the world…..that I could never figure out if its the same data he used or whether the data was changed or some other unknown issue.
And that is the heart of the matter. It really is that simple, so obfuscation by some others aside, lets just remember that simplicity is the key and to win this battle which obviously we learn day after day is not about science but about politics, that if we keep the message simple and tell the truth we will win eventually win. Again, telling the truth is the key there. Keep it simple, and sceptics win. Explain the complicated questions as approrpiate, but keep discussions simple and concise and not confuse people, but rather educate them.
Its only in education that our species advances, through obfuscation and other morally questionable techniques we will sink down to a lower level of intellectual discourse everytime. Case in point, Phil Jones. Notice how the more he lies the more stupid he sounds because he makes it more and more complicated and as its said: “Oh the webs we weave when we first try to deceive. ” This could be a case study in why lying is bad for children. Point at the funny man children, because this is what you become if you lie!!!
Perhaps that last part is a little over the top, but I have wondered myself where do climate scientists see themselves in 10 years? They have to realize by now that they are just jokes to the common man who if you even mention global warming to them, they start telling a joke….and it has only gotten worse recently…
Why do you think that President Obama will not even mention climate change or global warming anymore?
Part of the problem is the lack of politicians with a science background in the west. They are largely lawyers who are too easily bamboozled by what looks, on the surface, to be rigorous scientific research. That and the fact that the ‘scientists’ are telling them what they want to hear. The electorates are waking up to the scam because they now realise how much it is costing them. To speed up the process of killing this climate hysteria, all it would take would be for one brave government to call out that the emperor has no clothes.
The reputation of science is being undermined by eco-pseudoscientists. Try reading the environment sections of the Guardian/BBC (same site really) to get an insight of the ignorance fuelled drivel being pushed by these lunatics that want to send us back to the dark ages. More worrying is the shrill response of the acolytes.
We want two things. First : Any paper that is not completely reproducible with the information available should not be allowed to be published and should not be allowed to be funded by money from the public. Second : All papers and data funded by the public should always be freely available,
Talk about job security…
Nick Stokes says:
June 1, 2012 at 5:30 pm
Well, you can. And individuals have done it. You can get raw data from GHCN, the various SST sources and the NMS’s. But CRU doesn’t use it. It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?
Yes, they are. Being publicly funded (in part or in whole is irrelevant), and having already done the work, there is no valid reason they shouldn’t. As my lecturers used to say, “show me your work”. That phrase does not mean “I disbelieve you” – it means “prove that you understand, and so that I may replicate what you have done”. If the work is solid, good, if there are unnoticed errors, they can be fixed, and everyone will benefit from an improved understanding of the field. It is to everyone’s advantage. Where is the problem in that? Why is it necessary to make it difficult for others to – let’s use another word here – confirm their work? It doesn’t make sense, and it leads to suspicion as to motive, which is thoroughly counter-productive.
It is now a common comment that climate science is simply politics. OK, then, rather than complain, consider political action. For example, there is one senator, James Inhofe, who is a strong skeptic —
http://inhofe.senate.gov/public/
At the moment the senate is controlled by the left, but that could change next year — the electoral chances of the current administration are declining. Next year Sen Inhofe could submit a data transparency bill with a chance of passage.
This blog is a good example of “new media power” and could consider ideas for Sen. Inhofe on appropriate language and sanctions. The obvious idea is that any tax-supported non-defense data should be available to anyone in a raw state, and that all scientific journals establish an appropriate policy to support the concept.
How absurd!
As an ex-Federal employee I can speak to FOI requests. They are NOT an audit, ever!
They are a request for information. If you are organized at all, e.g. Gail in her post above, complying with a FOI is a snap.
If there are questions, say some information might/does contain propriety or personal information, you calculate about how long it will take to redact, estimated cost and hand that back to see if the cost to retrieve is acceptable. If you not have the information requested, you state that fact, why and if anyone else might be the correct source. Yes, it is an irritating interruption of your daily work, but that is all it is.
If you are unorganized as philly J. states, then you’ve screwed the pooch and you have to report back how long it will take to find and review the requested information. Be advised, the courts do not accept “I’m unorganized and it took forever to find” as an excuse for high FOI charges. They’ll decide in favor of the requestor if they believe the costs are absurd. You’re actions are your bosses problem as you explain why you are in non-compliance and over budget.
Well, that’s how it works where there is accountability. Something that the CAGW alarmists and trolls lack.
Arguing that you shouldn’t have to release something for whatever reason should be easily decided at a meeting with the FOI staff in your organization. The FOI staff are trained in what may be denied. Any further disagreement gets sent to the lawyers for a final say.
Reading the climategate emails, you can see where Phill J. ignores, delays, denies, ignores FOI requests. You can understand the frustration of the FOI officer as he first tries to get FOIs completed properly and eventually joins with the dark side. My perspective is that the FOI officer just wanted to make his job easier by not fighting Phil.
From Phil Js statement above; he has relinquished all claims to science. Phil’s denial of following proper science practice or accepting responsibility for maintaining his source material means he is not a scientist and is not producing a product worth using. CRU’s climate department is now moot.
haven’t these guys been caught cooking the books before?…temperature varies by day and by night, from season to season, and from place to place….most weather stations record only the max and min temps for the day…the midpoint of these is not the average for the day…also weather stations are very unevenly distributed…this whole concept of average global temperature is screwed up
Jones has done all that people like me asked of him.
he has produced the data as he used it, and the code used to produce the results.
His choice to use adjusted data is entirely rational.
Further he lists the source so you can check that as well.
Basically, CRU do no adjusting.
Now, comes the question about raw data.
1. GHCN provides unadjusted data
2. GHCN daily provides unadjusted data
3. an entirely new data base is being built that wil document data all the way back to paper forms
where they exist.
If you havent worked with raw data, get your hip waders on. When you finish slogging thru it
you’ll discover that homogenization makes little changes here and there, (canada is interesting )
but overall, its warmer now than in the LIA. If you focus on small areas you might find more substantial changes, but its still warmer now than when washington crossed the delware.