Over at Climate Audit there is renewed interest in data availability with McIntyre asking whether journals that don’t guarantee data archiving (The Holocene in this case) should be cited in IPCC reports.
It happened that yesterday Phil Jones of CRU gave a talk at KNMI in De Bilt, The Netherlands, where he also talked about availability of data, in this case the data behind the recently published Crutem4 and Hadcrut4 graphs. The talk itself was pretty neutral, just explaining what had been done to produce these two datasets. However at the end Jones made a statement that is relevant to the long lasting discussions about data availability:
For raw temperature data you have to contact the NMS’s.
NMS’s stands for National Meteorological Services, like the Met Office in the UK or KNMI in The Netherlands. The good news is, as we can also read in his latest Crutem4 paper, that CRU will make all data available. However the bad news is that these data have already been homogenized by the NMS’s and the original data are not available at the Crutem4 webpage.
Jones explained that he and Moberg concluded already in 2003 that the homogenisation of temperature data is best done at the NMS’s. So for Crutem4 whenever possible they used these homogenized data of the NMS’s directly, as can be seen at their webpage.
Now in itself this is a fair approach. If the NMS’s cannot figure out what happened with their stations in the past and how best to control for station moves and instrument changes who else can?
…
We know that in some countries adjustments to raw data determine a large part of the trend. In New Zealand sceptics fight (see also here) with NIWA (the NMS of New Zealand) over the adjustments made to the raw data. The temperature trend in the raw data is only 0,3 degrees per century while the adjusted data show a trend of 1 degree per century. Jones uses the adjusted NIWA data in Crutem4. Later this year the High Court in New Zealand will consider this case.
Jones seemed satisfied with the new situation. Anyone asking him for the raw data in the future will be referred to the NMS’s.
================================================================
Anthony: This is particularly worrisome, because as we’ve seen, metadata for GHCN global stations is very poor to virtually non-existent, and from what we know, GHCN and CRU takes very little metadata into account. While the NMS’s may have a better handle on metadata, given the disparity of quality of met services globally, this pretty much ensures that no individual researcher is going to get their hands on a complete set of all data. Phil Jones is essentially blowing off the issue saying “let them figure it out, not our responsibility”. Whatta guy!
================================================================
Crok continues:
In his future answers to sceptics asking for data he can almost copy this paragraph of Joelle Gergis blowing off McIntyre when he requested some tree ring data from her:
This list allows any researcher who wants to access non publically available records to follow the appropriate protocol of contacting the original authors to obtain the necessary permission to use the record, take the time needed to process the data into a format suitable for data analysis etc, just as we have done. This is commonly referred to as ‘research’.
In the case of Crutem4 the raw data in many cases is also not publicly available and anyone interested has to contact each of the NMS’s trying to get these data. There is no guarantee at all that they will release the data.
Now although Jones was very obstructive to data requests from sceptics in the past, I don’t say that Jones is to blame for the current situation. At least he tried to get permission from the NMS’s to release the data as he promised in this Nature article in 2009, which was also covered in several Climate Audit posts (see here for example). In the Nature article Jones said:
“We’re trying to make them all available,” says Jones. “We’re consulting with all the meteorological services — about 150 members [of the World Meteorological Organization] — and will ask them if they are happy to release the data.” A spokesperson for the Met Office confirmed this, saying “we are happy for CRU to take the lead on this, as they are their data”.
But getting the all-clear from other nations won’t be without its challenges, says Jones, who estimates that it could take several months. In addition, some nations may object if they make money by selling their wind, sunshine and precipitation data.
In his new paper on Crutem4 he reports back on this attempt:
In November 2009, the UK Met Office wrote on our behalf to all NMSs to determine if we could release the versions of their monthly temperature series that we held. Of the about 180 letters, we received 62 positive replies, 5 negative replies, and the remainder did not reply.
These results are worrisome in itself. Almost two-thirds of the NMS’s didn’t even bother to answer to a request concerning one of the most important climate graphs in the world. For these countries Crutem4 uses the GHCN data.
===============================================================
The need for a journal that demands all data, (used and excluded) up front, along with methodology, code, and supplementary material to ensure the work is reproducible, before even considering a paper for review is becoming clearly obvious. – Anthony
h/t to Dr. Roger Pielke Sr.
There’s real science and then there’s ‘climate science’.
Two completely different rules and practices.
Peter Miller
I would argue that the use of Climate and Science in a sentence is definitionally not possible.
Without the raw data being available to all, no reliance can be placed on any conclusion drawn from any study using the unavailable data,
End of story.
“Jones explained that he and Moberg concluded already in 2003 that the homogenisation of temperature data is best done at the NMS’s.”
Once again it should be recalled what the homogenizations are. These are primarily techniques for fully restoring the effect of disturbances on trends. Thanks to stations move, raw series are indeed partially cleared of that.
“Now in itself this is a fair approach. If the NMS’s cannot figure out what happened with their stations in the past and how best to control for station moves and instrument changes who else can?”
Well, the various NMSes have themselves come up with several different and evolving approaches to determining adjusted data -without- knowing systematic and detailed station moves and instrument change data.
Preventing the non-governmental scientists from accessing the raw data can then be reasonably assumed to inhibit yet-more methods of interpretation. Which is a key part in how you realize ‘Hey, that method is better.’
I’ve never read, seen or heard anything from Dr. Phil Jones that shows him anything but a team player. I think this is a weasel move.
Why do they fear FOI?
@ Peter Smith
I’d ammend that to: there’s science and there’s politics. The doomsayers have been around since the begining of human existence, and their goal has never been the advancement of knowledge, and has always been the advancement of an agenda. The agenda usually consists of, “Give us more money and more power and we will wave our magic wands and make all your troubles go away.” Climate Science is just politics. They apparently follow few if any of the rules and practices associated with traditional scientific inquiry, and follow all the rules of the game of politics. Might as well call a spade a spade. Michael Mann and his ilk need not even do any more research, all they have to do is decide which conclusion they want and put it on the ballot. If people vote for it, it must be true!
Without verifiable data, all published theories and conclusions are in essence useless as the burden of proof for their validity is on the author and not met. Nobody should as much as buy a pack of gum based on them.
I don’t know. We all dump on climate science, but I have heard some disgruntled grumbling from other fields that journals are not doing a good job of making sure the work covered in the articles is reproducible. I think the problems may be systemic in the sciences right now and that has me worried. Of course the only places that I see wild claims bandied about are climate and medicine, but what if some of the other articles that seem to be noncontroversial are based on just as bad analysis methods, but because the conclusions seemed within experience we never questioned them? I don’t know. I hope other researchers have more integrity than what has been on display by some in climate though, because in the end, displays of integrity are all that will save science in the eyes of the public.
Homogenization prevents the contents of the mix from rising to the top.
This position liberates them from using any reliable source at all. The reports are unquestionable, the methods beyond reproduction. This is Phil Jones saying “trust me – you have no choice, actually”. Wrong guy for the job, wrong institution for the responsibility. Pull the plug.
What an utter crock! The absence of readily being able to undertake repeatability stinks, if this is truly the case. It’s an absolute ‘red flag’. I simply fail to comprehend how climate ‘science’ organises its affairs. How do these folk sleep at night?
In my branch of science, one usually welcomes such requests as both an opportunity to promote one’s ‘important’ findings and to demonstrate the ‘elegance’ of one’s method. One always goes to publication ready to provide such things upon request.
Does anyone know the status of the FOI request from U of Penn/Mann? Here’s to hoping that they reveal some really horrifying behavior on the parts of “Team”. It seems to me that these guys are always “weasling” when it comes to the release of all data. A comuppance would seem in order.
TB
Chuck Nolan says:
June 1, 2012 at 1:35 pm
http://dilbert.com/strips/comic/2007-10-31/
Climate science is an oxymoron. It is better described as climastrology.
The letter from Professor P Jones to the Swedish “Met Office” was written as to ensure a negative response:
Can we publish your data ? Please note that we may have homogenized it, so it will probably be quite different from your records!.
Not surprisingly, the answer was: No, you must not publish such data as ours.
At that time, it was important for CRU to have such a non-permission because that was what they have said.
http://www.theclimatescam.se/2010/03/06/phil-jones-och-smhis-temperaturdata/
Swedish site. Relevant letters in english.
Shows Professor P Jones: “At least he tried to get permission.. ” Not so – he tried hard to get a NO.
Climate science is oxymoronic!
I completely agree with the need for a new journal requiring total transparency. Dr Felicity Mellor during a talk at the BBC, of all places, suggested that anonymous peer review reports should be made available also, partly to help journalists appreciate any inherent weaknesses in a paper and perhaps as a disincentive to pal review.
Metadata and raw data together with homogenised data from National Meteorological Services should ideally be sent to a central digital repository, available over the internet to anyone, free of charge. Although I distrust the UN, the World Meteorological Association have published a good report on ‘guidelines on climate data and homogenization’:
“When data are quality controlled and/or homogenized, it is always important to document the procedures applied in the metadata. When transmitting a dataset it is desirable to provide both the original and the adjusted data. This will be very helpful for other users who may need to apply a different homogenization approach to meet their particular requirements.
“To recap, homogenization is a difficult but unavoidable task. By properly adjusting a station or dataset we gain a better understanding of climate and especially of climate variability and change.”
http://www.wmo.int/pages/prog/wcp/wcdmp/wcdmp_series/documents/WCDMP-53.pdf
If the CRU can acquire and analyse the data then anyone else can too. The fact that we now have numerous global temperature series which all agree very closely doesn’t mean that other teams can’t do their own research and potentially come to different conclusions, so I don’t see that there’s a problem.
REPLY: Go ahead then and try, see how far you get trying to get the base temperature and the base metadata rather than all of the “adjusted value added” products. Let us know how it goes – Anthony
Anthony: This is particularly worrisome, because as we’ve seen, metadata for GHCN global stations is very poor to virtually non-existent, …. Phil Jones is essentially blowing off the issue saying “let them figure it out, not our responsibility”.
I agree this is very worrisome. What CRU and Phil Jones is saying is that “our work will be a big part of global environmental policy, but do not hold us accountable for the raw data.” Meanwhile, the NMS’s can say “we aren’t responsible for policy, so go fish!”
However, I think this is “too clever by half”. At least in some countries there are Inspectors General that can throw a monkey-wrench into the works if such shenanigans come to light and focus.
This is a time for a well-crafted “elevator speech” that can capture the attention of honest politicians and a distracted public being hoodwinked.
REPLY: Go ahead then and try, see how far you get trying to get the base temperature and the base metadata rather than all of the “adjusted value added” products. Let us know how it goes – Anthony
Nothing galls me more than the image of dutiful data collectors in the 1930s squinting at 0.05°F plus minus day after day having their direct calibrated observations adjusted upwards to account for UHI effects in the 1990s.
Owen in GA says:
June 1, 2012 at 2:09 pm
I don’t know. We all dump on climate science, but I have heard some disgruntled grumbling from other fields that journals are not doing a good job of making sure the work covered in the articles is reproducible. I think the problems may be systemic in the sciences right now and that has me worried.
_____________________________
BINGO!
The problem has metatasized to most fields in science. Science is getting a black eye and it is fully justified.
How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data
72% questionable research practices and 14% out right misconduct does not say much for the holier than thou attitude of these research scientists now does it?
As far as I am concerned THAT study should be sent to every politician and every scientist with a note that say “SHOW ME YOUR DATA or we must assume you are lying”
At this point that is they only attitude an intelligent person can take.
You already know the Warmistas’ excuse here:
We won’t have any time for our science if we’re doing nothing but handling an avalanche of harassing requests for data, code, or procedures.
After all, we have a world to save and those darned skeptics are slowing us down!
Instead of all these attempts to adjust the data, why not just adjust the confidence intervals? Of course this wouldn’t show the warming which they would never admit to. Doing magic adjustments that assume you know what the values are is ridiculous. Without a way of knowing the actual value you need to adjust the data you’re just guessing. That way you could give an honest assessment of the temps taking into account siting, moves and coverage without the fudge factors.
In a 21, 2005 reply to a request for data from Warwick Hughes Jones refused. “We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.” (Apparently Jones confirmed his reply when asked by Von Storch).
Presumably Jones is buoyed by the self-serving investigations by the University of East Anglia and has moved on from his suicidal nadir.
http://www.dailymail.co.uk/news/article-1249134/Climate-change-scientist-I-considered-suicide.html
Willis Eschenbach did a detailed dissection of similar requests for information.
http://wattsupwiththat.com/2009/11/29/when-results-go-bad/
It’s no surprise that a PR company called “Outside Organization” was brought in to manage Climategate. The person in charge of the investigation was arrested in July 2011 for involvement in the shameful phone hacking operations at the News of the World. His partner described the CRU contract as “more covert” than others they handled.
http://climateaudit.org/2011/07/14/covert-operations-by-east-anglias-cru/
Maurice Strong established the IPCC through the World Meteorological Organization. This means the IPCC is effectively the NMS, because each member nation appoints the IPCC members. Jones used the lack of ability to disclose because of control by the NMS members previously, but it was not the first line of defense. He is using it again. The question is how much can a UN member nation refuse to provide data paid for by the taxpayer of the nation? What do they have to hide? National security? I remember stories about using weather as a weapon of war. Is it another case of Weather of Mass Destruction?
It is time other climate related people start looking at what has and is going on. I was impressed by German physicist and meteorologist Klaus-Eckart Plus’ confession, but shocked by the tacit acceptance by someone working in the field.
“Ten years ago I simply parroted what the IPCC told us. One day I started checking the facts and data – first I started with a sense of doubt but then I became outraged when I discovered that much of what the IPCC and the media were telling us was sheer nonsense and was not even supported by any scientific facts and measurements. To this day I still feel shame that as a scientist I made presentations of their science without first checking it.”
Did Jones and others rely on this complacency? It’s time more weather and climate people feel shame and examine what the IPCC have done?
Clicking on “Hadcrut4 graphs”, we find this sentence:
Fitted linear trends in temperature anomalies are approximately 0.07°C/decade from 1901 to 2010 and 0.17°C/decade from 1979 to 2010 globally.
So the impression is given that global warming has been accelerating over the last 30 years. However in another article, we read:
No statistically significant warming over 15 years on Hadcrut4.
http://icecap.us/index.php/go/political-climate/no_global_warming_for_15_years/
Is this last statement with regards to 15 years true or not?
Hadcrut4 only goes to December 2010 so what I did was get the slope of Hadcrut4 from April, 1997 to the end of December 2010. Then I got the slope of Hadcrut3 from April, 1997 to December 2010. Then I got the slope of Hadcrut3 from April, 1997 to the present. The difference in slope was that the slope was 0.0037 lower for the total period. The positive slope for Hadcrut4 was 0.0077 from April, 1997 . So IF Hadcrut4 were totally up to date, and IF it then were to trend like Hadcrut3, I conclude it would show a positive slope of 0.0040/year over the last 15 years. See the graphs below to illustrate this.
http://www.woodfortrees.org/plot/hadcrut3gl/from:1997.25/plot/hadcrut3gl/from:1997.25/to:2011/trend/plot/hadcrut3gl/from:1997.25/trend/plot/hadcrut4gl/from:1997.25/plot/hadcrut4gl/from:1997.25/trend
Now as to whether 0.0040/year over 15 years is significant or not, I would like to quote Phil Jones from an earlier interview where he said: “I also calculated the trend for the period 1995 to 2009. This trend (0.12C per decade) is positive, but not significant at the 95% significance level.” So while the numbers are a bit different now, if 0.12 C per decade was not significant then over a 15 year period, there is no way that 0.040 C/decade is significant now over a 15 year period.
Yet researchers are glad to use the Surface Stations dataset, ready or not. I guess some people are more equal than others.
Icarus62 says:
June 1, 2012 at 2:43 pm
If the CRU can acquire and analyse the data then anyone else can too. The fact that we now have numerous global temperature series which all agree very closely doesn’t mean that other teams can’t do their own research and potentially come to different conclusions, so I don’t see that there’s a problem.
===============================================
Uhmm….. no, they don’t really. Sure, the final global temp trends agree, but, they don’t collaborate on lying very well. Now, this is very preliminary, but the “new” warming found in HadCrut4 vs HadCrut3 was thought (at least by me) to be because of the addition of Hansen’s imaginary thermometers in the polar regions. And, in part, it probably is. But, a large part of the warming according to HadCrut4 is in the tropics. 30°S – 30°N. Click here to see HC3 vs HC4 vs HC4 tropics, from 1998 to the end of HC4 2010.92. Clearly Jonesy is stating that the tropics are warming faster than the rest of the globe.
But, that’s not what GISS tells us. Jimmy H says the warming is occurring towards the poles. Click here for a trend map for the period 1998-2011. Scroll down to see the latitudinal distribution.
I’ve already written one post on this, but I’ll probably clean it up and write another. https://suyts.wordpress.com/2012/05/28/is-phil-and-jim-geographically-challenged-as-well/
Oddly, HC3 and HC4 from 1965-2000 are nearly identical…. as if they weren’t altered at all.
We have two separate temp series showing warming in very different places and, yet, coming to essentially the same total trend. Which brings me to ask the obvious question. What are the odds that either one of these series are an accurate representation of our global temps? They’d be pretty negligible, in my view. But, what are the odds that both of these series were artificially manipulated to render the same final global trend? Pretty good, in my estimation. It simply stretches credulity too far to believe the outcome wasn’t predetermined, and that the numbers were made to fit the outcome.
That’s the thing about serial liars. The act of serial lying implies a deficient intellect. It’s much easier to be truthful. Intelligent people understand this. Lies have a tendency to cause more lying. Soon, they lies are too difficult to manage. And this is what happened.
As has been said so many times before, if the world was entering a crisis and you needed everyone to be convinced that this was so in order to take avoiding action, why would you not give them ALL the information required to convince them of the urgency? Given the outright criminal activity that has been perpetrated ‘for the cause’, who gives a stuff about the ‘ownership’ of some of the data? Surely it is in the best interests of all to share the data, so there is no reason not to do this.
Add to that the fact that most if not all of these NMS’s are publicly funded, how could it not be made available. I do work for the WHO, and they do not believe in copyright of their materiel. As the public essentially pays for it, the public owns it. How I wish this was a more widely held idea!
Those guys just don’t want the public to see how that came up with their results. They think that we should just trust them.
Hopefully after November there can be federal legislation to require that the raw data, source code, and any other pertinent information be available on a publicly accessible FTP server no later than the date of publication when such research is funded directly or indirectly by taxpayers. There should be penalties if it isn’t readily available. It should be in a format that would permit any competent researcher to quickly verify the result If there are adjustments, then we need to know just how those adjustments were done.
Research that has not been independently verified [I’m not talking about pal review] should not be considered in formulating policy. Research not funded by our taxpayers should not be considered in formulating policy unless they show their work as with taxpayer funded research. is time to derail this gravy train.
I can’t see what the criticism of Jones is here (“blowing off the issue”). It seems what he is saying is – we have and use homogenized data from the NMS’s. That’s available here. We don’t use the raw data. For that you’ll have to go to the NMS’s.
What should he have said?
In fact most of the unadjusted data is available in GHCN V3 unadjusted.
REPLY: The issue is reproducibility. Nick you’re a smart guy, if it is so easy to get all the raw data, and fully, exactly, reproduce what CRU does, then I suggest you do it and report back here. Partial reproduction would not be tolerated from us if we were criticizing the data set/methods, so you’ll need the full set. In the meantime, you should recall that CRU uses a mixture of raw and adjusted data. See the HARRY README file for some examples of the mishmash.
Lucia on her blog wrote: http://rankexploits.com/musings/2009/cru-data-lost-or-to-be-posted-in-a-few-months/
…at Roger Pielke Jr’s blog, I read Roger asked for CRU data and CRU told him this:
This strongly suggest their reason for not giving Roger raw data is it doesn’t exist.
In contrast, an article in Nature says this:
Jones says he can’t fulfil the requests because of confidentiality agreements signed in the 1990s with some nations, including Spain, Germany, Bahrain and Norway, that restrict the data to academic use. In some cases, says Jones, the agreements were made verbally, and in others the written records were mislaid during a move.
He says he is now working to make the data publicly available online. As Nature went to press, Jones was expected to post a statement on the CRU website to that effect, including any existing confidentiality agreements. Jones says any such data release “needs to be done in a systematic way”.
(Mind you, my understanding is that, because data were requested under FOI, the Met Office is required by law to contact those countries and ask them if they are willing to release the data. Happiness or unhappiness on the part of the Met Office is irrelevant. I am also under the impression that several of the FOI requests made this clear to the Met office back when they did not seem quite so happy to undertake this effort. )
But, I digress. . .
What I was wondering was this: Which data is Jones planning to release? The value added data they told Roger they still have? Or the raw data they told Roger they lost? Or am I totally confused? Does Jones have a personal archive of raw data that CRU lost? (This would not surprise me. I’ve known lots of researchers who have data on their how machines, back up zip disks etc.)
Anyone who can untangle this, please enlighten me!
===========================================
So it seems CRU had/may still have the raw data, but didn’t keep it, or they still have it and won’t share. Who knows? They are responsive to FOIA requests on the matter.
– Anthony
This is a shell game. Suppose you wrote to all the NMS’s and they all provided you with their data. Then you trend it using the same techniques that Hadcrut uses, and suppose you get a different answer. Without having the data directly from HadCrut, the NMS data is actually useless because you don’t know:
1. If the data the NSM sent you is identical to the data that they sent HadCrut.
2. If HadCrut made any mistakes transposing from the format the NSM sent to the format that HadCrut uses.
3. If HadCrut used all the data they were sent, or only some of it.
4. If the NMS sent you a superset of the data they sent to HadCrut.
….and so on. Just a shell game to prevent anyone from looking behind the curtain and discovering that the floating bloated head is just a wee mann pretenting to be a giant.
“I don’t say that Jones is to blame for the current situation. At least he tried to get permission from the NMS’s to release the data as he promised…”
Or says he did? I have zero confidence that this is true unless it’s confirmed from a reliable source – and I don’t include most people connected with climate science.
That’s what you’ve helped do for the integrity of your field, Jones.
Some skiiled statistician could compile a complete estimated reconstruction of the raw data of the Blimate Stations based on the likely distortions anticipated in the “homogenized” data, publish it and wait for “The Warmists’ ” objections. Then they could be challenged to prove the compilation wrong by bringing forward the real raw data for everyone to look at.
In the UK we had an expenses scandal with MPs.
What Jones is advocating is the same as each MP being soley responsible for their own expenses with absolutely no auditing or anything from a central group.
Jones couldn’t care less if people just made up data … in other words: “not my responsibility”.
So whose responsibility is it?
If Jones is not prepared to do the job properly have have properly audited data then he shouldn’t get a penny in grants.
His attitude is totally disgusting.
We need to turn over the temperature record authority to a statistical agency who has the expertise and technological power to make it an easy objective process.
There are billions being spent on climate research. Government research funding agencies should re-allocate $30 million or so of that to a consortium of national statistical agencies who will go back in and redo the series and keep track of it from now on.
That would have been quite a feat for thermometers graduated in whole degree increments. Drop a decimal point and I’ll agree.
Nick Stokes says:
June 1, 2012 at 4:10 pm
I can’t see what the criticism of Jones is here (“blowing off the issue”). It seems what he is saying is – we have and use homogenized data from the NMS’s. That’s available here. We don’t use the raw data. For that you’ll have to go to the NMS’s.
What should he have said?
In fact most of the unadjusted data is available in GHCN V3 unadjusted.
Because it isn’t simply a question of the data, homogenised or otherwise. It’s a question of the methodology used to arrive at the end result. If an individual cannot replicate the results, how can they be confirmed or improved upon? Not refuted, please understand, but replicated. Where, exactly, is the problem with that?
I am a bit naive here, I am not a scientist
but even if Jones did the homogenisation
or even if the sources did it in a consistant manner
and then it was all published.
so what ? ok, so it will give us ammo to shoot at the other side,
but where does it actually get us ?
we are still trying to measure the weight of an elephant by looking at one of its footprints
Chuck Nolan on June 1, 2012 at 1:36 pm said:
Why do they fear FOI?
—————-
Let’s consider this scenario.
You have filled out your tax records in a way that is entirely ethical and honest.
But you had a personal disagreement with a tax inspector who demonstrated complete ignorance of how your business operates. And is likely incompetent in any case.
That tax inspector has indicated up front he’s gonna nail you. He has an agenda.
And then he initiates a complete tax audit of your business that is going to take 3 months to complete and which is going to keep you at his beck and call for that time.
How cooperative are you going to be. Are you going to tell him to get f@&$d?
Elftone says: June 1, 2012 at 5:10 pm
“If an individual cannot replicate the results, how can they be confirmed or improved upon?”
Well, you can. And individuals have done it. You can get raw data from GHCN, the various SST sources and the NMS’s. But CRU doesn’t use it. It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?
So, the future of mankind hangs in the balance, but the data needed to understand the processes leading to the catastrophe, have been denied to those not within the current cabal.
History is full of such stories, that never turn out well.
Nick Stokes says:
June 1, 2012 at 4:10 pm
“I can’t see what the criticism of Jones is here”
Nick Stokes – defender of the indefensible.
Nick, maybe you should spend a little time to reacquaint yourself with the way real science works. When I was at school it went something like this: Do the experiment, observe the results, write about it explaining your methods in detail, outlining the results and draw your conclusions. Then hand it to the teacher (or peer review and publish depending how old you are!) and she/he would say whether it was any good.
May I draw your attention to the ‘explain your method in detail’ phrase in the above paragraph. This is the bit missing from Master Philip Jones report and teacher is not happy about it. May I also point out that teacher is not interested in the ‘post normal’ method of writing about the experiment, which is to leave out the ‘explain your method in detail’ part, and does in fact require Master Jones to tell the world exactly how he conducted his experiment, because otherwise the results are not worth a jot.
Back to school for you I’m afraid Nick as it seems your first spell at school was a complete and utter waste of time!
Nick Stokes;
I can’t see what the criticism of Jones is here (“blowing off the issue”). It seems what he is saying is – we have and use homogenized data from the NMS’s. That’s available here. >>>>
No. He’s saying where he got his data from and directing you to obtain it from the same source if you want to. What he is skipping over is that without seeing what data he actually used as an input, we don’t actually know who sent him what or if he used it or not. Shell game.
Why don’t we have a principle that any paper, study or major document paid for by public funds must include all supporting data, codes, and other backup?
Owen in GA says:
June 1, 2012 at 2:09 pm
I think the problems may be systemic in the sciences right now and that has me worried
========
The problem of false positives is widespread at present because nobody pay money to by a newspaper that says “nothing happened today”. Folks only want to pay money for results. So, if you want to have a job tomorrow, you select the data that shows a result. If someone asks to see your work, of course you are not going to give it to them. You are going to come up with every excuse in the book to keep it away from them.
Nick Stokes says:
June 1, 2012 at 5:30 pm
It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?
=====
It is zero work for CRU to archive their compiled NMS data so the results can be replicated. Their unwillingness to archive their data is part of a long standing patter of junk science labelled climate science.
It makes the whole profession look bad. The same thing happened to other professionals that didn’t clear up their acts. There was a time that lawyers, bankers and politicians were considered honorable professions. Now they rate lower on the truth scale than used car salesmen.
Scientists are quickly heading into the same toilet. It only takes one bad apple to spoil the barrel. Unless scientists as a whole speak up and place integrity ahead of money and fame, they too will go down the drain.
Dale Hartz says:
June 1, 2012 at 5:44 pm
Why don’t we have a principle that any paper, study or major document paid for by public funds must include all supporting data, codes, and other backup?
=========
Not if the results will tend to show that the scientists involved are incompetent to the point of fraudulent. No university is ever going to willingly release such information as it will be financially devastating.
davidmhoffer says: June 1, 2012 at 5:36 pm
“No. He’s saying where he got his data from and directing you to obtain it from the same source if you want to. “
Well, I may be reading it wrongly, but this post seems to say that he got homogenized data from the NMS’s, and you can get that data, which was actually used,, from CRU. What people here seem to want is raw data, which is not what CRU used.
Nick Stokes;
Well, you can. And individuals have done it. You can get raw data from GHCN, the various SST sources and the NMS’s. But CRU doesn’t use it. It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?>>>>
1. They had to compile it for their own purposes, so suggesting there is ANY additional work to be done on that score is silly; and
2. Yes. Already being compiled, all theny need do is release it.
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says.
D. J. Hawkins says: @ June 1, 2012 at 5:07 pm
….That would have been quite a feat for thermometers graduated in whole degree increments. Drop a decimal point and I’ll agree.
_______________________________
And that is why this whole “adjustment” of historical data is so much nonsense. The numbers are good to +/- 1 degree at best and these idiots are trying to squeeze a 0.07°C/decade warming out of the data.
They all need remedial math classes in my opinion. Taken while enjoying the hospitality of the government.
See: http://joannenova.com.au/2012/03/australian-temperature-records-shoddy-inaccurate-unreliable-surprise/
http://strata-sphere.com/blog/index.php/archives/11420
Richard Lawson says: June 1, 2012 at 5:35 pm
‘Nick Stokes says: June 1, 2012 at 4:10 pm
“I can’t see what the criticism of Jones is here”’
“Nick Stokes – defender of the indefensible.”
In fact I’m not defending anything. The reason is that people can’t specify the accusation. What is this post saying that Jones has done wrong? What should he have done?
davidmhoffer says: June 1, 2012 at 6:24 pm
“They had to compile it for their own purposes”
How do you know that? It seems quite contrary to what the post says. CRU obtained homogenized data direct from the NMS’s. They did not use raw data.
LazyTeenager says: @ June 1, 2012 at 5:27 pm
You have filled out your tax records in a way that is entirely ethical and honest…..
_________________________________________
YOU have never dealt with an IRS audit have you?
What you do is hand over your information and smile sweetly as you do so. Then you call your tax account and and if necessary your lawyer.
I have 20 years of tax data for my business including every single receipt stored in such a way that I can hand them the correct boxes within minutes of being asked for it. EVERYTHING is stored by line item per the tax code and a copy of the tax code for that year is included in the boxes with the receipts, journals and my copy of my tax filings.
So in other words it would take me less than 10 minutes to hand over the tax information. In the case of large businesses the companies provide offices for the IRS tax auditors in their office buildings. That is from a friend who was CFO for a large corporation.
Sorry Lazy, That dog don’t hunt either.
I find it sad that so many scientists have such an unhealthy attitude to errors.
All humans make mistakes. If you publish something, you can be fairly sure it contains errors. Sometimes the errors are significant, sometimes they are not — but they are always there.
If you find a mistake in my work, that isn’t a failure. It’s a success. Every error found and fixed improves the quality of the work.
With this in mind, why would you make it hard for people to reproduce your work? You should make it as easy as possible. If more people check the work, more errors will be found and fixed.
LazyTeenager;
You have filled out your tax records in a way that is entirely ethical and honest.
But you had a personal disagreement with a tax inspector who demonstrated complete ignorance of how your business operates. And is likely incompetent in any case.
>>>>>>>>>>>>
The tax inspector doesn’t NEED any understanding of how your business operates. What he NEEDS is your financial information provided to him based on GAAP (Generaly Accepted Accounting Practices). If you did that, there’s no problem.
LazyTeenager;
That tax inspector has indicated up front he’s gonna nail you.
>>>>>>>>>>>>
You should be very excited if this is the case because as a government official announcing such a thing he has basically convicted himself of abuse of official powers and the judiciary will take a very dim view of that when your lawyer hauls the issue into court.
LazyTeenager;
And then he initiates a complete tax audit of your business that is going to take 3 months to complete and which is going to keep you at his beck and call for that time.
How cooperative are you going to be. Are you going to tell him to get f@&$d?
>>>>>>>>>>>>>>>>
I’m going to cooperate as is required of me by law. If the tax inspector abuses his authority, he will pay the price. In your example however, the shoe is on the other foot. An audit has been requested of Jones and the CRU, that audit has a basis in law as well as accepted scientific principles, complying with the audit is not onerous as the data has already been compiled and exists in electronic format which can be shared with but a few mouse clicks, and yet Jones and the CRU continue to hide behind legal processes in defiance of the law of the land. Why?
The only logical conclusion is that their data was not assembled in an ethical and honest manner. Defend away buddy, all they have to do to exonerate themselves is release the data.
Phil. says:
June 1, 2012 at 6:26 pm
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says;
>>>>>>>>>>
Complere with sources and metadata?
If so, I stand corrected.
Manfred says:
How do these folk sleep at night?
Soundly. Whatever happens, their asses are covered. Even if the entire gig is busted a decade from now, Phil can simply point the finger at the NMS’s and their “bad data”. His analysis was, of course, perfectly sound.
Gail Combs says:
BINGO!
The problem has metatasized to most fields in science. Science is getting a black eye and it is fully justified.
It’s true. But part of the problem lies outside of the science and the scientists. A lot has to do with the manner in which science is funded. Increasingly, government as well as non-governmental agencies are only willing to fund research which supports preordained conclusions. As we’ve seen with climate work in particular, scientists are under enormous pressure to ensure their studies are fully compliant with the party line. It should not be a shock that such funding policies drive an increase in corruption and fraud.
I am not excusing the behavior for a second. But we should be clear that the solution lies less with science/scientists and more with the politicians and others in public administration.
[b]The need for a journal that demands all data, (used and excluded) up front, along with methodology, code, and supplementary material to ensure the work is reproducible, before even considering a paper for review is becoming clearly obvious.[/b] – [i]Anthony[/i]
NEEDs Repeating for Mann, Hansen, Gore, Clarke, Jones, Briffa, et.al.
David, there’s a link to the data with description in the original post, so yes you stand corrected.
Tom in Worcester: June 1, 2012 at 2:22 pm
“Does anyone know the status of the FOI request from U of Penn/Mann? Here’s to hoping that they reveal some really horrifying behavior on the parts of “Team”. It seems to me that these guys are always “weasling” when it comes to the release of all data. A comuppance would seem in order.”
Well, it would hardly be very effective to submit a FOI to U of Penn regarding Michael Mann because, for everyone who does not know the difference (BUT SHOULD), U of Penn is a private, Ivy-League School of exceptional repute located in Philadelphia, PA that has NO association whatsoever with Michael Mann of PENN STATE infamy (Penn State being a pseudo-public PA state institution located in University Park, PA, about 200 miles northwest of Philadelphia, PA.). Please do not disparage an institute of outstanding academic merit and integrity by falsely associating it with the likes of Michael Mann!
Phil. says:
June 1, 2012 at 7:32 pm
David, there’s a link to the data with description in the original post, so yes you stand corrected.
>>>>>>>>>>>>>>>
Well not so fast….
I followed the link.
1. It lists a whole bunch of sources, about 1/4 of which have no link. No link = you have to request it yourself from that source, and you have no way of verifying if what they sent you and what they sent Jones are the same thing.
2. Of the sources that do have links, so what? We’re trying to VERIFY work here. Unless we see the actual grid data on a grid by grid basis, we don’t know that the numbers Jones used match those in the link to the source.
3. Some of the source descriptions are plain silly. Like “Data from various NMSs received in 2010”. Yeah, I can track that down in a snap….NOT!
4. At the bottom of the list of sources is a link to the a UK MET Office formatted version of the station data. I followed that too. First of all, it is for HadCrut3, not HAdCrut4. Second, the first line in the description says “subset” so it isn’t even complete and hence useless.
To be fair, there is also a link to HadCrut4 station data in a zip file. I didn’t bother opening it, but will take it at face value that this is the actual data they used. My arguments above still stand. We have no way to verify if the numbers they used match a considerable amount of the data they claim they were sent, we don’t know what they discarded and what they didn’t, we don’t know if they screwed up when they transposed the data they were sent into a format that would match the rest of their data…. In brief, if I missed something obvious, please correct me (yet again) but I just do not see how this amounts to disclosing their data. Itz just a bunch of numbers that they claim to be accurate with no way on earth to check the accuracy.
LazyTeenager says:
June 1, 2012 at 5:27 pm
Chuck Nolan on June 1, 2012 at 1:36 pm said:
Why do they fear FOI?
—————-
Let’s consider this scenario.
You have filled out your tax records in a way that is entirely ethical and honest.
But you had a personal disagreement with a tax inspector who demonstrated complete ignorance of how your business operates. And is likely incompetent in any case.
That tax inspector has indicated up front he’s gonna nail you. He has an agenda.
And then he initiates a complete tax audit of your business that is going to take 3 months to complete and which is going to keep you at his beck and call for that time.
How cooperative are you going to be. Are you going to tell him to get f@&$d?
======================================================================
Sooo …. what you’re saying is, “Only “The Team” has the secret recipe for making Hockey Sticks so leave them alone!”?
Phil. says:
June 1, 2012 at 6:26 pm
Not true David, Jones is making available the data which he used so anyone who wishes to replicate the CRU’s work can do so, exactly as Nick says.
=========================================================
“For raw temperature data you have to contact the NMS’s.”
Sooo … He’s making available the data he has by telling people to get it from someplace else. Got it!
Just two thoughts for everyone:
Simplicity and replication.
Ask anyone who has taught anyone something. If you can not dumb a subject down and teach an advanced subject to a dumb person, how can you possibly think you are smart enough to control science which is about simplicity and replication?
Make your work easy to understand and make the information necessary to replicate it easy to find and easy to understand. If you can’t do that for climate sceptics and the rest of us who are advanced in the understanding of this field, how can you possibly think that the average person can likewise follow what you are doing?
I just want to know how I can obtain the same information P. Jones got when its likely that even if I could somehow magically obtain all of his data from over 100 sources all over the world…..that I could never figure out if its the same data he used or whether the data was changed or some other unknown issue.
And that is the heart of the matter. It really is that simple, so obfuscation by some others aside, lets just remember that simplicity is the key and to win this battle which obviously we learn day after day is not about science but about politics, that if we keep the message simple and tell the truth we will win eventually win. Again, telling the truth is the key there. Keep it simple, and sceptics win. Explain the complicated questions as approrpiate, but keep discussions simple and concise and not confuse people, but rather educate them.
Its only in education that our species advances, through obfuscation and other morally questionable techniques we will sink down to a lower level of intellectual discourse everytime. Case in point, Phil Jones. Notice how the more he lies the more stupid he sounds because he makes it more and more complicated and as its said: “Oh the webs we weave when we first try to deceive. ” This could be a case study in why lying is bad for children. Point at the funny man children, because this is what you become if you lie!!!
Perhaps that last part is a little over the top, but I have wondered myself where do climate scientists see themselves in 10 years? They have to realize by now that they are just jokes to the common man who if you even mention global warming to them, they start telling a joke….and it has only gotten worse recently…
Why do you think that President Obama will not even mention climate change or global warming anymore?
Part of the problem is the lack of politicians with a science background in the west. They are largely lawyers who are too easily bamboozled by what looks, on the surface, to be rigorous scientific research. That and the fact that the ‘scientists’ are telling them what they want to hear. The electorates are waking up to the scam because they now realise how much it is costing them. To speed up the process of killing this climate hysteria, all it would take would be for one brave government to call out that the emperor has no clothes.
The reputation of science is being undermined by eco-pseudoscientists. Try reading the environment sections of the Guardian/BBC (same site really) to get an insight of the ignorance fuelled drivel being pushed by these lunatics that want to send us back to the dark ages. More worrying is the shrill response of the acolytes.
We want two things. First : Any paper that is not completely reproducible with the information available should not be allowed to be published and should not be allowed to be funded by money from the public. Second : All papers and data funded by the public should always be freely available,
Talk about job security…
Nick Stokes says:
June 1, 2012 at 5:30 pm
Well, you can. And individuals have done it. You can get raw data from GHCN, the various SST sources and the NMS’s. But CRU doesn’t use it. It may be inconvenient to compile NMS data, but is CRU obliged to do it for you?
Yes, they are. Being publicly funded (in part or in whole is irrelevant), and having already done the work, there is no valid reason they shouldn’t. As my lecturers used to say, “show me your work”. That phrase does not mean “I disbelieve you” – it means “prove that you understand, and so that I may replicate what you have done”. If the work is solid, good, if there are unnoticed errors, they can be fixed, and everyone will benefit from an improved understanding of the field. It is to everyone’s advantage. Where is the problem in that? Why is it necessary to make it difficult for others to – let’s use another word here – confirm their work? It doesn’t make sense, and it leads to suspicion as to motive, which is thoroughly counter-productive.
It is now a common comment that climate science is simply politics. OK, then, rather than complain, consider political action. For example, there is one senator, James Inhofe, who is a strong skeptic —
http://inhofe.senate.gov/public/
At the moment the senate is controlled by the left, but that could change next year — the electoral chances of the current administration are declining. Next year Sen Inhofe could submit a data transparency bill with a chance of passage.
This blog is a good example of “new media power” and could consider ideas for Sen. Inhofe on appropriate language and sanctions. The obvious idea is that any tax-supported non-defense data should be available to anyone in a raw state, and that all scientific journals establish an appropriate policy to support the concept.
How absurd!
As an ex-Federal employee I can speak to FOI requests. They are NOT an audit, ever!
They are a request for information. If you are organized at all, e.g. Gail in her post above, complying with a FOI is a snap.
If there are questions, say some information might/does contain propriety or personal information, you calculate about how long it will take to redact, estimated cost and hand that back to see if the cost to retrieve is acceptable. If you not have the information requested, you state that fact, why and if anyone else might be the correct source. Yes, it is an irritating interruption of your daily work, but that is all it is.
If you are unorganized as philly J. states, then you’ve screwed the pooch and you have to report back how long it will take to find and review the requested information. Be advised, the courts do not accept “I’m unorganized and it took forever to find” as an excuse for high FOI charges. They’ll decide in favor of the requestor if they believe the costs are absurd. You’re actions are your bosses problem as you explain why you are in non-compliance and over budget.
Well, that’s how it works where there is accountability. Something that the CAGW alarmists and trolls lack.
Arguing that you shouldn’t have to release something for whatever reason should be easily decided at a meeting with the FOI staff in your organization. The FOI staff are trained in what may be denied. Any further disagreement gets sent to the lawyers for a final say.
Reading the climategate emails, you can see where Phill J. ignores, delays, denies, ignores FOI requests. You can understand the frustration of the FOI officer as he first tries to get FOIs completed properly and eventually joins with the dark side. My perspective is that the FOI officer just wanted to make his job easier by not fighting Phil.
From Phil Js statement above; he has relinquished all claims to science. Phil’s denial of following proper science practice or accepting responsibility for maintaining his source material means he is not a scientist and is not producing a product worth using. CRU’s climate department is now moot.
haven’t these guys been caught cooking the books before?…temperature varies by day and by night, from season to season, and from place to place….most weather stations record only the max and min temps for the day…the midpoint of these is not the average for the day…also weather stations are very unevenly distributed…this whole concept of average global temperature is screwed up
Jones has done all that people like me asked of him.
he has produced the data as he used it, and the code used to produce the results.
His choice to use adjusted data is entirely rational.
Further he lists the source so you can check that as well.
Basically, CRU do no adjusting.
Now, comes the question about raw data.
1. GHCN provides unadjusted data
2. GHCN daily provides unadjusted data
3. an entirely new data base is being built that wil document data all the way back to paper forms
where they exist.
If you havent worked with raw data, get your hip waders on. When you finish slogging thru it
you’ll discover that homogenization makes little changes here and there, (canada is interesting )
but overall, its warmer now than in the LIA. If you focus on small areas you might find more substantial changes, but its still warmer now than when washington crossed the delware.
“The need for a journal that demands all data, (used and excluded) up front, along with methodology, code, and supplementary material to ensure the work is reproducible, before even considering a paper for review is becoming clearly obvious.”
Capable parties only need the data.
I argued many years ago for the need for a climatological equivalent of CERN, the global nuclear physics community.
I suspect very strongly that effective data storage, access and FOI processing would be more effective in such an organisation.
What is critical, however, is that oversight is carried out by non-scientists and that policy making is separated healthily from scientists.
They have managed to duck the issue of ‘raw’ data and ‘availability’ by passing the buck to supplier NMS’s.
This doesn’t help anyone – and moreover, does this imply that any ‘supplied’ data, as in not directly under a researchers/government departments control cannot really be verified?
So, for example, if the nice folk at the Maldives supply met data that shows a massive warming trend – does that simply get taken as read?
The picture I’m getting is that the ability for fudging/manipulating has just been increased multifold…….
worse still, it only takes a few ‘SNAFU’s’ for any and all raw data to suddenly be ‘lost’ forever within supplier NMS’s! wink, wink, nudge, nudge…..
ed said:
“@Peter Miller
I would argue that the use of Climate and Science in a sentence is definitionally not possible.”
How about: “I am going to work in the Climate field, so I won’t need to know any Science”?
Given the issues the Kiwis are having with their NMS, how can Jones et al rely on any of the data that is being provided by them?
It’s all very well coming up with a graph of the data you have been provided but you (Jones) have a duty to test that data to ensure its veracity.
Has Jones ever been asked to justify his trust in the data being provided?
My old science teachers had it dead right. If you don’t show your data and working, you get zero marks. We need to agree on a (preferably common) word,which will immediately convey that the “paper” being used to “support” some rubbish claim is null and void and not to be trusted because of any such data hiding / cheating / dishonesty in its production. We also need to be 100% inflexible in applying that word publicly whenever such a “paper” is referenced. No data, no credibility. No argument.
I would suggest that Anthony could have another page listing all such null papers, except that in climate science it would be a darn sight quicker to list the valid ones.
Steven Mosher so were is Jone’s Chinese UHI data ; last we heard ‘the dog had eaten it ‘
And that is before we get to Jones deleting information and asking others to delete information and of course its view that he does not like given information out becasue others may find something wrong with it , which ironically is what ‘critical review ‘ is supposed to do .
The idea Jones follows good practice is frankly a joke , his long sold out doing science to doing advocacy which he has found far more ‘rewarded’
Nick Stokes largely what is being asked is that these ‘professional scientists’ should meet a mini the standard that is expected of their own students, which is actual rather low given the great claims of urgency and certainty they make .
Is it really OK that Jones and co , self proclaimed leaders of their field , cannot work at a standard which we expect a undergraduate to be able to do to pass their degree course ?
Steven Mosher,
“His choice to use adjusted data is entirely rational.”
No, it isn’t rational. Homogénlisations have a huge unexplained bias (0.5 ° C for the twentieth century).
My apologies to the mods for the length of this. It comprises a long note to myself that will be the basis of a future article. As such it is episodic rather than a narrative, but the gist of it can be readily seen that there has been systematic adjustment of raw historic temperature data (perhaps for good reason) of virtually all the old records. The pdf’s referred to were sent to me privately so do not appear.
—– ——- —
Note to myself Feb 2012
Four 17th and 18th Century Metereological associations
Brief extract from these notes;
“The research papers detailing the systematic correction of early temperature records- which are invariably found to be ‘warm biased’ and subsequently adjusted downwards as they did not agree with the model expectations- are detailed in the contents page of the Camuffo/Jones book
‘ Improved understanding of past climatic variability from early daily European instrumental sources.’ Which was produced as a result of the IMPROVE project.
As well as those detailed here, many of the old records seem to be contained in the book ‘ History and Climate’ by Jones and Briffa. They would be worth investigating further to determine the reasons for the downwards adjustments.”
In my article ‘The Long Slow Thaw I made this note, intending to return to it at some point as it covered a subject of which I knew nothing;
http://judithcurry.com/2011/12/01/the-long-slow-thaw/
Note 4
It is interesting to note that in 1780 the Societas Meteorological Palatine of Mannheim, a standardised network of 40 observing stations, was set up in Germany and other European countries and a small number in the USA, all equipped with comparable instrumentation with standard instructions for use. This came to an end in 1795 with the siege and capture of Mannheim.’
Its subsequent investigation has allowed a number of pieces of the historic jigsaw to fall together for me, by putting this paragraph from one of my earlier articles into context,
‘Frederik became King of Prussia in 1701 and immediately set up a measuring station that became Berlin Tempelhof, one of our oldest records and this started a rash of similar stations that caused Samuel Horsley to comment in 1774: ‘The practice of keeping meteorological journals is of late years becoming very general’.
There was a dramatic upturn in weather station numbers and the period was characterised by attempts in the 17th and 18th Century to obtain consistently derived weather data as part of international collaborations, of which four were of some note and are covered in these brief notes. The establishment of the collaborations was largely driven by royal patronage, an increasing fascination with science, the birth of various scientific institutions such as the Royal Society and the invention and development of a number of related instruments including the barometer and thermometer.
The international collaborations were generally made in an attempt to ensure weather information was derived in a consistent manner with instructions given by the promoters to the various subscribers as to where instruments should be placed, what type should be used and frequency and time of reading.
The first brief international brief efforts were began in Italy 1653/1654 and was called Rete medicea with stations in Italy, France, Austria, Germany and Poland all reporting to agreed parameters, including placing the specified thermometer on an external north facing wall. It ended in 1667.
The second association was set up in 1723 by James Jurin secretary of the Royal Society according to strict parameters and with instruction to fill in all six columns of a journal recording everything from wind speed, temperature, precipitation , pressure, with a section for general weather conditions. It was suggested the thermometer be kept indoors in a north facing unheated room as had been the custom in England( by some) since 1660. There were some 20 stations in England, Europe- including Italy and Austria- North America and India. The journals were sent annually to the Royal Society for collation and still exist. The network ceased in 1735
The third was a French sponsored association which commenced 1776 and was called Societe Royale de Medecine of Paris. It was particularly well organised, had some 50 international and French participants and lasted until 1793 .
The fourth and best organised of the lot was The Societas Meteorologica Palatine of Mannheim which lasted from 1781 to 1792. There were some 39 sites including Greenland most of Europe, Russia, and North America . There were strict parameters about the placing of thermometers outside and at an elevation, and there seems to have been considerable use of screens which were first mentioned in regards to the first association in the mid 17th century.
D Camuffo wrote succinctly of these various associations in his chapter ‘History and correction of long temperature series.’ It is worth reading the section from page 9 to 12 although it is Italian centric and all the details perhaps do not seem to quite match up to original sources.
http://books.google.co.uk/books?id=eEOFQy86zzEC&pg=PA11&lpg=PA11&dq=james+jurin+an+invitation+to+form&source=bl&ots=3Lr0jYCSFq&sig=Bc0xTiWxrlXv_-DjUVdSYGQXhZU&hl=en&sa=X&ei=uwRFT-StFMex0AWnuuSJBA&ved=0CCgQ6AEwAA#v=onepage&q=james%20jurin%20an%20invitation%20to%20form&f=false
Detailed information on each of the Associations
Greater details can be obtained from the series of attached pdf’s which were compiled by the Met office some thirty years ago, which in turn were often derived from the original documents translated from Latin.
1) There are no further details available at present on the Early Italian association as this was identified only after research for the rest of this article.
2) The Royal Society . Attached is the pdf ‘James Jurin 1722’ which is a copy of the original letter from James Jurin of the Royal Society in 1722 asking observers in other countries to participate in an international effort to record weather data in a consistent fashion.
Title; ‘An invitation to form a meteorological system of observations by world-wide
Agreement’
Abstract; ‘But just suppose there were observers in suitable numbers, appropriately distributed over a large area of the earth’s surface; and, ultimately, someone to collate their various diaries, and make notes of the agreements and discrepancies; we should then before long have a Meteorological History covering a number of years, of a kind that could hardly be imagined or even dreamed about today. This would help to discover in what regions the winds originated and their subsequent pattern over the earth.’
The Royal Society lived up to its motto ‘nobody’s word is final’ as their aim was to either prove or disprove this theory, by the use of reliable observation. Edmund Halley had been a particular proponent of the theory of wind.
It can be usefully read in conjunction with the pdf ‘Haverfordwest weather register’ which provides a great deal of further background information, although it concerns itself mainly with the English readings All the weather data from Jurins international association still exists.
3) The French association is well described in the pdf ‘ eighteenth century source of meteorological data’.
‘The Society had over seventy correspondents in 1784 and this might well have been the maximum as a slight decrease occurred in 1786, and in 1793 the Society, like many other learned bodies of the Ancien Regime’, was suppressed by the revolutionary decree which abolished ”
Toutes les academies el socieles lilleraires patentees on dotees par la nation.’
The French are attempting to put these records on line-see http://meteo.academie-medecine.fr/
The Met Office are assisting them. I have attached an email from the archivist at the Met Office at the end of this document describing the current situation.
4) The Societas Palatine of Mannheim was the best organised and most consistent of all, lasting from 1781 to 1795 when it came to an end following the siege of Mannheim during the wide ranging Napoleonic wars.. At its peak it had either 39 stations (Camuffo) or 50 (Kington of CRU) observation stations. The history and the locations of stations is very well described in the fascinating pdf ‘societas meterologic palatina’.
The station locations are also described by the personal email from the Met Office quoting David Parker, who was responsible for setting up CET from 1772. (See foot of this paper under ‘emails’.)
I understand the Mannheim data still exists and indeed is much scrutinised and adjusted by such as Camuffo and Phil Jones separately, and also jointly through an EU funded collaboration between them in 2002 in the IMPROVE project when they examined 7 long European data sets, of which four were from the Mannheim association.
Other associations
There were others who drifted in and out of these organised efforts together with some who formed less formal associations such as within the Scandinavian countries. The records of Stockholm from 1750 and Uppsala 1720 are especially interesting. Uppsala records the great warming of the climate at its start date by noting that mulberry trees were grown in the botanic gardens. By the time Stockholm started reporting thirty years later the warm spell had subsided and the mulberry trees destroyed by the returning cold. Botanic gardens are a good source of background information of periods of warmth and cold due to introductions of new plants.
However, many stations never became part of any association.
Could the existing global records be extended?
A reconstruction of a ‘global’ temperature to 1750 could reasonably be made-although with a strong European bias and large error bars-however there are several stations within the groups, including CET- reckoned to be a good proxy for world temperatures. A reconstruction to 1720 could also be attempted to identify the widespread warming trend at the start of the century but this would need to be augmented by documentary records and could do nothing more than demonstrate the temperature ‘tendency but not the precision’ (Lamb)
General comment on the early historic records
The research papers detailing the systematic correction of early temperature records- which are invariably found to be ‘warm biased’ and subsequently adjusted downwards as they did not agree with the model expectations- are detailed in the contents page of the Camuffo/Jones book
‘ Improved understanding of past climatic variability from early daily European instrumental sources.’ Which was produced as a result of the IMPROVE project.
As well as those detailed here, many of the old records seem to be contained in the book ‘ History and Climate’ by Jones and Briffa. They would be worth investigating further to determine the reasons for the downwards adjustments.
Many of the sites and the instrumental readings of these and many other historic stations are recorded in my web site.
http://climatereason.com/LittleIceAgeThermometers/
There certainly seems scope for an independent review of these historic records as the examination of old records and their subsequent reconstruction seems to be very specialised, conducted by some 20 regulars of which Camuffo and Jones have been to the fore.
How accurate are historic records?
I wrote at some length of the evolution of weather recording and of the inherent problems of recording temperatures, in two articles which can both be accessed from this link;
http://wattsupwiththat.com/2011/05/23/little-ice-age-thermometers-%E2%80%93-history-and-reliability-2/
I have come to believe that there is some case to be made that the older historic readings are more accurate than those from the mid 19th to mid 20th century- especially when consistent standards and methodology have been laid down , as with associations, as instruments were of scientific quality, observers often well trained, and a great deal of prestige attached to scientific accuracy due to the patronage of royalty and the wealthy. In addition, many of the old records have been very thoroughly scrutinised and (over?) evaluated and amended. Having said that there are many inherent problems in temperature readings as described in part two of little ice age thermometers linked above, and bearing in mind their crucial importance to determining policy matters , it is doubtful if a good case can be made for using any instrumental records until modern times-1980’s- with the advent of digital stations and satellites, although each of these methods have their own inherent problems.
tonyb
Is your irony/hyperbole detector busted? Try reading old Dilbert strips for an hour a day for a month. That should help.
Steven Mosher says:
June 1, 2012 at 10:32 pm
,,,its still warmer now than when washington crossed the delware.
I hope so — it’s almost summertime, for gosh sakes…
Dale Hartz says:
June 1, 2012 at 5:44 pm
Why don’t we have a principle that any paper, study or major document paid for by public funds must include all supporting data, codes, and other backup?
_________________________________
We do. It is called the Scientific Method and that is the problem. Jones et al are not following the Scientific Method and the editors of the Scientific Journals should have tossed the papers back at them and told them SHOW YOUR WORK
If you do not want to show the work it is not published and it is a “Trade Secret” but it is NOT SCIENCE
You might know that the Australian Bureau of Meteorology was asked by New Zealand to review their temperature data a couple of years ago. So far, we have not been able to obtain more than a paragraph or so of comment.
Meantime, the Australian BoM has moved through at least 4 phases in the last 20-30 years.
1. A version largely compiled from previous publications, some weather stations and little metadata.
2. A homogenised version, started by Simon Torok as a PhD thesis.
3. A reduced set of about 100 stations in a High Quality data set, again with little metadata for sites in general.
4. A version of about 100 stations, not many overlapping with the HQ set, released a few months ago under the title Acorn-SAT.
For a preliminary look at the latter two of these versions do visit Ken Stewart’s works:
http://kenskingdom.wordpress.com/2012/03/13/near-enough-for-a-sheep-station/
http://kenskingdom.wordpress.com/2012/05/14/acorn-sat-a-preliminary-assessment/
You might then form an opinion, not only about Phil leaving it to the NMSs, but also about the need to recalibrate proxies as applicable in recent papers like Gergis et al 2012 that Steve Mc noted.
I look forward to a Daly-type study that simply selects a limited number of weather stations worldwide, where the metadata of each can reasonably be checked. I still refuse to believe that we need thousands of weather stations for a global picture. Ilarionov at Heartland 2010 showed that Russian records could be captured with good accuracy by just four stations.
Without individual clean bills of health for weather stations used, we will never solve the UHI. But there are many clues on the way, that suggest the homogenized records have NOT got UHI adequately sorted. Have a look here particularly at Ilarionov’s superb presentation
Owen in GA says:
June 1, 2012 at 2:09 pm
I don’t know. We all dump on climate science, but I have heard some disgruntled grumbling from other fields that journals are not doing a good job of making sure the work covered in the articles is reproducible. I think the problems may be systemic in the sciences right now and that has me worried.
_____________________________
I think you’re right to be worried, Owen.
I think it would be helpful if reviewers pre-publication comments and questions were also made available for general consumption.
I don’t think this would compromise anonymity more than it already is. [Such as it is, where the identities are often knowable by authors]. It would be an aid to readers, and also allow them to make a judgement about the quality of the Journal and the reviewers. It might also prompt reviewers to greater diligence.
If public sources of data (such as NOAA) do not include the raw data for public scrutiny, we are just reshuffling second had merchandise and I think are targeting the wrong folks here. Those that have taken the raw data and homogenized it, sanitized it, and then slow cooked it are just doing what researchers do with raw data. Those whom we have payed to collect the raw data from surface stations and to keep it safe and in original condition are the ones who should be investigated. This is a plain and simple fact.
The next president of the United States should force these government funded agencies such as NOAA to account for where the untouched raw data is (or has gone to) and then be held accountable on a yearly basis for a data audit to insure we will not be losing raw data from here on out. This is a reasonable request from every private tax paying citizen. We pay you to keep that raw data in pristine condition. And it appears we must now force you, the government, to show us whether or not you have done that. It is our due diligence.
Has science and the scientific method really changed since I was a physics undergraduate? I remember the huge amount of work that went into producing your results from the lab work. Preparing the data for presentation took far longer than the actual experiment. (1) The method had to be described in detail, (2) it was mandatory to quote the errors in all measurements taken and (3) the method of calculating the errors in the final results also had to be shown.
Miss out any of (1)(2) or (3) and your work would be returned for a re-show, no exceptions. That rigour is what instilled in me respect for the scientific method. What the ‘Team’ is doing to science makes me sick because, in the public’s mind, it tarnishes all science. The sooner these frauds are held to account the better.
Anthony, I have a suggestion.
We need two on-line publications: the Book of Reproducible Results and the Book of Unreproducible Results.
It is hard to keep track which papers are published with no supporting data or methods, or with partial data and incomplete methods (whether by design or incompetence). The point is that there are too many important-to-the-field papers being published reporting results are not reproducible by other researchers.
When one gets into a discussion there are frequent references to papers that are definitely in the Journals, but whose result are not reproducible. This is an affront to science but the authors do no seem to be bothered by this, for whatever reasons they have.
On your links to the right of these blogs there is a list of sites binned according to temperament and content. Would it be possible to create there a link that goes to a list, at least two lists actually, of important-to-the-discussion papers divided into two major groups: those which contain archived data and methods and which can be used to reproduce results, and those which do not. A third list might be papers for which the data is promised or is available on request.
Before allowing debaters to get away with citing-in-argument the result of works that are not reproducible, we can have a quick look-up of the paper by simple CTRL+F for part of the name. If it appears in the list of ‘papers with unreproducible results’ then we can legitimately respond with a comment that the works may be important but that the results are not reproducible and are thus suspect until independently verified.
The list might be in the form of a simple chat so anyone posting on that chat can simply post the link to the paper and its official name. That chat can be searched by anyone clicking on the list.
The motivation here is to take a long term position on this issue: the gradual erosion of scientific norms for the acceptance and publication of papers which show how the researcher got the results must be addressed by the academic community at large. With the entry into the field of Climate by many well-informed non-specialists, we should establish the tools needed to assist their activities.
Given the very wide influence of WUWT, it may have inflience in the long term if a large number of people start insisting in debate that only works which meet the norms of formal science be considered.
Yet researchers are glad to use the Surface Stations dataset, ready or not. I guess some people are more equal than others.
Follow up correction: it would be better to say:
“papers with unreproduced results” not “unreproducible”. There is a big difference, the point being to encourage people to include their data and methods when the publish.
It is quite possible that pressure on major journals like Science and Nature will see them stop the silliness of promoting AGW with reams of papers filled with un-reproduced and un-reproducible results. The effect of the present system is to stifle debate because if results are not reproducible, one can hardly engage in scientific debate about them. That was I believe the whole point of having a data archiving policy in the first place (now in tatters).
Dr. Phil Jones: “This is how I will avoid responding to requests to see my data from now on”
These are not the climate frauds you are seeking. Nothing to see here, move along, move along.
It is interesting that the Iceland Met certainly make homogeneity adjustments to their raw data, and yet GHCN still insist on making further adjustments. Presumably this applies to many other countries.
(I actually have access to the original data here)
http://notalotofpeopleknowthat.wordpress.com/2012/03/14/reykjavik-temperature-adjustments/
There has been enough trouble lately in various areas of science with “scientists” who fabricated results just to fit their pet theories and done enough harm with it.
Not showing the raw data and the methodology in detail to make the work reproducible should be reason enough to remove grants to those scientists. Yes, they may do private work and research but not on public money.
And indeed this should be the criteria to admit or not the papers for IPCC submission.
@Larry Plume P.
I agree completely. This is a basic rule of engagement – you take public money, you show what you did with it. To hide behind ‘we are still working on a new publication’ does not entitle one to the private use of public goods (the data and methods developed with public funds). It is patently obvious that some of this ‘science’ is defective and the authors are either misguided, incompetent or crooked. Either way, they are ruling themselves out of receiving further grants. If you look at the terms of, for example, Department of Energy funding documents, it is quite clear that one is not entitled to hide the methods and data while still producing results that will be accepted.
There is a Data Quality Act in the USA that can be applied to these cases. The methods used to make scientific determinations have to themslves be valid and provably so. “Climate science” is not exempted from complying with the requirements of the Data Quality Act with respect to methods. If you produce works, you have to use methods that are valid. If you do not show your methods then it is obvious your methods cannot be checked for validity. If you method is invalid, or cannot be validated, you can’t be compliant with the (very reasonable) mandated need for due diligence.
It surprises me that so many US gov’t Departments have not demanded authors receiving public grants comply with the Data Quality Act. I guess it is optional for Journals (in spite of published editorial policies that turned out to be ‘flexible’) but it is not, in the case of the expenditure of public funds.
Stephen Brown says:
June 1, 2012 at 1:24 pm
Without the raw data being available to all, no reliance can be placed on any conclusion drawn from any study using the unavailable data,
End of story.
++++++++++++++++++++++++++++++++++++++++++++++
Now they can do what they want, say what they want, and NO ONE CAN PROVE THEM WRONG…
Just like Obama’s EO on scientific integrity, no scientist (or fraud) who is employed by the government need fear an FOIA request… Agenda trumps Truth….
WE ARE THE GOVERNMENT AND YOUR THINKING HAS BEEN DONE….
now do as we say slaves….
Who says the UN Agenda 21 can not be forced up everyone…
scary indeed.!
IT just dawned on me… The Government can adjust the data at will because it is no longer in the public eye as an unchanged file…
this is much worse than hide the decline… IT’s HIDE THE DATA
I thought the berkeley study did include all data. The criticisms that some data was cherry picked was something that this study was supposed to prove. The author of the study was reputed to be a skeptic of the pre-existing studies for having cherry picked data.
In any case, 0.6 or 1.0 or 0.3 whatever the rise it isn’t enough to be worried about because the trend has been essentially zero for 20 years it may be another degree in the 21st century but it is hard to get anybody convinced the temperature will rise 3 degrees since that would require that temperatures rise every decade for the rest of the 21st century the amount of the entire 20th century’s rise. It’s hard to imagine that happening with no evidence temperatures have ever gone up that much and not during this time when every molecule of co2 will have much more effect than a molecule of co2 100 years from now will have.
If everyone in the whole world swears it is true, if it is based on a thousand papers, if it is known to be true for a thousand years but is a lie, it will be found out. A fool whispering the truth can bring down the greatest theories of the most renown thinkers and show them to be the real fools. The truth will out! It is just that it takes so darn long.
“Because it isn’t simply a question of the data, homogenised or otherwise. It’s a question of the methodology used to arrive at the end result. If an individual cannot replicate the results, how can they be confirmed or improved upon? Not refuted, please understand, but replicated. Where, exactly, is the problem with that?
#############
many people here fail to understand or remember the issues involved in the request for CRU data.
The issue was twofold
1. CRU claimed to make “value added adjustments” so we asked for the raw data, and all the data that went into the final numbers.
2. we asked for their code.
CRU has responded with CRU4. in CRU4 they use the data AS PROVIDED after homogenization.
This effectively removes the question about which adjustments CRU makes. They dont anymore.
If you compare the CRU versions of data with those provided by NWS they match.
The question about code is also answered. Actually that was answered a long time ago when I replicated their method and match their answers exactly.
There are NO issues of interest surrounding various methodologies. Changing methodologies has little impact on numbers. There is no issue surrounding CRU adjustments. They dont. They use homogenized data as provided.
Now, the issue is the homogenization process. Folks here will not like what has been found over and over again.
1. Homogenization is required
2. Different methods produce the same results
3. the job of checking got much harder
The raw data is out there. its a royal pain to work with because it is full of gross errors.
Steven,
Anthony already countered this in his original thoughts on this.
Anthony: This is particularly worrisome, because as we’ve seen, metadata for GHCN global stations is very poor to virtually non-existent, and from what we know, GHCN and CRU takes very little metadata into account. While the NMS’s may have a better handle on metadata, given the disparity of quality of met services globally, this pretty much ensures that no individual researcher is going to get their hands on a complete set of all data. Phil Jones is essentially blowing off the issue saying “let them figure it out, not our responsibility”. Whatta guy!
Not sure what you are arguing against, as the original post by elfstone is still correct despite these wild thoughts and rather off-topic strawmen you threw up.
And as a side note, I would love to see how you obtained the data from over 100 Government agencies (MET’s) and were able to reconcile this using what is provided and otherwise create the same output as Dr. Jones using his methods as provided.
Let me give you a hint on your problem here, you are claiming here: Actually that was answered a long time ago when I replicated their method and match their answers exactly.
that you basically replicated their methods. So how did you get them all to respond to you since you “replicated their method and matched their answers exactly.”
Please, feel free to prove me wrong and provide all the data and methodology used and how you were able to communicate with the MET’s so I could duplicate the experiment. I would love to know how to obtain raw “homogenized” data to work with. So feel free to share with us all.
Steven Mosher says:
June 2, 2012 at 5:11 pm
many people here fail to understand or remember the issues involved in the request for CRU data.
The issue was twofold
1. CRU claimed to make “value added adjustments” so we asked for the raw data, and all the data that went into the final numbers.
2. we asked for their code.
CRU has responded with CRU4. in CRU4 they use the data AS PROVIDED after homogenization.
This effectively removes the question about which adjustments CRU makes. They dont anymore.
If you compare the CRU versions of data with those provided by NWS they match.
The question about code is also answered. Actually that was answered a long time ago when I replicated their method and match their answers exactly.
There are NO issues of interest surrounding various methodologies. Changing methodologies has little impact on numbers. There is no issue surrounding CRU adjustments. They dont. They use homogenized data as provided.
Now, the issue is the homogenization process. Folks here will not like what has been found over and over again.
1. Homogenization is required
2. Different methods produce the same results
3. the job of checking got much harder
The raw data is out there. its a royal pain to work with because it is full of gross errors.
Let’s take this a step at a time.
Did they respond with the code they had used, together with links to the actual data they had used?
Did they explain why they had used someone else’s homogenised data?
Did they examine the methodologies used in homogenising said data?
Why, exactly, should any researcher have to reproduce, from scratch, their methodology?
Yes, homogenisation is required. Do all methods produce the same results? (I know, a broad question, but I believe it to be valid.) Checking what, precisely?
If the raw data is full of gross errors, why is one method of determining what is correct and what isn’t better than another? Who determines what errors are gross or otherwise?
Matching answers is a good thing, as long as it’s reproducible again and again. Methodology is important, however, and without the means to reproduce the “experiment”, it seems to be reduced to the level of wiggle-matching.
In the end, it comes down to whether or not one is able to reproduce the results, using the same methods originally employed. If that is not possible, then one has to assume that there’s something missing… simple logic dictates that. Openness is the only way, surely?
Homogenization of data should produce a neutral result on trends. This is not the case (0.5 ° C additional warming for the twentieth century on the data of NMS). This bias must absolutely be explained so that the methodology has any validity. Steven Mosher, instead of always saying the same thing, explain us this bias.
Steven Mosher could be inspired by Hansen et al 2001 Chapter 4 and Figure 1:
http://pubs.giss.nasa.gov/docs/2001/2001_Hansen_etal.pdf
Incomplete but a good start to explain.
Crispin in Waterloo says:
June 2, 2012 at 9:18 am
“Anthony, I have a suggestion.
We need two on-line publications: the Book of Reproducible Results and the Book of Unreproducible Results.
It is hard to keep track which papers are published with no supporting data or methods, or with partial data and incomplete methods (whether by design or incompetence). The point is that there are too many important-to-the-field papers being published reporting results are not reproducible by other researchers.
When one gets into a discussion there are frequent references to papers that are definitely in the Journals, but whose result are not reproducible. This is an affront to science but the authors do no seem to be bothered by this, for whatever reasons they have.
On your links to the right of these blogs there is a list of sites binned according to temperament and content. Would it be possible to create there a link that goes to a list, at least two lists actually, of important-to-the-discussion papers divided into two major groups: those which contain archived data and methods and which can be used to reproduce results, and those which do not. A third list might be papers for which the data is promised or is available on request.
Before allowing debaters to get away with citing-in-argument the result of works that are not reproducible, we can have a quick look-up of the paper by simple CTRL+F for part of the name.”
Crispin, that is a very good idea. Sooner or later research should move in this direction.
Such list would greatly facilitate real science advance and better understanding but would require real work to be maintained.
Nevertheless even if in the beginning it would contain 1 or 2 articles it would still be a strong reference for those articles.
Crispin in Waterloo says: @ June 2, 2012 at 9:18 am
Anthony, I have a suggestion.
We need two on-line publications: the Book of Reproducible Results and the Book of Unreproducible Results.
It is hard to keep track which papers are published with no supporting data or methods, or with partial data and incomplete methods (whether by design or incompetence). The point is that there are too many important-to-the-field papers being published reporting results are not reproducible by other researchers…..
_____________________________________
THAT is what the Union of Concerned Scientists SHOULD be doing instead of using the name to promote “Post Normal Psycience”
It is also what scientific societies such as ACS (AMERICAN CHEMICAL SOCIETY) should be doing for chemical papers. ACS should not be a CAGW advocacy group. It is NOT their field and they should just keep their mouth SHUT.
So if you get adjusted data from 150+ NMS and each NMS uses its own method of adjustment, how can you combine that data into a global temperature mean with any confidence?
I would think you would need the adjusted and unadjusted data from each NMS so you could make sure that the adjustment methods used were at least reasonable. I mean if Outer Frajakastan, where astrology is the national religion, adjusts their data based on the position of Venus would you reasonably use that adjusted data in your research?
ddpalmer: Only if it agreed with my preconceived grant application guaranteed result /sarc (for those who need it)
“Get the data directly from every NMS yourself.”
Translation:
“Go pound sand.”