From the Inconvenient Sceptic, word that the new Global Historical Climatological Network (GHCN) Version 3 may not have significant quality control.
He writes:
I have been looking at tweaking the blended temperature set that I have been using. My intent has been to replace the Hadley data with the NCDC data (Global, Land & Sea). While I was looking through the data I found some bizarre discrepancies. The problem might be due to the presentation, but so far I have not found another source to get their data, so I cannot verify this.
The data from their site is tedious to extract as it is available by month from 1880-current. After I built my tables of data I did a comparison between the Beta-V3 and the V2 data. The primary difference I have found is that the difference between the two sets is not being calculated correctly. This problem is most evident in April, September and December. Sometimes enormous differences between the versions exist, but the differences are not being seen in the stated difference.
For example, April of 1996 has V2: 0.71 °C and Beta-V3: 0.24 °C. The calculated difference is 0.02 °C instead of the correct 0.47 °C difference. April and December are full of such errors. Above is a chart of the errors for these two months.
Needless to say it is difficult to trust the value of data that cannot even correctly state the difference between the two sets they are using.
Read more here
Here’s the announcement Monday for dataset availability of GHCN V3:
—–Original Message—–
Date: Mon, 22 Nov 2010 09:05:05 -0600
From: CLIMLIST <climlist@wku.edu>
To: <climlist@lists.wku.edu>
Subject: [CLIMLIST] Announcement: Global Historical Climatology
Network-Monthly (GHCNM) v3 beta dataset announcement
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
CLIMLIST Mailing Number 10-11-58
Origin: Byron.Gleason <Byron.Gleason [at] noaa.gov>
***** DO NOT USE REPLY FUNCTION *****
***** REPEAT – DO NOT USE REPLY! *****
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Dataset Announcement:
The U.S. National Climatic Data Center in Asheville, NC has released
Version 3.0 beta of the Global Historical Climatology Network-Monthly
(GHCNM).
This dataset currently consists of monthly mean temperature data for
7280 stations and includes both raw and adjusted data. The dataset is
hosted at the following web sites:
<http://www.ncdc.noaa.gov/ghcnm/>
and
<ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/>
Users are encouraged to provide feedback for this beta version at:
Thank You,
Byron Gleason
Physical Scientist
Climate Analysis Branch
National Climatic Data Center
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
at least you can see the ( susposedly) raw data versus adjusted and be aware of the problems. previosuly it was all a black box of faith .
Have a look at maybe time to report? AMSU temps going way way down to below anomaly
That’s cold.
Speaking of cold…looks like the ASMU global temperatures for 14K feet just slipped below 2007 and the average…
Would I be correct in saying that successive data sets have shown increased warming?
DaveE.
What we have here is incompetence piled upon incompetence. They were caught out
with their first attmepts at global temperature measurements; they try again and demonstrate why they should just be ignored. Shameful, given the amount of
governmentour money they get to produce the politically desired results.Only an agency of the Federal government could handle this junk.
You can’t get away with that kind of stuff these days. We’re watching…
One of the sites I’ve studied in a fair amount of detail has data available in both raw & GHCN versions. The homogenization procedure makes assumptions about the relationship between TMax & TMin that are beyond ridiculous. At this stage it is not clear what should be done about this, in part because it appears that the folks implementing these algorithms lack the awareness & judgement needed to understand how & why they are going wrong. Sorting this business out is going to be an exceptionally tedious exercise in data analysis, diagnostics, and communication. My advice in the meantime: Consider completely avoiding the GHCN data unless you are both equipped & prepared to run painstaking conditional multivariate diagnostics.
@ur momisugly Robert of Ottawa: November 23, 2010 at 5:47 pm
Maybe they should follow the New Zealand data collecting authority (NIWA) and simply declare that they don’t have to follow best practice and in fact their temperature record is not an official temperature record.
Is V3 raw data the same as V2 & V1 raw data?
Is there an independant analysis mechanism to which this information and your concerns can be addressed? We have a similar problem in Australia with BoM and their adjustments that just go one way, up. The only complaints department is BoM itself and the responsible minister who would simply forward the request to , you guessed it, BoM.
I was not even aware they had just released a new data set, I was just comparing the data I could get access to and nothing was making sense. I was baffled when the charted difference between the sets didn’t match. I kept double checking to find where I made my mistake.
Finally I went back and started checking the website. Sure enough, there was no error in what I did, it was all on their side. It makes me hesitant to use any of their data if they can’t even subtract two numbers correctly before releasing the data.
The new set does show more warming than the last set. That is also no surprise.
John Kehr
The Inconvenient Skeptic
Great work John!
Mike Jowsey
How can NIWA follow best practice in temperature data archiving and analysis when it has been proven time and again that those who should be setting the standards of best practice continue to fail the target of being the best at anything. How can NIWA be expected to follow “best practice” when the Australian BOM can’t get within a verifiable sight of it? When NOAA can’t achieve it. When the Hadley Centre can’t achieve it. When CRU and GISS never got close enough to read the cover on the manual.
Sadly it appears that NIWA have a point that could almost be valid. There is no such standard in climate science. But at least NIWA does have a database that is reasonably well maintained, open, and fully searchable via the internet and at no charge. The database contains detailed data on a whole host of climatic variables. For NZ it is possible for the public to retrieve virtually all the monthly data (unfortunately not the daily data). The basic data does not appear to have been manipulated in secret and seems to be the same version as it was decades ago.
So for NZ, if an individual wants to invest the time, then the data can be accessed and analysed independently. Infact even though NIWA does not say so in an explicit manner, they are effectively inviting others to have a crack at it. So I would suggest to any complainants that they stop grousing about it and actually do someting about it, regardless of whether NIWA (National Institute of Water and Atmospheric Science) are being funded to carry out this function.
Anthony, I’m sure your readers have heard about the New Zealand coal mining disaster
http://www.climateconversation.wordshine.co.nz/open-threads/climate/climate-science/energy-and-fuel/
please leave a message
Condolences for the Pike River victims
Send your condolences to the families of the victims of the Pike River coal mine disaster.
To leave a message for their loved ones please use the comment field at the foot of this page.
24/11/2010 – Stuff
The Pike River coal mine victims: Conrad John Adams, Malcolm Campbell, Glen Peter Cruse, Allan John Dixon, Zen Wodin Drew, Christopher Peter Duggan, Joseph Ray Dunbar, John Leonard Hale, Daniel Thomas Herk, David Mark Hoggart, Richard Bennett Holling, Andrew David Hurren, Jacobus (Koos) Albertus Jonker, William John Joynson, Riki Steve Keane, Terry David Kitchin, Samuel Peter Mackie, Francis Skiddy Marden, Michael Nolan Hanmer Monk, Stuart Gilbert Mudge, Kane Barry Nieper, Peter O’Neill, Milton John Osborne, Brendan John Palmer, Benjamin David Rockhouse, Peter James Rodger, Blair David Sims, Joshua Adam Ufer, Keith Thomas Valli.
The GHCN version seems to rely upon countries sending raw data to be pooled into a claculation. However, it is very difficult to discover which version of data are being sent. The Australian data are being upgraded continuously as more old paper metadata sheets are processed, but the processing at many sites goes back only a few years. Thus, there is the possibility that the BoM in Australia is sending revised data from time to time. I don’t now if this is the case or not and I cannot get a straight answer.
Presumably the same applies to many other countries. They might have been caught with records in a poor state when global warming took off a few years ago and are still only part way through doing the necessary quality control to make the data good enough for example, to calibrate proxies. And of course, this could be an explanation for the changes shown above.
I keep raining the matter that proxy analysis calibrated on one set of temperature data will not give the same result as proxy data calibrated on another set. Who knows if we have a single reliable set? Where are the corrections to past proxy papers where the authors now should know that their calibrations were wrong?
How wrong? Here, as a reminder, is one reconstruction of Darwin, Australia. A person calibrating proxies might, unwittingly, choose any one of these public graph lines.
http://i260.photobucket.com/albums/ii14/sherro_2008/Darwindifferencespaghetti.jpg
looks like a boo boo in their web presentation
http://www.ncdc.noaa.gov/ghcnm/time-series/index.php?surface=land_ocean®ion=90S.90N&month=1&beg_trend_year=1880&end_trend_year=2010&submitted=Submit#time-series
pick different months and scroll around.
They do at least ask for feedback. I know they should have done a better job before release but sometimes it’s quicker and easier to get someone you know will find the errors to do it for you. AND boy will you find the errors
the errors are not “in the temperatures” the error is in the web presentation of the difference between the two data sets.
I expect you will find there are errors “in the temperatures” after they have been reviewed for awhile. They are still applying adjustments and homogenizations as part of their quality assurance procedures. The dataset overview says in part: “This new versioning format will facilitate improved documentation and communication of updates and modifications that occur as a normal part of the life of a climate dataset.” The dataset is not a fixed set of data memorializing exactly what was observed in the past, unlike what the general public may imagine. Instead, the dataset is under continuing change and “development” with an automated adjustments life of its own. Suffice it to observe that experimenters still do not have access to the original raw data needed to independently validate and verify the origins, identities, quality, and extent of the original observational data records.
“Finally I went back and started checking the website. Sure enough, there was no error in what I did, it was all on their side.”
So, V2 was much higher than the new V3 set but they’re claiming only minor adjustment? Sounds like they fixed some “problems” with the data/calculations but basically don’t want to admit it.
Note in April, the v2 line skipped 1901, so data values are not aligned properly. It’s simply a web issue.
“the errors are not “in the temperatures”
I submit that Steven Mosher couldn’t detect errors in the temperature, because he didn’t collect the data.
Andrew
…may not have significant quality control.
That’s a joke, right? These guys have NO quality control and have never had any quality control. In fact proper QC would destroy their ability to continue the big lie they are pushing.
The blogosphere is the only QC going on for this crap which is why they despise it so much.
Some of the “errors” could very well be a numerical rounding issue. For example, .044 and .036 will both round to .04. However, the difference, .008 will round to .01.
There seem to be many places where this occurs in the link given by steven above.