Errors in GHCN metadata inventories show stations off by as much as 300 kilometers
Guest post by Steven Mosher
In the debate over the accuracy of the global temperature nothing is more evident than errors in the location data for stations in the GHCN inventory. That inventory is the primary source for all the temperature series.
One question is “do these mistakes make a difference?” If one believes as I do that the record is largely correct, then it’s obvious that these mistakes cannot make a huge difference. If one believes, as some do, that the record is flawed, then it’s obvious that these mistakes could be part of the problem. Up until know that is where these two sides of the debate stand.
Believers convinced that the small mistakes cannot make a difference; and dis-believers holding that these mistakes could in fact contribute to the bias in the record. Before I get to the question of whether or not these mistakes make a difference, I need to establish the mistakes, show how some of them originate, correct them where I can and then do some simple evaluations of the impact of the mistakes. This is not a simple process. Throughout this process I think we can say two things that are unassailable:
1. the mistakes are real. 2. we simply don’t know if they make a difference. Some believe they cannot (but they haven’t demonstrated that) and some believe they will (but they haven’t demonstrated that). The demonstration of either position requires real work. Up to now no one has done this work.
This matters primarily because to settle the matter of UHI stations must be categorized as urban or rural. That entails collecing some information about the character of the station, say its population or the characteristics of the land surface. So, location matters. Consider Nightlights which Hansen2010 uses to categorize stations into urban and rural. That determination is made by looking up the value of a pixel in an image. If it is bright, the site is urban. If it’s dark (mis-located in the ocean) the site is rural.
In the GHCN metadata the station may be reported at location xyz.xyN yzx.yxE. In reality it can be many miles from this location. That means the nightlights lookup or ANY georeferenced data ( impervious surfaces, gridded population, land cover) may be wrong. One of my readers alerted me to a project to correct the data. That project can be found here. That resource led to other resources including a 2 year long project to correct the data for all weather stations. Its a huge repository. That led to the WMO documents one of the putative sources for GHCN. This source also has errors. Luckily the WMO has asked all member nations to report more accurate data back in 2009. That process has yet to be completed and when it is done we should have data that is reported down to the arc second. Until then we are stuck trying to reconcile various sources.
The first problem to solve is the loss of precision problem. The WMO has reports that are down to the arc minute. It’s clear that when GHCN uses this data and transforms it into decimal degrees that they round and truncate. These truncations, on occasion, will move a station. I’ve documented that by examining the original WMO documents and the GHCN documents. In other cases it hard to see the exact error in GHCN, but they clearly dont track with WMO. First the WMO coordinates for WMO 60355 and then the GHCN coordinates:
WMO: 60355 SKIKDA 36 53N 06 54E [36.8833333, 6.9000]
GHCN: 10160355000 SKIKDA 36.93 6.95
GHCN places the station in the ocean. WMO places it on land as seen above.
To start correcting these locations I started working through the various sources. In this post I will start the work by correcting the GHCN inventory using WMO information as the basis. Aware, of course that WMO may have it own issue. The task is complicated by the lack of any GHCN documents showing how they used WMO documents. In the first step I’ve done this. I compared the GHCN inventory with the WMO inventory and looked at those records where GHCN and WMO have the same station number and station name. That is difficult in itself because of the way GHCN truncates names to fit a data field. It’s also complicated by the issue of re spelling, multiple names for each site and the issue of GHCN Imod flags and WMO station index sub numbers.
Here is what we find. If we start with the 7200 stations in the GHCN inventory and use the WMO identifier to look up the same stations in the WMO official inventory we get roughly 2500 matches. Here are the matching rules I used.
1. the WMO number must be the same
2. The GHCN name must match the WMO name (or alternate names match).
3. The GHCNID must not have any Imod variants. (no multiple stations per WMO)
4. The WMO station must not have any sub index variants. (107 WMO numbers have subindexes)
That’s a bit hard to explain but in short I try to match the stations that are unique in GHCN with those that are unique in the WMO records. Here is what a sample record looks like.WMO positions are translated from degrees and minutes to decimal degrees and the full precision is retained. You can check that against GHCN rounding. As we saw in previous posts slight movements in stations can move them from Bright to dark and from dark to bright pixels.
63401001000 JAN MAYEN 70.93 -8.67 1001 JAN MAYEN 70.93333 -8.666667
63401008000 SVALBARD LUFT 78.25 15.47 1008 SVALBARD AP 78.25000 15.466667
63401025000 TROMO/SKATTO 69.50 19.00 1025 TROMSO/LANGNES 69.68333 18.916667
63401028000 BJORNOYA 74.52 19.02 1028 BJORNOYA 74.51667 19.016667
63401049000 ALTA LUFTHAVN 69.98 23.37 1049 ALTA LUFTHAVN 69.98333 23.366667
You also see some of the name matching difficulties where the two records have the same WMO and slightly different names. If we collate all differences on lat and lon in matching stations we get the following:
And when we check the worst record we find the following
WMO: 60581 HASSI-MESSAOUD 31.66667 6.15
GHCN: 10160581000 HASSI-MESSOUD 31.7 2.9
GHCN has the station at longitude [smm] 2.9. According to GHCN the station is an airport:
The location in the WMO file
And the difference is roughly 300km.WMO is more correct than GHCN. GHCN is off by 300km
An old picture of the approach (weather station is to the left)
Now, why does this matter. Giss uses GHCN inventories to get Nightlights. Nightlights uses the location information to determine if the pixel is dark (rural) or bright (urban)
NASA thinks this site is dark. They think it is pitch dark. Of course they are looking 300km away from the real site. From the inventory used in H2010.
10160581000 HASSI-MESSOUD 31.70 2.90 398 630R HOT DESERT A 0
No so sure that is the weather station to the left of the runway. might be the VOR/DME if I look at the airfield diagrams.
At the risk of climbing on a bandwagon here, i also don’t understand how there can be any questions left? Their conclusions are almost always long term projections, which means that the DATA as well as the methods of handling data must be faultless.
When you start with faulty data, extrapolating any conclusions from that is pointless? And the further away you are projecting, the larger the error. So climate warnings for a century from now are worthless, total imaginary garbage.
And on these they are basing “emergency” carbon emissions laws?
I’m not a scientist in any way, but even a layman can spot a con artist at work.
I’m a biological scientist interested in genetics generally, and the genetics of human disease specifically. I can see a broadly comparable situation in my world to what appears to be happening here. I’m a data generator – I do experiments and generate my own raw data to look at different diseases and their genetics. However, increasingly in science we’re being encouraged to collaborate and in my field, that means collaborating with medically qualified staff. To be clear, I hvae nothing against medically qualified people, but it must be said that there is a clear view amongst a lot of people that medically qualified people are the best people to do “medical” research are the medics. This is, of course, nonsense – properly-trained scientists are equally capable and in many cases, better equipped to do so. Anyway, repeatedly in my experience, medical researchers want genetic data and want to analyse and publish that data without ever wanting to understand how it was generated and what the issues and concerns might be with the underlying data (ie. it’s accuracy and reliability etc etc). I’ve also had many cases where I’ve raised concerns about the quality of underlying data, only to be told not to worry about it. In one case I’ve asked to have my name taken off a publication where I felt this resulted in flaws in the data interpretation, which was the catalyst for my concerns to be taken seriously.
Its just an example to say there are other places where people place a great deal of “faith” in the quality of the data they’re relying on to say something significant, without actually having the tiniest clue about its validity or reliability.
Makes one wonder if the meaning of ‘robust’ needs to be amended.
The WMO coordinates for Skikda are at an airport, as well as Weather underground’s are using the roughly the same lat/long.
http://www.maplandia.com/algeria/airports/skikda-airport/
GHCN database likely has the wrong coordinates, but if they are using WMO data, it should have three zeros at the end.
Variable Definitions:
ID: 11 digit identifier, digits 1-3=Country Code, digits 4-8 represent
the WMO id if the station is a WMO station. It is a WMO station if
digits 9-11=”000″.
LATITUDE: latitude of station in decimal degrees
LONGITUDE: longitude of station in decimal degrees
STELEV: is the station elevation in meteres
NAME: station name
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/README
As for HASSI-MESSAOUD, it may be someone with a”fat” finger. Even MESSAOUD is spelled wrong in GHCN.
John Kerr
I don’t think it matter that much as there is enough data to support that the Earth is warmer now than it has for most of the last few hundred years.
This is another of those rather silly arm waving statements. When the science is bad and the measurement techniques are bad and the data collection are bad how can you make a statement like that?
Steve Mosher.
I just wonder what this mis-location problem has when GISS do their 1200 km aggregation technique. They are 25% out from there centre point point.
Questions of the veracity of the temperature record is what initially made me suspicious that many climate scientists are using data extracted from noise to make a case for AGW. The more the topic of the actual collection of data is discussed, the more I become convinced that the data collection methadology is more than a little inconsistent; the final theoretical product seems (to me, at least) to be rather like a symphony played by an orchestra from a score that was accidentally shredded and then reassembled by a committee of statisticians who had no idea of what the symphony sounded like in the first place and whose final product is nothing more than a collection of random noise.
My initial doubts about methadology have not been assuaged!
Due to the ‘Nightlights’ light-level nature of city centers being rather gaussian, it’s not hard to fall off the bell curve when it comes to rating whether a station needs UHI correction or not. And since most rural stations have been abandoned in the record, uncorrected warming bias is a gimme.
It would be ramdomly poor for any given station to hit ‘pixel on’, and thus, the vast majority of stations are undercorrected.
This problem is highly embarassing and reeks of poor accuracy.
It says that the system was never obtained the precision which is necessary to determine the Global Temperature on which to accurately assess how much the Earth has warmed. It overshot the mark.
The error compounds the uncertainty of exactly how much warming C02 is responsible for.
Steven, there is a saying among educators who happen to believe that with proper scientific rigor, students can catch up to grade level. Here is the saying:
Belief trumps data. You can have pages and pages of data showing how your students are making “catch up” gains. Doesn’t matter. Most people believe that success in school is an individual trait that cannot be adjusted. Therefore it is the student’s responsibility to learn, not the teacher’s.
I think one of the reasons why this is the way it is is because people don’t want to start over learning something new. If the new way works against their beliefs, it makes all their years of work wrong and subject to “do over” status. So we stick to our beliefs and refuse to budge out of fear of being wrong.
I’m wondering if everybody reading this story knows what metadata is?
I’m curious about “nightlights”. From your description it appears that “bright” is based on visible light spectrum? Lotsa lights = urban and no lights = rural. Seems to me that’s an over simplification. No lights on your average granary, but there’s heat coming off it and plenty of it. Septic field same. A snow fence can mess with temperature for miles. Seems to me that looking at a map for light sources is pretty primitive when one is trying to measure fractions of a degree over a century. You would need to do a physical survey of each station for nearby changes and keep it updated regularily. Of course then GISS would just extend the temps out 1200 km anyway.
And CO2 would still be logarithmic.
You’re not even trying Mosher, I found one that was several thousand kilometres out.
OK, Evan, I give up. What’s a Hazen screen?
Steven
It is NDB not VOR/DME. VOR/DME is located slightly to the left (it is not visible on this picture).
If you look for Met station check space between taxiways C and D.
Or about 50m North of NDB.
Tomasz Kornaszewski
…Skikda has an Algerian “Edmund Fitzgerald”, apparently:
1989 shipping disaster
The city has a commercial harbour with a gas and oil terminal. On 15th February 1989 the Dutch tanker Maassluis was anchored just outside the port, waiting to dock the next day at the terminal, when extreme weather broke out. The ship’s anchors didn’t hold and the ship smashed on the pier-head of the port. The disaster killed 27 of the 29 people on board.[4] [WIKIPEDIA] … Don’t be 29 aboard…[Edmund Fitzgerald
had 29 too, but no survivor…]
Steven:
This is a question that also came to me when you found the wandering temperature station. Could one or more of these misplaced stations cause them to wander from one 5°x5° Grid Box into another?
@Eric Anderson (first commenter) I agree with Eric Anderson. I am a different Eric Anderson. Which I found kind of spooky when I saw the first comment on this post. But clearly we Eric Andersons are a sound-minded lot. 🙂
Whenever I get into arguments with people about global warming, I urge them to go over to the surfacestations.org site and just peruse the pictures of the siting of the temperature sensing apparatus. Then look at the “corrections” to raw data. A few pictures are worth a thousand words. But I doubt many people actually take up my challenge.
OK, I was wrong
What you see on picture is NOT a NDB. And it is NOT a VOR/DME. According to . NDB is slightly to the South.
According to: it is GP/DME (antenna for ILS system).
Tomasz Kornaszewski
A good first cut would be to check the station location on Google Earth or similar. This would catch rural stations in the middle of a Walmart parking lot or urban stations adjcent to a farm house.
What is apparent is that global temperature records have unresloved QA/QC problems and until they are resolved any conclusions drawn from the data should be filed under fiction.
One would have thought that as a matter of fundamental pre-requisite when compiling the data to be used for the global data set that each and every station would have been audited by way of individual physical inspection ascertaining, amongst other matters, its location, all siting issues, station changes, dates when any station changes were made and precisely what these changes consisted of (eg., location changes, equipment changes etc) equipment used, how data is collected and recorded, when equipment was last calibrated.
It never ceases to amaze me the extent of potential errors that have been allowed to creep into the global data set through simple sloppiness and how those that advance the AGW theory are blind to the reality caused by this, namely that there can be no confidence in the data extracted through homogenisation from this data source and that it is incapable of isolating the signal from the noise.
As other have said, without an accurate data set, no reliable projections can be made and to make any meaningful extrapolation of data trends into the future is impossible given the poor quality (and potential unreliability) of the under lying data.
One needs to start a fresh. In my opinion, we should only be looking at sea temperatures and satellite collected temperatures, or sea temperatures and wholly unadjusted rural data sets. All other data sets should be disregarded.
Climate is mainly driven by the sea (which cover approx 70% of the Earth and the volume of which is a giant storage reservoir) . Thus sea temperature data is the most important single issue.
As regards land temperature, one only needs to look at unadjusted raw rural data from Canada, USA, UK, Sweden or Norway, Russia, Germany or France, and China to have a very good idea what has happened to the Northern hemisphere. As regards the Southern Hemisphere, there is little quality data but my understanding is that unadjusted rural data from New Zealand and possibly Australia suggests little warming. There is no reason to suspect that that data is not typical of the Southern Hemishpere.
I consider that the poles should be looked at seperately since these are their own micro climates and the effect of climate change taking place at the poles raises very different issues to climate change occurring at more temperate latitudes.
Wayne Gramlich says:
October 31, 2010 at 11:51 pm
I recently reread the paper Contiguous US Temperature Trends Using NCDC Raw and Adjusted Data for One-Per-State Rural and Urban Station Sets by Dr. Edward R. Long.
Anthony, this paper should be posted if it has not been already. It is very damning – definitely “worse than I thought”. It also raises again the question of when your paper will be published.
If you can get your fixed permenant location correct why should anyone expect you to be able to get correctly measure multiple temperature records daily ?
garbage in …
From Long’s paper referenced above, taking urban plus rural residential land area as 5% of total USA land area, we get an area weighted warming of .19 degrees C for the USA, using the raw data, ie about 1/4 of the claimed warming. We also have the recent warm peak slightly lower than the late 1930s warm peak. Long’s analysis has at least one other advantage – it is not distorted by “the march of the thermometers” toward the equator that has been analyzed by Chiefio.
It is at least possible that real global warming in the 20th century was no more than 0.2 degrees C.
I should have said “toward lower latitudes and elevations”.
I apologize for messing up links. I forgot about closing tags.
Tomasz Kornaszewski