GISS & METAR – dial "M" for missing minus signs: it's worse than we thought

Here’s a story about how one missing letter, an M, can wreck a whole month’s worth of climate data. It is one of the longest posts ever made on WUWT, I spent almost my entire Saturday on it. I think it might also be one of the most important because it demonstrates a serious weakness in surface data reporting.

In my last post, we talked about the  a curious temperature anomaly that Jean S. found in the March GISS data and posted at Climate Audit:

http://climateaudit.files.wordpress.com/2010/04/ghcn_giss_hr2sst_250km_anom03_2010_2010_1971_2000.gif?w=520&h=307&resize=520%2C307

The anomaly over Finland has an interesting signature to it, and the correction that GISS posted on their website confirms something I’ve been looking at for a few months.

The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.

With some work I started back in late December and through January, and with GISS putting stamp of approval on “missing minus signs” I can now demonstrate that missing minus signs aren’t just an odd event, they happen with regularity, and the effect is quite pronounced when it does happen. This goes to the very heart of data gathering integrity and is rooted in simple human error. The fault lies not with GISS (though now they need a new quality control feature) but mostly with NOAA/NCDC who manages the GHCN and who also needs better quality control. The error originates at the airport, likely with a guy sitting in the control tower. Readers who are pilots will understand this when they see what I’m talking about.

I’ve seen this error happen all over the world. Please read on and be patient, there is a lot of minutiae that must be discussed to properly frame the issue. I have to start at the very bottom of the climate data food-chain and work upwards.

First, a discussion about the root of error and the differences between the surface and satellite dataset. I should mention that in the satellite image from NASA’s Earth Observations (NEO), we don’t see the same error as we see in the GISTEMP map above.

NASA NEO March 1-31 2010 day satellite measured temp anomaly – click for larger image

Why? Better sensors, maybe, but mostly it has to do with a different data gathering methodology. In the surface data sets, including land and ocean data, most every datapoint is touched by a human hand, even airport data that gets done by automated airport sensors sometimes gets transcribed manually (often in third world and technologically stunted countries). In the surface data, thousands of sensors are spread across the globe, many different designs, many different exposures, many different people with different standards of measurement and reporting. The precision, accuracy, and calibration of the vast surface network varies, especially when we have broad mix of instrumentation types.For example in the US Historical Climatological Network the equipment varies significantly.

In satellite data, the data is measured at a single point with one sensor type, the Microwave Sounder Unit on the satellite, calibrated to a precision source on-board. On-board redundant precision platinum resistance thermometers (PRTs) carried on the satellite radiometers. The PRT’s are individually calibrated in a laboratory before being installed in the instruments. The satellite data is automatically measured and  transmitted. In contrast to the surface temperature record, no human hands touch the data gathering or data reporting process. Satellite data generation is far more homogeneous than the mish-mash of surface data.

I think it would be safe to say that the chances of human error in raw surface data are at least an order of magnitude greater (if not several) than error in raw satellite data. Post measurement processing is another issue, but for the purposes of this essay, I’m focusing only on raw data gathering and transmittal.

As mentioned in the recently updated compendium of issues with the surface temperature data by Joe D’Aleo and myself, there has been a move in the Global Historical Climatological Network (GHCN) to rely more and more on airports for climate data. This, in my opinion, is a huge mistake because in addition to those issues

E.M. Smith aka “Chiefio” reports that in GISS (which uses GHCN) worldwide, there has been a wholesale migration towards airport weather data as a climatic data source.  In an email sent to me on Jan 20, 2010 he says that

Look at:

http://chiefio.wordpress.com/2009/08/26/agw-gistemp-measure-jet-age-airport-growth/

which as a fairly good descriptions of the problems in the data, we have a global report for GHCN as of that August data.  There is more deail in the link, but I think you care about “now”:

Percentage of sites that are AIRPORTS NOW, by decade of record

Year   S.P   S.C   S.T   S.W   EQ.   N.W   N.T   N.C   N.P  Total

1909   0.0  42.0  15.1  28.2  29.2  36.7  22.8  33.3  44.4  25.4

1919   0.0  36.4  12.8  23.5  25.1  37.7  20.9  35.0  39.8  24.1

1929   0.0  37.0  11.9  27.4  27.7  32.7  20.4  35.9  56.4  24.1

1939   0.0  43.9  17.6  32.0  33.8  29.1  20.2  36.2  51.0  25.1

1949   0.0  32.3  24.4  37.6  44.4  31.8  23.3  39.3  60.9  29.1

1959   0.0  24.0  35.0  50.0  59.4  39.4  30.9  41.0  62.9  37.3

1969   0.0  18.1  39.3  53.2  63.2  40.2  31.4  41.1  61.5  39.0

1979   0.0  17.9  39.1  52.0  64.2  40.7  28.8  41.1  62.3  37.7

1989   0.0  20.7  41.5  52.5  67.8  41.9  29.1  40.8  64.9  37.7

1999   0.0  21.0  53.5  57.4  68.0  53.0  32.6  49.0  59.0  41.6

2009   0.0  17.9  74.0  64.7  66.5  51.5  30.2  45.4  57.3  41.0
This is by major climate latitude band, the total is 41% for the globe (and rising daily 😉
Also in:

I do break outs by continent and by some countries.  For the USA, I further do a specific with / without USHCN (the older version, not the USHCN.v2 put in 15Nov09) and findFor COUNTRY CODE: 425

But it masks the rather astounding effect of deletions in GHCN without the USHCN set added in:
LATpct: 2006  3.7 18.3 29.5 33.2 14.4  0.0  0.4  0.3  0.1  0.1 100.0

AIRpct:       1.3  4.0  6.3  6.7  3.2  0.0  0.4  0.3  0.1  0.1 22.4

LATpct: 2007  8.2 17.2 28.4 26.9 11.2  0.0  3.7  3.0  0.7  0.7 100.0

AIRpct:       8.2 15.7 27.6 23.1  9.0  0.0  3.7  3.0  0.7  0.7 91.8

LATpct: 2008  8.8 16.9 28.7 26.5 11.0  0.0  3.7  2.9  0.7  0.7 100.0

AIRpct:       8.8 15.4 27.9 22.8  8.8  0.0  3.7  2.9  0.7  0.7 91.9

LATpct: 2009  8.1 17.8 28.1 26.7 11.1  0.0  3.7  3.0  0.7  0.7 100.0

AIRpct:       8.1 16.3 27.4 23.0  8.9  0.0  3.7  3.0  0.7  0.7 91.9

DLaPct: 2009  4.3 18.4 29.5 32.5 13.6  0.0  0.7  0.9  0.2  0.1 100.0

DArPct:       2.1  5.7  8.8  8.9  3.7  0.0  0.6  0.8  0.2  0.1 30.7

That in the YEAR 2009 the USA has almost 92% airports in GHCN.

So clearly, airports make up a significant portion of the climate data.

On the issues of airports as climate station, obvious issues with siting, UHI, failing ASOS instrumentation, and conflicting missions (aviation safety -vs-climate) aside, I’m going to focus on one other thing unique to airports: METAR

What is METAR you ask? Well in my opinion, a government invented mess.

When I was a private pilot (which I had to give up due to worsening hearing loss – tower controllers talk like auctioneers on the radio and one day I got the active runway backwards and found myself head-on to traffic. I decided then I was a danger to myself and others.) I learned to read SA reports from airports all over the country. SA reports were manually coded teletype reports sent hourly worldwide so that pilots could know what the weather was in airport destinations. They were also used by the NWS to plot synoptic weather maps. Some readers may remember Alden Weatherfax maps hung up at FAA Flight service stations which were filled with hundreds of plotted airport station SA (surface aviation) reports.

The SA reports were easy to visually decode right off the teletype printout:

From page 115 of the book "Weather" By Paul E. Lehr, R. Will Burnett, Herbert S. Zim, Harry McNaught - click for source image

Note that in the example above, temperature and dewpoint are clearly delineated by slashes. Also, when a minus temperature occurs, such as -10 degrees Fahrenheit, it was reported as “-10”. Hang on to that, it is important.

The SA method originated with airmen and teletype machines in the 1920’s and lasted well into the 1990’s. But like anything these days, government stepped in and decided it could do it better. You can thank the United Nations, the French, and the World Meteorological Organization (WMO) for this one. SA reports were replaced by METAR in 1996.

From Wikipedia’s section on METAR

METAR reports typically come from airports or permanent weather observation stations. Reports are typically generated once an hour; if conditions change significantly, however, they can be updated in special reports called SPECIs. Some reports are encoded by automated airport weather stations located at airports, military bases, and other sites. Some locations still use augmented observations, which are recorded by digital sensors, encoded via software, and then reviewed by certified weather observers or forecasters prior to being transmitted. Observations may also be taken by trained observers or forecasters who manually observe and encode their observations prior to transmission.

History

The METAR format was introduced 1 January 1968 internationally and has been modified a number of times since. North American countries continued to use a Surface Aviation Observation (SAO) for current weather conditions until 1 June 1996, when this report was replaced with an approved variant of the METAR agreed upon in a 1989 Geneva agreement. The World Meteorological Organization‘s (WMO) publication No. 782 “Aerodrome Reports and Forecasts” contains the base METAR code as adopted by the WMO member countries.[1]

Naming

The name METAR is commonly believed to have its origins in the French phrase message d’observation météorologique pour l’aviation régulière (“Aviation routine weather observation message” or “report”) and would therefore be a contraction of MÉTéorologique Aviation Régulière. The United States Federal Aviation Administration (FAA) lays down the definition in its publication the Aeronautical Information Manual as aviation routine weather report[2] while the international authority for the code form, the WMO, holds the definition to be aerodrome routine meteorological report. The National Oceanic and Atmospheric Administration (part of the United States Department of Commerce) and the United Kingdom‘s Met Office both employ the definition used by the FAA. METAR is also known as Meteorological Terminal Aviation Routine Weather Report or Meteorological Aviation Report.

I’ve always thought METAR coding was a step backwards, for reasons I’ll discuss shortly.

But first, quick! Spot the temperature and dewpoint in this METAR report:

The following is an example METAR from Burgas Airport in Burgas, Bulgaria, and was taken on 4 February 2005 at 16:00 Coordinated Universal Time (UTC).

METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 M04/M07 Q1020 NOSIG 9949//91=

Could you read this and know what the weather is in Burgas? I can, only becuase I’ve looked at hundreds the past few months, but I still have to pick through the report to find it. The reason is that METAR is a variable field reporting format. Data isn’t always in the same position.

In the report above. The temperature and dewpoint is: M04/M07

M04/M07 indicates the temperature is −4 °C (25 °F) and the dewpoint is −7 °C (19 °F). An M in front of the number indicates that the temperature/dew point is below zero (0) Celsius.

Notice also that the entire METAR report is visually more complex. This is fine if you are having computers code it, but many METAR reports are still hand coded by technicians at airports, and thus begins the introduction of human error into the climate data process. Complexity is not a good thing when manual labor is involved as it increases the likelihood of error.

Here is where METAR coding departs from normal numeric convention. SA reports did not have this problem.

In the METAR report above, instead of using the normal way we treat and write negative numbers, some policy wonk decided that we’ll use the letter “M” to report a negative number. Only a bureaucrat could think like this.

So instead of a below zero Centigrade temperature and dewpoint looking like this:

-04/-07

in the “new and improved” METAR coding, it looks like this:

M04/M07

OK not a problem you say? Well I beg to differ, because it forces technicians who manually code METAR reports for transmission to do something they would not do anywhere else, and that’s write down an “M” instead of a minus sign. Using an M is totally counter-intuitive and against basic math training, and increases the likelihood of error.

It gets worse. Let’s say the technician makes a boo-boo and puts a minus sign instead of an “M” in front of the numbers for temperature/dewpoint. You’d think this would be alright, and the system would correctly interpret it, right?

Let’s put the METAR report from Burgas Airport into an online METAR decoder.

http://www.wx-now.com/Weather/MetarDecode.aspx

Here’s the report with the easy to make mistake, using minus sign instead of M for the temperature.

METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 -04/M07 Q1020 NOSIG 9949//91=

The output from the online METAR decoder reads:

Hey look at that, the temperature is 39°F (3.8°C). Minus signs are discarded from METAR decoding. Note that decoded METAR temperature also comes out the same if the “M” is missing in front of the 04/-07 or 04/M07

If it had been decoded correctly we would have gotten:

(-4) degrees Celsius = 24.8 degrees Fahrenheit

A whole 14.2 degrees F difference!

Reference for METAR decoding:

http://www.met.tamu.edu/class/METAR/quick-metar.html

Also note that METAR data has no year stamp component to the data, so the METAR decoder has no way of knowing this was a report from 2005, not 2010. Since each METAR report is essentially disposable within 24 hours, this presents no problem for pilots, they don’t care. But if you are tracking climate over years using METAR data, not having a year time stamp increases the likelihood of error.

Also the temperature error itself in this case has no bearing on a pilot’s decision to takeoff or land. Unless they are worried about density altitude on hot humid days, the temperature is a throwaway datum. They are mostly concerned about winds, sky conditions, and barometer (altimeter setting). In fact cool/cold days are far better for aviators; see AOPA’s Why Airplanes Like Cool Days Better.

My point here is this:

If a pilot or tower controller sees an erroneous METAR report like this:

METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 -04/M07 Q1020 NOSIG 9949//91=

Or this:

METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 04/M07 Q1020 NOSIG 9949//91=

Pilots/controllers/dispatchers aren’t likely to care, since current temperature and dewpoint are not important to them at these cooler temperatures. They also aren’t likely to call up the tower and holler at the technician to say “Hey, the temperature is wrong!”. Further, since the METAR report may be reissued sometime within the hour if somebody DOES spot the error, problem solved.

Point is that updates/corrections to METAR data may not be logged for climate purposes, since they are likely to be seen as duplicate reports because of the hourly timestamp.

So, in the case of M’s and minus signs, the propensity exists for erroneous METAR reports to not get corrected and to stay logged in the system, eventually finding their way into the climate database if that airport happens to be part of GISS, CRU, or GHCN datasets.

Maddeningly, even when egregious errors in aviation weather data are pointed out and even acknowledged by the reporting agency,  NOAA keep them in the climate record as was demonstrated last year in Honolulu, HI International Airport when a string of new high temperature records were set by a faulty ASOS reporting station. NOAA declined to fix the issue in the records:

NOAA: FUBAR high temp/climate records from faulty sensor to remain in place at Honolulu

The key sentence from that story from KITV-TV:

The National Weather Service said that is not significant enough to throw out the data and recent records.

Hmmm, look at another nearby station and compare the differences. You be the judge.

Does NOAA consider this a climate reporting station? Yes according to NCDC MMS database, it is part of the “A” network, designated for climate:

Clearly, NOAA simply doesn’t seem to care that erroneous records finds their way into the climatic database.

OK back to the METAR issue.

The problem with METAR reporting errors is worldwide. I’ve found many examples easily in my spare time. Let’s take for example, a station in Mirnvy, Russia. It is in Siberia at 62.5° N 113.9° E and has an airport, is part of GHCN, and reports in METAR format.

Weather Underground logs and plots METAR reports worldwide, and these METAR reports are from their database on November 11th, 2009.

It shows a clear error in the 12:30PM (330Z) and 1 PM (400Z) METAR report for that day:

http://www.wunderground.com/history/airport/UERR/2009/11/1/DailyHistory.html?req_city=NA&req_state=NA&req_statename=NA

UERR 010330Z    22005G08MPS 9999 -SN 21/M23 Q1026 NOSIG RMK QFE738 24450245

UERR 010400Z    22005G08MPS 9999 -SN SCT100 OVC200 20/M22 Q1025 NOSIG RMK QFE737 24450245

UERR 010430Z    21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245

UERR 010430Z    21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245

UERR 010500Z    21005G08MPS 4000 -SN SCT100 OVC200 20/M22 Q1023 NOSIG RMK QFE736 24450245

Note the missing ” M” on the 12:30PM (330Z) and 1 PM (400Z). It happens again at 2PM (500Z). Of course it isn’t very noticeable looking at the METAR reports, but like the GISS plot of Finland, stands out like a sore thumb when plotted visually thanks to Weather Underground:

Mirnvy, Russia

The effect of the missing “M” is plotted above, which coincidentally looks like an “M”.

Put those METAR reports in this online METAR decoder: http://www.wx-now.com/Weather/MetarDecode.aspx and you get 70F for 12:30PM and 68F for 1PM

What do you think 70 degree F spike this will do to monthly averaged climate data in a place where the temperature stays mostly below freezing the entire month?

http://www.wunderground.com/history/airport/UERR/2009/11/1/MonthlyHistory.html?MR=1

Does NOAA log METAR data from Mirnvy Russia (ICAO code UERR)?

Yes, they do. Plus many other METAR reporting stations discussed here.

Does NCDC classify it as a climate station?

Yep, it is part of the “A” network. Which means it either directly reports climate data and/or is used to adjust data at other stations, such as GHCN stations.

List of GHCN stations:

http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt

It is not however, part of GHCN. But there are plenty stations that have this error that are part of GHCN. Yakutsk, Russia, also in Siberia is part of GHCN and has a METAR reporting error. Here’s an example what one off-coded hourly reading will do to the climate database.

The city of Yakutsk, one of the coldest cities on earth, reported a high of 79˚F on November 14th with a low of -23˚F.

Weather Underground seems to have done some quality control to the METAR reports, but the erroneous high temp remains in the daily  and monthly report:

http://www.wunderground.com/history/station/24959/2009/11/14/DailyHistory.html

http://www.wunderground.com/history/station/24959/2009/11/14/MonthlyHistory.html

A month later, it happened again reporting a high of 93˚F on December 14th with a low of -34˚F

And the erroneous 93F high temp remains in both the daily and monthly reports, but has been removed from the METAR report, so I can’t show you the missing “M” I observed back in January. I wish I had made a page screen cap.

http://www.wunderground.com/history/station/24959/2009/12/14/DailyHistory.html

http://www.wunderground.com/history/station/24959/2009/12/14/MonthlyHistory.html

When the temperature data was calculated with that error then, this was found:

The average for the day,  30˚F, was some 67˚F above normal, pushing the anomaly for the month of December from 3.6˚F above normal to 5.9˚F above normal… quite a shift!

More examples:

Here’s an example of a properly coded METAR report from Nizhnevartovsk, Russia,  for December 11, 2009, but the data is wrong. I’m thinking it was supposed to be M30 but came out M13. The dewpoint value M16 is also erroneous.

Nizhnevartovsk, Russia Dec 7, 2009

METAR USNN 111230Z 00000MPS P6000 SCT026 OVC066 M27/M29 Q1014 NOSIG RMK QFE755 SC062

METAR USNN 111300Z 12005G08MPS P6000 SCT066 OVC200 M13/M16 Q1035 NOSIG RMK QFE772 SC063

METAR USNN 111330Z 12005G08MPS P6000 SCT066 OVC200 M13/M16 Q1035 NOSIG RMK QFE772 SC063

METAR USNN 111400Z 00000MPS P6000 SCT020 OVC066 M28/M29 Q1014 NOSIG RMK QFE755 SC065

And it was not a one time occurrence, happening again on Dec 25th as shown in the Monthly graph:

Nizhnevartovsk, Russia, December 2009

http://www.wunderground.com/history/airport/USNN/2009/12/25/MonthlyHistory.html

The daily graph and METAR reports, notice it happened about the same time (1300Z) and in the same way (M27 then M13) , perhaps pointing to the same technician on duty making the same habitual mistake again. Maybe too much Vodka having to work the Xmas night shift?

Nizhnevartovsk, Russia Dec 25, 2009

METAR USNN 251230Z 11006MPS 2200 -SN SCT014 OVC066 M27/M30 Q1015 NOSIG RMK QFE757 SC055

METAR USNN 251300Z 35002MPS 6000 -SN SCT015 OVC066 M13/M15 Q1010 NOSIG RMK QFE752 SC03

METAR USNN 251330Z 12006MPS 4100 -SN SCT015 OVC066 M27/M29 Q1014 NOSIG RMK QFE756 SC055

http://www.wunderground.com/history/airport/USNN/2009/12/25/DailyHistory.html

It did not appear initially to be in the GHCN list or on the GISS list, but I’ve found that some of the names on Weather Underground are different from the place names in the GHCN and GISS lists. It turns out that if you search in Weather Underground for the station ALEKSANDROVSKOE it will point you to use the data from Nizhnevartovsk. ALEKSANDROVSKOE is a GHCN/GISS station.

I found other instance of METAR errors for that station, this one was quite pronounced on Jan 16th, 2009, lasting for 7 hours before it was corrected.

Nizhnevartovsk, Russia Jan 16, 2009

Here’s the METAR reports

METAR USNN 151800Z 23002MPS P6000 BKN066 OVC200 M22/M24 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 151830Z 23002MPS 2900 -SHSN SCT020CB OVC066 22/M23 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 151900Z 23002MPS 2100 -SHSN SCT019CB OVC066 21/M23 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 152000Z 24001MPS 5000 -SHSN SCT022CB OVC066 21/M22 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 152030Z 24002MPS 4300 -SHSN SCT020CB OVC066 21/M22 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 152100Z 24002MPS 6000 -SHSN SCT018CB OVC066 20/M22 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 152130Z 25002MPS P6000 SCT020CB OVC066 20/M22 Q1009 NOSIG RMK QFE751 SC038

METAR USNN 152200Z 28002MPS P6000 SCT022CB OVC066 20/M22 Q1009 NOSIG RMK QFE752 SC038

METAR USNN 152300Z 27003MPS P6000 -SHSN SCT016CB OVC066 M19/M21 Q1010 NOSIG RMK QFE752 SC038

http://www.wunderground.com/history/airport/USNN/2009/1/16/DailyHistory.html

The monthly report shows the event:

Nizhnevartovsk, Russia, January 2009

http://www.wunderground.com/history/airport/USNN/2009/1/16/MonthlyHistory.html

It happened twice on Feb 2nd, 2007, and with a space added between the M and 09 on the 0300Z report, it is a clear case of human error:

METAR USNN 020100Z 11010G15MPS 0500 R03/1200 +SN +BLSN VV002 M09/M11 Q1003 TEMPO 0400 +SN +BLSN VV002 RMK QFE748 QWW060 MOD ICE MOD TURB S

METAR USNN 020200Z 12009G14MPS 0500 R03/1200 +SN +BLSN VV002 M09/M10 Q1001 TEMPO 0400 +SN +BLSN VV002 RMK QFE747 QWW060 MOD ICE MOD TURB S

METAR USNN 020300Z 11008G13MPS 1100 R03/1200 SN +BLSN BKN004 OVC066 M 09/M10 Q1000 NOSIG RMK QFE745 QRD120 MOD ICE MOD TURB SC045

...

METAR USNN 021200Z 18009MPS P6000 -SHSN DRSN SCT017CB OVC066 M07/M09 Q0989 TEMPO 2000 SHSN RMK QFE736 MOD ICE MOD TURB SC042

METAR USNN 021300Z 16009MPS P6000 DRSN SCT016CB OVC066 08/M11 Q0989 NOSIG RMK QFE736 MOD ICE MOD TURB SC042

METAR USNN 021400Z 16008MPS P6000 DRSN SCT016CB OVC066 M08/M11 Q0989 NOSIG RMK QFE736 MOD ICE MOD TURB SC042

http://www.wunderground.com/history/airport/USNN/2007/2/2/DailyHistory.html

The monthly data shows the double peak:

http://www.wunderground.com/history/airport/USNN/2007/2/2/MonthlyHistory.html

I’m sure many more can be found, I invite readers to have a look for themselves by looking for such events at Weather Underground

It is not just Russia that has METAR reporting errors

Lest you think this a fault of Russia exclusively, it also happens in other northern hemisphere Arctic site and also in Antarctica.

Svalbard, Oct 2, 2008

METAR ENSB 020550Z 13012KT 6000 -SN FEW010 SCT015 BKN030 M04/M06 Q1013 TEMPO 4000 SN BKN012

METAR ENSB 020650Z 14013KT 9000 -SN FEW010 SCT018 BKN040 03/M06 Q1013 TEMPO 4000 SN BKN012

METAR ENSB 020750Z 15011KT 9999 -SN FEW015 SCT025 BKN040 M03/M07 Q1013 TEMPO 4000 SN BKN012

http://www.wunderground.com/history/airport/ENSB/2008/10/2/DailyHistory.html

Eureka, Northwest Territory, Canada March 3 2007

It hit 109.4 F (43C) there on March 3rd 2007 according to this METAR report. Eureka is the northernmost GHCN station remaining for Canada. It’s temperature gets interpolated into adjacent grid cells.

CWEU 031600Z 14004KT 15SM FEW065 BKN120 M43/M45 A2999 RMK ST1AS2 VIA YQD SLP150

CWEU 031700Z 15005KT 10SM FEW065 BKN012 43/46 A3000 RMK SF1AS1 VIA YQD SLP163

Decoded: 11:00 AM 109.4 °F 114.8 °F 100% 30.01 in 10.0 miles SSE 5.8 mph - Mostly Cloudy

CWEU 031800Z 11003KT 15SM FEW050 FEW065 OVC130 M43/M46 A3001 RMK SF2SC1AS1 VIA YQD SLP164

http://www.wunderground.com/history/airport/CWEU/2007/3/3/DailyHistory.html

In these cases below for Antarctic stations Dome C and Nico, the METAR reports seem to have all sorts of format issues and I’m not even sure how where the error occurs, except that Weather Underground reports a spike just like we see in Russia.

Dome C station Dec 9, 2009

AAXX 0900/ 89828 46/// ///// 11255 36514 4//// 5//// 90010

AAXX 0901/ 89828 46/// ///// 10091 36514 4//// 5////

AAXX 09014 89828 46/// /1604 11225 36480 4//// 5//// 9014

http://www.wunderground.com/history/station/89828/2009/12/9/DailyHistory.html

Nico Station,  University of Wisconsin Dec 9, 2009

AAXX 0920/ 89799 46/// ///// 11261 4//// 5//// 92030

AAXX 0920/ 89799 46/// ///// 11103 4//// 5//// 92040

AAXX 0921/ 89799 46/// ///// 11270 4//// 5////

http://www.wunderground.com/history/station/89799/2009/12/9/DailyHistory.html

Admusen Scott Station Jan 14th, 2003

Here’s generally properly formatted METAR data, but note where the technician coded the extra space, oops!

NZSP 131350Z GRID36007KT 9999 IC SCT020 BKN060 M31/ A2918 RMK SDG/HDG

NZSP 131450Z GRID36007KT 9999 IC FEW010 FEW020 SCT035 SCT050 M3 1/ A2918 RM K SDG/HDG

NZSP 131550Z GRID10008KT 9999 IC BCFG FEW010 SCT020 BKN050 M31/ A2919 RMK VIS E 2400 BCFG E SDG/HDG

http://www.wunderground.com/history/station/89009/2003/1/14/DailyHistory.html

And I’m sure there are many more METAR coding errors yet to be discovered. What you see above is just a sampling of a few likely candidates I looked at over a couple of hours.

Missing M’s – Instant Polar Amplification?

It has been said that the global warming signature will show up at the poles first. Polar Amplification is defined as:

“Polar amplification (greater temperature increases in the Arctic compared to the earth as a whole) is a result of the collective effect of these feedbacks and other processes.It does not apply to the Antarctic, because the Southern Ocean acts as a heat sink. It is common to see it stated that “Climate models generally predict amplified warming in polar regions”, e.g. Doran et al. However, climate models predict amplified warming for the Arctic but only modest warming for Antarctica.

Interestingly, the METAR coding error has its greatest magnitude at the poles, becuase the differences in the missing minus sign become larger as the temperature grows colder. Eureka, NWT is a great example, going from -43°C to +43°C (-45.4°F to 109.4°F) with one missing “M”.

You wouldn’t notice METAR coding errors at the equator, because the temperature never gets below 0°C. Nobody would have to code it. In middle latitudes, you might see it happen, but it is much more seasonal and the difference is not that great.

For example:

M05/M08  to 05/M08 brings the temp from -5°C to +5°C, but in a place like Boston, Chicago, Denver, etc a plus 5C temperature could easily happen in any winter month a -5C temperature occurred. So the error slips into the noise of “weather”, likely never to be noticed. But it does bump up the temperature average a little bit for the month if uncorrected.

But in the Arctic and Antarctic, a missing M on a M20/M25 METAR report makes a 40°C difference when it becomes +20°C. And it doesn’t seem likely that we’d see a winter month in Siberia or Antarctica that would normally hit 20°C, so it does not get lost in the “weather” noise, but becomes a strong signal if uncorrected.

Confirmation bias, expecting to see polar amplification may be one reason why until now, nobody seems to have pointed it out. Plus, the organizations that present surface derived climate data, GISS, CRU, only seem to deal in monthly and yearly averages. Daily or hourly data is not presented that I am aware of, and so if errors occur at those time scales, they would not be noticed. Obviously GISS didn’t notice the recent Finland error, even though it was glaringly obvious once plotted.

With NASA GISS admitting that missing minus signs contributed to the hot anomaly over Finland in March, and with the many METAR coding error events I’ve demonstrated on opposite sides of the globe, it seems reasonable to conclude that our METAR data from cold places might very well be systemically corrupted with instances of coding errors.

The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.

4/15/10 http://data.giss.nasa.gov/gistemp/

That darned missing M, or an extra space, or even writing “-” when you mean “M” (which is counterintuituve to basic math) all seem to have a factor in the human error contributing to data errors in our global surface temperature database. To determine just how much of a problem this is, a comprehensive bottom up review of all the data, from source to product is needed. This needs to start with NOAA/NCDC as they are ultimately responsible for data quality control.

It has been said that “humans cause global warming”. I think a more accurate statement would be “human error causes global warming”.

Note: In this post I’ve demonstrated the errors. In a later post, I hope to do some data analysis with the numbers provided to see how much effect these errors actually have. Of course anyone who wants to do this is welcome to leave links to graphics and tables. -Anthony

See these weather underground sites:
Yakutsk (Jakutsk)
and this one, is particularly interesting, because it shows a clear error in the 12:30PM and 1 PM METAR report for that day
12:00 PM -5.8 °F -9.4 °F 84% 30.33 in – SW 11.2 mph 17.9 mph N/A   Clear
UERR 010300Z 22005G08MPS CAVOK M21/M23 Q1027 NOSIG RMK QFE738 24450245
12:30 PM 69.8 °F -9.4 °F 4% 30.30 in 6.2 miles SW 11.2 mph 17.9 mph N/A   Unknown

UERR 010330Z 22005G08MPS 9999 -SN 21/M23 Q1026 NOSIG RMK QFE738 24450245

1:00 PM 68.0 °F -7.6 °F 5% 30.27 in 6.2 miles SW 11.2 mph 17.9 mph N/A   Unknown

UERR 010400Z 22005G08MPS 9999 -SN SCT100 OVC200 20/M22 Q1025 NOSIG RMK QFE737 24450245

1:30 PM -4.0 °F -7.6 °F 85% 30.24 in 2.5 miles SSW 11.2 mph 17.9 mph N/A Snow Light Snow

UERR 010430Z 21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245

Note the missing ” M” on the 1230 and 1PM reports 21/M23
Put that in this online METAR decoder:
and you get 70F for 12:30PM and 68F for 1PM man made global warming thanks to hand coded teletype report.
Pilots will know its wrong and disregard, they mostly worry about baro pressure/altimeter and winds. Temps on the ground are never as extreme as what aircraft experience in the air.
No incentive to correct this…not a big deal to aviation.
Here is what I think is going on:
1) Russian METAR is hand-coded from airports, thus prone to error. put in a minus sign for M or make an M+space+ temp/dp and you get the same thing. example:
UERR 010330Z 22005G08MPS 9999 -SN -21/M23 Q1026 NOSIG RMK QFE738 24450245 gives 70F in the online decoder above, later systems may strip the minus sign as being an invalid character in the report which is why we may not see it…or they just forget to add “M” either way, all we need is one of these per month.
2) or…character formatting western/cyrillic may contribute to missing or badly formatted characters in automated decoding.
Either way, there’s our spurious Russian warming, and why we seem to have a permanent red spot there.
Here’s what one off reading will do. The city of Yakutsk, one of the coldest cities on earth, reported a high of 94˚F on December 14th with a low of -35˚F. The average for the day, 30˚F, was some 67˚F above normal.. pushing the anomaly for the month of December from 3.6˚F above normal to 5.9˚F above normal… quite a shift!
It also happens in Antarctica:
Dome C station
Nico University of Wisconsin
Admusen Scott Station
Here is the list of sites with GHCN WMO numbers per this list:
WMO number    Station Name            Day/Month/Year of error
24266                Verhojansk                   13/11/2009
23955                ALEKSANDROVSKOE  11/12/2009
and also 01/16/2009
and 02/02/2007

Sponsored IT training links:

Ultimate VCP-410 practice files formulated by top experts to help you pass 220-702 and 640-822 exam in easy and fast way.


0 0 votes
Article Rating
296 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Ken Smith
April 17, 2010 6:56 pm

Fascinating post. Regarding the UHI effect, I have a question I hope someone can address.
Is it plausible that UHI might be increased by a _local_ greenhouse effect? What I mean is this: if carbon dioxide and other greenhouse gases are trapping heat, might they be trapping it to a greater extent at the locations where large amounts of those gases are emitted? Like urban areas, and airports particularly?
Or does the greenhouse effect (whatever one may reckon as it’s extent) only operate at upper atmosphere levels and over massive regional or global levels?

JDN
April 17, 2010 7:02 pm

Notice how the errors as shown ALWAYS result in higher temps.

Tom in Florida
April 17, 2010 7:03 pm

It’s GIGO all over again!

timhulsey
April 17, 2010 7:05 pm

Two wrongs (bad data plus bad models) did not make a right!

April 17, 2010 7:06 pm

Thanks, Anthony. Great post (with lots of graphics). This was enlightening.
REPLY: Coming from Mr. “Gobs of Graphics” himself, that is quite a complement. Thanks -A

April 17, 2010 7:08 pm

I noticed in February in Colorado that NCDC/NOAA was consistently showing above average temperatures along the Front Range – while Accuweather stats, the unusually prolonged frozen lakes, and practically everyone I talked to indicated that it was a very cold February. Accuweather showed it 2-4 degrees below normal.

April 17, 2010 7:15 pm

A cultural/social effect may also be at work. When I was the US Representative to the Soviet Union on the Protection of the Environment [included weather forecasting] back in 1976 http://www.leif.org/EOS/Nature/261546a0.pdf I noticed, both from verbal interchange and from the evening news on TV that in the middle of winter when the temperature was typically -30C, the Russians never mentioned the minus sign; they would simply state “it’s 30 today”.

April 17, 2010 7:16 pm

Anthony, thanks for your hard work on this! Excellent analysis, I’ve seen such errors of omission/coding in epidemiological studies of pathogen counts with similar results. Garbage in, garbage out!
With the powerful neural network and AI tools available, you’d think that someone who was serious about the integrity of their data would subject it to some level of peer review by statistical process control. I don’t think it would be hard at all.
And yet, the AGW train keeps on rolling down the tracks, with a Senate cap & trade/energy bill in the works. It will be interesting.

Nick
April 17, 2010 7:21 pm

That’s right,Anthony,accusations and extrapolation first,then ‘some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
REPLY: Nick, dial down your anger, then look at the hard data errors presented. Glaciers are mostly proxies for precipitation, so they don’t really care anyway. Surface project is coming along fine, a paper is being prepared. Thanks for asking. Of course if you just want to ignore what is presented, and rant, I suggest ClimateProgress, where ranting is an art form. -A

HankHenry
April 17, 2010 7:21 pm

wonderful work.

Keith Minto
April 17, 2010 7:25 pm

Excellent report, Anthony.
As Ric Werme said earlier, temperatures in the form of degrees Kelvin would solve the problem of negative temperatures.

Paul Vaughan
April 17, 2010 7:30 pm

I have had the experience of finding a weather station with all negative (degrees Celsius – so below freezing) temperatures SET TO ZERO deg C.
When I consulted officials, they admitted serious data quality issues for the main sites (the ones most valuable to me) at which I was looking.
They also thanked me for pointing out 200cm (over 6 feet) of snow that came and went in a single day. They guessed it was probably 2cm (under 1 inch) and changed it to that.
They didn’t seem too surprised to hear that I also noticed amounts like 400cm to 500cm of snow appearing, disappearing, appearing again, then disappearing, etc. The response was something like, “Yeah, the quality was so bad at that site – we almost shut it down.”
I encouraged them to keep the sites running since the sites were essential to my research. I’d much rather deal with bad data than no data.

Gary from Chicagoland
April 17, 2010 7:32 pm

Wow, outstanding work Anthony! This shows how important raw climate data is for the proper scientific method to work. You just showed us how a few hours of hard work from you uncovered a possible big flaw in the way climate data is recorded, and how important it is for this data to be corrected. Once the data is verified, then it should be compared to the theory, and if it is in conflict, then the theory gets modified not the valid data. Climategate showed how the correction clause was not achieved as the valid data was truncated to “hide the decline”. We need more eyes to find these temperature errors, and more open minded climate scientists and politicians to accept a modified global warming theory to better match the measured data!

Editor
April 17, 2010 7:33 pm

Leif Svalgaard (19:15:23) :
Dr. Leif:
Thank you for that. I was immediately reminded of the work of Benjamin Lee Whorf, an “amateur” anthropological linguistics enthusiast whose day job was Fire Safety Engineer for the Hartford Insurance Company. He noted that an awful lot of industrial fires started in “empty rooms”…. specifically, the “empty drum room”. What better place to sneak a smoke than in the “empty drum room”…. no danger there, right?
Gotta wonder how many foreign visitors who thought they knew Russian froze to death when they dressed for unexpectedly balmy weather.

Neel Rodgers
April 17, 2010 7:34 pm

While you have found a number of sites that have this error, I think there may be a slight exaggeration in how easy it is to do this. Does it create a significant jump in temps, yes. But that is why there should be some quality control of the data at the site. Every day, many of these places send 24-48 observations an hour (depending on hourly versus every half hour). And there is a huge number of these stations all over the world. It is kind of like saying “An Airplane crashed so flying isnt safe.”
As someone who deals with METAR day in and day out, I can say these errors arent quite as rampant as this leads to believe.
Also, when CORs (corrections to a wrong observation) are sent, it effectively overwrites the incorrect observation in the databases.
And lastly, could this be more of a problem of relying too much on computers as opposed to human interaction? In the old code, a space could just as easily be placed in the temp, or the minus sign could be forgotten. Human observers would catch and fix this, computers would not. And thus we would be in the same position we are now.

Sam Hall
April 17, 2010 7:35 pm

“The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.”
How do you download data on the day before it happens?

John Blake
April 17, 2010 7:41 pm

What would it take to reinstate the old system, or edit METAR to minimize error-prone transcriptions plus render the reporting exercise intelligible without abstruse bureaucratic nomenclature?
Really shouldn’t be that complicated to produce a halfway decent reprise of basic weather conditions, transcribed and reported in a format no more complex than various financial listings. Try omitting minus signs in trading data, and watch regulatory authorities go stark bonkers.

Editor
April 17, 2010 7:44 pm

Great job, you covered nearly all the nuances I’d be likely to think of.
This is sort of the ideal error – it’s more likely to make a temperature read too high, it’s more likely to affect recent temperatures than past ones, and being a warming error is less likely to attract the attention of people expecting to see warming.
If I had time, it would be fun to play with software to look for the problem in the records, even your +/- 5 error in temperate latitudes could be identified by looking at the time, sky cover, wind direction change, and even station history.

Benjamin
April 17, 2010 7:50 pm

@JDN
It can only lead to warmer temperature for the simple reason that it’s much much much likelier to forget/mistype an M than to add one where there isn’t supposed to be one !

Terry Jackson
April 17, 2010 7:54 pm

Nice work. The motto is “Garbage in Gospel out”.

R. Gates
April 17, 2010 7:54 pm

Excellent post. I’m sure (in fact, quite sure) that there will be many folks looking over your investigation very closely in the next few days…and of course, the bottom line of all this will be…
What difference or effect will this all make in the actual global temperature anomalies?
This was a great piece of work, and seriously you should at the very least be getting a big fat pay check from several agencies for doing such a thorough job of independent quality control work…
REPLY: Thank you sincerely. There’s always the tip jar 😉 – Anthony

Steve Koch
April 17, 2010 7:57 pm

It would not be difficult to program a comparison of satellite temps vs surface sensor temps. When they are sufficiently different for a particular location/time, it can be flagged by the program. Probably it would be possible to run historic data through this comparison program to find past mistakes.
Given the mess that is surface data, the papers based on this data should be rewritten based on satellite temps.

Editor
April 17, 2010 7:58 pm

Keith Minto (19:25:30) :

Excellent report, Anthony.
As Ric Werme said earlier, temperatures in the form of degrees Kelvin would solve the problem of negative temperatures.

Careful – that was a bit of a rant. (And an opportunity to say “Kelvins” instead of “degrees Kelvin” 🙂 .) For data in something like a METAR, expecting one of us Fahrenheit drooling Americans to accurately report in Kelvins would likely lead to all sorts of errors if any transcription is involved. Worse, it wouldn’t be immediately apparent, but it might have a better chance of being caught eventually, as the errors could be quite impressive.

Editor
April 17, 2010 7:59 pm

Impressive. You should write a book…

Benjamin
April 17, 2010 8:05 pm

@Sam
The map was online between 4/13 and 4/15, showing March average using data available on March 12th.
@Antony
If only high and low temperatures are kept for the day, then averaged to get the daily temperature, then averaged with the 30 other days of the month to get the monthly temperature, the impact can be quite big in cold regions.
Assume average temperature is -15 all month.
Assume just one mistake per month.
You get a monthly average of -14.5 instead of -15.
The thing is, to get the magnitude needed to reproduce Finland’s error (which is near zero therefore with less impact each type you flip the sign), you basically need to have the wrong side almost all month.
It would be interesting to check if they have a new employee feeding in the data…or maybe an intern… someone who wrote in minus signs all the time instead of M…. who knows ?

April 17, 2010 8:09 pm

This is an artifact of how I/O is normally done in Fortran. Many scientists program in Fortran because that is how they have always done it. Older versions of Fortran require fixed positions in data files, which allows for fast I/O but makes parsing prone to errors like Anthony has uncovered.
But who cares how fast file I/O occurs? The science world is locked into a poor programming paradigm which affects everyone’s productivity and accuracy.

April 17, 2010 8:11 pm

Wow. Who knew? And if I read your article right, since the records are discarded, you can’t go back year by year to correct them?
Reminded me of all the data corruption errors we used to get in the 80’s and 90’s converting data between different cpu architectures because people didn’t understand byte ordering differences (big endian processors vs little endian processors). Now I am certain when the climate centres converted their computer systems over time they would have caught that one. Certainly someone would have noticed 45 being changed to 54? Right? They caught that, right?
REPLY: I don’t know if NOAA archives all original METAR reports. If they don’t, they are in a pickle then. Maybe the FAA does. I don’t know that these things have been caught, because climate folks deal with monthly data, not hourly or daily. So if the data is automatically averaged as it comes in, chances are they’ll look at a month’s worth of data from a station and say “seems warm there”. If you are expecting it, like with polar amplification, then it may go unnoticed.
GISS missed an apparently identical error in Finland, and that probably would not have been caught if it were not a country that had a separate met service with a publication saying the country had been below normal in March 2009- Anthony

Alvin
April 17, 2010 8:12 pm

Neel Rodgers (19:34:22) :
While you have found a number of sites that have this error, I think there may be a slight exaggeration in how easy it is to do this. Does it create a significant jump in temps, yes. But that is why there should be some quality control of the data at the site. Every day, many of these places send 24-48 observations an hour (depending on hourly versus every half hour).

And yet, the errors persist…

Marvin
April 17, 2010 8:19 pm

Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
===
If people can’t understand what Anthony has written here in context then they aren’t going to understand anything anyway. This is quite interesting and I look forward to the real data crunching to find out what effect this is actually having. What I do love is how you make the extrapolation that glaciers are melting in your tongue in cheek manner after your disdain to the ‘accusations and extrapolation first’ of Anthony. Do we get to see your number crunching ever? I believe not.
We have however, seen the Arctic reach very close to normal ice levels with old ice being the major contributor. As far as deviation goes, it certainly deviated in the right direction, towards a large upswing recovering back to ~2003 levels. If there is anything you would like to actually contribute in the way of information about glaciers melting we’ll all be here.. waiting.

Kirk W. Hanneman
April 17, 2010 8:21 pm

Anthony, this is both fascinating and deeply disturbing. As you say, there needs to be some serious quality control done before any other analysis can be done with such a dataset, much less conclusions drawn. But the saving grace is that at least we have billions budgeted for climate research that can be used to fix these sorts of egregious and pervasive errors, right? We need to know urgently if the high temperatures consistently reported in places like SIberia are correct, or if they are significantly influenced by these mistakes.

Gerald Machnee
April 17, 2010 8:27 pm

I think there is also an additional problem. The USA is using Fahrenheit while most countries are using Celcius.
A number of years ago the reports were quality controlled by humans. When they retired they were not replaced. Now we see what the computers are doing.

kim
April 17, 2010 8:31 pm

Judith Curry, in comments at Bishop Hill and at Pielke Fils, is saying that the whole temperature and paleo records need to be reworked. I think she’s tired of the foolishness, and I think there are a lot of other people who are tired, too.
==================================

Rob Dawg
April 17, 2010 8:33 pm

Lord Kelvin smiles.

J.Peden
April 17, 2010 8:34 pm

Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
Bravo! Spoken like a true Post Normal Scientist, Nick. You need to ask for a raise.

Bob
April 17, 2010 8:35 pm

Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience. I was worried from the beginning of my interest in so-called AGW about the way data were being handled and discussed. I have been a daily (if not hourly) visitor to your site for about two and a half years. I feel in my gut that you may have found a major source of the confusion which arises when one tries to follow the arguments for increasing temperatures. This may be true not only for the how supposed global “rise” in temperature is occurring, but also for how the arguments have never reached firm conclusions, even among the AWG supporters. If the data are subject to (possibly) random influences which have a major effect, then the foundation will never be solid and one can never feel truly grounded in the conclusions. As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry. I am eager to read the responses to this post. Thanks for this information.

rbateman
April 17, 2010 8:36 pm

Leaves little doubt about starting over.
Now we know how bad it can get when you have grid points depending on a single station, and that station has METAR data errors.
I pretty much have an idea of how a single day can mess up a month, as I do as much station graphs from daily data.

mpaul
April 17, 2010 8:36 pm

Once the Finland error was corrected, what was the effect on the average monthly temp for Finland?

April 17, 2010 8:39 pm

REPLY: I don’t know if NOAA archives all original METAR reports. If they don’t, they are in a pickle then. Maybe the FAA does. I don’t know that these things have been caught, because climate folks deal with monthly data, not hourly or daily.>>
If that error crept in undetected for so long I would have to believe there are others, and if the original records no longer exist, then there is no way to know. I was being a bit fascetious on the big endian little endian thing, but I would have to believe they caught that one, the total of the errors introduced would be so enormous it would be unlikely to go undetected. I think. I hope.
But there are other issues and given the amount of lost data, undocumented code, even their backup systems don’t sound like they were properly maintained. Remember when computer viruses first started appearing? Most of them were instantly noticeable so people had to deal with them. There was one very nasty little monster named “Ripper”. It would sit on your computer and do mostly nothing. Every 1024th write to the hard drive it would swap two bytes, and then do nothing again. Word documents would suddenly have spelling errors. Spread sheet formulas would develop errors. But table data? No one would notice for a long long time. I cleaned up a mess at a research facility once where the researcher had 10 years of clinical trial data corrupted, and no way to recover it because there was no backup system and the paper records had been trashed as the data was entered.
Given the gaping hole you just exposed, and the clear lack of professionnal standards when it came to not just writing and documenting code, but managing the integrity of the data itself, I can’t but help thinking that it is all junk and we need to start with the paper records themselves. again.

Michael Larkin
April 17, 2010 8:40 pm

Terrific post, Anthony. Every new wrinkle we hear about just makes it more and more obvious that temperature data is suspect.
Terry Jackson (19:54:00) :
The motto is “Garbage in Gospel out”.
QOTW?

Jim Arndt
April 17, 2010 8:42 pm

Hi,
Here is global warming for you. I think the ice sheets will melt soon. This is for real.
Vostok, Antarctica
http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-78.44999695,106.87000275
Tuesday Night
Overcast. Low: -103 °F . Wind West 17 mph . Windchill: -148 °F .
Wednesday
Overcast. High: -81 °F . Wind West 17 mph . Windchill: -139 °F .
Wednesday Night
Overcast. Low: -110 °F . Wind SW 17 mph . Windchill: -155 °F .
Thursday
Partly Cloudy. High: -77 °F . Wind WSW 20 mph . Windchill: -158 °F .
Thursday Night
Overcast. Low: -104 °F . Wind SSW 6 mph . Windchill: -133 °F .

Wren
April 17, 2010 8:46 pm

There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Switching from requiring a minus sign to requiring an “M” in front of the value to indicate it is negative could cause a one-time step up in a trend, if people are more likely to forget an M than a minus sign. But after that it should not affect the trend, unless again there was an increasing tendency to forget to use the M.
Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value.

Mike McMillan
April 17, 2010 8:49 pm

Sam Hall (19:35:58) :
How do you download data on the day before it happens?

Use climate models.
This seems like a fertile area for research, for those of us less technologically inclined. Just download a station’s data, plot it somehow, and look for spikes.

Tim F
April 17, 2010 8:51 pm

I have seen this effect myself. Some years ago when I was in the AF we were flying through the Med in Jan or Feb and we asked the weather briefer to pull up the METAR for home–Grand Forks, ND. He read us the observation and when we asked how cold it was he said that “it looks like the mean temperature is 20 degrees.” We had to tell him that the letter M meant a negative temp–part of daily life in ND, not so much in Crete.
Good piece of detective work Mr. Watts.

mindbuilder
April 17, 2010 8:52 pm

Has anybody noticed that in mid 1966 the US started rounding negative temperatures hotter? From the Wikipedia article on rounding:

In a guideline issued in mid-1966,[8] the U.S. Office of the Federal Coordinator for Meteorology determined that weather data should be rounded to the nearest round number, with the “round half up” tie-breaking rule. For example, 1.5 rounded to integer should become 2, and −1.5 should become −1. Prior to that date, the tie-breaking rule was “round half away from zero”.

Michael Larkin
April 17, 2010 8:56 pm

PS – Anthony: $50 on its way to you via the tip jar – wish it could be more. Hope it’s a useful contribution to your upcoming trip.

sierra117
April 17, 2010 8:59 pm

An outstanding piece of scientific forensic investigation Anthony. If the archived reports are available, it wont be difficult to develop some code to correct the record.
Looking at the March GISS its obvious NASA doesnt pay any attention to the material its producing. The Finland data is obviously wrong.
More importantly, this post just highlights the appalling state of our temperature record. Our policy makers would do well to take note.

JRR Canada
April 17, 2010 9:03 pm

I agree with KIrk.The deeper we get to look the sadder this science seems.Is it because these people are all govt employees?I mean the errors are so pathetic its like they just do not care, no repercussions for poor work no consequences for mistakes.Lets save billions, their work can not be trusted? Stop funding.

GregO
April 17, 2010 9:04 pm

Great work.
Reported thermometer readings matter. Instrument bias and uncertainty matter. Missing minus signs matter. They matter because we are attempting to measure a temperature rise allegedly caused by man-made CO2 and said temperature rise is conjectured to threaten to destroy the world. In this debate, thermometer readings matter.
The conjecture that world-wide AGW is from man-made CO2 is weak due to a paucity of observable evidence. Melting ices caps? What melting ice caps? Rising sea levels? Really? Disappearing glaciers? So what? They cover only a fraction of Earth’s surface and though they shrink and grow, are hardly disappering (much to the embarassment to IPCC). Ifs, coulds, maybes. On and on.
Isn’t this all about (global) catastrophic warming? If it’s really getting catastrophically hotter here; show me the thermometer readings. They will show distinct unmistakable catastrophic heating if the readings are any good and if it is really getting catastrophicall hotter, the thermometers will show it’s getting catastrophically hotter. Period.
What we actually see in surface thermometer temperature records is inconsistency, instrument bias, human error, and this engineer has had it up to his keester with wishful thinking, madness of crowds, with frauds and charlatans showing up to the debate with a suitcase full of political baggage, advocacy, and empty rhetoric. All we need in this debate is the temperature readings.
Kindly, Anthony, you and others have shown me the thermometers. I’m sure the analysis to follow will be interesting; but to me not as interesting as this post. I’ve seen enough to know that any global catastorphic warming from man-made CO2 is either being smothered by negative feedback within the climate system itself, and/or is so minute as to be within the measurement error and of little consequence in comparason to world destruction from AGW.
What is really fascinating to me is how a weak conjecture like AGW has grown into a mass delusion driving drastic political policy change despite a complete lack of observable results/catastrophes.

Sera
April 17, 2010 9:04 pm

I’m having the same problem, sort of, using the drifting bouys in the arctic. Missing minus signs are screwing up the daily averages. All of them- none seem to be exempt.
http://www.sailwx.info/shiptrack/shipposition.phtml?call=25626
As you can see, the temp goes from -5 to 5, and then back to -5. You can also export this page into Excell easily.
Missing minus signs- been there, doing that.

Cement a friend
April 17, 2010 9:04 pm

Great post Anthony. Please get it published in a peer review paper so the AGW crowd can not dispute it.
Ken Smith (18:56:17) : The answer about local effect from CO2 is no. There is plenty of evidence about very high local CO2 levels. I have personally measured over 1000ppm around a furnace exhaust stack. No one has been able to demonstrate a change to local temperatures resulting from high CO2. W Kreutz (1941) made measurements of CO2 at four heights in association with climate data over 1.5 years. The data in his paper shows that solar radiation leads local air temperature which in turn leads CO2 levels. I have noted some criticisms of Kreutz but it would appear they these critics a) did not read the paper b) can not understand (technical) German or c)deliberately ignored parts of the paper and his intent. The paper shows that he, from time to time, during the 1.5 year period made CO2 readings at 1 or 2 hr intervals and his figure 1 (24 Sep 1940 to 28 Sep 1940) shows an example of a daily measurement. In this example lower temperatures are associated with higher CO2.
However, clouds obviously have a local effect. Humidity and wind chill have a personal temperature feel effect. Convective heat transfer is an important effect for local near surface air temperatures. This is particularly noticeable near the sea. We have a thermometer in the car (under the bonnet near the front) giving outside air temperature.
We noted on Friday night around 6PM (at sunset) the temperature where we live (bush and 10 km from the coast) at 22C (slight wind from west) while the temperature at the coast opposite a well known beach was 26C (slight wind from east over the sea)
Beside the effect of wrong data inclusion shown above and UHI effects, it has been pointed out on this site that the GHCN database also includes incorrect data splicing from a number of sites now being included as airports eg data on Darwin. Most airports did not exist before 1942, yet they have spliced data. Then many airports have been substantially upgrade in the last 20 years, causing UHI effects and maybe splicing. This is shown by the Chiefo’s dT/dt analyes method http://chiefio.wordpress.com/2010/04/11/the-world-in-dtdt-graphs-of-temperature-anomalies/
Anthony, I welcome your blog and regularly look at this site when I look at the internet. Great posts from Willis, Dr Roy Spencer, and many others. Thanks for your efforts

Capn Jack.
April 17, 2010 9:11 pm

The M in this case stands for Mother.
The Mother of all outliers. It was long read but worth the effort.
Do people even do any eyeball checks, what about data integrity analysis, those sub zero anomalies would stand out. They are so far off the curve, they must skew any analysis.
But as Anthony points out they are used primarily as aero warning stations, not climate stations. But people using this for climate modelling that is another story.
Thanks.

Sera
April 17, 2010 9:12 pm

You wanna see missing values for air temp?
http://www.sailwx.info/shiptrack/shipposition.phtml?call=48534

Sera
April 17, 2010 9:14 pm

I meant ‘missing minus signs’, not values. Oops.

Capn Jack.
April 17, 2010 9:15 pm

Oops she be the mother of outliers, me mateys, aargh, memo to head remember persona.

Capn Jack.
April 17, 2010 9:18 pm

Sera I think you just found ol matey Kevins missing ocean heat, you sluethed the buoys stole it.

Mike G
April 17, 2010 9:19 pm

You must really be on to something here for Nick to come like he did with the venom. Just who are the deniers anyway?
First thing I noticed is this would tend to produce more warming the farther north you go. How convienient is that?

Sera
April 17, 2010 9:28 pm

Hi Capn Jack- yes, someone should tell him that his missing heat is at N 84, W 040. It is currently 14.6C. That’s pretty hot this time of year…

Mark
April 17, 2010 9:29 pm

Another error that works in favor of the believers. The odds of this happening are remote. And it’s not just us noticing this:
Professor Watson, who served as chairman of the IPCC from 1997-2002, said: “The mistakes all appear to have gone in the direction of making it seem like climate change is more serious by overstating the impact. That is worrying. The IPCC needs to look at this trend in the errors and ask why it happened.”
http://www.timesonline.co.uk/tol/news/environment/article7026932.ece

jack morrow
April 17, 2010 9:33 pm

Good work sir! As a pilot I always hated those metar reports compared with what I first learned to read. I’m glad not to have to decipher them anymore. I can see how easily people could misread them and give bad reports.

pat
April 17, 2010 9:38 pm

Wow. And this is very serious. As bad as the mathematics for the Mars probes which caused calamitous results . The Western world is in the hands of scientific and political dolts.

wmsc
April 17, 2010 9:40 pm

http://www.wunderground.com/history/airport/PANC/2009/12/8/DailyHistory.html
I’m guessing this is the tower at the airport, but wow, what a temperature variation, +25F in 9 minutes.
12:00 PM 15.8 °F
12:08 PM 41.0 °F
12:17 PM 15.8 °F
REPLY: Yep. you found another METAR coding error where the M was left off M05 on a METAR report issued at 12:08PM
SPECI PANC 082108Z 00000KT 3SM R07R/0600V1200FT BR FEW001 05/M12 A3028 RMK AO2 VIS N 1 BR FEW000
– Anthony

kim
April 17, 2010 9:44 pm

GregO 21:04:11
Good rant there. AGW is indeed an ‘Extraordinary Popular Delusion and Madness of the Crowd’. The herd has been whipped along by a few unscrupulous scientists, a few greedy speculators, and a few power-hungry politicians. Most of the rest of us have just been along for the ride.
======================

jaypan
April 17, 2010 9:44 pm

There are well funded institutions, getting all these raw data, processing them in order to “enhance their value”, then tell the world about the dangerous findings and projections, feed politicians with action plans and even get heard.
Why can’t these highflyers not do their homework first, like check the siting of their reporting stations, doublecheck the numbers they report in case of suspicious results, just be sceptic, as a pre-post-normal scientist has to be?
Instead the people doing this work voluntarily are called deniers, flat-earthers, and looked at downward of their ivory tower.
I am getting sick of all these a……..

Leon Brozyna
April 17, 2010 9:45 pm

A superb bit of detective work.
This is a fine concretized and particularized example of the data difficulties highlighted in the famed Harry_Read_Me file. Brings it into focus for the average layman.
BTW, thanks for the link to the updated work you’ve done with Joseph D’Aleo. I’m sure it’ll make for some fine reading tomorrow morning.

GregO
April 17, 2010 9:46 pm

Bob (20:35:01) :
“Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience… As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry.”
I too in an earlier lifetime had the opportunity to deal with a similar type of problem in precise data transcription. What we found is that to assure better than 99% data integrity in data transcription the following guidelines needed to be understood and followed: 1) Limit the number of individuals making the transcriptions, make them responsible for error rates, reward accuracy. 2) Audit transcriptions in as close to real time as possible – provide immediate feedback to transcribers, make corrections as immediately as possible. (In this particular situation, to find an error it took 24 hours for it to show up, some delta T to fix it, another 24 hours for the “fix” to show up. 3) error, if left to accumulate, was logarithmic in growth and created a constant data offset.

GregO
April 17, 2010 9:51 pm

Cementafriend,
“There is plenty of evidence about very high local CO2 levels. I have personally measured over 1000ppm around a furnace exhaust stack. No one has been able to demonstrate a change to local temperatures resulting from high CO2. W Kreutz (1941) made measurements of CO2 at four heights in association with climate data over 1.5 years. The data in his paper shows that solar radiation leads local air temperature which in turn leads CO2 levels. I have noted some criticisms of Kreutz but it would appear they these critics a) did not read the paper b) can not understand (technical) German or c)deliberately ignored parts of the paper and his intent. The paper shows that he, from time to time, during the 1.5 year period made CO2 readings at 1 or 2 hr intervals and his figure 1 (24 Sep 1940 to 28 Sep 1940) shows an example of a daily measurement. In this example lower temperatures are associated with higher CO2.”
Fascinating. I would not have thought to ask this question as I know quite well that localized CO2 can be quite high and not change local temperature almost intuitively from my experience – but I am impressed with your quantitative answer.

April 17, 2010 9:51 pm

Excellent sleuthing Anthony!
Where is the data cleansing? Data cleansing is common in a host of disciplines. In my mundane field, forestry, we call measuring forests “mensuration” or “cruising”. Data is either hand-written in field notes and key-punched in later, or entered in a data logger in the field. At some point in the data gathering process, it is HIGHLY RECOMMENDED that some sort of data cleansing/editing software be applied. Because mistakes are common. A decent data cleansing algorithm will note that a 3 inch diameter tree cannot be 100 feet tall. Or that it is unlikely that a 30 inch diameter tree is only 10 feet tall.
Since THE FATE OF THE WORLD depends (these days) on taking the globe’s temperature, it might be handy if the GENIUS CLIMATE SCIENTISTS take the same precautions with their data that a lowly forester does.
Note to NOAA. Please clean your data. If the job is too tough, too confusing, and you don’t even know where to begin, call a forester. They can help you out.

Editor
April 17, 2010 9:55 pm

stevengoddard (20:09:09) :

This is an artifact of how I/O is normally done in Fortran. Many scientists program in Fortran because that is how they have always done it
But who cares how fast file I/O occurs?

The supercomputer community cares. When I worked for a mini-supercomputer company Amdahl’s Law was frequestly quoted – “Overall system speed is governed by the slowest component.” If you spend 40% of the time reading data in, 40% writing processed data out, and 20% of the time crunching numbers, then speeding up the compute phase by 2X will reduce the runtime by 10%. Among the answers are striped file systems spread across multi-node file servers linked together with multi-GB networks.
Binary I/O in modern Fortran applications can be extremely fast. Decoding free form text input is much slower, but that has always been the case.
Besides, people can drop the M or use ‘-‘ just as easily in fixed format records as they can in free format records.
————–
davidmhoffer (20:11:46) :

Wow. Who knew? And if I read your article right, since the records are discarded, you can’t go back year by year to correct them?

Reminded me of all the data corruption errors we used to get in the 80’s and 90’s converting data between different cpu architectures because people didn’t understand byte ordering differences (big endian processors vs little endian processors). Now I am certain when the climate centres converted their computer systems over time they would have caught that one. Certainly someone would have noticed 45 being changed to 54? Right? They caught that, right?

Endian problems for integer data generally resulted in extremely wrong data that would be impossible to miss, and even worse for floating point data.
Your “45” value is hex “0x2d”. Byte swapped in a 32 bit integer, it would be 0x2d000000, or decimal 754974720. Hard to miss, unless Al Gore is looking at temperatures recorded underground.

REPLY: I don’t know if NOAA archives all original METAR reports. If they don’t, they are in a pickle then.

This is exactly why the original data must be preserved. Warts and all.
Hmm, here’s a thought – global warming is said to impact the winter the most. Winter is when the negative temperatures occur (°C or °F). Wouldn’t it be embarrassing if a substantial portion of that impact was really due to dropped M-signs?

Frank K.
April 17, 2010 10:13 pm

Excellent post, Anthony!
Jim Arndt (20:42:11) :
The coldest temperature ever recorded on earth was at Vostok:
http://en.wikipedia.org/wiki/Coldest_temperature_recorded_on_Earth
That temperature was -129F. The forecast low for Wednesday is not too far from that. And that’s mind-numbingly cold!

April 17, 2010 10:15 pm

Ric Werme (21:55:21) :
File I/O is very slow compared to other computer operations. Any app which is dominated by file I/O, either does little computing or is very poorly written.
GCMs might do sixty seconds of file I/O, and then computations for three days. The time required to parse the input files is unimportant.

KimW
April 17, 2010 10:24 pm

Many years ago, in the late 1960’s, I toured the NZ Defence pay section, which was changing over to computer payroll for about 13 000 personnel. They commented that the entire exercise was being done twice by two separate teams of data entry operators and separated by a day. Each teams efforts were then run in parallel through readers and entry errors found. I recall the comment was that they caught a lot of entry errors, but these were trivial compared with those found on the original

Anthony Scalzi
April 17, 2010 10:30 pm

Sera (21:12:55) :
There are 856 measurements listed for the buoy you posted. Of those 856 measurements, 42 were missing, and 60 had sign errors. Correcting for the sign errors changed the average from -17.0 to -20.1, a 3 degree change.

Eric Anderson
April 17, 2010 10:43 pm

I thought I was beyond being astonished, but this leaves me speechless. Great investigative first steps.

April 17, 2010 10:47 pm

Anthony,
Thank you for sharing this original work in real time. An incrementally growing warm bias like a virus, has been growing inside of the data. A month of averaged data is worthless, if it can be yanked around like this, via a single error for a few hours or even days.
The implications of the impact are clear, this is a long lived and growing problem for the data. That is, as the number of *total* stations goes down in number, and at the same time, the number of airports as a % of the total is raised, this bias would continue to grow inside of the data. I am starting to wonder if GISS should drop its Southern most stations completely, and no longer report Antarctic temps.
I hate dropping any station, however when you look at their maps of world, it never has looked right to have the coldest place on the planet, glowing RED all the time.

April 17, 2010 10:49 pm

This may have something to do with the trend we have seen in the data for less cold winters rather than warmer summers.

Anticlimactic
April 17, 2010 10:51 pm

This is a real can of worms! The data is not fit for purpose.
It is very difficult to program round the inventiveness of human error. A lot of the METAR data can be flagged with ‘if position 4=M’ as a lot seems to be the first M is missing. Possibly if the value of the previous reading is different by an unusual amount it should be flagged [10 degrees?]. If data is flagged it may still be correct but should be checked by a human.
I hope all the source data is still available as ALL the results are suspect and need checking. If the source is not available then in scientific terms all the results are invalid and should be scrapped.
If the source data is available then, as you show, the anomalies are easy to spot on a graph. With the web it should be easy to get volunteers to visually check these graphs for anomalies, perhaps in batches of fifty, and checked by more than one person, starting from the poles and working towards the equator. A human should be able to process a normal graph in under 2 seconds.
The long term solution would be to have the thermometers showing the temperature in degrees Kelvin, and recorded in Kelvin.
It is ridiculous to have the fate of the western world riding on such an amateur shoestring operation!

Sera
April 17, 2010 10:56 pm

Anthony Scalzi (22:30:12) :
Yep- lots of buoys, lots of errors…but the nearby stations aren’t so bad.

Louis Hissink
April 17, 2010 11:02 pm

I am fast beginning to think this global warming issue isn’t so much a situation of a bunch of wannabe’s manipulating data for nefarious purposes as simple institutionalised incompetence at work, with the data accepted at face value and, for reasons yet to be worked out, biassed to the high side.
In addition the mindset behind AGW also seem oblivious to the principles underlying the scientific method – that when a hypothesis is falsified by the data, then it’s the hypothesis which has to be relooked at. But do Trenberth and others do? Postulate that some hidden, unknown factor is operating in the climate cooling things down.
Absolutely no thought seems to be directed to the possibility that the AGW hypothesis might be flawed, or just plain wrong. No, it’s accepted as being true not from experiment or data but by peer consensus.
No wonder countering science with science doesn’t work – the AGW isn’t a scientific hypothesis in the first place and out mistake was assuming it was.

Theo Goodwin
April 17, 2010 11:03 pm

I have a complaint about climate scientists’ use of the word “anomaly.” The word has a very clear and distinct meaning, despite all the efforts of Thomas Kuhn. An anomaly is a fact of experience that does not fit with the accepted world view and stands as a challenge to it. So, to say that temperatue measurements are “anomalies” is bizarre. It is tantamount to saying that they are revolutionary. That is nonsense. Could not climate scientists call them by another name? Why not call them temperature measurements? Isn’t that what they are? Whooh! Climate science is wierd, even among the critical. Who parented these people?

Stephan
April 17, 2010 11:04 pm

The ONLY data you can trust is RSS and UAH and yes it shows considerable warming BTW for the first 3 months of 2010. The rest is absolute trash. The warmistas should support it hahahaha

jknapp
April 17, 2010 11:09 pm

If a station reports hourly and only makes 1 drop a minus mistake that day and the daily average is calculated by averaging high low then it almost certain that that mistake will be the high. In otherwords using high low averaging selects the bad data to use from the available data. So even low error rates could have siginificant impact. If there is a 1% error rate, that would be about 1 error every 4 days and that erroneous high reading would be the one used on those days. Suppose a station had a daily low of -20 and a daily high of -10 for 4 days, The average tempertature for the 4 days is -15. Now suppose one of the temps in the 4 days had a drop minus error. On average that would change a -15 to a 15. That would be that days high. So that day would average (-10 + 15)/2 = 2.5 Averaging the 3 -15 ‘s and the 2.5 gives -10.6. That is 4.4 degrees above that actual average of -15. Quite a difference.
Wren, You state that this should not effect the trend. However, Anthony showed that the percent of airport stations being used is increasing. If there are a greater number of warm errors in the airport data and the percent of airports used is increasing then the trend will be effected.
By the way, Anthony, do you know if the data is gathered from METAR reports or is climate data sent in some other format. (Which if it is transcribed from the METAR might still contain the errors but also might be cleaned up a bit)

Bernd Felsche
April 17, 2010 11:14 pm

What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO
??

Al Gored
April 17, 2010 11:15 pm

Truly outstanding detective work here Anthony!
I learned a lot about a lot!
So, is what you are basically saying is that the whole surface data set is so riddled with errors that it is mush even before they deliberately tweak it?
Have you found any examples where the M was erroneously added to suggest a colder temperature? If not, it gets more convenient.

wayne
April 17, 2010 11:21 pm

Wonderful investigation Anthony.
Where are all of the errors to the negative side.
I’m really kidding because I am sure that if there are any at all then they are few and far between. That is the way all other “errors” in climate science being compiled are, positively skewed, always positive, always positive, always positive. Feed that into statistics and give me the verdict!

Konrad
April 17, 2010 11:23 pm

Anthony,
This is an excellent post. I look forward to your further analysis of the data. There are already many good reasons why temperature data from airports is problematic for climate analysis. This added human error could have been causing a significant warm bias in more polar surface station data for many years.

Not Again
April 17, 2010 11:31 pm

Wow! Great work Anthony!
Maybe the AGW crowd is just not as observant as they should be.
The AGW group seems to be attempting to prove GW exists based upon miniscule temperature increases – where you have show the errors alone are orders of magnitude greater.
So how do we really know what the truth is?

Jantar
April 17, 2010 11:32 pm

It comes back to data being fit the purpose. Aviation data is great for flight planning, but obviously not suitable to climate investigations. Its time to scrap land based thermometers for global climate and instead rely on satellite data. Land based thermomometers are still ideal for tracking local temperatures as no homogenisation is needed and errors like that pesky missing “M” are easily picked up.

Al Gored
April 17, 2010 11:36 pm

GregO (21:04:11) wrote: “What is really fascinating to me is how a weak conjecture like AGW has grown into a mass delusion driving drastic political policy change despite a complete lack of observable results/catastrophes.”
Just turn it around. AGW was just the chosen tool for “drastic political policy change.”
Its the ideal One World Watermelon/Wall Streeter trick. That has always been clear but now it is becoming screamingly obvious.
Just look at the media or the whitewashes, or follow the potential money or the power. This Big Lie is going to be like some kind of zombie which will take a lot more than rational scientific logic to kill off. If it was that simple it would already be buried. There’s too much riding on this AGW pony for them to just put it to pasture. But with information like what Anthony just came up with, and so much, much more lately, the tide is hopefully turning. It is getting so basic and obvious now that common sense takes over. I hope.

crosspatch
April 17, 2010 11:37 pm

So this is the second kind of reported systematic error. The first was the fact that sometimes a month’s mean temperature is duplicated from the previous month. This isn’t a problem in the middle of summer or middle of winter when average temperatures between two months might be the same (July and August or January and February). The problem was Steve M. discovered it happening in the fall so a month got reported much warmer than it should have.
Now we have this sign error on the temperature. While that doesn’t make much difference when temperatures are near zero, they make a huge difference in places where temperatures are far from zero.
I would love to see the record straightened out and see what the global service charts look like after correction.

E.M.Smith
Editor
April 17, 2010 11:52 pm

Ric Werme (19:58:33) : For data in something like a METAR, expecting one of us Fahrenheit drooling Americans to accurately report in Kelvins would likely lead to all sorts of errors if any transcription is involved. Worse, it wouldn’t be immediately apparent, but it might have a better chance of being caught eventually, as the errors could be quite impressive.
Of course, the rational answer is for America to report in degrees Rankine…

Doug in Seattle
April 17, 2010 11:53 pm

Anthony, the solution to the problem is clear. You and the other climate auditors must not be allowed to look at raw data, in any form at any time, from now on.
To paraphrase a famous climate “scientist” – All you want to do is find something wrong with it.

Jimmy Haigh
April 18, 2010 12:31 am

Again it just shows that the AGW peers aren’t very good….

DeNihilist
April 18, 2010 12:54 am

OMG, feeling like I did that fateful November evening when the Climate Research Unit mails were liberated.
As Kate Bush would say “I feel like I’m floating in Gaffa”

899
April 18, 2010 12:56 am

THAT, Anthony, was a tour de force!
Now I wonder: In the name of eliminating the ‘M’ errors, why does not the WMO adopt the Kelvin temperature scale?
That way there would be no signed error to deal with.
Now what bothers me otherwise is that there seems to be no computer program which rings bells and flashes lights when there are such anomalies (extremes) exhibited in the data sets.
You’d THINK that such a large variation would get ‘somebody’s’ attention at the time …

aurbo
April 18, 2010 12:56 am

re wmsc (21:40.45)
“http://www.wunderground.com/history/airport/PANC/2009/12/8/DailyHistory.html
I’m guessing this is the tower at the airport, but wow, what a temperature variation, +25F in 9 minutes.
12:00 PM 15.8 °F
12:08 PM 41.0 °F
12:17 PM 15.8 °F
REPLY: Yep. you found another METAR coding error where the M was left off M05 on a METAR report issued at 12:08PM
SPECI PANC 082108Z 00000KT 3SM R07R/0600V1200FT BR FEW001 05/M12 A3028 RMK AO2 VIS N 1 BR FEW000
– Anthony”
There’s more to this one than just a missing “M”. As all METAR temps are are coded in °C, the 41²F reading is certainly from an entry of 05, but the preceding and following temps of 15.8°F would have to have been entered as M09 to equate to 15.8°F. So there was more to it than the simple dropping of the letter “M”.
Errors were more frequent where observers would enter the data manually on a telautograph device which transmitted hand-written information which because of poor penmanship could easily be misinterpreted when it was read on the receiving end to be manually transcribed to the daily weather observation (WABAN) file.
I’ve seen many errors of the type noted by Anthony from the METARS still extant on supposedly QC checked data from NCDC in their monthly and historic summaries. The main relevance to climate distortion is that the
METAR errors with a missing M, and earlier SA records with a missing – sign, could only result in the reported temperatures being higher than the observed temps.
As to rounding up. Many daily mean temps at sites that do not take hourly readings are compiled from simple whole number max/min temps. (See the daily C6 local climate data sheets). Currently these averages ([Max+Min]/2)are rounded up to the nearest whole number to provide a daily whole number mean. If one asswumes that in a 30-day month, on average half of the Max+Min sums would be an odd number, and that the mean of these numbers would necessarily be rounded up half a degree, you would have 15 days with the reported mean temperature being half a degree too high which would raise the monthly sum of the means by 7.5 degrees finally resulting in a monthly mean 0.25 degrees too high. For this reason, when the full month’s data is recomputed, the algorithm sums all of the daily maxes and mins and for a 30-day month divides this sum by 60.
In earlier times, even the small rounding up errors were compensated for by
rounding up fractional numbers when the unit digit was odd, and rounding down when the unit digit was even.
Finally, in radiosonde data for upper air temperature observations, in order to maintain 5-digit coding groups and still be able to report temps in 0.2°C increments, a 3-digit coding is used for tempertures; When the temperatures are positive, the third digit (the decimal place) is an even number digit and when negative the decimal is given an odd number digit. (Dew point readings use a totally different code using 2 digits which represent dew point depression.) The only reason that the coded message is not completely chaotic is because human error is elimimated by having the encoding done automatically by the RAOB ground equipment.

April 18, 2010 12:58 am

Anthony: This is great work, many good comments too. No magic solution to the data entry error problem just lots of had work and constant checking. We have all probably have a raft of horror stories. That is why I never allow anything out of my office without someone checking my data, calculation and conclusions. Thanks for your efforts and all the good suggestions from everyone.

David, UK
April 18, 2010 1:04 am

One of the best posts I’ve read in a while, many thanks for your tireless efforts Anthony. It illustrates very well why human error, where mathematics are concerned, will always favour warmer temps, rather than cooler. And how convenient.

Frank Lansner
April 18, 2010 1:04 am

Extremely interesting, and I wonder how far the waves from this will go, thankyou from Denmark!
(It must have taken a looooong time to prepare!)

Robert Wykoff
April 18, 2010 1:06 am

That kind of thing makes my teeth itch. I recently started a custom software company. I take it as a point of pride that I make it impossible for operators to enter spurious data. In addition to that, I also take it as a point of pride that operators can also enter data in a variety of manners based on their personal preferences and however they enter data will be correctly interpreted. A simple thing as interpreting a “-” or an “M” is a joke. Who ever developed this system probably had a team of PHD’s and grants for eleventy billion dollars. Right now its just me, and I can do way better than that. That has been my experience with scientists writing software though. My day job is in automation, and from time to time we do projects that were initiated by our R&D department. For some reason PHD’s always assume everybody using their software will also be PHD’s. Attention to seemingly small details like that make your software usable or worthless. Keep the scientists away from coding, thats what engineers are for, we at least understand the target audience.

April 18, 2010 1:12 am

I wish someone could find a way to do something similar with the shift from manual to automatic measurements. I know for a fact that some station attendants simply used to make up measurements and these “estimated” (fraudulent) readings may well have been drastically overestimated. Then you’ve got water ingress (dripping from clothes) which could have reduced temperature.
Then you’ve got the timing issue which in hotter countries may have encouraged measurements to be made earlier in the morning and later at night. I.e. toward the colder periods.

Alan S. Blue
April 18, 2010 1:15 am

“Of course, the rational answer is for America to report in degrees Rankine…”
Bite your tongue. Hard.
Kelvin!

TerryS
April 18, 2010 1:32 am

It will be interesting to see what the quality of data was like for Europe over the past week. With practically all of Europe grounded I can imagine the data reporting being sloppier than usual. If the data is not being used for its primary purpose (aircraft) then those entering it will not take as much care and more errors will be probably be present.

Fudsdad
April 18, 2010 1:39 am

You only had to read the daily weather reports each day in the paper to see that Helsinki and nearby St Petersburg were well below freezing for most of March. Is there no common sense check by GISS or whoever?

crosspatch
April 18, 2010 1:44 am

“the rational answer is for America to report in degrees Rankine”
Shhh! They’ll hear you!

April 18, 2010 1:45 am

Wow!
This looks quite impressive. I’ve only had time to skim over it, but will read it carefully this evening.
A whole Saturday?
You must have some kind of distraction-proof bunker or something.
I’d need at least a week to write something this long!

Sera
April 18, 2010 1:46 am

Rankine? Do they still do that? Wait, let me go get my bamboo slide rule…

Anton
April 18, 2010 1:51 am

So much for the “hottest March on record.” What’s next, the “hottest April on record?” NOAA is ALWAYS wrong, but it always takes some unpaid and viciously-ridiculed “denier” to find the mistakes the so called experts continuously miss.
The AGW crowd lives and breathes false, or falsified, data, but claims skeptics are dishonest. More projection. Fortunately, the public is wising up. We know when it’s cold outside, no matter what NOAA claims with its doctored satellite maps. And older people can well remember much warmer Marches.
Can “climate scientists” do ANYTHING to change the climate? Can following any of their recommendations, or all of them combined, do ANYTHING to change the climate, or to stop climate from changing? If the entire world abandoned fossil fuels tomorrow, shut down every coal-powered electric plant, and turned off every electric appliance and light, would it make the slightest difference to the climate? If not, then all their huffing and puffing is for naught; they are wasting their lives on the acquisition of useless, and obviously largely erroneous, information. Meteorologists save lives by warning of actual coming storms. Who have “climate scientists” ever benefited other than themselves and investors?
Take away all the grant money, and climate scientists will become extinct within a single generation. They are nothing more or less than a reinvention of the alchemists of old, operating on the same inductive reasoning and consensus thinking.

Larry Fields
April 18, 2010 1:56 am

Great job, Anthony! I think that you should buy yourself a deerstalker cap. You know, the kind that Sherlock Holmes was fond of. And posting a photo of yourself wearing the new chapeau would be a nice touch. Indubitably!

Editor
April 18, 2010 1:58 am

Great work. I’m about to read this through again (now that I’ve finished my coffee).
KevinUK’s maps allow a visual ‘shock’ of the high proportion of airports:
Fully interactive map here: Climate Applications – Airports (note slow to load – keep hitting ‘no’)
Static picture (if I’ve got the link right):
here

Joshua Zuckerberg
April 18, 2010 2:00 am

Anthony, I noticed that in the original version you claimed that
“Around 1990, NOAA began weeding out more than three-quarters of the climate measuring stations around the world. They may have been working under the auspices of the World Meteorological Organization (WMO). It can be shown that they systematically and purposefully, country by country, removed higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”
whereas in the new version this paragraph was silently “adjusted”
“Around 1990, NOAA/NCDC’s GHCN dataset lost more than three-quarters of the climate measuring stations around the world. It can be shown that country by country, they lost stations with a bias towards higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”
[snip ~dbs]

Tenuc
April 18, 2010 2:04 am

Thanks Anthony for showing us yet another error effecting this data-set.
Errors in data tend only to be found if they conflict with what you believe, so I suspect a scientist who believes in CAGW wouldn’t notice a problem with a high temperature error, but would quickly spot one going the other way.
The foundations of climate science really are made of straw.

Ed J Zuiderwijk
April 18, 2010 2:05 am

Looking at the NASA NEO image, how on Earth can March 2010 have been the “hottest on record”??

April 18, 2010 2:06 am

so essentially what this means is that all of the METAR data used in climate research should be omitted as unreliable. It seems to me that everyone inside the new orthodoxy bubble that is the religion of AGW is so focussed on the prize, that they are unable to audit their data his any amount of scientific integrity. or maybe they are aware but it suits them to ignore. who knows? either way it stinks.

Lindsay H
April 18, 2010 2:06 am

many years ago I had to check thousands of invoices (pre computer stuff) for accuracy in pricing and extensions, ie 45 widgets @ 3.49 = ? add tax sum to total, to check the accuracy of the billson a major contract.
It transpired there was a 2% error rate in all the invoices checked, and a 90% bias in favor of the biller in the errors.
The errors on one major contract added up to 10’s of thousands of dollars of over charging.
I think anthony’s analysis shows the same potential for human error.
It would be interesting to know what the error rate in transcription over a period covering some thousands of METAR reports would be.
Based on my experience I would expect M to be miscoded 2% of the time unless there was a system of double checking which would reduce it to .04%
Great Work, Keep it up!!

HR
April 18, 2010 2:11 am

Does the homogenization process clear these sort of errors?

pwl
April 18, 2010 2:15 am

As a computer scientist this type of input error is all too common.
In a bridge design system that I wrote with a civil engineer we used graphs of the structure to ensure that this same type of error was caught before the parts of the bridge were cast in concrete. It does happen all the time during input to systems, the key is catching it as it happens or as soon as possible after before there are downstream consequences that cost a fortune to correct.
Protecting systems from bad input leading to “Garbage In – Garbage Out” is basic computer science 101.
The cultural bias of speaking of negative numbers by dropping the sign could lead to an increase of this type of error. It would be interesting to see the percentage of errors presented by country/cultural group.
The rounding error mentioned above (rounding -1.4 towards -1 instead of -2) will also create a noticeable warming bias towards warming on the order of the alleged warming signals seen. WHY ROUND THE NUMBERS AT ALL? It makes NO SENSE at all!

janama
April 18, 2010 2:17 am

Fantastic Anthony – we’ve all pondered over that hot spot in Russia for years without receiving any fallible explanation. Yours sure is.
If you get the chance could you look at January 2007 – it’s a classic
http://www.climate4you.com/images/EQUATOR%202007%2001%20vs%201998-2006.gif

Peter Miller
April 18, 2010 2:21 am

It will be truly scary if persistent human error turns out to be the principal reason for AGW in recent times. The alarmist whitewash that may be about to happen will be a wonder to behold.
Anthony, if your human error theory proves correct, then I doubt if Al Gore will return his Nobel Prize or that they will give you one.
I suppose one quick way in which it could be verified is by looking at the relative ‘global warming’ measured in land stations in recent times.
Theoretically, it should be most in polar regions, less in temperate regions and almost none in the tropics – I seem to remember reading somewhere that this is exactly what is happening – can anyone comment?

DaveF
April 18, 2010 2:27 am

Anthony Watts:
“I’ve found many examples easily in my spare time.”
You have spare time?
Best wishes.

Ben Lankamp
April 18, 2010 2:42 am

A nice story about METARs, however, they are not used by GISS nor CRU for their calculation of the monthly global temperature anomaly. All the NMIs put the monthly average temperatures of their synoptic stations on the GTS in CLIMAT bulletins, as clearly indicated in the documentation of the GHCN. The wrong temperatures in Finland were not the fault of GISS but the Finnish NMI. They forgot to put the sign indicator in their CLIMAT bulletins.
Regarding the impact of this error, GISS says: ,,The March 2010 global mean temperature was affected by about 2/100 of a degree Celsius, well below the margin of error (about 15/100 of a degree for monthly global means).”

Tony Hansen
April 18, 2010 2:43 am

‘Is it quality assured?’
‘….mmmmmmm…..sorry, did you say something?’
‘Is it quality assured?’
‘….mmmmmmm…… dunno, how can ya tell’.

E.M.Smith
Editor
April 18, 2010 2:47 am

Hmmmm Wonder if that is part of why Finland has very suppressed ‘low going’ anomalies after 1996 in this graph?
http://chiefio.files.wordpress.com/2010/03/finland_1825_hair_seg.png
It’s monthly anomalies for all the fine “hair” and the main line is the running total of the annual average of those monthly anomalies.

Shytot
April 18, 2010 2:48 am

Great work and it shows how poor the data gathering is for this serious science subject.
As with the CRU if you are generous (as the investigating committee was) it appears that the intentions of the scientists are good but they are just a bit too disorganised and should look at tightening up their procedures and data gathering.
Alternatively you could surmise that there is no interest in finding fault with or qualifying the data as long as it meets the target figures.
This once again shows that in very simple terms the whole Hockey team theories (along with those of warmists worldwide) are based on very poor quality data and questionable methods.
As for Nick’s comments “….accusations and extrapolation first,then ’some data analysis’ later …”- apart form deliberately missing the point, the so called accusations are valid since the people responsible for the data manipulation admitted their error (in this instance), extrapolition is fine for guidance if it is based on solid data (as opposed to the manipulation for the right answer – sole domain of Mann and co) and at least the analysis is being done with qualified / quality data which is a useful hint for others if they were really interested in the science and not the economics.
Nice work Anthony, I sincerely hope that some people in power actually have the balls to take this and challenge the IPCC and their Carbon Credit profiteering buddies.

Sean Inglis
April 18, 2010 2:50 am

A fascinating read, presented in a very accessible way.
Instructive to see quantitative example (and in particular the effect on the buoy data, a courtesy from the “other” Anthony that saved me from having to do it)
Although there is absolutely no doubt that these “perfect” errors could almost be designed by a thriller-writer to keep the reader guessing, I’d echo the sentiment that they seem to have been good enough for their intended purpose as disposable indicators.
The problem, as ever in Climate Science, is that far too many people are far too certain about too many things and far too willing to shout about it.
The good news is that even very simple heuristics in a recent snapshot of this data should provide a decent indicator how much of an effect this is likely to have overall, and provide solid “dig here” pointers.

TJA
April 18, 2010 2:52 am

If I found out that one of my programmers was using techniques that did not ensure data integrity, and delivered a solution without adequate testing of all of the cases, well, his or her name would be at the top of the list for the next RIF.
“Fahrenheit drooling Americans” Uh, maybe you should see a therapist about your self image.

Sean Inglis
April 18, 2010 2:56 am

Sorry, by way of a supplementary question, is there any way to get to the METAR data for the most recent week / month and see / download the data in it’s raw form? Or are individual records collated with some sort of back-end process that has no public interface?

Predicador
April 18, 2010 3:02 am

what is also disturbing about climate data is how often the very place names are misspelled.
it’s not ‘Mirnvy’ – Russian Мирный, usually transliterated as Mirniy, sometimes as Mirnij, Mirnyy or Mirnyj – but in no way there’s v in it. (the meaning of it is ‘peaceful’ or ‘peace-loving’ – several Soviet-era settlements carry this name)

Jordan
April 18, 2010 3:03 am

Wren (20:46:49) : Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value
Anthony’s post notes one of the issues as follows “conflicting missions (aviation safety -vs-climate)”.
METAR is designed and operated by the aviation industry, with no other use in mind. Without doubt, climatologists will be free to use publicly available data if they want, but the terms of the transaction will be “strictly as seen”.
These ownership and mission issues get things off to a poor start for objectives such as quality, accuracy and precision of climate trends.
And if I were to use the word “disgrace” it would be in the sense that GISS use this data, and then appear to have failed to identify sich a flaw. Is it not fair to consider it disgraceful that WUWT and CA appear to have become part of the QC/QA machinery of funded projects?
There should now be a detailed investigation of this issue. GISS has the most to gain and lose from this issue, so the onus falls to GISS to use some of its funding to get the matter sorted out.
For a quick indication of the magnitude of the problem, my suggestion would be examine the most recent data first. Assess the magnitude of the error at individual locatons and consider whether averaging reduces the error (averaging doesn’t necessarily reduce systematic error, and it is necessary to take into account extrapolation to neighbouring grids).
If this is combined with existing error analysis (which we assume does not account for any METAR error), the confidence interval of the GISS global data set would expand. The expansion would come into play around 1996 (introduction of METAR) and there would be continuing expansion with the increased use of airport data.
At that point Wren, GISS will have put you in a position where your could examine the lower bound of the confidence interval to see whether there is any reason to argue for a significant trend.

Stephan
April 18, 2010 3:11 am

We are hoping this posting stays up Numero 1 for at least 48 hours! (or stories directly related to it) VIP! (Very important Posting)

April 18, 2010 3:27 am

Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK

Frederick Davies
April 18, 2010 3:27 am

Why anyone still uses the Global Historical Climatological Network when better and more accurate satelite measurements exist is beyond belief. These people are a bunch of incompetents!

Graham Dick
April 18, 2010 3:29 am

A.W. You say
“In middle latitudes, you might see it (coding error) happen, but it is much more seasonal and the difference is not that great.”
True, but for that very reason, surely coding errors are more likely to slip through unnoticed. Conceivably, an operator then would be more likely to continue the habit, oblivious to the serial errors. The “difference is not that great”, but it can be significant if unchecked, especially in the context of just 0.6degC global warming in 30 years.

Peter_dtm
April 18, 2010 3:34 am

I am amazed.
Even unimportant factory records do basic confidence data checks when historizing process/production data. Normally we won’t acept data that is outside sane data-point to dat-point delta.
How on earth can this data be useful for anything ?
What sort of idiot/ignorant numptie doesn’t
do basic confidence checks ?
These people /claim/ to be scientists ? They demonstrate such an appalling lack of basic knowledge they shouldn’t be allowed near a keyboard, never mind a science based job.
Well done again Mr Watts,
Another problem ( which would never occur to either the French or English speakers )
In how many languages does the word for ‘minus’ start with ‘M’ ? ……..

kwik
April 18, 2010 3:39 am

Theo Goodwin (23:03:40) :
I have a complaint about climate scientists’ use of the word “anomaly.”
Maybe using “Absolute” and “Relative” would be better?

Duncan B (UK)
April 18, 2010 3:40 am

Outstanding work!

Shona
April 18, 2010 3:45 am

The more you look at this, the more you realise it’s impossible to do what they’re doing.
I cannot understand why one would base one’s life work on this. It cannot be done.
Possibly with sat data, we’ll start to get a handle on it, but as it stands now, the whole thing is rubbish.

H.Timman
April 18, 2010 3:54 am

Here is bit newer and sharper giff picture from that scandinavian error, was still having distinct error in it and almost half of europe.
Here in Finland we wheren´t sure should we laugh or go angry, because its been many degrees colder than normally in this winter and spring.
http://i756.photobucket.com/albums/xx206/MWPeriod/agw/GHCN_GISS_HR2SST_1200km_Anom03_2010.gif

Keith G
April 18, 2010 4:05 am

Anthony, I have no idea how you have time to do all this analysis and work. I barely get the kids to bed without falling asleep reading to them. Whatever your secret, thank goodness you keep at it. It must be exhausting, but I hope the exhilaration and other benefits are enough to keep you at it. You are making a positive difference in the world.
A more enlightened NASA or NOAA or NWS would hire you as a full time gadfly. Wouldn’t that be something? The government hires Inspectors General, offices to manage Equal Opportunity, has full time ethics agencies, all sort of checks and balances folks, but no gadflies. Just someone to say, “yeah, but what if…..”. Think of the time, effort and money to be saved if someone like you were there to simply ask something like, “but what if the data is wrong?”

Steve Keohane
April 18, 2010 4:19 am

Excellent post Anthony. This explains so much of the GISS deviation from what is observed. Great piece of work.
Wren, if you have a data set of 150 years, of which the last 16 are biased high, please explain how this does not affect the trend!

Chad Woodburn
April 18, 2010 4:20 am

<>
Anthony is clearly IDENTIFYING the problem here, and then talks about ANALYZING its extent and impact later. Well, duh, Nick! You can’t analyze the extent and impact of a problem until you have identified it. What scientist ever reverses this order? (“Hey guys, I’m gonna analyze the extent and impact of a problem even though I don’t know whether it exists.”)
Your scorn is clearly not scientific (it is based on an anti-scientific procedural error). So, I think I have fairly identified your problem. What remains to be analyzed is the extent and impact that your scorn has had in the development of your AGW paradigm. Your brief comment about the world’s glaciers and Anthony’s “US surface stations project” suggests that it is quite extensive.

Chad Woodburn
April 18, 2010 4:22 am

The previous comment by Chad Woodburn first quoted Nick’s comment, but I put the quote in angle brackets, not realizing that that would eliminate them from the post.
“Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. “

Geoff Sherrington
April 18, 2010 4:26 am

Anthony, this is a most important subject, well explained and documented. Congratulations.
The big unknown, of course, is whether the dropping of a negative sign happens enough times to swing the country or global data, or whether some countries have QQ that detects it. The Antarctic would seem particularly at risk.
In looking at data from one of the Australian Antarctic stations, Mawson WMO 94997, there was not enough metadata available to me to address the same points that you raise. However, the first look shows in essence that the extreme temperatures are somewhat ragged at the edges. (For the record, Mawson has an airstrip. Australia changed to decimal in Feb 1966, but as a scientific base, the records were possibly in C from the start in 1954). The data are from the Australian Bureau of Meteorology, duly acknowledged.
An initial look is presented now in the form of three spreadsheet portions and a graph plus a comment on each. I stress that we might be looking at different mechanisms, but what I present does not preclude errors of the type that you describe with METAR.
Overall, the data set had 19,419 days, with Tmax and Tmin. There were 87 days of missing Tmax and 22 days of missing Tmin, reason for missing unknown.
The mean of all Tmax values is -8.38 deg C. The mean of all Tmin is -14.27. The mean difference is 5.87 deg. Condider the sign dropping exercise with these figures, first with the Tmax going from -8.38 to +8.38, causing the Tmean to change from -11.33 to -2.93. If the sign was dropped from Tmin, the two Tmean would change from -11.33 to +2.93. As this shows, the dropping of sign on the minimum is a bigger problem than dropping the sign on the maximum, as you noted.
Mawson1 is a small part of the Mawson daily maximum temperatures sorted from highest down, for the first few weeks.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson1.jpg?t=1271577750
If there was a sign error for Tmax, because its average is close to -8 Deg C, there are 2 consequences. Where Tmax is already above zero, in about 20% of the cases, we are not subjected to missing a negative since there are none. Where Tmax is below zero and the negative sign is dropped, this leads to an apparent warming that can report in 80% of the data. Tmax can be as low as -32 deg C, so dropping the sign gives a healthy wrong warming.
Mawson2 is for the minimum temperatures sorted from the highest down.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson2.jpg
This spreadsheet partially shows that only a few days had Tmin above zero. The vast majority have a negative sign and dropping the negative sign would cause a strong warming impression. It is also more probable an error than with Tmax. The average of all readings is -14.2, the coldest reading is -36 deg C with a potential for a daily Tmean error of 36 deg C.
Mawson3 shows the difference between Tmax and Tmin.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson3.jpg?t=1271577538
The largest difference is 32 degrees. If there were large mistakes made from occasional sign dropping, there would be occasional large values that did not seem to be part of a smooth distribution. However, it is plausible that such “unlikely” figures would be culled by the hand of man, making this detective work less relevant.
Then there is a daily graph showing the differences calculated in Mawson3.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/MAWSONGRAPH.jpg?t=1271577707
It is apparent that the hand of man has been at work, because one can see distinct artefacts in the graph of the differences. See the group of parallel values around day 10,000. Of course, this graph must have no points below the zero X axis because this would mean that Tmin exceeded Tmax, creating a terminological problem.
At this stage, further work is not warranted because too many assumptions need to be made. Primarily, evidence of numerical reworking exists but the cause(s) is unknown. It is noted that Tmax-Tmin was rising from about 1965 to 1980, then there is the 10,000 day gap, then the difference mainly fell between 1986 and 2000, then levelled. What this means in Nature I know not.
With further work not reported here, there are possible indications from several outliers that dropping of a negative sign could have happened, but there is no definite evidence that it did happen. The day-to day variation of temperatures at these latitudes is large and there is a lack of rigour in plausible reconstructions to explain outliers.

Gail Combs
April 18, 2010 4:27 am

Just thinking….
Willis shows the deviation from “average” toward more warming occurs in the winter at night. http://wattsupwiththat.com/2010/04/16/wheres-the-climate-beef/
Anthony now show an error that can cause the deviation from “average” toward more warming when the temperature are below freezing.
The temperature are below freezing most often in the winter at night.
Human error is more apt to occur at night especially in a rotating shift type situation or when the shifts are longer than 8 hours.
A minus sign error has the most impact on data as you approach the poles. Not only does the sign switch cause a major deviation from the true value in the individual data point but the area around the poles is more likely to have sparse data causing the impact to be even greater and the incorrect data to be “smeared” to adjoining grids.
Definitely an area that needs more investigation.

stephen richards
April 18, 2010 4:30 am

Guys you have to realise that Nick, David X Benson et al are serious disciples of the RealClimate Church. They see only what their high priest tells them to see.
Sad very sad.

brodie
April 18, 2010 4:40 am

Those antarctic stations are not reporting in METAR format. That is called “ship synoptic” code. It has it’s own problems. The biggest being that it is done by quartermasters that have never been properly trained. I have seen them put a sling psychrometer into a bowl of water and read the temperature and dewpoint off of it from there.
You did a very good job here. I don’t think this is as big of a problem on land in the United States as this article makes it seem tho. For a couple of reasons. One is that we all talk to each other. If I am observing in Pax River and I see that Dover made a mistake I call them and tell them to send out a correction. Two is that we have a second group for temperatures. The “T” group, which you can see here. METAR KDOV 180855Z AUTO 30008KT 10SM CLR 05/01 A2991 RMK AO2 SLP130 T00520007 55002 $ This uses a 1 or 0 to denote positive and negative temperatures. Third is that when we go to insert this in the computer it checks it for us. Fourth, those online decoders do not work the same way that the NWS decoders do. Those are written in FORTRAN and pick up on some of that stuff, and always use the T group when available.
I do not mean to discount this work tho. You are pointing out a problem that is very real. Overall great job!

Methow Ken
April 18, 2010 4:49 am

Thank you for taking the time and making the effort to publish this detailed analysis. Excellent; and IMO qualifies as another ”smoking gun”.
But I almost dispair at finding a practical solution to one huge problem; i.e.:
How do we ever get even a concise summary of important information like this considered by the MSM, let alone the average citizen; who has NO IDEA what is actually involved in gathering and adjusting all the data used to (at least theoretically) come up with statements like ”this was the warmest month on record in Oz”, etcetera ??
Take just two key statement from thread start:
”. . . . the chances of human error in raw surface data are at least an order of magnitude greater (if not several) than error in raw satellite data.”
and:
”. . . . there has been a wholesale migration towards airport weather data as a climatic data source.”
If even 1 out of 1000 people in North America have any idea that last two above are true let alone why and impact of same, I’d be surprised. But without at least a basic grasp of these and other key facts laid out in this post, citizens (and Members of Congress) are unable to make informed judgements on key issues; and remain vulnerable to self-appointed ”leaders” who contine to demagogue on the ”urgent need” to address CAGW regardless of cost and other priorities.
I guess as long as the WUWT hit counter continues to grow there is hope that objective science will prevail. Jury is still out, I suppose. . . .

anticlimactic
April 18, 2010 4:50 am

I would agree with a previous comment that satellite data should now be the primary source of information as the other sources are compromised. This would make CRU and GISS redundant, so they could be terminated.
The NOAA/NCDC should continue to collect raw station data purely to make it available to researchers in an online database, with their remit just to make it as accurate as possible, but not to interpret it themselves. In addition they should be charged with trying to recover all historical data, and fully document any homogenisation.
Source data you can actually trust is paramount.
————
[Note : I would regard ‘raw’ data as being like that shown in the article – a variety of formats. I would regard ‘source’ data as being after the raw data has been formatted in to a standard record. If the source data needs a correction as per the article then the original data should be stored in the same record so that any correction is flagged and is reversible. I would want any adjustments [say, from homogenisation] in separate fields so the source data is always left intact, and, again, it is flagged and reversible.]

observa
April 18, 2010 4:50 am

Ken Smith (18:56:17) asks-‘Is it plausible that UHI might be increased by a _local_ greenhouse effect?’ I don’t know about that Ken but I’d suggest UHI may well be an increasing culprit for another reason, if as Anthony suggests the surface temp record is increasingly reliant on airport records. He points out the METAR conundrum but that raises another issue with airports, if like my local airport in Adelaide (State capital of South Australia), advancing aircraft technology instrumentation and flight control are having another specific impact on airports like mine. How so? You might like to ask yourself if development at your local airport in recent years matches that of mine. Let me describe Adelaide Airport which you might like to check out on Google maps for yourself as a useful exercise. Basically it sits close to the coast (St Vincents Gulf) between the city and the sea. Its western boundary was always Tapleys Hill Rd and for much of my life it consisted of 2 crossed runways amid a very large grassy plain with a smallish State terminal vis a vis Melbourne and Sydney international airports. Furthermore the land between Tapleys Hill Rd and the coast was also used for playing fields, golf course, horse agistment and largely undeveloped. I can tell you as a keen motorcyclist for many years, particularly on winter nights, you braced yourself for a significantly cooler ride along Taps as you left the built up suburbs either side of this natural plain, albeit a pleasant and welcome cool change in summer. Noticeably warmer nowadays with an airport that’s been privatised, an international terminal added and everything from a Harbourtown shopping centre with large bitumen carparks to logistics warehouses and even old folks home being built ever closer to the runways. That would have been unheard of for pilots in the past on safety grounds. Clearly better navigation and flight control has been the catalyst for a shrinking safety area around the runways and with it a rapidly increasing UHI effect over the last decade or so. Presumably Adelaide Airport is not alone due to such a technology factor?

April 18, 2010 4:57 am

It’s amazing what objectivity does for science. Oh wait, isn’t all science supposed to be objective, unbiased, seek the truth?
Sounds like some modern software design could help alot.

Dodgy Geezer
April 18, 2010 5:03 am

Interesting. I am certainly convinced that such errors exist, and will bias the station temperature data upwards.
However, a critical question is – ‘How prevalent are these errors?’. It is possible that they have a minor effect – we cannot tell unless, at vast expense, we redo ALL the collected data from scratch. I believe we should do this, but no warmist does…

DirkH
April 18, 2010 5:05 am

1) Make the assumption that data entered into the system is correct
2) Believe your assumption and don’t check
3) Never look back
Voila, global warming.

April 18, 2010 5:09 am

Regarding Wren (20:46:49) :
There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Wren, you may have demonstrated a classic case of confirmation bias. You wish to discount any possible effect on the trend, and so fail to note that this mistake is from “airport” stations, and there had been a dramatic move towards airport stations in the GISS data base; and therfore this would impact the trend, and even more so if these readings are extrapolated to dropped stations.

April 18, 2010 5:21 am

Regarding jim karlock (03:27:07) :
Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK
Good question and some curious posts about these buoy errors.
I hope some of the qualified posters give more detail on this.

DocMartyn
April 18, 2010 5:40 am

all flights from UK airports have been grounded. No planes.
Now would be a good time to compare the temperatures of the AIRPORTS (Heathrow, Gatwick) around London, with the stations around them.
If the airport temperatures show a drop in temperature, vs. nearby sites, then we have our smoking gun.

Editor
April 18, 2010 5:56 am

TJA (02:52:11) quoted me:
“Fahrenheit drooling Americans” Uh, maybe you should see a therapist about your self image.
Oh actually I have a pretty good self image and mental store of conversion factors. After my father retired, he spent some time as a substitute instructor at the computer and meteorology departments the newarby state college. He was quite amazed at how many students didn’t know basic stuff like 5,280 feet in a mile. I suggested that might have stemmed from the shift to metric track sporting events, e.g. the 440 yard run (1/4 mile).
I use 3 or 4 formulae for mental F <-> C conversions, including my favorite that Dad taught me. (Add 40, multiply by 9/5 if going to F, 5/9 if going to C, subtract 40.) Length stuff like 39.37 in/m, 2.54 cm/in, 1.6 km/mi, 0.6 mi/km. Volume stuff like about 8 lb/gal of H2O, about 8 gal/ft^2, 2 tablespoons per ounce (how many cooks don’t know that, probably a lot!) One of my favorites and used while lying on a housemate’s new water bed is 0.455 lb/ft^2 per vertical foot of H2O. I get to use that while looking into developing a spring uphill from our yurt on a New Hampshire mountain.
I do have very little patience for meat counter clerks when I ask for “eight tenths of a pound of ground beef.” They’re more comfortable with “three quarters”, but all their scales these days read in pounds and decimal fractions! Arrgh.

April 18, 2010 5:57 am

mindbuilder (20:52:22) :
Has anybody noticed that in mid 1966 the US started rounding negative temperatures hotter? From the Wikipedia article on rounding:
In a guideline issued in mid-1966,[8] the U.S. Office of the Federal Coordinator for Meteorology determined that weather data should be rounded to the nearest round number, with the “round half up” tie-breaking rule. For example, 1.5 rounded to integer should become 2, and −1.5 should become −1. Prior to that date, the tie-breaking rule was “round half away from zero”.


If that’s done in Celsius, where negative numbers are quite common for much of the US for a good part of the year, then would not that convention alone produce a lot of completely spurious ‘warming’?
/Mr Lynn

Rhys Jaggar
April 18, 2010 6:14 am

This accuracy of the data record is perhaps a useful point to be put to Professor Jones at CRU.
As he has been exonerated of all wrongdoing in a brief but concise establishment enquiry, he will be uniquely positioned as a reputable scientist to engage in the effects that such errors might impinge upon his scrupulously performed research.
Clearly, if his position is one of accepting the input data unchallenged, then he would need, in this domain, to be regarded as an extremely dexterous technician, since technicians in general do not question sources from on high and take what they are handed as gospel.
If of course his research had already quality controlled the GCHN, then we are in different territory…………….

April 18, 2010 6:15 am

Ric Werme (21:55:21)
. . . Hmm, here’s a thought – global warming is said to impact the winter the most. Winter is when the negative temperatures occur (°C or °F). Wouldn’t it be embarrassing if a substantial portion of that impact was really due to dropped M-signs?
Louis Hissink (23:02:03)
. . . Absolutely no thought seems to be directed to the possibility that the AGW hypothesis might be flawed, or just plain wrong. No, it’s accepted as being true not from experiment or data but by peer consensus.
No wonder countering science with science doesn’t work – the AGW isn’t a scientific hypothesis in the first place and our mistake was assuming it was.
Peter Miller (02:21:19) :
It will be truly scary if persistent human error turns out to be the principal reason for AGW in recent times. The alarmist whitewash that may be about to happen will be a wonder to behold. . .
DirkH (05:05:55) :
1) Make the assumption that data entered into the system is correct

2) Believe your assumption and don’t check

3) Never look back
Voila, global warming.

Another rip in the tattered fabric of global temperature data! Hasn’t this amazing work by Anthony Watts, along with EM Smith and Joseph D’Aleo, gotten us to the point where the ‘consensus’ that the Earth has warmed over the past century has to be questioned?
I notice that Willis Eschenbach, even while poking more holes in that cloth in many posts, still concludes (or assumes? or concedes, for the sake of argument?) that the putative warming has taken place. But has it?
Maybe it is time to challenge the basic premise of the Warmist doctrine, that there has been any significant warming at all since the end of the Little Ice Age back in the early part of the 20th century. And if so, it should be done in a dramatic fashion, at a press conference, and before Congress, with a huge pile of reports like Anthony’s here on the dias, saying simply, “The Warmists are wrong. Their data is corrupt and unusable, and their conclusions unjustified. Period. End of story.”
/Mr Lynn

Owen from Cornwall Ontario
April 18, 2010 6:17 am

A very interesting and very informing essay.
Just to be a little picky about temperature as a throw-away bit of information for pilots. I agree with that for most pilots of light aircraft, but for operators of larger transport type aircraft temperature plays a large roll in our daily operations, even on cold days. Engine thrust values at take-off are all calculated on outside air temperatures (OAT). The warmer the OAT the less engine thrust available. The less engine thrust available the less weight we are able to carry off a given runway. In airline terms less weight = less revenue. In most flight operations nowadays flight plans are created by computers automaticly gathering data online from various agencies and then reviewed by a flight dispatcher and the captain. If a 05 temp is used instead of a m05, the amount of weight we are able to lift will be affected. This error might or might not be noticed when we look at the maximum allowable take-off weight limitation, as the OAT (at least in my companies flight plans) is not presented in the same sections. Something I am going to ensure I compare from now on.
As for the METAR corrections overwriting the incorrect data, the incorrect METAR is still shown in most weather requests as usually (here in Canada at least) the corrected METAR is only a partial report changing the incorrect data. The reader of the corrected METAR must compare the two to determine where the mistake/correction is. Do the programs that collect the METAR information collect the corrected partial METAR issued anytime through out the hour or just the COMPLETE METAR issued at the start of each hour.

James
April 18, 2010 6:17 am

It only takes one METAR reporting error during the day to make a new massive maximum daily temp. The operators can do 23 correctly and one wrong one like this influences the daily max temp massively. Very good investigative analysis.
Sattelite data is oviously a more reliable data gathering technique.
why dont thosse who create the GHCN look for obvious anomalies in the data? Surely a one off hourly jump like that is obvious and could be filtered out? It would not get the habitual operator errors as reliably but still would be an improvement.

1DandyTroll
April 18, 2010 6:28 am

Nice write up. Although it does far as a case of compelling trollogic rather than just spelling human errors.
Anyone that knew of these errors afore hand, for instance, can always claim ignorance of the problem. Would such a person have been as ignorance of the errors if the error had been reversed, making positive numbers into negative numbers? Now, with M(ystery), the lack of QA and error control can just be used to reinforce AGW hypothesis wether knowingly or unwittingly. Talk about positive feedback. :p

starzmom
April 18, 2010 6:31 am

This post explains a lot. I am speechless that these types of errors are as systematic and uncorrected as they apparently are. Where are the adults?

GaryPearse
April 18, 2010 6:38 am

Anthony, were there not errors the other way,ie. wrongly encoding colder temps?
REPLY: I didn’t see any, but that does not mean they are not there. However, the only way to get colder temps, given the way the METAR decoder works, is to invert or mistype the data. For example 20/15 becomes 02/15, but in that case, the DP would be higher than the temp, and one would think they at least recognize that physical impossibility. – Anthony

pyromancer76
April 18, 2010 6:50 am

Three points:
1. Subscriptions, please.
2. Proposal from real scientists and investigative journalists re how to establish an accurate data base of the raw temperature readings — historical and contemporary) that remain and how to evaluate them with as much accuracy as is humanly (with computer assistance) possible.
3. Congratulations, Anthony, for an excellent overview of your research and investigation. This is at the cutting edge of “climate-global temperature” science. I look forward to further “publications”.
1. Michael Larkin (20:56:10) 4/17: “PS – Anthony: $50 on its way to you via the tip jar – wish it could be more. Hope it’s a useful contribution to your upcoming trip.”
I am sending my 2nd quarter $10/mo subscription to WUWT because Anthony provides some of the best real science and investigative reporting on any blog. I challenge regular readers to do the same — we need sites like WUWT and we need their creators to understand our approval of their entrepreneurial choices. I have taken these funds from my former subscriptions to pseudo-science publications (Nature, etc) and political rags (like the LA Times and NY Times).
I wish Anthony would create a list of regular, voluntary subscribers so the reading public could see the mass media/sci-journal transformation occuring. Even a place where the numbers of subscribers is made public would do — it might encourage others to subscribe as well.
2. Anthony, Steve(s), EM, Bob, Willis, Leif, and many others: please imagine and articulate how this mess can be cleaned up (including who should be fired) and suggest how government money for (its-our community, using our taxes) responsibility in weather/climate accounting and dangers could be spent REASONABLY. Imagine the decline in national debts when real scientists take over. I offer three quotes from commenters:
2a. kim (20:31:45) 4/17 :
Judith Curry, in comments at Bishop Hill and at Pielke Fils, is saying that the whole temperature and paleo records need to be reworked. I think she’s tired of the foolishness, and I think there are a lot of other people who are tired, too.
2b. Davidmhoffer 20:39:49 4/17
“Given the gaping hole you just exposed, and the clear lack of professionnal standards when it came to not just writing and documenting code, but managing the integrity of the data itself, I can’t but help thinking that it is all junk and we need to start with the paper records themselves. again.”
2c. JRR Canada (21:03:36) :
“I agree with KIrk.The deeper we get to look the sadder this science seems.Is it because these people are all govt employees?I mean the errors are so pathetic its like they just do not care, no repercussions for poor work no consequences for mistakes.Lets save billions, their work can not be trusted? Stop funding.”
2d. GregO (21:46:07) 4/17:
“‘Bob (20:35:01) :
‘Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience… As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry.’
“I too in an earlier lifetime had the opportunity to deal with a similar type of problem in precise data transcription. What we found is that to assure better than 99% data integrity in data transcription the following guidelines needed to be understood and followed: 1) Limit the number of individuals making the transcriptions, make them responsible for error rates, reward accuracy. 2) Audit transcriptions in as close to real time as possible – provide immediate feedback to transcribers, make corrections as immediately as possible. (In this particular situation, to find an error it took 24 hours for it to show up, some delta T to fix it, another 24 hours for the “fix” to show up. 3) error, if left to accumulate, was logarithmic in growth and created a constant data offset.”

Digsby
April 18, 2010 6:56 am

Bernd Felsche (23:14:08) wrote :
“What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO
??”
Yes, but, of course, it is common knowledge that human concentration deteriorates at higher temperatures, so I think it is more likely that we will see this headline:
INCREASING NUMBER OF TYPOS CAUSED BY GLOBAL WARMING SAY SCIENTISTS!
(Indeed, we may just have discovered here the most important positive feedback loop for notional AGW.)

Richard M
April 18, 2010 7:15 am

It seems like this error could be quantified. Couldn’t GIStemp be run against all non-airport stations and that result compared to their standard reports? Sounds like a job for Superman, er um, Chiefio.

harrywr2
April 18, 2010 7:20 am

Hmm,
The minus sign problem is only a problem when temps are cold and the vast majority of the global warming in the Northern hemisphere has occurred in the Winter.

Crusty the Clown
April 18, 2010 7:23 am

Don’t have time right now to read all the comments, so apologies if I’m repeating something already said. Before my promotion to clown I was one time a programmer for an analytical lab. It was a rule to have input data validation checks and either reject or flag any apparently (obviously?) spurious data delivered to the program. You’d have to be nuts to skip that first sanity check.
C the C

tommy
April 18, 2010 7:31 am

I have seen this issue alot at wunderground myself.

Alexej Buergin
April 18, 2010 7:33 am

While we may disagree about the relative merits of METAR and SA-USA (deceased), it seems very reasonable that in an international business like aviation the whole world uses one and the same system.
It makes no sense to measure windspeed in knots and/or mph, reduced air pressure (QNH) in hPa or inch Hg, temperature in °C and °F and so on.

Jeff L
April 18, 2010 7:38 am

Another factor that probably amplifies the error further is the griding process because there are less stations in these remote northern areas, thus less data is used to generate a grid value – essentially less data averaging to generate the grid value, thus any error will have any even greater effect than it would in a region with more dense station spacing.
That being said, until we know how widespread this problem may be, it is hard to quantify what the effect is in the global temp record.

Marlene Anderson
April 18, 2010 7:39 am

A thoroughly investigated and analyzed piece of work. Now the question is, will the legislators hell bent on taxing their citizenry to oblivion see or even care there’s a warming bias built into the temperature records through persistent entry error? The political desire for AGW theory to be true far surpasses the climate scientists who made proof of this theory their life’s work. We may need to grab them by the collar and make them look.

Brent Matich
April 18, 2010 7:40 am

I think I actually understood this! Good job!
Brent in Calgary

John Bunt
April 18, 2010 7:41 am

After reading your post last night, with no comments, and now all of the comments by this morning, I think that you should try to find a sleep disorder clinic to advertise on this site. Great work.

Dave Springer
April 18, 2010 7:46 am

As I noted in the earlier article about the missing minus sign in Finland I was more concerned about how many times this error occured in the past before automation took pencils out of the loop and before there was such intense scrutiny of the records. The surface station records are a bad joke. Spotty coverage, error prone, lack of enforced quality controls, and never designed to track long term changes in average temperature in tenths of degrees. The only reliable record we have with the required accuracy, reliability, and global coverage begins in 1979 with satellite data. Unfortunately 30 years of data is still more a record of weather than it is of climate.
By the way, it’s been twenty years since I was behind a yoke, but I was a private pilot flying in southern California in the distant past. I seldom worried about low temperatures there but high altitude destinations were popular where one needed to be concerned about higher temperatures adversely effecting takeoff distance and climb rate in normally aspirated piston singles. That said, aren’t low temperatures a concern when it makes icing possible? A missing minus sign in that case could spell the difference between deciding to stay safely on the ground or end up crashing into it inside a flying ice cube.

April 18, 2010 7:49 am

Wow. Thanks to Anthony for a very clear explanation of what’s going on. The potential explanation for why the GISS anomolies are often strongly positive at the poles is particularly interesting! How amazing, that such errors have been in the system for such a long time, clearly visible to anybody who bothered to look.

April 18, 2010 7:51 am

Wow, now Phil Jones and Hansen & Co. have a second reason for their Preferential siting of temperature stations at airports!:
http://hockeyschtick.blogspot.com/2010/03/another-awful-email-from-phil-jones.html

Bruckner8
April 18, 2010 8:00 am

Neel Rodgers (19:34:22) :
And lastly, could this be more of a problem of relying too much on computers as opposed to human interaction? In the old code, a space could just as easily be placed in the temp, or the minus sign could be forgotten. Human observers would catch and fix this, computers would not. And thus we would be in the same position we are now.

Anthony’s entire point is that human error is causing this problem. That said, if I were in charge of the technology where this data is ultimately stored, I’d write a computer program with algorithms for sniffing out the simplest anomalies…the most obvious errors could be reviewed and corrected very quickly.
I say “more computers and less humans, please” realizing that humans create computers and the programs. It all comes down to good science.

ron from Texas
April 18, 2010 8:08 am

I commented in another thread about the problem with Finland and supposed it to be the result of station deletion as done by GISS and NOAA, compounded by averaging algorithms from distant stations. I still think that is a factor. And to see this article, wherein non-competent transcribers are recording data and not in a rigidly defined format is creating screwy data sets. Of course, evidently, expecting scientific rigor and quality control from NASA/GISS and NOAA is akin to expecting congress to represent the people. Oops, I think I said that out loud and used my outside voice. As for non-competent, that is not an insult. I borrow it from OSHA. OSHA defines the competent person on a job as the one trained in the rigors of the safety processes. So, a competent data collector would be trained in the proper way to take measurements and encode them in a defined format that is instantly readable and reduces errors. Apparently, we don’t have that.

A C Osborn
April 18, 2010 8:22 am

Sera (21:04:16) :
Sera (21:12:55) :
Can anybody get this data?
If it as bad as Sera illustrates it should be added to Anthony’s findings, especially as it is in the Arctic.

April 18, 2010 8:28 am

Not a single one of the miscoded METAR stations records at Weather Underground have bled through into the GHCN Daily station data.
See post here: Watts: Dogs barking, can’t fly without umbrella
REPLY: GISS uses GHCN plus other stations that are not part of GHCN. CRU also had their own list that chose selected stations plus GHCN. If the issue is not a problem, how then do you explain this statement from GISS?
The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.
My point is that since GISS has shown it to be a problem in one case, there were probably others. I demonstrated that there were. GHCN might do quality control that solves the issue, OTOH GISS admits to a quality control problem involving missing minus signs. Can you say that GISS and CRU are both unaffected by the issue? We know for certain at least GISS is.
My whole point here is that the minus signs issue is real and observable over different locations and time frames, and that since GISS admits to exactly this type of error affecting the final climate product. The fact that it does bears further investigation. – Anthony

A C Osborn
April 18, 2010 8:31 am

Ben Lankamp (02:42:01) :
Regarding the impact of this error, GISS says: ,,The March 2010 global mean temperature was affected by about 2/100 of a degree Celsius, well below the margin of error (about 15/100 of a degree for monthly global means).”
But of course that was the only error that was quite so obvious and found not by GISS but the home country who new it couldn’t be true.
How many other less noticable errors are there.
To have no error checking on something supposedly so important is a disgrace, but then again using Airport data is as well.

April 18, 2010 8:36 am

My previous comments should read: Not a single one of the miscoded METAR stations records identified by Anthony have bled through into the GHCN Daily station data.
I am not excluding the possibility that there has been some bleed-through of miscoding errors. Just not the ones listed in the OP.

April 18, 2010 8:56 am

Watts: My whole point here is that the minus signs issue is real and observable over different locations and time frames, and that since GISS admits to exactly this type of error affecting the final climate product. The fact that it does bears further investigation.
So if I’m reading you right, Anthony, when you subtitled this thread “its worse than we thought”, what you intended to say was something more along the lines of “I have identified an issue in METAR encoding that might be problem in GHCN and GISS data sets, but we don’t know that yet”
REPLY: It is worse than we thought. GISS Identified one instance of minus sign error, I showed there were many more. Hence, “worse than we thought”. Since you are harping on titles, would you care to explain yours?
“Watts: Dogs barking, can’t fly without umbrella”
Perhaps it was designed just to portray me negatively and that my points about the other instances of missing minus signs have no merit, even though GISS admits to the problem and some of the station used in my post appear in GISS, even though they are not part of GHCN?
If you want to look at data, fine. If you want to tell the world you don’t like me, fine too. But please don’t mix the two and then complain that I’m the one with a negativity (or minus sign if you prefer) problem. 😉
– Anthony

A C Osborn
April 18, 2010 9:07 am

Ron Broberg (08:36:28) : Not a single one of the miscoded METAR stations records identified by Anthony have bled through into the GHCN Daily station data.
So you are saying Anthony is lying then!
Let’s take for example, a station in Mirnvy, Russia. It is in Siberia at 62.5° N 113.9° E and has an airport, is part of GHCN,

April 18, 2010 9:13 am

MMMM — Mighty interesting!

April 18, 2010 9:17 am

Anthony GREAT piece of detective work. My sense is that the warmist side looks at these errors like so:
Opps, lets fix this error here and move on.
They dont bother to investigate the CAUSE of the error and see how pervasive it may or may not be.
They also tend to assume that all such human errors balance out. This one is clearly biased toward heating.

Roy Hogue
April 18, 2010 9:23 am

You make a strong case for automating the temperature measurements. And we have the satellite data that shows a quite different picture than the ground based observations. Why then do NASA and NOAA continue to rely on airport weather reports?
You pointed out that pilots would understand what you’re talking about. As a pilot myself I know the temperature isn’t of much interest unless density altitude is a problem. But I can attest to some errors made by FAA tower and ground controllers. For instance, being cleared to turn to right base leg only to find I’m staring at another aircraft on left base for the same runway. Human error is everywhere.
If data can be obtained by automated means then the human error in data entry is simply eliminated.
Good detective work!

David Segesta
April 18, 2010 9:27 am

Very nice detective work Anthony. Although I also have to wonder why the mistakes always increase warming.

Evan Jones
Editor
April 18, 2010 9:28 am

(Sing a song, brother)
If the sun refused to shine,
I don’t mind, I don’t mind.
(Yeah)
If the mountains fell in the sea,
Let it be, it ain’t me.
Got my own world to live through
And I ain’t gonna copy you.
Now, if 6 turned up to be 9,
I don’t mind, I don’t mind.
If all the hippies cut off their hair,
I don’t care, I don’t care.
Did, ‘cos I got my own world to live through
And I ain’t gonna copy you.

Rod Smith
April 18, 2010 9:31 am

This is a very astute article Anthony.
However as a guy who ran weather editing sites for years even before METAR code became common, these and other errors are not new, nor generally in my day unknown to handlers/users. Maybe some of the old AWN editors will chime in here.
First I will remark that, as far as I know, all weather reports, in whatever code, have for decades only been transmitted with day-of-month and Zulu hours and minutes. The networks passing this “operational” data therefore have few (but not zero!) problems with date/time groups. It has always been the responsibility of the archiving agency, for example NCDC, to sort and archive this data in the right year and month bins. This is NOT a new problem. Do they do it with 100% accuracy? Your guess is as good (or better) than mine.
The RULE was always that an observation MUST have a decodable location and time as a minimum.
Since we MAILED 24 hours worth of data to Asheville daily, this is likely to be more of a problem for non-US archivers. I always thought that Asheville got the dates right, but that may not be correct.
I will comment that there are observation decoders, and there are observation decoders. Some errors are not detectable/correctable without reference to the last observation or analysis, and sometimes that may not even cause it to be either rejected or corrected.
The former USAF forecast center (AFGWC) ran “validators” which, among other thing compared current observations to recent observations. This (hopefully) caught the notorious problems in Siberian reports in coding very cold temperatures. (A long story — but in Synoptic, not METAR code!) There were also problems decoding dew-points from Mexico for a long time. I remember talking to our WMO rep about this. I believe this may have been in Airways code — which you identify as SA. Airways and AERO preceded METAR, which the US fought for years before its adoption. I have even seen library problems causing confusion in what units some reports were coded.
I would suspect that very few airports consider their primary function to be climatology rather than safety.
I would also suspect that most climatologists have essentially zero experience with observations beyond the archives. As a matter of fact, it would not surprise me if most code formats were unknown to these folks.
I might also make the point that when we first started automating weather relays (in the 60’s) we found that about 30% of the world wide traffic needed manual assistance.
No mater how it is sliced, you have just shown that real-world experience is invaluable.
Thank you, thank you, thank you.

Dave S
April 18, 2010 9:37 am

JDN (19:02:42) :
Notice how the errors as shown ALWAYS result in higher temps.
————–
That seems to be the natural consequence of (i) the unusual (in the context of normal mathematical/scientific usage) format – is someone more likely to inadvertently drop an M where there should be one, or inadvertently include an M where there shouldn’t? I know where I’d put my money – and (ii) that dropping the M still leaves a valid data format. If the rules required a P prefix for positives as well as M for negatives, then there would presumably be equal chance of accidentally dropping either prefix, and the resultant number would show up as invalid format.

April 18, 2010 9:38 am

A C Osborn (09:07:40) : So you are saying Anthony is lying then!
I am not saying that Anthony is lying.
I will leave it up to Anthony to decide if he wants to further clarify the bound and scope of the claims that can accurately be made given the current state of his investigation.
REPLY: So no comment then on the questions I posed to you? – Anthony

WTF
April 18, 2010 10:02 am

The thing that jumps out at me is how could the map showing Finland burning while all around is freezing have been published without someone (besides Anthony) saying wait a minute and looking ito it further. No QC obviously. Any computer program worth anything should flag these sort of errors for investigation.
Also how many conversion errors thoughout the temperature record are there? Didn’t NASA crash a Billion dollar probe into the surface of Mars a few years ago due to a conversion error?

Daniel H
April 18, 2010 10:14 am

How do you decode the AAXX formatted lines? I was looking at Calgary and saw a strange temp reading for Feb 21, 2010 at 5:00 PM:
4:00 PM 6.0 °C
5:00 PM -4 °C
5:00 PM 4.0 °C
The first 5:00 PM reading is based on the AAXX printout while the second is based on the METAR printout. I found this to be very confusing.
http://www.wunderground.com/history/airport/CYYC/2010/2/21/DailyHistory.html

April 18, 2010 10:31 am

Class act, Anthony.
“Since you are harping on titles, would you care to explain yours?”
Heads up to the Tip Jar!
Incommmminnng!!

Rod Smith
April 18, 2010 10:48 am

From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?

April 18, 2010 10:57 am

Anthony, is it possible for you to create a pdf of this post?—there is a lot to go through and being old school I’d like to read a hard copy and keep it in my files.

E.M.Smith
Editor
April 18, 2010 11:01 am

Richard M (07:15:46) : It seems like this error could be quantified. Couldn’t GIStemp be run against all non-airport stations and that result compared to their standard reports?
Um, the problem these days is finding NON-airport stations in GHCN to use.
For France, there was ONE station not flagged as an airport (and even that one had a commenter assert that it really IS at an airport and the meta-data were just failing to report that…
See this comment:
Demesure
“But at least Perpignan is not an Airport… ”
@chefio,
Err… untrue. It is at an airport : http://maps.google.fr/maps?ll=42.729468,2.8703895&z=13&t=h&hl=fr
Except Mont Aigoual, there is not a single French GHCN station with some decent data (continuous temperatures for more than 2 decades) which is NOT at airport,
Here is the list of the 6 (six !) “reference” stations for Meteo France, with precise coordinates, all are at airports (except Mont Aigoual).

From this article:
http://chiefio.wordpress.com/2010/03/30/france-hide-the-decline/
In the entire Pacific Ocean basin the Islands are headed for 100% Airports. The USA in GHCN was about 92%.
The bottom line is that looking for current non-airport sites in the “Global” data is like looking for a Ham Sandwich at a Kosher Deli… Sometimes you can find one, but it’s not at the best deli, it’s over to the side away from everything else, and close inspection will often show it’s really a turkey ham…
The other problem is that it’s a bit hard to run GIStemp on “variety data” with parts left out. It tends to suffer ‘brittle failure’ as it expects exact station matches at different points in the process (it matches files line by line on station ID from several different files. Yeah, 1970’s old school style…)
Sounds like a job for Superman, er um, Chiefio.
Wow! Er, um, stammer (Blush)… Golly….
I think it might be easier to run a report of the MIN temps and the MAX temps. What you ought to see is the MIN taking an upturn after 1996 while the MAX does not. Also, if done “by month”, there ought to be more “impact” in the “shoulder months” where temps could plausibly be either plus or minus C than in the dead of winter high latitude when the guys coding the reports ought to get used to putting an M on everything.
So get some Finish or Russian MIN / MAX data and look for differential behaviour with some kind of a ‘knee’ about the onset of the change to M flags. Ought to flag an “issue” pretty quickly…

Editor
April 18, 2010 11:04 am

Ron Broberg (08:36:28) :

My previous comments should read: Not a single one of the miscoded METAR stations records identified by Anthony have bled through into the GHCN Daily station data.
I am not excluding the possibility that there has been some bleed-through of miscoding errors. Just not the ones listed in the OP.

You could be doing a better service if you found why GISS used bad Finland data instead of saying Anthony’s examples are not the data points at fault.
BTW, I am intrigued that several records you found reported no high temperature for the day. That causes the obvious and other problems (like the perpetual restatement of the monthly temp for the indefinite future).
The bottom line in all this our raw data is bad in more ways that we thought and the errors make it into the final products.

April 18, 2010 11:12 am

This is excellent analysis and I accept Anthony’s position regarding the airport temps being temporary and disposable data and therefore not subjected to the scrutiny a scientific data point should be.
However, when that is combined with what Sera discovered in the Argo buoy data, I have to agree with:
===============
jim karlock (03:27:07) :
Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK
===============
Could it be possible that there are some renegade ghosts in the machine? Knowingly dropping M’s from time to time, and knowingly switching signs on data that is surely never transcribed by a human (buoy data)?
If they knew that daily averages would become monthly averages and the initial daily/hourly data was scrapped, it would be easy to nudge the numbers up.
I’m not a paranoid conspiracy theorists, but one or two zealots ‘doing the right thing – because you *know* it’s warming’ could do real harm to the temperature record, given the obvious lack of statistical quality control.
How can buoy data be acceptable that varies by ten degrees C back and forth 42 times?

Milwaukee Bob
April 18, 2010 11:16 am

kwik (03:39:32) :
Theo Goodwin (23:03:40) :
“I have a complaint about climate scientists’ use of the word “anomaly.””
Maybe using “Absolute” and “Relative” would be better?
Actually the correct “term” should be (is)- New Average.
Logically, scientifically, statistically you CAN NOT have an “anomaly” from/between averages. And this so called monthly global temperature is an average of averages of averages of averages. Even the satellite measurements as are an average AT the point in time of acquisition that are then averaged and summed and averaged.
When once again you “take” multiple measurements, average by area, then “fill” holes with averages, then average by geo region and “correct” by averaging, then sum and divide – – all you have is a new average, and the difference between it and the old average is NOT an anomaly.
You can ONLY have an anomaly from a “normal” or “fixed” or “set” condition. There is no such thing as global “normal” temperature.
While semantically we can collectively agree to “call” that difference anything we want, i.e., “anomaly”, you are correct kwik, for a scientist or statistician or for us here, for that matter, to use that term instead of calling it what it really is, A NEW Average, is technically wrong and I believe highly inappropriate AND plays into the argument from the AGW/CO2 side, which is of course why they use that term.
By identifying it as an entity on to its self and specifically calling it an anomaly, it gains stature and therefore meaning because people instinctively (even in other languages than English) think of it as a change from normal which implicitly means there IS a normal.
To simply call it what it is – an increase/decrease to the previous average would put it in the “almost meaningless” category.
We couldn’t have that now, could we…..

Daniel H
April 18, 2010 11:29 am

Anthony,
I found an example of the reported temp for Repulse Bay in Nunavut, Canada flipping the sign as well:
4:00 PM -22.0 °C -25.0 °C
METAR CYUT 112200Z 00000KT 25SM FEW014 FEW015 M22/M25
5:00 PM 25.0 °C 25.0 °C
METAR CYUT 112300Z CCA 00000KT 25SM FEW014 FEW015 25/25
http://www.wunderground.com/history/airport/YUT/2010/2/11/DailyHistory.html

Keith .
April 18, 2010 11:41 am

I find the data trail from Nizhnevartovsk to be interesting in a minor way. Same consistent error at roughly the same time on the two different days, an addition of 14-15 degrees Celsius to the previous values. The timing suggests someone giving the regular reporter a dinner break (7:00 p.m. local time) which would be consistent with a 3 to 11 shift. So that would suggest some form of conversion process that the breaker did not know was done by the regular person.
If the addition had been 17-18 degrees, I would say that it was likely that they were converting from a Fahrenheit thermometer to the Celsius scale, but forgetting the 32 degree offset. But 14-15 degrees C is only 25-27 degrees F, and I cannot think of anyway that that could be the conversion problem. Anyone have some ideas that could be associated with that temperature difference?

E.M.Smith
Editor
April 18, 2010 11:43 am

Crusty the Clown (07:23:09) : It was a rule to have input data validation checks and either reject or flag any apparently (obviously?) spurious data delivered to the program. You’d have to be nuts to skip that first sanity check.
There are a bunch of “QA Checks”, but they are not (IMHO) as well thought out as they might be. They use absolute temperature values for limit tests (instead of variable by altitude and latitude) so have differential error bands as those things change (high lats and altitudes have wider dynamic ranges, the filter doesn’t…)
from:
http://chiefio.wordpress.com/2010/04/11/qa-or-tossing-data-you-decide/
2) Climate division consistency. Departures of a station’s data from the monthly average of the data are calculated for all stations within a climatic division [see Guttman and Quayle (1996) for a description and history of the 344 climatic divisions in the contiguous United States]. The average departure for each day is then calculated. A datum is flagged for further review if the departure for a given station and day differs from the divisional average for the day by more than ±10°F. For a given day, temperature means and variances are estimated from all the divisional data that have not been flagged for further review. Any flagged data that exceed ±3 standard deviations from the mean for the day are then flagged for replacement. Replacement values are calculated from data for up to six nearby stations by the following procedure.
There are a couple of “issues” here. First off, the 10 F fixed limit. Normal LOW excursions will be further than NORMAL high excursions (convection limits highs, not lows) so more valid lows will be dropped than valid highs.
Then the dropped values are replaced by an AVERAGE. So if you drop a valid low excursion, you replace it with an average that will never be as low as a low excursion (averages are like that…).
Also, the “reference” is the accepted data… so errors that ‘make it’ can help assure more errors are ‘acceptable’ and make it too… As you add more (warmer) airports, more warm data is ‘acceptable’ and less cool data is ‘acceptable’.
There’s a lot more, but this is just a comment and I don’t want to reprise the whole article. There are some reasonable checks that would tend to catch the worst of these excursions (such as rate of change checks – but those also toss some valid data). But even those “have issues”. Not the least of which is that the data which makes it into GHCN come from the various National Met. Services already computed. Those NMS use their own procedures that may not include the QA steps used by NOAA on the US data. So this problem will have high variability by country…
The bottom line is that by the time we’ve made this Mulligan Stew of bits found by the airport all over the globe, it’s a real mess.

A C Osborn (08:22:16) :
Sera (21:04:16) :
Sera (21:12:55) :
Can anybody get this data?
If it as bad as Sera illustrates it should be added to Anthony’s findings, especially as it is in the Arctic.

Well, you can get the METAR and CLIMAT reports from Ogimet here:
English: http://www.ogimet.com/index.phtml.en
En Español: http://www.ogimet.com/
there are also other sources anyone can use on the net.

April 18, 2010 12:03 pm

Leif Svalgaard (19:15:23) :
A cultural/social effect may also be at work. When I was the US Representative to the Soviet Union on the Protection of the Environment [included weather forecasting] back in 1976 http://www.leif.org/EOS/Nature/261546a0.pdf I noticed, both from verbal interchange and from the evening news on TV that in the middle of winter when the temperature was typically -30C, the Russians never mentioned the minus sign; they would simply state “it’s 30 today”.
Leif you have it!
Hay we live at the 45th parallel and even here we DO NOT REPEAT over and over it’s bellow freezing! When you have 20 feet of snow and it is jan-feb when some one say’s it made it to 60 at night, You know what they mean ! know one repeats over and over the same thing, like, minus, below, under, see how this sounds;
well it is a minus, 10 degrees Fahrenheit below the point of water turns to ice.
what is the temperature? 22F? 10 below a point of scale? minus 10?
most people do not like the Kalvin scale but this would salve this problem.
TX A for the post
Tim L
10 degrees Fahrenheit below (the point of water turns to ice)
here we say most often 10 below 20 below 30 below, 20 below today and 40’s tonight!

bubbagyro
April 18, 2010 12:04 pm

As a scientist dealing with regulatory bodies, we have been forced to use redundancy in measurement and recording of data. This was a pain, but well justified. In USA, the CFR (Compendia of Federal Regulations) dictates that we must use both redundancy and witnessing of recorded data. It was educational to see how, even with a witness, data are recorded sometimes erroneously, especially with regard to introduction of systematic error. On top of redundancy and witnessing, a QA (Quality Assurance) Officer, reporting to the CEO not to QC, is employed. The reporting line function to the top, not an employee reporting to the QC (Quality Control) Officer is mandated so that there is no collusion to make the QC function look good, or pressure the subordinate to make the data look good. This insurance is also the basis for ISO (International Standard Operation) manufacturing guidelines, so that manufactured goods can be trusted for their quality. This methodology is costly, and a pain in the rear end for scientists, but with the billions being thrown around today in climate science, this should be the minimum standard insisted upon.
Another point – apart from inadvertent error, there is pressure to tow the company (AGW) line, that may lead to funny business, as we have seen occurring with Jones and Mann et.al. Who can rule out skulduggery in the current environment? Anthony is too much the gentleman to bring this up, but I don’t suffer from the same affliction!
Another problem, the practice of interpolation and extrapolation in the climate world of data processing has me very nervous. If an erroneous peak is introduced, and Anthony has shown how easy this is, how do the final masseuses massage that into the mean? If a final arbiter at CRU, GISS, or NASA sees a high peak, does he interpolate the peaks around that high to be intermediate and then does he/she employ a “correction”? IS a QC witness or a QA officer watching? (Not!) Does the data logger interpolate if that happens; if so, then the error becomes compounded as follows: How does that result affect the neighboring station that does not have a number for whatever the reason? Does the data handler extrapolate the number to the blank station? Interpolate with a distant station? What are the SOPs for these situations? Does this affect the Arctic more, since there are dismally few stations there?

April 18, 2010 12:17 pm

Quite a piece of work, Anthony. Very good and thanks!

wmsc
April 18, 2010 12:23 pm

GaryPearse (06:38:40) :
Anthony, were there not errors the other way,ie. wrongly encoding colder temps?
REPLY: I didn’t see any, but that does not mean they are not there. However, the only way to get colder temps, given the way the METAR decoder works, is to invert or mistype the data. For example 20/15 becomes 02/15, but in that case, the DP would be higher than the temp, and one would think they at least recognize that physical impossibility. – Anthony
======
http://www.wunderground.com/history/airport/KGYH/2009/11/18/DailyHistory.html
Doing a little bit more digging around on my own, I went looking through the records of a nearby airport, of which I know their procedures for getting data to the NWS, and I found a negative-going error (I know, shocking). This actually seems to have been a misplaced decimal on the part of the NWS when they typed it into the system:
Time Temp Dewpt
1:47 PM 57.2 °F
2:47 PM 33.8 °F 39.2 °F
3:47 PM 57.2 °F
At this facility, during those hours, the controllers are reading data from an AWOS, and calling it into the NWS. I would seriously doubt that the controllers would botch the numbers that way, and by that much.
It would seem to me, that there is no quick check going on at the NWS at their data entry stations, or by the folks entering the data and that is a travesty. As Anthony pointed out, DP isn’t going to be higher than Temp, unless something has gone horribly wrong with our understanding of physics 🙂
I wouldn’t expect, that it would slow things down too excessively, for the data entry terminals to do a comparison with the last hour’s data to see if the data was seriously wrong, and give the operator the option of passing the data through if it has indeed changed that much (ie the volcano started erupting, etc).

Glenn McKee
April 18, 2010 12:34 pm

Dear Sir,
In case you don’t have enough volunteers already, I have the technical knowledge and experience to organize a system to pull the METAR information across the internet, check for data issues and handle them appropriately. I could also provide statistical data summaries for each file processed. (I believe that those summaries would form the basis for doing a unsupervised learning in hopes of catching more than just a missing minus sign, the “black swan issue”.)
I am a dinosaur who has made a concerted effort to keep up with improvements programming in the areas computer science, and computational science. I have worked a lot with FORTRAN, C/C++, Python, and R (in both Linux/UNIX and WIN32). My first professional job included data collection and processing at a hydrodynamics lab.
I have done literate programming and have no issues with providing the code and supporting information for this project. I have always believed that I should be prepared to be challenged on my work and how it was done.
I have been between day jobs for a year. I would like to contribute something positive. Applying statistical learning techniques to my existing knowledge of reviewing and cleaning data would be a payback for me at this point. I also understand a new subject once I can get an interesting project going.
If you would like further information on my technical abilities, or a more detailed project design; please feel free to let me know. I have not read the entire thread yet because I wanted to get my offer into the queue while it might still be relevant.
I will understand if you already have all the resources that you can handle for now (the “Mythical Man-Month”, I have also been there and done that.)
Glenn McKee

FrankK
April 18, 2010 12:52 pm

I notice that the GISS diagram has the minus signs missing in the legend.
I commented earlier about por old Tasmania just below Australia being a nonsense “hotspot” in the GISS figure. I notice there is no such hotspot but normal temps in the satellite diagram for Tasmania. That makes much more sense.
Comparing the two diagrams I’d say GISS needs to do some urgent quality control of their data. Their grid resolution is also very poor.

JimAsh
April 18, 2010 1:03 pm

“Eureka, NWT is a great example, going from -43°C to +43°C (-45.4°F to 109.4°F) with one missing “M”.”
Holy Crud !
Keep up the good work. Surely some of this must be penetrating somewhere !

kim
April 18, 2010 1:10 pm

Whoa! Judith Curry just now on Bishop Hill: ‘I am no longer substituting the IPCC’s judgment for my own judgment in this matter’. The world in a tiny little sentence.
=============

E. Mitchell
April 18, 2010 1:16 pm

My daughter is a Russian linguist and I asked her about Leif Svalgaard’s comments about Russian media and others leaving out the “minus” or “negative” in weather reports during the winter. She said that yes, this would be likely – that it is common to leave out things that are “duh obvious” such as saying “30 degrees” when it was “-30 degrees”. She had seen that when she lived in Russian. While she has no specific expertise in meteorology she also said that she believes many of the reporting stations very likely continue to record data manually, not automatically. Consequently, simple manual recording error might be a factor in not coding as an “M”.

899
April 18, 2010 1:37 pm

Rod Smith: April 17, 2010 at 6:42 pm
From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?
~~~~~~~~~~~~~~~~~
Yes: FIGURES SHIFT (FIGS)
http://nemesis.lonestar.org/reference/telecom/codes/baudot.html

April 18, 2010 1:58 pm

observa (04:50:57) :
” Noticeably warmer nowadays with an airport that’s been privatised, an international terminal added and everything from a Harbourtown shopping centre with large bitumen carparks to logistics warehouses and even old folks home being built ever closer to the runways. That would have been unheard of for pilots in the past on safety grounds. Clearly better navigation and flight control has been the catalyst for a shrinking safety area around the runways and with it a rapidly increasing UHI effect over the last decade or so. Presumably Adelaide Airport is not alone due to such a technology factor?”
There’s an even simpler explanation than “better navigation and flight control”. Simple greed and the knowledge that nobody will be called to account when the inevitable instant urban renewal program occurs.

VicV
April 18, 2010 2:15 pm

What Digsby accentuated with a good laugh in his response to Felsche seems important to me:
Digsby (06:56:49):
Bernd Felsche (23:14:08) wrote :
“What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO ??”
… I think it is more likely that we will see this headline:
INCREASING NUMBER OF TYPOS CAUSED BY GLOBAL WARMING SAY SCIENTISTS!

For me, this echoes Methow Ken (04:49:09):
But I almost dispair at finding a practical solution to one huge problem; i.e.:
How do we ever get even a concise summary of important information like this considered by the MSM, let alone the average citizen; who has NO IDEA what is actually involved in gathering and adjusting all the data used to (at least theoretically) come up with statements like ”this was the warmest month on record in Oz”, etcetera ?? …
If even 1 out of 1000 people in North America have any idea that last two above are true let alone why and impact of same, I’d be surprised. But without at least a basic grasp of these and other key facts laid out in this post, citizens (and Members of Congress) are unable to make informed judgements on key issues; and remain vulnerable to self-appointed ”leaders” who contine to demagogue on the ”urgent need” to address CAGW regardless of cost and other priorities.

Is there anyone out there who gets it (Anthony’s terrific work) who can use their 6 degrees (or less) of separation to get a little quality time with a national morning television show weather man (or woman, if that’s the case) to present this? (I’d try to go through John Coleman in San Diego, but people have probably already been warned by their masters not to talk to him.)
With the Democratic leadership still experiencing inner gloat from “healthcare”, I think it may be approaching *urgent* that the Cap and Tax be shot dead in its tracks.

David, UK
April 18, 2010 2:15 pm

What with all these disappearing Ms, I bet Mr McIntyre and McKitrick are feeling mighty nervous right now. And Mr Watts, you be careful – a W is only an M upside down. And you know how good these guys are at flipping things upside down.

Wren
April 18, 2010 2:26 pm

David A (05:09:59) :
Regarding Wren (20:46:49) :
There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Wren, you may have demonstrated a classic case of confirmation bias. You wish to discount any possible effect on the trend, and so fail to note that this mistake is from “airport” stations, and there had been a dramatic move towards airport stations in the GISS data base; and therfore this would impact the trend, and even more so if these readings are extrapolated to dropped stations.
=====
There is a limit to how much a systematic bias can affect a trend. For example, I can gain weight faster if, in addition to eating more, I drink more and more bottles of beer each day immediately before weighing. But eventually I won’t be able to hold any more beers to give an upward bias to my weight gain trend.
If the recorders at airports are more likely to forget minus signs( whether indicated by “-” or “M”) than recorders not at airports, a move toward airport stations as a data base could introduce an upward bias in temperature records until there was no longer a move toward airport stations.
However, I doubt recorders at airports are any more likely to forget minus signs than other recorders. I think I am demonstrating a classic case of skepticism rather than a classic case of confirmation bias.

aurbo
April 18, 2010 2:42 pm

Re :”Rod Smith (10:48:05) :
From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?”
You are touching on the basis for why such a problem in coding exists today.
A little history:
Until the implementation of high-speed data networks from about the late 1960s forward, most WX data was transmitted by teletype. In the US their were three basic circuits used by the NWS for the dissemination of WX data.
Service “A” for the aviation hourly data, Service “C” for the National and regional synoptic reports, upper air (RAOB) data and NWS text information, and Service “O” for international data content similar to the domestic Service “C” data. The major constraint on these systems was the time it took for data transmission. The characters were 5 bits each. This would allow for only 31 characters except that one character was a shift character which was equivalent to shifting the keyboard into one of two cases (letters or figures).
All of the letters were in upper case while the figures case consisted of numerals, punctuation and special characters and non-printing characters for functions like CR, LF, Space, FIGS, LTRS, etc. The standard teletype speed was originally 45.5 baud or about 60 words per minute, by the 1950s it increased to 57 baud, roughly equivalent to 75 wpm per in baudot code. Service “A” machines had special weather charcters for their hourly messages which included arrows for wind direction and transmitted in the SA format which used these symbols. The shift to the METAR codingeliminated these characters and replaced these wind direction symbols using either numbers for 10s of degrees (32 = NW) or one or more combinations of the cardinal points (WNW = west-northwest). In the standard keyboard, the minus sign was in the Figures mode where the upper case “A” lay in the Letters mode. In the older SA code, tyhe minus sign was the equivalent of an open circuit signal when in the fugures mode. The differences in these two formats led to the removal of the minus sign as a useable character since its buadot code was different for each model configuration. Thus the infamous letter “M” was introduced to replace it. Today that would be unnecessary as the 8-level ASCII code allows for many more characters to be used unambiguously.
The highly abbreviated nature of the messages were essential because of the constraints of time at these low baud rates. In fact, experienced map-plotters could manually plot the data fast enough to stay ahead of the machines including the Model-15 and Model-28 page printers generally in use. So blame the genesis of the minus sign problems on the need for the highly abbreviated coding used in the hourly and synoptic data transmissionsduringb the mid 20th Century.
By the time I stopped plotting data in real time manually and moved on to using the facsimile charts and then computerized plotting, the transmission speed of NWS data had increased in stages from 100 baud, to 300, then 1200 and finally 9600 baud for digital data transmission to printers.
BTW, When I loooked at the Finnish CLIMAT transmission, they had it coded correctly, and this is what GISS should have been ingesting. What might have been misleading was that in the color coded portion of their message which was produced for public consumption, the temperature departures from normal for each station did not have the minus sign, although the accompanying color coding of the entries for those stations clearly used the dark blue indicating much below normal!

Pompous Git