Here’s a story about how one missing letter, an M, can wreck a whole month’s worth of climate data. It is one of the longest posts ever made on WUWT, I spent almost my entire Saturday on it. I think it might also be one of the most important because it demonstrates a serious weakness in surface data reporting.
In my last post, we talked about the a curious temperature anomaly that Jean S. found in the March GISS data and posted at Climate Audit:
The anomaly over Finland has an interesting signature to it, and the correction that GISS posted on their website confirms something I’ve been looking at for a few months.
The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.
With some work I started back in late December and through January, and with GISS putting stamp of approval on “missing minus signs” I can now demonstrate that missing minus signs aren’t just an odd event, they happen with regularity, and the effect is quite pronounced when it does happen. This goes to the very heart of data gathering integrity and is rooted in simple human error. The fault lies not with GISS (though now they need a new quality control feature) but mostly with NOAA/NCDC who manages the GHCN and who also needs better quality control. The error originates at the airport, likely with a guy sitting in the control tower. Readers who are pilots will understand this when they see what I’m talking about.
I’ve seen this error happen all over the world. Please read on and be patient, there is a lot of minutiae that must be discussed to properly frame the issue. I have to start at the very bottom of the climate data food-chain and work upwards.
First, a discussion about the root of error and the differences between the surface and satellite dataset. I should mention that in the satellite image from NASA’s Earth Observations (NEO), we don’t see the same error as we see in the GISTEMP map above.
Why? Better sensors, maybe, but mostly it has to do with a different data gathering methodology. In the surface data sets, including land and ocean data, most every datapoint is touched by a human hand, even airport data that gets done by automated airport sensors sometimes gets transcribed manually (often in third world and technologically stunted countries). In the surface data, thousands of sensors are spread across the globe, many different designs, many different exposures, many different people with different standards of measurement and reporting. The precision, accuracy, and calibration of the vast surface network varies, especially when we have broad mix of instrumentation types.For example in the US Historical Climatological Network the equipment varies significantly.
In satellite data, the data is measured at a single point with one sensor type, the Microwave Sounder Unit on the satellite, calibrated to a precision source on-board. On-board redundant precision platinum resistance thermometers (PRTs) carried on the satellite radiometers. The PRT’s are individually calibrated in a laboratory before being installed in the instruments. The satellite data is automatically measured and transmitted. In contrast to the surface temperature record, no human hands touch the data gathering or data reporting process. Satellite data generation is far more homogeneous than the mish-mash of surface data.
I think it would be safe to say that the chances of human error in raw surface data are at least an order of magnitude greater (if not several) than error in raw satellite data. Post measurement processing is another issue, but for the purposes of this essay, I’m focusing only on raw data gathering and transmittal.
As mentioned in the recently updated compendium of issues with the surface temperature data by Joe D’Aleo and myself, there has been a move in the Global Historical Climatological Network (GHCN) to rely more and more on airports for climate data. This, in my opinion, is a huge mistake because in addition to those issues
E.M. Smith aka “Chiefio” reports that in GISS (which uses GHCN) worldwide, there has been a wholesale migration towards airport weather data as a climatic data source. In an email sent to me on Jan 20, 2010 he says that
Look at:
http://chiefio.wordpress.com/2009/08/26/agw-gistemp-measure-jet-age-airport-growth/
which as a fairly good descriptions of the problems in the data, we have a global report for GHCN as of that August data. There is more deail in the link, but I think you care about “now”:
Percentage of sites that are AIRPORTS NOW, by decade of record Year S.P S.C S.T S.W EQ. N.W N.T N.C N.P Total 1909 0.0 42.0 15.1 28.2 29.2 36.7 22.8 33.3 44.4 25.4 1919 0.0 36.4 12.8 23.5 25.1 37.7 20.9 35.0 39.8 24.1 1929 0.0 37.0 11.9 27.4 27.7 32.7 20.4 35.9 56.4 24.1 1939 0.0 43.9 17.6 32.0 33.8 29.1 20.2 36.2 51.0 25.1 1949 0.0 32.3 24.4 37.6 44.4 31.8 23.3 39.3 60.9 29.1 1959 0.0 24.0 35.0 50.0 59.4 39.4 30.9 41.0 62.9 37.3 1969 0.0 18.1 39.3 53.2 63.2 40.2 31.4 41.1 61.5 39.0 1979 0.0 17.9 39.1 52.0 64.2 40.7 28.8 41.1 62.3 37.7 1989 0.0 20.7 41.5 52.5 67.8 41.9 29.1 40.8 64.9 37.7 1999 0.0 21.0 53.5 57.4 68.0 53.0 32.6 49.0 59.0 41.6 2009 0.0 17.9 74.0 64.7 66.5 51.5 30.2 45.4 57.3 41.0
I do break outs by continent and by some countries. For the USA, I further do a specific with / without USHCN (the older version, not the USHCN.v2 put in 15Nov09) and findFor COUNTRY CODE: 425
LATpct: 2006 3.7 18.3 29.5 33.2 14.4 0.0 0.4 0.3 0.1 0.1 100.0 AIRpct: 1.3 4.0 6.3 6.7 3.2 0.0 0.4 0.3 0.1 0.1 22.4 LATpct: 2007 8.2 17.2 28.4 26.9 11.2 0.0 3.7 3.0 0.7 0.7 100.0 AIRpct: 8.2 15.7 27.6 23.1 9.0 0.0 3.7 3.0 0.7 0.7 91.8 LATpct: 2008 8.8 16.9 28.7 26.5 11.0 0.0 3.7 2.9 0.7 0.7 100.0 AIRpct: 8.8 15.4 27.9 22.8 8.8 0.0 3.7 2.9 0.7 0.7 91.9 LATpct: 2009 8.1 17.8 28.1 26.7 11.1 0.0 3.7 3.0 0.7 0.7 100.0 AIRpct: 8.1 16.3 27.4 23.0 8.9 0.0 3.7 3.0 0.7 0.7 91.9 DLaPct: 2009 4.3 18.4 29.5 32.5 13.6 0.0 0.7 0.9 0.2 0.1 100.0 DArPct: 2.1 5.7 8.8 8.9 3.7 0.0 0.6 0.8 0.2 0.1 30.7
That in the YEAR 2009 the USA has almost 92% airports in GHCN.
So clearly, airports make up a significant portion of the climate data.
On the issues of airports as climate station, obvious issues with siting, UHI, failing ASOS instrumentation, and conflicting missions (aviation safety -vs-climate) aside, I’m going to focus on one other thing unique to airports: METAR
What is METAR you ask? Well in my opinion, a government invented mess.
When I was a private pilot (which I had to give up due to worsening hearing loss – tower controllers talk like auctioneers on the radio and one day I got the active runway backwards and found myself head-on to traffic. I decided then I was a danger to myself and others.) I learned to read SA reports from airports all over the country. SA reports were manually coded teletype reports sent hourly worldwide so that pilots could know what the weather was in airport destinations. They were also used by the NWS to plot synoptic weather maps. Some readers may remember Alden Weatherfax maps hung up at FAA Flight service stations which were filled with hundreds of plotted airport station SA (surface aviation) reports.
The SA reports were easy to visually decode right off the teletype printout:
Note that in the example above, temperature and dewpoint are clearly delineated by slashes. Also, when a minus temperature occurs, such as -10 degrees Fahrenheit, it was reported as “-10”. Hang on to that, it is important.
The SA method originated with airmen and teletype machines in the 1920’s and lasted well into the 1990’s. But like anything these days, government stepped in and decided it could do it better. You can thank the United Nations, the French, and the World Meteorological Organization (WMO) for this one. SA reports were replaced by METAR in 1996.
From Wikipedia’s section on METAR
METAR reports typically come from airports or permanent weather observation stations. Reports are typically generated once an hour; if conditions change significantly, however, they can be updated in special reports called SPECIs. Some reports are encoded by automated airport weather stations located at airports, military bases, and other sites. Some locations still use augmented observations, which are recorded by digital sensors, encoded via software, and then reviewed by certified weather observers or forecasters prior to being transmitted. Observations may also be taken by trained observers or forecasters who manually observe and encode their observations prior to transmission.
History
The METAR format was introduced 1 January 1968 internationally and has been modified a number of times since. North American countries continued to use a Surface Aviation Observation (SAO) for current weather conditions until 1 June 1996, when this report was replaced with an approved variant of the METAR agreed upon in a 1989 Geneva agreement. The World Meteorological Organization‘s (WMO) publication No. 782 “Aerodrome Reports and Forecasts” contains the base METAR code as adopted by the WMO member countries.[1]
Naming
The name METAR is commonly believed to have its origins in the French phrase message d’observation météorologique pour l’aviation régulière (“Aviation routine weather observation message” or “report”) and would therefore be a contraction of MÉTéorologique Aviation Régulière. The United States Federal Aviation Administration (FAA) lays down the definition in its publication the Aeronautical Information Manual as aviation routine weather report[2] while the international authority for the code form, the WMO, holds the definition to be aerodrome routine meteorological report. The National Oceanic and Atmospheric Administration (part of the United States Department of Commerce) and the United Kingdom‘s Met Office both employ the definition used by the FAA. METAR is also known as Meteorological Terminal Aviation Routine Weather Report or Meteorological Aviation Report.
I’ve always thought METAR coding was a step backwards, for reasons I’ll discuss shortly.
But first, quick! Spot the temperature and dewpoint in this METAR report:
The following is an example METAR from Burgas Airport in Burgas, Bulgaria, and was taken on 4 February 2005 at 16:00 Coordinated Universal Time (UTC).
METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 M04/M07 Q1020 NOSIG 9949//91=
Could you read this and know what the weather is in Burgas? I can, only becuase I’ve looked at hundreds the past few months, but I still have to pick through the report to find it. The reason is that METAR is a variable field reporting format. Data isn’t always in the same position.
In the report above. The temperature and dewpoint is: M04/M07
M04/M07 indicates the temperature is −4 °C (25 °F) and the dewpoint is −7 °C (19 °F). An M in front of the number indicates that the temperature/dew point is below zero (0) Celsius.
Notice also that the entire METAR report is visually more complex. This is fine if you are having computers code it, but many METAR reports are still hand coded by technicians at airports, and thus begins the introduction of human error into the climate data process. Complexity is not a good thing when manual labor is involved as it increases the likelihood of error.
Here is where METAR coding departs from normal numeric convention. SA reports did not have this problem.
In the METAR report above, instead of using the normal way we treat and write negative numbers, some policy wonk decided that we’ll use the letter “M” to report a negative number. Only a bureaucrat could think like this.
So instead of a below zero Centigrade temperature and dewpoint looking like this:
-04/-07
in the “new and improved” METAR coding, it looks like this:
M04/M07
OK not a problem you say? Well I beg to differ, because it forces technicians who manually code METAR reports for transmission to do something they would not do anywhere else, and that’s write down an “M” instead of a minus sign. Using an M is totally counter-intuitive and against basic math training, and increases the likelihood of error.
It gets worse. Let’s say the technician makes a boo-boo and puts a minus sign instead of an “M” in front of the numbers for temperature/dewpoint. You’d think this would be alright, and the system would correctly interpret it, right?
Let’s put the METAR report from Burgas Airport into an online METAR decoder.
http://www.wx-now.com/Weather/MetarDecode.aspx
Here’s the report with the easy to make mistake, using minus sign instead of M for the temperature.
METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 -04/M07 Q1020 NOSIG 9949//91=
The output from the online METAR decoder reads:
Hey look at that, the temperature is 39°F (3.8°C). Minus signs are discarded from METAR decoding. Note that decoded METAR temperature also comes out the same if the “M” is missing in front of the 04/-07 or 04/M07
If it had been decoded correctly we would have gotten:
(-4) degrees Celsius = 24.8 degrees Fahrenheit
A whole 14.2 degrees F difference!
Reference for METAR decoding:
http://www.met.tamu.edu/class/METAR/quick-metar.html
Also note that METAR data has no year stamp component to the data, so the METAR decoder has no way of knowing this was a report from 2005, not 2010. Since each METAR report is essentially disposable within 24 hours, this presents no problem for pilots, they don’t care. But if you are tracking climate over years using METAR data, not having a year time stamp increases the likelihood of error.
Also the temperature error itself in this case has no bearing on a pilot’s decision to takeoff or land. Unless they are worried about density altitude on hot humid days, the temperature is a throwaway datum. They are mostly concerned about winds, sky conditions, and barometer (altimeter setting). In fact cool/cold days are far better for aviators; see AOPA’s Why Airplanes Like Cool Days Better.
My point here is this:
If a pilot or tower controller sees an erroneous METAR report like this:
METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 -04/M07 Q1020 NOSIG 9949//91=
Or this:
METAR LBBG 041600Z 12003MPS 310V290 1400 R04/P1500N R22/P1500U +SN BKN022 OVC050 04/M07 Q1020 NOSIG 9949//91=
Pilots/controllers/dispatchers aren’t likely to care, since current temperature and dewpoint are not important to them at these cooler temperatures. They also aren’t likely to call up the tower and holler at the technician to say “Hey, the temperature is wrong!”. Further, since the METAR report may be reissued sometime within the hour if somebody DOES spot the error, problem solved.
Point is that updates/corrections to METAR data may not be logged for climate purposes, since they are likely to be seen as duplicate reports because of the hourly timestamp.
So, in the case of M’s and minus signs, the propensity exists for erroneous METAR reports to not get corrected and to stay logged in the system, eventually finding their way into the climate database if that airport happens to be part of GISS, CRU, or GHCN datasets.
Maddeningly, even when egregious errors in aviation weather data are pointed out and even acknowledged by the reporting agency, NOAA keep them in the climate record as was demonstrated last year in Honolulu, HI International Airport when a string of new high temperature records were set by a faulty ASOS reporting station. NOAA declined to fix the issue in the records:
NOAA: FUBAR high temp/climate records from faulty sensor to remain in place at Honolulu
The key sentence from that story from KITV-TV:
The National Weather Service said that is not significant enough to throw out the data and recent records.
Hmmm, look at another nearby station and compare the differences. You be the judge.
Does NOAA consider this a climate reporting station? Yes according to NCDC MMS database, it is part of the “A” network, designated for climate:
Clearly, NOAA simply doesn’t seem to care that erroneous records finds their way into the climatic database.
OK back to the METAR issue.
The problem with METAR reporting errors is worldwide. I’ve found many examples easily in my spare time. Let’s take for example, a station in Mirnvy, Russia. It is in Siberia at 62.5° N 113.9° E and has an airport, is part of GHCN, and reports in METAR format.
Weather Underground logs and plots METAR reports worldwide, and these METAR reports are from their database on November 11th, 2009.
It shows a clear error in the 12:30PM (330Z) and 1 PM (400Z) METAR report for that day:
UERR 010330Z 22005G08MPS 9999 -SN 21/M23 Q1026 NOSIG RMK QFE738 24450245 UERR 010400Z 22005G08MPS 9999 -SN SCT100 OVC200 20/M22 Q1025 NOSIG RMK QFE737 24450245 UERR 010430Z 21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245 UERR 010430Z 21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245 UERR 010500Z 21005G08MPS 4000 -SN SCT100 OVC200 20/M22 Q1023 NOSIG RMK QFE736 24450245
Note the missing ” M” on the 12:30PM (330Z) and 1 PM (400Z). It happens again at 2PM (500Z). Of course it isn’t very noticeable looking at the METAR reports, but like the GISS plot of Finland, stands out like a sore thumb when plotted visually thanks to Weather Underground:
Mirnvy, Russia
The effect of the missing “M” is plotted above, which coincidentally looks like an “M”.
Put those METAR reports in this online METAR decoder: http://www.wx-now.com/Weather/MetarDecode.aspx and you get 70F for 12:30PM and 68F for 1PM
What do you think 70 degree F spike this will do to monthly averaged climate data in a place where the temperature stays mostly below freezing the entire month?
http://www.wunderground.com/history/airport/UERR/2009/11/1/MonthlyHistory.html?MR=1
Does NOAA log METAR data from Mirnvy Russia (ICAO code UERR)?
Yes, they do. Plus many other METAR reporting stations discussed here.
Does NCDC classify it as a climate station?
Yep, it is part of the “A” network. Which means it either directly reports climate data and/or is used to adjust data at other stations, such as GHCN stations.
List of GHCN stations:
http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
It is not however, part of GHCN. But there are plenty stations that have this error that are part of GHCN. Yakutsk, Russia, also in Siberia is part of GHCN and has a METAR reporting error. Here’s an example what one off-coded hourly reading will do to the climate database.
The city of Yakutsk, one of the coldest cities on earth, reported a high of 79˚F on November 14th with a low of -23˚F.
Weather Underground seems to have done some quality control to the METAR reports, but the erroneous high temp remains in the daily and monthly report:
http://www.wunderground.com/history/station/24959/2009/11/14/DailyHistory.html
http://www.wunderground.com/history/station/24959/2009/11/14/MonthlyHistory.html
A month later, it happened again reporting a high of 93˚F on December 14th with a low of -34˚F
And the erroneous 93F high temp remains in both the daily and monthly reports, but has been removed from the METAR report, so I can’t show you the missing “M” I observed back in January. I wish I had made a page screen cap.
http://www.wunderground.com/history/station/24959/2009/12/14/DailyHistory.html
http://www.wunderground.com/history/station/24959/2009/12/14/MonthlyHistory.html
When the temperature data was calculated with that error then, this was found:
The average for the day, 30˚F, was some 67˚F above normal, pushing the anomaly for the month of December from 3.6˚F above normal to 5.9˚F above normal… quite a shift!
More examples:
Here’s an example of a properly coded METAR report from Nizhnevartovsk, Russia, for December 11, 2009, but the data is wrong. I’m thinking it was supposed to be M30 but came out M13. The dewpoint value M16 is also erroneous.
Nizhnevartovsk, Russia Dec 7, 2009
METAR USNN 111230Z 00000MPS P6000 SCT026 OVC066 M27/M29 Q1014 NOSIG RMK QFE755 SC062 METAR USNN 111300Z 12005G08MPS P6000 SCT066 OVC200 M13/M16 Q1035 NOSIG RMK QFE772 SC063 METAR USNN 111330Z 12005G08MPS P6000 SCT066 OVC200 M13/M16 Q1035 NOSIG RMK QFE772 SC063 METAR USNN 111400Z 00000MPS P6000 SCT020 OVC066 M28/M29 Q1014 NOSIG RMK QFE755 SC065
And it was not a one time occurrence, happening again on Dec 25th as shown in the Monthly graph:
Nizhnevartovsk, Russia, December 2009
http://www.wunderground.com/history/airport/USNN/2009/12/25/MonthlyHistory.html
The daily graph and METAR reports, notice it happened about the same time (1300Z) and in the same way (M27 then M13) , perhaps pointing to the same technician on duty making the same habitual mistake again. Maybe too much Vodka having to work the Xmas night shift?
Nizhnevartovsk, Russia Dec 25, 2009
METAR USNN 251230Z 11006MPS 2200 -SN SCT014 OVC066 M27/M30 Q1015 NOSIG RMK QFE757 SC055 METAR USNN 251300Z 35002MPS 6000 -SN SCT015 OVC066 M13/M15 Q1010 NOSIG RMK QFE752 SC03 METAR USNN 251330Z 12006MPS 4100 -SN SCT015 OVC066 M27/M29 Q1014 NOSIG RMK QFE756 SC055
http://www.wunderground.com/history/airport/USNN/2009/12/25/DailyHistory.html
It did not appear initially to be in the GHCN list or on the GISS list, but I’ve found that some of the names on Weather Underground are different from the place names in the GHCN and GISS lists. It turns out that if you search in Weather Underground for the station ALEKSANDROVSKOE it will point you to use the data from Nizhnevartovsk. ALEKSANDROVSKOE is a GHCN/GISS station.
I found other instance of METAR errors for that station, this one was quite pronounced on Jan 16th, 2009, lasting for 7 hours before it was corrected.
Nizhnevartovsk, Russia Jan 16, 2009
Here’s the METAR reports
METAR USNN 151800Z 23002MPS P6000 BKN066 OVC200 M22/M24 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 151830Z 23002MPS 2900 -SHSN SCT020CB OVC066 22/M23 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 151900Z 23002MPS 2100 -SHSN SCT019CB OVC066 21/M23 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 152000Z 24001MPS 5000 -SHSN SCT022CB OVC066 21/M22 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 152030Z 24002MPS 4300 -SHSN SCT020CB OVC066 21/M22 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 152100Z 24002MPS 6000 -SHSN SCT018CB OVC066 20/M22 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 152130Z 25002MPS P6000 SCT020CB OVC066 20/M22 Q1009 NOSIG RMK QFE751 SC038 METAR USNN 152200Z 28002MPS P6000 SCT022CB OVC066 20/M22 Q1009 NOSIG RMK QFE752 SC038 METAR USNN 152300Z 27003MPS P6000 -SHSN SCT016CB OVC066 M19/M21 Q1010 NOSIG RMK QFE752 SC038
http://www.wunderground.com/history/airport/USNN/2009/1/16/DailyHistory.html
The monthly report shows the event:
Nizhnevartovsk, Russia, January 2009
http://www.wunderground.com/history/airport/USNN/2009/1/16/MonthlyHistory.html
It happened twice on Feb 2nd, 2007, and with a space added between the M and 09 on the 0300Z report, it is a clear case of human error:
METAR USNN 020100Z 11010G15MPS 0500 R03/1200 +SN +BLSN VV002 M09/M11 Q1003 TEMPO 0400 +SN +BLSN VV002 RMK QFE748 QWW060 MOD ICE MOD TURB S METAR USNN 020200Z 12009G14MPS 0500 R03/1200 +SN +BLSN VV002 M09/M10 Q1001 TEMPO 0400 +SN +BLSN VV002 RMK QFE747 QWW060 MOD ICE MOD TURB S METAR USNN 020300Z 11008G13MPS 1100 R03/1200 SN +BLSN BKN004 OVC066 M 09/M10 Q1000 NOSIG RMK QFE745 QRD120 MOD ICE MOD TURB SC045 ... METAR USNN 021200Z 18009MPS P6000 -SHSN DRSN SCT017CB OVC066 M07/M09 Q0989 TEMPO 2000 SHSN RMK QFE736 MOD ICE MOD TURB SC042 METAR USNN 021300Z 16009MPS P6000 DRSN SCT016CB OVC066 08/M11 Q0989 NOSIG RMK QFE736 MOD ICE MOD TURB SC042 METAR USNN 021400Z 16008MPS P6000 DRSN SCT016CB OVC066 M08/M11 Q0989 NOSIG RMK QFE736 MOD ICE MOD TURB SC042
http://www.wunderground.com/history/airport/USNN/2007/2/2/DailyHistory.html
The monthly data shows the double peak:
http://www.wunderground.com/history/airport/USNN/2007/2/2/MonthlyHistory.html
I’m sure many more can be found, I invite readers to have a look for themselves by looking for such events at Weather Underground
It is not just Russia that has METAR reporting errors
Lest you think this a fault of Russia exclusively, it also happens in other northern hemisphere Arctic site and also in Antarctica.
Svalbard, Oct 2, 2008
METAR ENSB 020550Z 13012KT 6000 -SN FEW010 SCT015 BKN030 M04/M06 Q1013 TEMPO 4000 SN BKN012 METAR ENSB 020650Z 14013KT 9000 -SN FEW010 SCT018 BKN040 03/M06 Q1013 TEMPO 4000 SN BKN012 METAR ENSB 020750Z 15011KT 9999 -SN FEW015 SCT025 BKN040 M03/M07 Q1013 TEMPO 4000 SN BKN012
http://www.wunderground.com/history/airport/ENSB/2008/10/2/DailyHistory.html
Eureka, Northwest Territory, Canada March 3 2007
It hit 109.4 F (43C) there on March 3rd 2007 according to this METAR report. Eureka is the northernmost GHCN station remaining for Canada. It’s temperature gets interpolated into adjacent grid cells.
CWEU 031600Z 14004KT 15SM FEW065 BKN120 M43/M45 A2999 RMK ST1AS2 VIA YQD SLP150 CWEU 031700Z 15005KT 10SM FEW065 BKN012 43/46 A3000 RMK SF1AS1 VIA YQD SLP163 Decoded: 11:00 AM 109.4 °F 114.8 °F 100% 30.01 in 10.0 miles SSE 5.8 mph - Mostly Cloudy CWEU 031800Z 11003KT 15SM FEW050 FEW065 OVC130 M43/M46 A3001 RMK SF2SC1AS1 VIA YQD SLP164
http://www.wunderground.com/history/airport/CWEU/2007/3/3/DailyHistory.html
In these cases below for Antarctic stations Dome C and Nico, the METAR reports seem to have all sorts of format issues and I’m not even sure how where the error occurs, except that Weather Underground reports a spike just like we see in Russia.
Dome C station Dec 9, 2009
AAXX 0900/ 89828 46/// ///// 11255 36514 4//// 5//// 90010 AAXX 0901/ 89828 46/// ///// 10091 36514 4//// 5//// AAXX 09014 89828 46/// /1604 11225 36480 4//// 5//// 9014
http://www.wunderground.com/history/station/89828/2009/12/9/DailyHistory.html
Nico Station, University of Wisconsin Dec 9, 2009
AAXX 0920/ 89799 46/// ///// 11261 4//// 5//// 92030 AAXX 0920/ 89799 46/// ///// 11103 4//// 5//// 92040 AAXX 0921/ 89799 46/// ///// 11270 4//// 5////
http://www.wunderground.com/history/station/89799/2009/12/9/DailyHistory.html
Admusen Scott Station Jan 14th, 2003
Here’s generally properly formatted METAR data, but note where the technician coded the extra space, oops!
NZSP 131350Z GRID36007KT 9999 IC SCT020 BKN060 M31/ A2918 RMK SDG/HDG NZSP 131450Z GRID36007KT 9999 IC FEW010 FEW020 SCT035 SCT050 M3 1/ A2918 RM K SDG/HDG NZSP 131550Z GRID10008KT 9999 IC BCFG FEW010 SCT020 BKN050 M31/ A2919 RMK VIS E 2400 BCFG E SDG/HDG
http://www.wunderground.com/history/station/89009/2003/1/14/DailyHistory.html
And I’m sure there are many more METAR coding errors yet to be discovered. What you see above is just a sampling of a few likely candidates I looked at over a couple of hours.
Missing M’s – Instant Polar Amplification?
It has been said that the global warming signature will show up at the poles first. Polar Amplification is defined as:
“Polar amplification (greater temperature increases in the Arctic compared to the earth as a whole) is a result of the collective effect of these feedbacks and other processes.” It does not apply to the Antarctic, because the Southern Ocean acts as a heat sink. It is common to see it stated that “Climate models generally predict amplified warming in polar regions”, e.g. Doran et al. However, climate models predict amplified warming for the Arctic but only modest warming for Antarctica.
Interestingly, the METAR coding error has its greatest magnitude at the poles, becuase the differences in the missing minus sign become larger as the temperature grows colder. Eureka, NWT is a great example, going from -43°C to +43°C (-45.4°F to 109.4°F) with one missing “M”.
You wouldn’t notice METAR coding errors at the equator, because the temperature never gets below 0°C. Nobody would have to code it. In middle latitudes, you might see it happen, but it is much more seasonal and the difference is not that great.
For example:
M05/M08 to 05/M08 brings the temp from -5°C to +5°C, but in a place like Boston, Chicago, Denver, etc a plus 5C temperature could easily happen in any winter month a -5C temperature occurred. So the error slips into the noise of “weather”, likely never to be noticed. But it does bump up the temperature average a little bit for the month if uncorrected.
But in the Arctic and Antarctic, a missing M on a M20/M25 METAR report makes a 40°C difference when it becomes +20°C. And it doesn’t seem likely that we’d see a winter month in Siberia or Antarctica that would normally hit 20°C, so it does not get lost in the “weather” noise, but becomes a strong signal if uncorrected.
Confirmation bias, expecting to see polar amplification may be one reason why until now, nobody seems to have pointed it out. Plus, the organizations that present surface derived climate data, GISS, CRU, only seem to deal in monthly and yearly averages. Daily or hourly data is not presented that I am aware of, and so if errors occur at those time scales, they would not be noticed. Obviously GISS didn’t notice the recent Finland error, even though it was glaringly obvious once plotted.
With NASA GISS admitting that missing minus signs contributed to the hot anomaly over Finland in March, and with the many METAR coding error events I’ve demonstrated on opposite sides of the globe, it seems reasonable to conclude that our METAR data from cold places might very well be systemically corrupted with instances of coding errors.
The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.
That darned missing M, or an extra space, or even writing “-” when you mean “M” (which is counterintuituve to basic math) all seem to have a factor in the human error contributing to data errors in our global surface temperature database. To determine just how much of a problem this is, a comprehensive bottom up review of all the data, from source to product is needed. This needs to start with NOAA/NCDC as they are ultimately responsible for data quality control.
It has been said that “humans cause global warming”. I think a more accurate statement would be “human error causes global warming”.
Note: In this post I’ve demonstrated the errors. In a later post, I hope to do some data analysis with the numbers provided to see how much effect these errors actually have. Of course anyone who wants to do this is welcome to leave links to graphics and tables. -Anthony
UERR 010330Z 22005G08MPS 9999 -SN 21/M23 Q1026 NOSIG RMK QFE738 24450245
UERR 010400Z 22005G08MPS 9999 -SN SCT100 OVC200 20/M22 Q1025 NOSIG RMK QFE737 24450245
UERR 010430Z 21005G08MPS 4000 -SN SCT100 OVC200 M20/M22 Q1024 NOSIG RMK QFE737 24450245
Sponsored IT training links:
Ultimate VCP-410 practice files formulated by top experts to help you pass 220-702 and 640-822 exam in easy and fast way.
Fascinating post. Regarding the UHI effect, I have a question I hope someone can address.
Is it plausible that UHI might be increased by a _local_ greenhouse effect? What I mean is this: if carbon dioxide and other greenhouse gases are trapping heat, might they be trapping it to a greater extent at the locations where large amounts of those gases are emitted? Like urban areas, and airports particularly?
Or does the greenhouse effect (whatever one may reckon as it’s extent) only operate at upper atmosphere levels and over massive regional or global levels?
Notice how the errors as shown ALWAYS result in higher temps.
It’s GIGO all over again!
Two wrongs (bad data plus bad models) did not make a right!
Thanks, Anthony. Great post (with lots of graphics). This was enlightening.
REPLY: Coming from Mr. “Gobs of Graphics” himself, that is quite a complement. Thanks -A
I noticed in February in Colorado that NCDC/NOAA was consistently showing above average temperatures along the Front Range – while Accuweather stats, the unusually prolonged frozen lakes, and practically everyone I talked to indicated that it was a very cold February. Accuweather showed it 2-4 degrees below normal.
A cultural/social effect may also be at work. When I was the US Representative to the Soviet Union on the Protection of the Environment [included weather forecasting] back in 1976 http://www.leif.org/EOS/Nature/261546a0.pdf I noticed, both from verbal interchange and from the evening news on TV that in the middle of winter when the temperature was typically -30C, the Russians never mentioned the minus sign; they would simply state “it’s 30 today”.
Anthony, thanks for your hard work on this! Excellent analysis, I’ve seen such errors of omission/coding in epidemiological studies of pathogen counts with similar results. Garbage in, garbage out!
With the powerful neural network and AI tools available, you’d think that someone who was serious about the integrity of their data would subject it to some level of peer review by statistical process control. I don’t think it would be hard at all.
And yet, the AGW train keeps on rolling down the tracks, with a Senate cap & trade/energy bill in the works. It will be interesting.
That’s right,Anthony,accusations and extrapolation first,then ‘some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
REPLY: Nick, dial down your anger, then look at the hard data errors presented. Glaciers are mostly proxies for precipitation, so they don’t really care anyway. Surface project is coming along fine, a paper is being prepared. Thanks for asking. Of course if you just want to ignore what is presented, and rant, I suggest ClimateProgress, where ranting is an art form. -A
wonderful work.
Excellent report, Anthony.
As Ric Werme said earlier, temperatures in the form of degrees Kelvin would solve the problem of negative temperatures.
I have had the experience of finding a weather station with all negative (degrees Celsius – so below freezing) temperatures SET TO ZERO deg C.
When I consulted officials, they admitted serious data quality issues for the main sites (the ones most valuable to me) at which I was looking.
They also thanked me for pointing out 200cm (over 6 feet) of snow that came and went in a single day. They guessed it was probably 2cm (under 1 inch) and changed it to that.
They didn’t seem too surprised to hear that I also noticed amounts like 400cm to 500cm of snow appearing, disappearing, appearing again, then disappearing, etc. The response was something like, “Yeah, the quality was so bad at that site – we almost shut it down.”
I encouraged them to keep the sites running since the sites were essential to my research. I’d much rather deal with bad data than no data.
Wow, outstanding work Anthony! This shows how important raw climate data is for the proper scientific method to work. You just showed us how a few hours of hard work from you uncovered a possible big flaw in the way climate data is recorded, and how important it is for this data to be corrected. Once the data is verified, then it should be compared to the theory, and if it is in conflict, then the theory gets modified not the valid data. Climategate showed how the correction clause was not achieved as the valid data was truncated to “hide the decline”. We need more eyes to find these temperature errors, and more open minded climate scientists and politicians to accept a modified global warming theory to better match the measured data!
Leif Svalgaard (19:15:23) :
Dr. Leif:
Thank you for that. I was immediately reminded of the work of Benjamin Lee Whorf, an “amateur” anthropological linguistics enthusiast whose day job was Fire Safety Engineer for the Hartford Insurance Company. He noted that an awful lot of industrial fires started in “empty rooms”…. specifically, the “empty drum room”. What better place to sneak a smoke than in the “empty drum room”…. no danger there, right?
Gotta wonder how many foreign visitors who thought they knew Russian froze to death when they dressed for unexpectedly balmy weather.
While you have found a number of sites that have this error, I think there may be a slight exaggeration in how easy it is to do this. Does it create a significant jump in temps, yes. But that is why there should be some quality control of the data at the site. Every day, many of these places send 24-48 observations an hour (depending on hourly versus every half hour). And there is a huge number of these stations all over the world. It is kind of like saying “An Airplane crashed so flying isnt safe.”
As someone who deals with METAR day in and day out, I can say these errors arent quite as rampant as this leads to believe.
Also, when CORs (corrections to a wrong observation) are sent, it effectively overwrites the incorrect observation in the databases.
And lastly, could this be more of a problem of relying too much on computers as opposed to human interaction? In the old code, a space could just as easily be placed in the temp, or the minus sign could be forgotten. Human observers would catch and fix this, computers would not. And thus we would be in the same position we are now.
“The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.”
How do you download data on the day before it happens?
What would it take to reinstate the old system, or edit METAR to minimize error-prone transcriptions plus render the reporting exercise intelligible without abstruse bureaucratic nomenclature?
Really shouldn’t be that complicated to produce a halfway decent reprise of basic weather conditions, transcribed and reported in a format no more complex than various financial listings. Try omitting minus signs in trading data, and watch regulatory authorities go stark bonkers.
Great job, you covered nearly all the nuances I’d be likely to think of.
This is sort of the ideal error – it’s more likely to make a temperature read too high, it’s more likely to affect recent temperatures than past ones, and being a warming error is less likely to attract the attention of people expecting to see warming.
If I had time, it would be fun to play with software to look for the problem in the records, even your +/- 5 error in temperate latitudes could be identified by looking at the time, sky cover, wind direction change, and even station history.
@JDN
It can only lead to warmer temperature for the simple reason that it’s much much much likelier to forget/mistype an M than to add one where there isn’t supposed to be one !
Nice work. The motto is “Garbage in Gospel out”.
Excellent post. I’m sure (in fact, quite sure) that there will be many folks looking over your investigation very closely in the next few days…and of course, the bottom line of all this will be…
What difference or effect will this all make in the actual global temperature anomalies?
This was a great piece of work, and seriously you should at the very least be getting a big fat pay check from several agencies for doing such a thorough job of independent quality control work…
REPLY: Thank you sincerely. There’s always the tip jar 😉 – Anthony
It would not be difficult to program a comparison of satellite temps vs surface sensor temps. When they are sufficiently different for a particular location/time, it can be flagged by the program. Probably it would be possible to run historic data through this comparison program to find past mistakes.
Given the mess that is surface data, the papers based on this data should be rewritten based on satellite temps.
Keith Minto (19:25:30) :
Careful – that was a bit of a rant. (And an opportunity to say “Kelvins” instead of “degrees Kelvin” 🙂 .) For data in something like a METAR, expecting one of us Fahrenheit drooling Americans to accurately report in Kelvins would likely lead to all sorts of errors if any transcription is involved. Worse, it wouldn’t be immediately apparent, but it might have a better chance of being caught eventually, as the errors could be quite impressive.
Impressive. You should write a book…
@Sam
The map was online between 4/13 and 4/15, showing March average using data available on March 12th.
@Antony
If only high and low temperatures are kept for the day, then averaged to get the daily temperature, then averaged with the 30 other days of the month to get the monthly temperature, the impact can be quite big in cold regions.
Assume average temperature is -15 all month.
Assume just one mistake per month.
You get a monthly average of -14.5 instead of -15.
The thing is, to get the magnitude needed to reproduce Finland’s error (which is near zero therefore with less impact each type you flip the sign), you basically need to have the wrong side almost all month.
It would be interesting to check if they have a new employee feeding in the data…or maybe an intern… someone who wrote in minus signs all the time instead of M…. who knows ?
This is an artifact of how I/O is normally done in Fortran. Many scientists program in Fortran because that is how they have always done it. Older versions of Fortran require fixed positions in data files, which allows for fast I/O but makes parsing prone to errors like Anthony has uncovered.
But who cares how fast file I/O occurs? The science world is locked into a poor programming paradigm which affects everyone’s productivity and accuracy.
Wow. Who knew? And if I read your article right, since the records are discarded, you can’t go back year by year to correct them?
Reminded me of all the data corruption errors we used to get in the 80’s and 90’s converting data between different cpu architectures because people didn’t understand byte ordering differences (big endian processors vs little endian processors). Now I am certain when the climate centres converted their computer systems over time they would have caught that one. Certainly someone would have noticed 45 being changed to 54? Right? They caught that, right?
REPLY: I don’t know if NOAA archives all original METAR reports. If they don’t, they are in a pickle then. Maybe the FAA does. I don’t know that these things have been caught, because climate folks deal with monthly data, not hourly or daily. So if the data is automatically averaged as it comes in, chances are they’ll look at a month’s worth of data from a station and say “seems warm there”. If you are expecting it, like with polar amplification, then it may go unnoticed.
GISS missed an apparently identical error in Finland, and that probably would not have been caught if it were not a country that had a separate met service with a publication saying the country had been below normal in March 2009- Anthony
Neel Rodgers (19:34:22) :
While you have found a number of sites that have this error, I think there may be a slight exaggeration in how easy it is to do this. Does it create a significant jump in temps, yes. But that is why there should be some quality control of the data at the site. Every day, many of these places send 24-48 observations an hour (depending on hourly versus every half hour).
And yet, the errors persist…
Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
===
If people can’t understand what Anthony has written here in context then they aren’t going to understand anything anyway. This is quite interesting and I look forward to the real data crunching to find out what effect this is actually having. What I do love is how you make the extrapolation that glaciers are melting in your tongue in cheek manner after your disdain to the ‘accusations and extrapolation first’ of Anthony. Do we get to see your number crunching ever? I believe not.
We have however, seen the Arctic reach very close to normal ice levels with old ice being the major contributor. As far as deviation goes, it certainly deviated in the right direction, towards a large upswing recovering back to ~2003 levels. If there is anything you would like to actually contribute in the way of information about glaciers melting we’ll all be here.. waiting.
Anthony, this is both fascinating and deeply disturbing. As you say, there needs to be some serious quality control done before any other analysis can be done with such a dataset, much less conclusions drawn. But the saving grace is that at least we have billions budgeted for climate research that can be used to fix these sorts of egregious and pervasive errors, right? We need to know urgently if the high temperatures consistently reported in places like SIberia are correct, or if they are significantly influenced by these mistakes.
I think there is also an additional problem. The USA is using Fahrenheit while most countries are using Celcius.
A number of years ago the reports were quality controlled by humans. When they retired they were not replaced. Now we see what the computers are doing.
Judith Curry, in comments at Bishop Hill and at Pielke Fils, is saying that the whole temperature and paleo records need to be reworked. I think she’s tired of the foolishness, and I think there are a lot of other people who are tired, too.
==================================
Lord Kelvin smiles.
Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. Better pass this information along to the world’s glaciers. How’s the US surface stations project coming along?
Bravo! Spoken like a true Post Normal Scientist, Nick. You need to ask for a raise.
Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience. I was worried from the beginning of my interest in so-called AGW about the way data were being handled and discussed. I have been a daily (if not hourly) visitor to your site for about two and a half years. I feel in my gut that you may have found a major source of the confusion which arises when one tries to follow the arguments for increasing temperatures. This may be true not only for the how supposed global “rise” in temperature is occurring, but also for how the arguments have never reached firm conclusions, even among the AWG supporters. If the data are subject to (possibly) random influences which have a major effect, then the foundation will never be solid and one can never feel truly grounded in the conclusions. As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry. I am eager to read the responses to this post. Thanks for this information.
Leaves little doubt about starting over.
Now we know how bad it can get when you have grid points depending on a single station, and that station has METAR data errors.
I pretty much have an idea of how a single day can mess up a month, as I do as much station graphs from daily data.
Once the Finland error was corrected, what was the effect on the average monthly temp for Finland?
REPLY: I don’t know if NOAA archives all original METAR reports. If they don’t, they are in a pickle then. Maybe the FAA does. I don’t know that these things have been caught, because climate folks deal with monthly data, not hourly or daily.>>
If that error crept in undetected for so long I would have to believe there are others, and if the original records no longer exist, then there is no way to know. I was being a bit fascetious on the big endian little endian thing, but I would have to believe they caught that one, the total of the errors introduced would be so enormous it would be unlikely to go undetected. I think. I hope.
But there are other issues and given the amount of lost data, undocumented code, even their backup systems don’t sound like they were properly maintained. Remember when computer viruses first started appearing? Most of them were instantly noticeable so people had to deal with them. There was one very nasty little monster named “Ripper”. It would sit on your computer and do mostly nothing. Every 1024th write to the hard drive it would swap two bytes, and then do nothing again. Word documents would suddenly have spelling errors. Spread sheet formulas would develop errors. But table data? No one would notice for a long long time. I cleaned up a mess at a research facility once where the researcher had 10 years of clinical trial data corrupted, and no way to recover it because there was no backup system and the paper records had been trashed as the data was entered.
Given the gaping hole you just exposed, and the clear lack of professionnal standards when it came to not just writing and documenting code, but managing the integrity of the data itself, I can’t but help thinking that it is all junk and we need to start with the paper records themselves. again.
Terrific post, Anthony. Every new wrinkle we hear about just makes it more and more obvious that temperature data is suspect.
Terry Jackson (19:54:00) :
The motto is “Garbage in Gospel out”.
QOTW?
Hi,
Here is global warming for you. I think the ice sheets will melt soon. This is for real.
Vostok, Antarctica
http://www.wunderground.com/cgi-bin/findweather/getForecast?query=-78.44999695,106.87000275
Tuesday Night
Overcast. Low: -103 °F . Wind West 17 mph . Windchill: -148 °F .
Wednesday
Overcast. High: -81 °F . Wind West 17 mph . Windchill: -139 °F .
Wednesday Night
Overcast. Low: -110 °F . Wind SW 17 mph . Windchill: -155 °F .
Thursday
Partly Cloudy. High: -77 °F . Wind WSW 20 mph . Windchill: -158 °F .
Thursday Night
Overcast. Low: -104 °F . Wind SSW 6 mph . Windchill: -133 °F .
There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Switching from requiring a minus sign to requiring an “M” in front of the value to indicate it is negative could cause a one-time step up in a trend, if people are more likely to forget an M than a minus sign. But after that it should not affect the trend, unless again there was an increasing tendency to forget to use the M.
Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value.
Sam Hall (19:35:58) :
How do you download data on the day before it happens?
Use climate models.
This seems like a fertile area for research, for those of us less technologically inclined. Just download a station’s data, plot it somehow, and look for spikes.
I have seen this effect myself. Some years ago when I was in the AF we were flying through the Med in Jan or Feb and we asked the weather briefer to pull up the METAR for home–Grand Forks, ND. He read us the observation and when we asked how cold it was he said that “it looks like the mean temperature is 20 degrees.” We had to tell him that the letter M meant a negative temp–part of daily life in ND, not so much in Crete.
Good piece of detective work Mr. Watts.
Has anybody noticed that in mid 1966 the US started rounding negative temperatures hotter? From the Wikipedia article on rounding:
PS – Anthony: $50 on its way to you via the tip jar – wish it could be more. Hope it’s a useful contribution to your upcoming trip.
An outstanding piece of scientific forensic investigation Anthony. If the archived reports are available, it wont be difficult to develop some code to correct the record.
Looking at the March GISS its obvious NASA doesnt pay any attention to the material its producing. The Finland data is obviously wrong.
More importantly, this post just highlights the appalling state of our temperature record. Our policy makers would do well to take note.
I agree with KIrk.The deeper we get to look the sadder this science seems.Is it because these people are all govt employees?I mean the errors are so pathetic its like they just do not care, no repercussions for poor work no consequences for mistakes.Lets save billions, their work can not be trusted? Stop funding.
Great work.
Reported thermometer readings matter. Instrument bias and uncertainty matter. Missing minus signs matter. They matter because we are attempting to measure a temperature rise allegedly caused by man-made CO2 and said temperature rise is conjectured to threaten to destroy the world. In this debate, thermometer readings matter.
The conjecture that world-wide AGW is from man-made CO2 is weak due to a paucity of observable evidence. Melting ices caps? What melting ice caps? Rising sea levels? Really? Disappearing glaciers? So what? They cover only a fraction of Earth’s surface and though they shrink and grow, are hardly disappering (much to the embarassment to IPCC). Ifs, coulds, maybes. On and on.
Isn’t this all about (global) catastrophic warming? If it’s really getting catastrophically hotter here; show me the thermometer readings. They will show distinct unmistakable catastrophic heating if the readings are any good and if it is really getting catastrophicall hotter, the thermometers will show it’s getting catastrophically hotter. Period.
What we actually see in surface thermometer temperature records is inconsistency, instrument bias, human error, and this engineer has had it up to his keester with wishful thinking, madness of crowds, with frauds and charlatans showing up to the debate with a suitcase full of political baggage, advocacy, and empty rhetoric. All we need in this debate is the temperature readings.
Kindly, Anthony, you and others have shown me the thermometers. I’m sure the analysis to follow will be interesting; but to me not as interesting as this post. I’ve seen enough to know that any global catastorphic warming from man-made CO2 is either being smothered by negative feedback within the climate system itself, and/or is so minute as to be within the measurement error and of little consequence in comparason to world destruction from AGW.
What is really fascinating to me is how a weak conjecture like AGW has grown into a mass delusion driving drastic political policy change despite a complete lack of observable results/catastrophes.
I’m having the same problem, sort of, using the drifting bouys in the arctic. Missing minus signs are screwing up the daily averages. All of them- none seem to be exempt.
http://www.sailwx.info/shiptrack/shipposition.phtml?call=25626
As you can see, the temp goes from -5 to 5, and then back to -5. You can also export this page into Excell easily.
Missing minus signs- been there, doing that.
Great post Anthony. Please get it published in a peer review paper so the AGW crowd can not dispute it.
Ken Smith (18:56:17) : The answer about local effect from CO2 is no. There is plenty of evidence about very high local CO2 levels. I have personally measured over 1000ppm around a furnace exhaust stack. No one has been able to demonstrate a change to local temperatures resulting from high CO2. W Kreutz (1941) made measurements of CO2 at four heights in association with climate data over 1.5 years. The data in his paper shows that solar radiation leads local air temperature which in turn leads CO2 levels. I have noted some criticisms of Kreutz but it would appear they these critics a) did not read the paper b) can not understand (technical) German or c)deliberately ignored parts of the paper and his intent. The paper shows that he, from time to time, during the 1.5 year period made CO2 readings at 1 or 2 hr intervals and his figure 1 (24 Sep 1940 to 28 Sep 1940) shows an example of a daily measurement. In this example lower temperatures are associated with higher CO2.
However, clouds obviously have a local effect. Humidity and wind chill have a personal temperature feel effect. Convective heat transfer is an important effect for local near surface air temperatures. This is particularly noticeable near the sea. We have a thermometer in the car (under the bonnet near the front) giving outside air temperature.
We noted on Friday night around 6PM (at sunset) the temperature where we live (bush and 10 km from the coast) at 22C (slight wind from west) while the temperature at the coast opposite a well known beach was 26C (slight wind from east over the sea)
Beside the effect of wrong data inclusion shown above and UHI effects, it has been pointed out on this site that the GHCN database also includes incorrect data splicing from a number of sites now being included as airports eg data on Darwin. Most airports did not exist before 1942, yet they have spliced data. Then many airports have been substantially upgrade in the last 20 years, causing UHI effects and maybe splicing. This is shown by the Chiefo’s dT/dt analyes method http://chiefio.wordpress.com/2010/04/11/the-world-in-dtdt-graphs-of-temperature-anomalies/
Anthony, I welcome your blog and regularly look at this site when I look at the internet. Great posts from Willis, Dr Roy Spencer, and many others. Thanks for your efforts
The M in this case stands for Mother.
The Mother of all outliers. It was long read but worth the effort.
Do people even do any eyeball checks, what about data integrity analysis, those sub zero anomalies would stand out. They are so far off the curve, they must skew any analysis.
But as Anthony points out they are used primarily as aero warning stations, not climate stations. But people using this for climate modelling that is another story.
Thanks.
You wanna see missing values for air temp?
http://www.sailwx.info/shiptrack/shipposition.phtml?call=48534
I meant ‘missing minus signs’, not values. Oops.
Oops she be the mother of outliers, me mateys, aargh, memo to head remember persona.
Sera I think you just found ol matey Kevins missing ocean heat, you sluethed the buoys stole it.
You must really be on to something here for Nick to come like he did with the venom. Just who are the deniers anyway?
First thing I noticed is this would tend to produce more warming the farther north you go. How convienient is that?
Hi Capn Jack- yes, someone should tell him that his missing heat is at N 84, W 040. It is currently 14.6C. That’s pretty hot this time of year…
Another error that works in favor of the believers. The odds of this happening are remote. And it’s not just us noticing this:
Professor Watson, who served as chairman of the IPCC from 1997-2002, said: “The mistakes all appear to have gone in the direction of making it seem like climate change is more serious by overstating the impact. That is worrying. The IPCC needs to look at this trend in the errors and ask why it happened.”
http://www.timesonline.co.uk/tol/news/environment/article7026932.ece
Good work sir! As a pilot I always hated those metar reports compared with what I first learned to read. I’m glad not to have to decipher them anymore. I can see how easily people could misread them and give bad reports.
Wow. And this is very serious. As bad as the mathematics for the Mars probes which caused calamitous results . The Western world is in the hands of scientific and political dolts.
http://www.wunderground.com/history/airport/PANC/2009/12/8/DailyHistory.html
I’m guessing this is the tower at the airport, but wow, what a temperature variation, +25F in 9 minutes.
12:00 PM 15.8 °F
12:08 PM 41.0 °F
12:17 PM 15.8 °F
REPLY: Yep. you found another METAR coding error where the M was left off M05 on a METAR report issued at 12:08PM
SPECI PANC 082108Z 00000KT 3SM R07R/0600V1200FT BR FEW001 05/M12 A3028 RMK AO2 VIS N 1 BR FEW000
– Anthony
GregO 21:04:11
Good rant there. AGW is indeed an ‘Extraordinary Popular Delusion and Madness of the Crowd’. The herd has been whipped along by a few unscrupulous scientists, a few greedy speculators, and a few power-hungry politicians. Most of the rest of us have just been along for the ride.
======================
There are well funded institutions, getting all these raw data, processing them in order to “enhance their value”, then tell the world about the dangerous findings and projections, feed politicians with action plans and even get heard.
Why can’t these highflyers not do their homework first, like check the siting of their reporting stations, doublecheck the numbers they report in case of suspicious results, just be sceptic, as a pre-post-normal scientist has to be?
Instead the people doing this work voluntarily are called deniers, flat-earthers, and looked at downward of their ivory tower.
I am getting sick of all these a……..
A superb bit of detective work.
This is a fine concretized and particularized example of the data difficulties highlighted in the famed Harry_Read_Me file. Brings it into focus for the average layman.
BTW, thanks for the link to the updated work you’ve done with Joseph D’Aleo. I’m sure it’ll make for some fine reading tomorrow morning.
Bob (20:35:01) :
“Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience… As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry.”
I too in an earlier lifetime had the opportunity to deal with a similar type of problem in precise data transcription. What we found is that to assure better than 99% data integrity in data transcription the following guidelines needed to be understood and followed: 1) Limit the number of individuals making the transcriptions, make them responsible for error rates, reward accuracy. 2) Audit transcriptions in as close to real time as possible – provide immediate feedback to transcribers, make corrections as immediately as possible. (In this particular situation, to find an error it took 24 hours for it to show up, some delta T to fix it, another 24 hours for the “fix” to show up. 3) error, if left to accumulate, was logarithmic in growth and created a constant data offset.
Cementafriend,
“There is plenty of evidence about very high local CO2 levels. I have personally measured over 1000ppm around a furnace exhaust stack. No one has been able to demonstrate a change to local temperatures resulting from high CO2. W Kreutz (1941) made measurements of CO2 at four heights in association with climate data over 1.5 years. The data in his paper shows that solar radiation leads local air temperature which in turn leads CO2 levels. I have noted some criticisms of Kreutz but it would appear they these critics a) did not read the paper b) can not understand (technical) German or c)deliberately ignored parts of the paper and his intent. The paper shows that he, from time to time, during the 1.5 year period made CO2 readings at 1 or 2 hr intervals and his figure 1 (24 Sep 1940 to 28 Sep 1940) shows an example of a daily measurement. In this example lower temperatures are associated with higher CO2.”
Fascinating. I would not have thought to ask this question as I know quite well that localized CO2 can be quite high and not change local temperature almost intuitively from my experience – but I am impressed with your quantitative answer.
Excellent sleuthing Anthony!
Where is the data cleansing? Data cleansing is common in a host of disciplines. In my mundane field, forestry, we call measuring forests “mensuration” or “cruising”. Data is either hand-written in field notes and key-punched in later, or entered in a data logger in the field. At some point in the data gathering process, it is HIGHLY RECOMMENDED that some sort of data cleansing/editing software be applied. Because mistakes are common. A decent data cleansing algorithm will note that a 3 inch diameter tree cannot be 100 feet tall. Or that it is unlikely that a 30 inch diameter tree is only 10 feet tall.
Since THE FATE OF THE WORLD depends (these days) on taking the globe’s temperature, it might be handy if the GENIUS CLIMATE SCIENTISTS take the same precautions with their data that a lowly forester does.
Note to NOAA. Please clean your data. If the job is too tough, too confusing, and you don’t even know where to begin, call a forester. They can help you out.
stevengoddard (20:09:09) :
The supercomputer community cares. When I worked for a mini-supercomputer company Amdahl’s Law was frequestly quoted – “Overall system speed is governed by the slowest component.” If you spend 40% of the time reading data in, 40% writing processed data out, and 20% of the time crunching numbers, then speeding up the compute phase by 2X will reduce the runtime by 10%. Among the answers are striped file systems spread across multi-node file servers linked together with multi-GB networks.
Binary I/O in modern Fortran applications can be extremely fast. Decoding free form text input is much slower, but that has always been the case.
Besides, people can drop the M or use ‘-‘ just as easily in fixed format records as they can in free format records.
————–
davidmhoffer (20:11:46) :
Endian problems for integer data generally resulted in extremely wrong data that would be impossible to miss, and even worse for floating point data.
Your “45” value is hex “0x2d”. Byte swapped in a 32 bit integer, it would be 0x2d000000, or decimal 754974720. Hard to miss, unless Al Gore is looking at temperatures recorded underground.
This is exactly why the original data must be preserved. Warts and all.
Hmm, here’s a thought – global warming is said to impact the winter the most. Winter is when the negative temperatures occur (°C or °F). Wouldn’t it be embarrassing if a substantial portion of that impact was really due to dropped M-signs?
Excellent post, Anthony!
Jim Arndt (20:42:11) :
The coldest temperature ever recorded on earth was at Vostok:
http://en.wikipedia.org/wiki/Coldest_temperature_recorded_on_Earth
That temperature was -129F. The forecast low for Wednesday is not too far from that. And that’s mind-numbingly cold!
Ric Werme (21:55:21) :
File I/O is very slow compared to other computer operations. Any app which is dominated by file I/O, either does little computing or is very poorly written.
GCMs might do sixty seconds of file I/O, and then computations for three days. The time required to parse the input files is unimportant.
Many years ago, in the late 1960’s, I toured the NZ Defence pay section, which was changing over to computer payroll for about 13 000 personnel. They commented that the entire exercise was being done twice by two separate teams of data entry operators and separated by a day. Each teams efforts were then run in parallel through readers and entry errors found. I recall the comment was that they caught a lot of entry errors, but these were trivial compared with those found on the original
Sera (21:12:55) :
There are 856 measurements listed for the buoy you posted. Of those 856 measurements, 42 were missing, and 60 had sign errors. Correcting for the sign errors changed the average from -17.0 to -20.1, a 3 degree change.
I thought I was beyond being astonished, but this leaves me speechless. Great investigative first steps.
Anthony,
Thank you for sharing this original work in real time. An incrementally growing warm bias like a virus, has been growing inside of the data. A month of averaged data is worthless, if it can be yanked around like this, via a single error for a few hours or even days.
The implications of the impact are clear, this is a long lived and growing problem for the data. That is, as the number of *total* stations goes down in number, and at the same time, the number of airports as a % of the total is raised, this bias would continue to grow inside of the data. I am starting to wonder if GISS should drop its Southern most stations completely, and no longer report Antarctic temps.
I hate dropping any station, however when you look at their maps of world, it never has looked right to have the coldest place on the planet, glowing RED all the time.
This may have something to do with the trend we have seen in the data for less cold winters rather than warmer summers.
This is a real can of worms! The data is not fit for purpose.
It is very difficult to program round the inventiveness of human error. A lot of the METAR data can be flagged with ‘if position 4=M’ as a lot seems to be the first M is missing. Possibly if the value of the previous reading is different by an unusual amount it should be flagged [10 degrees?]. If data is flagged it may still be correct but should be checked by a human.
I hope all the source data is still available as ALL the results are suspect and need checking. If the source is not available then in scientific terms all the results are invalid and should be scrapped.
If the source data is available then, as you show, the anomalies are easy to spot on a graph. With the web it should be easy to get volunteers to visually check these graphs for anomalies, perhaps in batches of fifty, and checked by more than one person, starting from the poles and working towards the equator. A human should be able to process a normal graph in under 2 seconds.
The long term solution would be to have the thermometers showing the temperature in degrees Kelvin, and recorded in Kelvin.
It is ridiculous to have the fate of the western world riding on such an amateur shoestring operation!
Anthony Scalzi (22:30:12) :
Yep- lots of buoys, lots of errors…but the nearby stations aren’t so bad.
I am fast beginning to think this global warming issue isn’t so much a situation of a bunch of wannabe’s manipulating data for nefarious purposes as simple institutionalised incompetence at work, with the data accepted at face value and, for reasons yet to be worked out, biassed to the high side.
In addition the mindset behind AGW also seem oblivious to the principles underlying the scientific method – that when a hypothesis is falsified by the data, then it’s the hypothesis which has to be relooked at. But do Trenberth and others do? Postulate that some hidden, unknown factor is operating in the climate cooling things down.
Absolutely no thought seems to be directed to the possibility that the AGW hypothesis might be flawed, or just plain wrong. No, it’s accepted as being true not from experiment or data but by peer consensus.
No wonder countering science with science doesn’t work – the AGW isn’t a scientific hypothesis in the first place and out mistake was assuming it was.
I have a complaint about climate scientists’ use of the word “anomaly.” The word has a very clear and distinct meaning, despite all the efforts of Thomas Kuhn. An anomaly is a fact of experience that does not fit with the accepted world view and stands as a challenge to it. So, to say that temperatue measurements are “anomalies” is bizarre. It is tantamount to saying that they are revolutionary. That is nonsense. Could not climate scientists call them by another name? Why not call them temperature measurements? Isn’t that what they are? Whooh! Climate science is wierd, even among the critical. Who parented these people?
The ONLY data you can trust is RSS and UAH and yes it shows considerable warming BTW for the first 3 months of 2010. The rest is absolute trash. The warmistas should support it hahahaha
If a station reports hourly and only makes 1 drop a minus mistake that day and the daily average is calculated by averaging high low then it almost certain that that mistake will be the high. In otherwords using high low averaging selects the bad data to use from the available data. So even low error rates could have siginificant impact. If there is a 1% error rate, that would be about 1 error every 4 days and that erroneous high reading would be the one used on those days. Suppose a station had a daily low of -20 and a daily high of -10 for 4 days, The average tempertature for the 4 days is -15. Now suppose one of the temps in the 4 days had a drop minus error. On average that would change a -15 to a 15. That would be that days high. So that day would average (-10 + 15)/2 = 2.5 Averaging the 3 -15 ‘s and the 2.5 gives -10.6. That is 4.4 degrees above that actual average of -15. Quite a difference.
Wren, You state that this should not effect the trend. However, Anthony showed that the percent of airport stations being used is increasing. If there are a greater number of warm errors in the airport data and the percent of airports used is increasing then the trend will be effected.
By the way, Anthony, do you know if the data is gathered from METAR reports or is climate data sent in some other format. (Which if it is transcribed from the METAR might still contain the errors but also might be cleaned up a bit)
What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO
??
Truly outstanding detective work here Anthony!
I learned a lot about a lot!
So, is what you are basically saying is that the whole surface data set is so riddled with errors that it is mush even before they deliberately tweak it?
Have you found any examples where the M was erroneously added to suggest a colder temperature? If not, it gets more convenient.
Wonderful investigation Anthony.
Where are all of the errors to the negative side.
I’m really kidding because I am sure that if there are any at all then they are few and far between. That is the way all other “errors” in climate science being compiled are, positively skewed, always positive, always positive, always positive. Feed that into statistics and give me the verdict!
Anthony,
This is an excellent post. I look forward to your further analysis of the data. There are already many good reasons why temperature data from airports is problematic for climate analysis. This added human error could have been causing a significant warm bias in more polar surface station data for many years.
Wow! Great work Anthony!
Maybe the AGW crowd is just not as observant as they should be.
The AGW group seems to be attempting to prove GW exists based upon miniscule temperature increases – where you have show the errors alone are orders of magnitude greater.
So how do we really know what the truth is?
It comes back to data being fit the purpose. Aviation data is great for flight planning, but obviously not suitable to climate investigations. Its time to scrap land based thermometers for global climate and instead rely on satellite data. Land based thermomometers are still ideal for tracking local temperatures as no homogenisation is needed and errors like that pesky missing “M” are easily picked up.
GregO (21:04:11) wrote: “What is really fascinating to me is how a weak conjecture like AGW has grown into a mass delusion driving drastic political policy change despite a complete lack of observable results/catastrophes.”
Just turn it around. AGW was just the chosen tool for “drastic political policy change.”
Its the ideal One World Watermelon/Wall Streeter trick. That has always been clear but now it is becoming screamingly obvious.
Just look at the media or the whitewashes, or follow the potential money or the power. This Big Lie is going to be like some kind of zombie which will take a lot more than rational scientific logic to kill off. If it was that simple it would already be buried. There’s too much riding on this AGW pony for them to just put it to pasture. But with information like what Anthony just came up with, and so much, much more lately, the tide is hopefully turning. It is getting so basic and obvious now that common sense takes over. I hope.
So this is the second kind of reported systematic error. The first was the fact that sometimes a month’s mean temperature is duplicated from the previous month. This isn’t a problem in the middle of summer or middle of winter when average temperatures between two months might be the same (July and August or January and February). The problem was Steve M. discovered it happening in the fall so a month got reported much warmer than it should have.
Now we have this sign error on the temperature. While that doesn’t make much difference when temperatures are near zero, they make a huge difference in places where temperatures are far from zero.
I would love to see the record straightened out and see what the global service charts look like after correction.
Ric Werme (19:58:33) : For data in something like a METAR, expecting one of us Fahrenheit drooling Americans to accurately report in Kelvins would likely lead to all sorts of errors if any transcription is involved. Worse, it wouldn’t be immediately apparent, but it might have a better chance of being caught eventually, as the errors could be quite impressive.
Of course, the rational answer is for America to report in degrees Rankine…
Anthony, the solution to the problem is clear. You and the other climate auditors must not be allowed to look at raw data, in any form at any time, from now on.
To paraphrase a famous climate “scientist” – All you want to do is find something wrong with it.
Again it just shows that the AGW peers aren’t very good….
OMG, feeling like I did that fateful November evening when the Climate Research Unit mails were liberated.
As Kate Bush would say “I feel like I’m floating in Gaffa”
THAT, Anthony, was a tour de force!
Now I wonder: In the name of eliminating the ‘M’ errors, why does not the WMO adopt the Kelvin temperature scale?
That way there would be no signed error to deal with.
Now what bothers me otherwise is that there seems to be no computer program which rings bells and flashes lights when there are such anomalies (extremes) exhibited in the data sets.
You’d THINK that such a large variation would get ‘somebody’s’ attention at the time …
re wmsc (21:40.45)
“http://www.wunderground.com/history/airport/PANC/2009/12/8/DailyHistory.html
I’m guessing this is the tower at the airport, but wow, what a temperature variation, +25F in 9 minutes.
12:00 PM 15.8 °F
12:08 PM 41.0 °F
12:17 PM 15.8 °F
REPLY: Yep. you found another METAR coding error where the M was left off M05 on a METAR report issued at 12:08PM
SPECI PANC 082108Z 00000KT 3SM R07R/0600V1200FT BR FEW001 05/M12 A3028 RMK AO2 VIS N 1 BR FEW000
– Anthony”
There’s more to this one than just a missing “M”. As all METAR temps are are coded in °C, the 41²F reading is certainly from an entry of 05, but the preceding and following temps of 15.8°F would have to have been entered as M09 to equate to 15.8°F. So there was more to it than the simple dropping of the letter “M”.
Errors were more frequent where observers would enter the data manually on a telautograph device which transmitted hand-written information which because of poor penmanship could easily be misinterpreted when it was read on the receiving end to be manually transcribed to the daily weather observation (WABAN) file.
I’ve seen many errors of the type noted by Anthony from the METARS still extant on supposedly QC checked data from NCDC in their monthly and historic summaries. The main relevance to climate distortion is that the
METAR errors with a missing M, and earlier SA records with a missing – sign, could only result in the reported temperatures being higher than the observed temps.
As to rounding up. Many daily mean temps at sites that do not take hourly readings are compiled from simple whole number max/min temps. (See the daily C6 local climate data sheets). Currently these averages ([Max+Min]/2)are rounded up to the nearest whole number to provide a daily whole number mean. If one asswumes that in a 30-day month, on average half of the Max+Min sums would be an odd number, and that the mean of these numbers would necessarily be rounded up half a degree, you would have 15 days with the reported mean temperature being half a degree too high which would raise the monthly sum of the means by 7.5 degrees finally resulting in a monthly mean 0.25 degrees too high. For this reason, when the full month’s data is recomputed, the algorithm sums all of the daily maxes and mins and for a 30-day month divides this sum by 60.
In earlier times, even the small rounding up errors were compensated for by
rounding up fractional numbers when the unit digit was odd, and rounding down when the unit digit was even.
Finally, in radiosonde data for upper air temperature observations, in order to maintain 5-digit coding groups and still be able to report temps in 0.2°C increments, a 3-digit coding is used for tempertures; When the temperatures are positive, the third digit (the decimal place) is an even number digit and when negative the decimal is given an odd number digit. (Dew point readings use a totally different code using 2 digits which represent dew point depression.) The only reason that the coded message is not completely chaotic is because human error is elimimated by having the encoding done automatically by the RAOB ground equipment.
Anthony: This is great work, many good comments too. No magic solution to the data entry error problem just lots of had work and constant checking. We have all probably have a raft of horror stories. That is why I never allow anything out of my office without someone checking my data, calculation and conclusions. Thanks for your efforts and all the good suggestions from everyone.
One of the best posts I’ve read in a while, many thanks for your tireless efforts Anthony. It illustrates very well why human error, where mathematics are concerned, will always favour warmer temps, rather than cooler. And how convenient.
Extremely interesting, and I wonder how far the waves from this will go, thankyou from Denmark!
(It must have taken a looooong time to prepare!)
That kind of thing makes my teeth itch. I recently started a custom software company. I take it as a point of pride that I make it impossible for operators to enter spurious data. In addition to that, I also take it as a point of pride that operators can also enter data in a variety of manners based on their personal preferences and however they enter data will be correctly interpreted. A simple thing as interpreting a “-” or an “M” is a joke. Who ever developed this system probably had a team of PHD’s and grants for eleventy billion dollars. Right now its just me, and I can do way better than that. That has been my experience with scientists writing software though. My day job is in automation, and from time to time we do projects that were initiated by our R&D department. For some reason PHD’s always assume everybody using their software will also be PHD’s. Attention to seemingly small details like that make your software usable or worthless. Keep the scientists away from coding, thats what engineers are for, we at least understand the target audience.
I wish someone could find a way to do something similar with the shift from manual to automatic measurements. I know for a fact that some station attendants simply used to make up measurements and these “estimated” (fraudulent) readings may well have been drastically overestimated. Then you’ve got water ingress (dripping from clothes) which could have reduced temperature.
Then you’ve got the timing issue which in hotter countries may have encouraged measurements to be made earlier in the morning and later at night. I.e. toward the colder periods.
“Of course, the rational answer is for America to report in degrees Rankine…”
Bite your tongue. Hard.
Kelvin!
It will be interesting to see what the quality of data was like for Europe over the past week. With practically all of Europe grounded I can imagine the data reporting being sloppier than usual. If the data is not being used for its primary purpose (aircraft) then those entering it will not take as much care and more errors will be probably be present.
You only had to read the daily weather reports each day in the paper to see that Helsinki and nearby St Petersburg were well below freezing for most of March. Is there no common sense check by GISS or whoever?
“the rational answer is for America to report in degrees Rankine”
Shhh! They’ll hear you!
Wow!
This looks quite impressive. I’ve only had time to skim over it, but will read it carefully this evening.
A whole Saturday?
You must have some kind of distraction-proof bunker or something.
I’d need at least a week to write something this long!
Rankine? Do they still do that? Wait, let me go get my bamboo slide rule…
So much for the “hottest March on record.” What’s next, the “hottest April on record?” NOAA is ALWAYS wrong, but it always takes some unpaid and viciously-ridiculed “denier” to find the mistakes the so called experts continuously miss.
The AGW crowd lives and breathes false, or falsified, data, but claims skeptics are dishonest. More projection. Fortunately, the public is wising up. We know when it’s cold outside, no matter what NOAA claims with its doctored satellite maps. And older people can well remember much warmer Marches.
Can “climate scientists” do ANYTHING to change the climate? Can following any of their recommendations, or all of them combined, do ANYTHING to change the climate, or to stop climate from changing? If the entire world abandoned fossil fuels tomorrow, shut down every coal-powered electric plant, and turned off every electric appliance and light, would it make the slightest difference to the climate? If not, then all their huffing and puffing is for naught; they are wasting their lives on the acquisition of useless, and obviously largely erroneous, information. Meteorologists save lives by warning of actual coming storms. Who have “climate scientists” ever benefited other than themselves and investors?
Take away all the grant money, and climate scientists will become extinct within a single generation. They are nothing more or less than a reinvention of the alchemists of old, operating on the same inductive reasoning and consensus thinking.
Great job, Anthony! I think that you should buy yourself a deerstalker cap. You know, the kind that Sherlock Holmes was fond of. And posting a photo of yourself wearing the new chapeau would be a nice touch. Indubitably!
Great work. I’m about to read this through again (now that I’ve finished my coffee).
KevinUK’s maps allow a visual ‘shock’ of the high proportion of airports:
Fully interactive map here: Climate Applications – Airports (note slow to load – keep hitting ‘no’)
Static picture (if I’ve got the link right):
here
Anthony, I noticed that in the original version you claimed that
“Around 1990, NOAA began weeding out more than three-quarters of the climate measuring stations around the world. They may have been working under the auspices of the World Meteorological Organization (WMO). It can be shown that they systematically and purposefully, country by country, removed higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”
whereas in the new version this paragraph was silently “adjusted”
“Around 1990, NOAA/NCDC’s GHCN dataset lost more than three-quarters of the climate measuring stations around the world. It can be shown that country by country, they lost stations with a bias towards higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.”
[snip ~dbs]
Thanks Anthony for showing us yet another error effecting this data-set.
Errors in data tend only to be found if they conflict with what you believe, so I suspect a scientist who believes in CAGW wouldn’t notice a problem with a high temperature error, but would quickly spot one going the other way.
The foundations of climate science really are made of straw.
Looking at the NASA NEO image, how on Earth can March 2010 have been the “hottest on record”??
so essentially what this means is that all of the METAR data used in climate research should be omitted as unreliable. It seems to me that everyone inside the new orthodoxy bubble that is the religion of AGW is so focussed on the prize, that they are unable to audit their data his any amount of scientific integrity. or maybe they are aware but it suits them to ignore. who knows? either way it stinks.
many years ago I had to check thousands of invoices (pre computer stuff) for accuracy in pricing and extensions, ie 45 widgets @ 3.49 = ? add tax sum to total, to check the accuracy of the billson a major contract.
It transpired there was a 2% error rate in all the invoices checked, and a 90% bias in favor of the biller in the errors.
The errors on one major contract added up to 10’s of thousands of dollars of over charging.
I think anthony’s analysis shows the same potential for human error.
It would be interesting to know what the error rate in transcription over a period covering some thousands of METAR reports would be.
Based on my experience I would expect M to be miscoded 2% of the time unless there was a system of double checking which would reduce it to .04%
Great Work, Keep it up!!
Does the homogenization process clear these sort of errors?
As a computer scientist this type of input error is all too common.
In a bridge design system that I wrote with a civil engineer we used graphs of the structure to ensure that this same type of error was caught before the parts of the bridge were cast in concrete. It does happen all the time during input to systems, the key is catching it as it happens or as soon as possible after before there are downstream consequences that cost a fortune to correct.
Protecting systems from bad input leading to “Garbage In – Garbage Out” is basic computer science 101.
The cultural bias of speaking of negative numbers by dropping the sign could lead to an increase of this type of error. It would be interesting to see the percentage of errors presented by country/cultural group.
The rounding error mentioned above (rounding -1.4 towards -1 instead of -2) will also create a noticeable warming bias towards warming on the order of the alleged warming signals seen. WHY ROUND THE NUMBERS AT ALL? It makes NO SENSE at all!
Fantastic Anthony – we’ve all pondered over that hot spot in Russia for years without receiving any fallible explanation. Yours sure is.
If you get the chance could you look at January 2007 – it’s a classic
http://www.climate4you.com/images/EQUATOR%202007%2001%20vs%201998-2006.gif
It will be truly scary if persistent human error turns out to be the principal reason for AGW in recent times. The alarmist whitewash that may be about to happen will be a wonder to behold.
Anthony, if your human error theory proves correct, then I doubt if Al Gore will return his Nobel Prize or that they will give you one.
I suppose one quick way in which it could be verified is by looking at the relative ‘global warming’ measured in land stations in recent times.
Theoretically, it should be most in polar regions, less in temperate regions and almost none in the tropics – I seem to remember reading somewhere that this is exactly what is happening – can anyone comment?
Anthony Watts:
“I’ve found many examples easily in my spare time.”
You have spare time?
Best wishes.
A nice story about METARs, however, they are not used by GISS nor CRU for their calculation of the monthly global temperature anomaly. All the NMIs put the monthly average temperatures of their synoptic stations on the GTS in CLIMAT bulletins, as clearly indicated in the documentation of the GHCN. The wrong temperatures in Finland were not the fault of GISS but the Finnish NMI. They forgot to put the sign indicator in their CLIMAT bulletins.
Regarding the impact of this error, GISS says: ,,The March 2010 global mean temperature was affected by about 2/100 of a degree Celsius, well below the margin of error (about 15/100 of a degree for monthly global means).”
‘Is it quality assured?’
‘….mmmmmmm…..sorry, did you say something?’
‘Is it quality assured?’
‘….mmmmmmm…… dunno, how can ya tell’.
Hmmmm Wonder if that is part of why Finland has very suppressed ‘low going’ anomalies after 1996 in this graph?
http://chiefio.files.wordpress.com/2010/03/finland_1825_hair_seg.png
It’s monthly anomalies for all the fine “hair” and the main line is the running total of the annual average of those monthly anomalies.
Great work and it shows how poor the data gathering is for this serious science subject.
As with the CRU if you are generous (as the investigating committee was) it appears that the intentions of the scientists are good but they are just a bit too disorganised and should look at tightening up their procedures and data gathering.
Alternatively you could surmise that there is no interest in finding fault with or qualifying the data as long as it meets the target figures.
This once again shows that in very simple terms the whole Hockey team theories (along with those of warmists worldwide) are based on very poor quality data and questionable methods.
As for Nick’s comments “….accusations and extrapolation first,then ’some data analysis’ later …”- apart form deliberately missing the point, the so called accusations are valid since the people responsible for the data manipulation admitted their error (in this instance), extrapolition is fine for guidance if it is based on solid data (as opposed to the manipulation for the right answer – sole domain of Mann and co) and at least the analysis is being done with qualified / quality data which is a useful hint for others if they were really interested in the science and not the economics.
Nice work Anthony, I sincerely hope that some people in power actually have the balls to take this and challenge the IPCC and their Carbon Credit profiteering buddies.
A fascinating read, presented in a very accessible way.
Instructive to see quantitative example (and in particular the effect on the buoy data, a courtesy from the “other” Anthony that saved me from having to do it)
Although there is absolutely no doubt that these “perfect” errors could almost be designed by a thriller-writer to keep the reader guessing, I’d echo the sentiment that they seem to have been good enough for their intended purpose as disposable indicators.
The problem, as ever in Climate Science, is that far too many people are far too certain about too many things and far too willing to shout about it.
The good news is that even very simple heuristics in a recent snapshot of this data should provide a decent indicator how much of an effect this is likely to have overall, and provide solid “dig here” pointers.
If I found out that one of my programmers was using techniques that did not ensure data integrity, and delivered a solution without adequate testing of all of the cases, well, his or her name would be at the top of the list for the next RIF.
“Fahrenheit drooling Americans” Uh, maybe you should see a therapist about your self image.
Sorry, by way of a supplementary question, is there any way to get to the METAR data for the most recent week / month and see / download the data in it’s raw form? Or are individual records collated with some sort of back-end process that has no public interface?
what is also disturbing about climate data is how often the very place names are misspelled.
it’s not ‘Mirnvy’ – Russian Мирный, usually transliterated as Mirniy, sometimes as Mirnij, Mirnyy or Mirnyj – but in no way there’s v in it. (the meaning of it is ‘peaceful’ or ‘peace-loving’ – several Soviet-era settlements carry this name)
Wren (20:46:49) : Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value
Anthony’s post notes one of the issues as follows “conflicting missions (aviation safety -vs-climate)”.
METAR is designed and operated by the aviation industry, with no other use in mind. Without doubt, climatologists will be free to use publicly available data if they want, but the terms of the transaction will be “strictly as seen”.
These ownership and mission issues get things off to a poor start for objectives such as quality, accuracy and precision of climate trends.
And if I were to use the word “disgrace” it would be in the sense that GISS use this data, and then appear to have failed to identify sich a flaw. Is it not fair to consider it disgraceful that WUWT and CA appear to have become part of the QC/QA machinery of funded projects?
There should now be a detailed investigation of this issue. GISS has the most to gain and lose from this issue, so the onus falls to GISS to use some of its funding to get the matter sorted out.
For a quick indication of the magnitude of the problem, my suggestion would be examine the most recent data first. Assess the magnitude of the error at individual locatons and consider whether averaging reduces the error (averaging doesn’t necessarily reduce systematic error, and it is necessary to take into account extrapolation to neighbouring grids).
If this is combined with existing error analysis (which we assume does not account for any METAR error), the confidence interval of the GISS global data set would expand. The expansion would come into play around 1996 (introduction of METAR) and there would be continuing expansion with the increased use of airport data.
At that point Wren, GISS will have put you in a position where your could examine the lower bound of the confidence interval to see whether there is any reason to argue for a significant trend.
We are hoping this posting stays up Numero 1 for at least 48 hours! (or stories directly related to it) VIP! (Very important Posting)
Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK
Why anyone still uses the Global Historical Climatological Network when better and more accurate satelite measurements exist is beyond belief. These people are a bunch of incompetents!
A.W. You say
“In middle latitudes, you might see it (coding error) happen, but it is much more seasonal and the difference is not that great.”
True, but for that very reason, surely coding errors are more likely to slip through unnoticed. Conceivably, an operator then would be more likely to continue the habit, oblivious to the serial errors. The “difference is not that great”, but it can be significant if unchecked, especially in the context of just 0.6degC global warming in 30 years.
I am amazed.
Even unimportant factory records do basic confidence data checks when historizing process/production data. Normally we won’t acept data that is outside sane data-point to dat-point delta.
How on earth can this data be useful for anything ?
What sort of idiot/ignorant numptie doesn’t
do basic confidence checks ?
These people /claim/ to be scientists ? They demonstrate such an appalling lack of basic knowledge they shouldn’t be allowed near a keyboard, never mind a science based job.
Well done again Mr Watts,
Another problem ( which would never occur to either the French or English speakers )
In how many languages does the word for ‘minus’ start with ‘M’ ? ……..
Theo Goodwin (23:03:40) :
I have a complaint about climate scientists’ use of the word “anomaly.”
Maybe using “Absolute” and “Relative” would be better?
Outstanding work!
The more you look at this, the more you realise it’s impossible to do what they’re doing.
I cannot understand why one would base one’s life work on this. It cannot be done.
Possibly with sat data, we’ll start to get a handle on it, but as it stands now, the whole thing is rubbish.
Here is bit newer and sharper giff picture from that scandinavian error, was still having distinct error in it and almost half of europe.
Here in Finland we wheren´t sure should we laugh or go angry, because its been many degrees colder than normally in this winter and spring.
http://i756.photobucket.com/albums/xx206/MWPeriod/agw/GHCN_GISS_HR2SST_1200km_Anom03_2010.gif
Anthony, I have no idea how you have time to do all this analysis and work. I barely get the kids to bed without falling asleep reading to them. Whatever your secret, thank goodness you keep at it. It must be exhausting, but I hope the exhilaration and other benefits are enough to keep you at it. You are making a positive difference in the world.
A more enlightened NASA or NOAA or NWS would hire you as a full time gadfly. Wouldn’t that be something? The government hires Inspectors General, offices to manage Equal Opportunity, has full time ethics agencies, all sort of checks and balances folks, but no gadflies. Just someone to say, “yeah, but what if…..”. Think of the time, effort and money to be saved if someone like you were there to simply ask something like, “but what if the data is wrong?”
Excellent post Anthony. This explains so much of the GISS deviation from what is observed. Great piece of work.
Wren, if you have a data set of 150 years, of which the last 16 are biased high, please explain how this does not affect the trend!
<>
Anthony is clearly IDENTIFYING the problem here, and then talks about ANALYZING its extent and impact later. Well, duh, Nick! You can’t analyze the extent and impact of a problem until you have identified it. What scientist ever reverses this order? (“Hey guys, I’m gonna analyze the extent and impact of a problem even though I don’t know whether it exists.”)
Your scorn is clearly not scientific (it is based on an anti-scientific procedural error). So, I think I have fairly identified your problem. What remains to be analyzed is the extent and impact that your scorn has had in the development of your AGW paradigm. Your brief comment about the world’s glaciers and Anthony’s “US surface stations project” suggests that it is quite extensive.
The previous comment by Chad Woodburn first quoted Nick’s comment, but I put the quote in angle brackets, not realizing that that would eliminate them from the post.
“Nick (19:21:26) :
That’s right,Anthony,accusations and extrapolation first,then ’some data analysis’ later. “
Anthony, this is a most important subject, well explained and documented. Congratulations.
The big unknown, of course, is whether the dropping of a negative sign happens enough times to swing the country or global data, or whether some countries have QQ that detects it. The Antarctic would seem particularly at risk.
In looking at data from one of the Australian Antarctic stations, Mawson WMO 94997, there was not enough metadata available to me to address the same points that you raise. However, the first look shows in essence that the extreme temperatures are somewhat ragged at the edges. (For the record, Mawson has an airstrip. Australia changed to decimal in Feb 1966, but as a scientific base, the records were possibly in C from the start in 1954). The data are from the Australian Bureau of Meteorology, duly acknowledged.
An initial look is presented now in the form of three spreadsheet portions and a graph plus a comment on each. I stress that we might be looking at different mechanisms, but what I present does not preclude errors of the type that you describe with METAR.
Overall, the data set had 19,419 days, with Tmax and Tmin. There were 87 days of missing Tmax and 22 days of missing Tmin, reason for missing unknown.
The mean of all Tmax values is -8.38 deg C. The mean of all Tmin is -14.27. The mean difference is 5.87 deg. Condider the sign dropping exercise with these figures, first with the Tmax going from -8.38 to +8.38, causing the Tmean to change from -11.33 to -2.93. If the sign was dropped from Tmin, the two Tmean would change from -11.33 to +2.93. As this shows, the dropping of sign on the minimum is a bigger problem than dropping the sign on the maximum, as you noted.
Mawson1 is a small part of the Mawson daily maximum temperatures sorted from highest down, for the first few weeks.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson1.jpg?t=1271577750
If there was a sign error for Tmax, because its average is close to -8 Deg C, there are 2 consequences. Where Tmax is already above zero, in about 20% of the cases, we are not subjected to missing a negative since there are none. Where Tmax is below zero and the negative sign is dropped, this leads to an apparent warming that can report in 80% of the data. Tmax can be as low as -32 deg C, so dropping the sign gives a healthy wrong warming.
Mawson2 is for the minimum temperatures sorted from the highest down.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson2.jpg
This spreadsheet partially shows that only a few days had Tmin above zero. The vast majority have a negative sign and dropping the negative sign would cause a strong warming impression. It is also more probable an error than with Tmax. The average of all readings is -14.2, the coldest reading is -36 deg C with a potential for a daily Tmean error of 36 deg C.
Mawson3 shows the difference between Tmax and Tmin.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson3.jpg?t=1271577538
The largest difference is 32 degrees. If there were large mistakes made from occasional sign dropping, there would be occasional large values that did not seem to be part of a smooth distribution. However, it is plausible that such “unlikely” figures would be culled by the hand of man, making this detective work less relevant.
Then there is a daily graph showing the differences calculated in Mawson3.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/MAWSONGRAPH.jpg?t=1271577707
It is apparent that the hand of man has been at work, because one can see distinct artefacts in the graph of the differences. See the group of parallel values around day 10,000. Of course, this graph must have no points below the zero X axis because this would mean that Tmin exceeded Tmax, creating a terminological problem.
At this stage, further work is not warranted because too many assumptions need to be made. Primarily, evidence of numerical reworking exists but the cause(s) is unknown. It is noted that Tmax-Tmin was rising from about 1965 to 1980, then there is the 10,000 day gap, then the difference mainly fell between 1986 and 2000, then levelled. What this means in Nature I know not.
With further work not reported here, there are possible indications from several outliers that dropping of a negative sign could have happened, but there is no definite evidence that it did happen. The day-to day variation of temperatures at these latitudes is large and there is a lack of rigour in plausible reconstructions to explain outliers.
Just thinking….
Willis shows the deviation from “average” toward more warming occurs in the winter at night. http://wattsupwiththat.com/2010/04/16/wheres-the-climate-beef/
Anthony now show an error that can cause the deviation from “average” toward more warming when the temperature are below freezing.
The temperature are below freezing most often in the winter at night.
Human error is more apt to occur at night especially in a rotating shift type situation or when the shifts are longer than 8 hours.
A minus sign error has the most impact on data as you approach the poles. Not only does the sign switch cause a major deviation from the true value in the individual data point but the area around the poles is more likely to have sparse data causing the impact to be even greater and the incorrect data to be “smeared” to adjoining grids.
Definitely an area that needs more investigation.
Guys you have to realise that Nick, David X Benson et al are serious disciples of the RealClimate Church. They see only what their high priest tells them to see.
Sad very sad.
Those antarctic stations are not reporting in METAR format. That is called “ship synoptic” code. It has it’s own problems. The biggest being that it is done by quartermasters that have never been properly trained. I have seen them put a sling psychrometer into a bowl of water and read the temperature and dewpoint off of it from there.
You did a very good job here. I don’t think this is as big of a problem on land in the United States as this article makes it seem tho. For a couple of reasons. One is that we all talk to each other. If I am observing in Pax River and I see that Dover made a mistake I call them and tell them to send out a correction. Two is that we have a second group for temperatures. The “T” group, which you can see here. METAR KDOV 180855Z AUTO 30008KT 10SM CLR 05/01 A2991 RMK AO2 SLP130 T00520007 55002 $ This uses a 1 or 0 to denote positive and negative temperatures. Third is that when we go to insert this in the computer it checks it for us. Fourth, those online decoders do not work the same way that the NWS decoders do. Those are written in FORTRAN and pick up on some of that stuff, and always use the T group when available.
I do not mean to discount this work tho. You are pointing out a problem that is very real. Overall great job!
Thank you for taking the time and making the effort to publish this detailed analysis. Excellent; and IMO qualifies as another ”smoking gun”.
But I almost dispair at finding a practical solution to one huge problem; i.e.:
How do we ever get even a concise summary of important information like this considered by the MSM, let alone the average citizen; who has NO IDEA what is actually involved in gathering and adjusting all the data used to (at least theoretically) come up with statements like ”this was the warmest month on record in Oz”, etcetera ??
Take just two key statement from thread start:
”. . . . the chances of human error in raw surface data are at least an order of magnitude greater (if not several) than error in raw satellite data.”
and:
”. . . . there has been a wholesale migration towards airport weather data as a climatic data source.”
If even 1 out of 1000 people in North America have any idea that last two above are true let alone why and impact of same, I’d be surprised. But without at least a basic grasp of these and other key facts laid out in this post, citizens (and Members of Congress) are unable to make informed judgements on key issues; and remain vulnerable to self-appointed ”leaders” who contine to demagogue on the ”urgent need” to address CAGW regardless of cost and other priorities.
I guess as long as the WUWT hit counter continues to grow there is hope that objective science will prevail. Jury is still out, I suppose. . . .
I would agree with a previous comment that satellite data should now be the primary source of information as the other sources are compromised. This would make CRU and GISS redundant, so they could be terminated.
The NOAA/NCDC should continue to collect raw station data purely to make it available to researchers in an online database, with their remit just to make it as accurate as possible, but not to interpret it themselves. In addition they should be charged with trying to recover all historical data, and fully document any homogenisation.
Source data you can actually trust is paramount.
————
[Note : I would regard ‘raw’ data as being like that shown in the article – a variety of formats. I would regard ‘source’ data as being after the raw data has been formatted in to a standard record. If the source data needs a correction as per the article then the original data should be stored in the same record so that any correction is flagged and is reversible. I would want any adjustments [say, from homogenisation] in separate fields so the source data is always left intact, and, again, it is flagged and reversible.]
Ken Smith (18:56:17) asks-‘Is it plausible that UHI might be increased by a _local_ greenhouse effect?’ I don’t know about that Ken but I’d suggest UHI may well be an increasing culprit for another reason, if as Anthony suggests the surface temp record is increasingly reliant on airport records. He points out the METAR conundrum but that raises another issue with airports, if like my local airport in Adelaide (State capital of South Australia), advancing aircraft technology instrumentation and flight control are having another specific impact on airports like mine. How so? You might like to ask yourself if development at your local airport in recent years matches that of mine. Let me describe Adelaide Airport which you might like to check out on Google maps for yourself as a useful exercise. Basically it sits close to the coast (St Vincents Gulf) between the city and the sea. Its western boundary was always Tapleys Hill Rd and for much of my life it consisted of 2 crossed runways amid a very large grassy plain with a smallish State terminal vis a vis Melbourne and Sydney international airports. Furthermore the land between Tapleys Hill Rd and the coast was also used for playing fields, golf course, horse agistment and largely undeveloped. I can tell you as a keen motorcyclist for many years, particularly on winter nights, you braced yourself for a significantly cooler ride along Taps as you left the built up suburbs either side of this natural plain, albeit a pleasant and welcome cool change in summer. Noticeably warmer nowadays with an airport that’s been privatised, an international terminal added and everything from a Harbourtown shopping centre with large bitumen carparks to logistics warehouses and even old folks home being built ever closer to the runways. That would have been unheard of for pilots in the past on safety grounds. Clearly better navigation and flight control has been the catalyst for a shrinking safety area around the runways and with it a rapidly increasing UHI effect over the last decade or so. Presumably Adelaide Airport is not alone due to such a technology factor?
It’s amazing what objectivity does for science. Oh wait, isn’t all science supposed to be objective, unbiased, seek the truth?
Sounds like some modern software design could help alot.
Interesting. I am certainly convinced that such errors exist, and will bias the station temperature data upwards.
However, a critical question is – ‘How prevalent are these errors?’. It is possible that they have a minor effect – we cannot tell unless, at vast expense, we redo ALL the collected data from scratch. I believe we should do this, but no warmist does…
1) Make the assumption that data entered into the system is correct
2) Believe your assumption and don’t check
3) Never look back
Voila, global warming.
Regarding Wren (20:46:49) :
There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Wren, you may have demonstrated a classic case of confirmation bias. You wish to discount any possible effect on the trend, and so fail to note that this mistake is from “airport” stations, and there had been a dramatic move towards airport stations in the GISS data base; and therfore this would impact the trend, and even more so if these readings are extrapolated to dropped stations.
Regarding jim karlock (03:27:07) :
Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK
Good question and some curious posts about these buoy errors.
I hope some of the qualified posters give more detail on this.
all flights from UK airports have been grounded. No planes.
Now would be a good time to compare the temperatures of the AIRPORTS (Heathrow, Gatwick) around London, with the stations around them.
If the airport temperatures show a drop in temperature, vs. nearby sites, then we have our smoking gun.
Anthony, my compliments from Italy!
In my blog we are doing your same battle!
It’s time to stop this false people!
http://translate.googleusercontent.com/translate_c?hl=it&ie=UTF-8&sl=it&tl=en&u=http://daltonsminima.wordpress.com/2009/08/20/la-green-list-ma-verde-e-leconomia-o-i-dollari-tutti-gli-uomini-dellipcc/&prev=_t&rurl=translate.google.it&usg=ALkJrhjJrH484m2TGgxlbHtX8XkSgAL08Q
Simon
TJA (02:52:11) quoted me:
“Fahrenheit drooling Americans” Uh, maybe you should see a therapist about your self image.
Oh actually I have a pretty good self image and mental store of conversion factors. After my father retired, he spent some time as a substitute instructor at the computer and meteorology departments the newarby state college. He was quite amazed at how many students didn’t know basic stuff like 5,280 feet in a mile. I suggested that might have stemmed from the shift to metric track sporting events, e.g. the 440 yard run (1/4 mile).
I use 3 or 4 formulae for mental F <-> C conversions, including my favorite that Dad taught me. (Add 40, multiply by 9/5 if going to F, 5/9 if going to C, subtract 40.) Length stuff like 39.37 in/m, 2.54 cm/in, 1.6 km/mi, 0.6 mi/km. Volume stuff like about 8 lb/gal of H2O, about 8 gal/ft^2, 2 tablespoons per ounce (how many cooks don’t know that, probably a lot!) One of my favorites and used while lying on a housemate’s new water bed is 0.455 lb/ft^2 per vertical foot of H2O. I get to use that while looking into developing a spring uphill from our yurt on a New Hampshire mountain.
I do have very little patience for meat counter clerks when I ask for “eight tenths of a pound of ground beef.” They’re more comfortable with “three quarters”, but all their scales these days read in pounds and decimal fractions! Arrgh.
This accuracy of the data record is perhaps a useful point to be put to Professor Jones at CRU.
As he has been exonerated of all wrongdoing in a brief but concise establishment enquiry, he will be uniquely positioned as a reputable scientist to engage in the effects that such errors might impinge upon his scrupulously performed research.
Clearly, if his position is one of accepting the input data unchallenged, then he would need, in this domain, to be regarded as an extremely dexterous technician, since technicians in general do not question sources from on high and take what they are handed as gospel.
If of course his research had already quality controlled the GCHN, then we are in different territory…………….
Another rip in the tattered fabric of global temperature data! Hasn’t this amazing work by Anthony Watts, along with EM Smith and Joseph D’Aleo, gotten us to the point where the ‘consensus’ that the Earth has warmed over the past century has to be questioned?
I notice that Willis Eschenbach, even while poking more holes in that cloth in many posts, still concludes (or assumes? or concedes, for the sake of argument?) that the putative warming has taken place. But has it?
Maybe it is time to challenge the basic premise of the Warmist doctrine, that there has been any significant warming at all since the end of the Little Ice Age back in the early part of the 20th century. And if so, it should be done in a dramatic fashion, at a press conference, and before Congress, with a huge pile of reports like Anthony’s here on the dias, saying simply, “The Warmists are wrong. Their data is corrupt and unusable, and their conclusions unjustified. Period. End of story.”
/Mr Lynn
A very interesting and very informing essay.
Just to be a little picky about temperature as a throw-away bit of information for pilots. I agree with that for most pilots of light aircraft, but for operators of larger transport type aircraft temperature plays a large roll in our daily operations, even on cold days. Engine thrust values at take-off are all calculated on outside air temperatures (OAT). The warmer the OAT the less engine thrust available. The less engine thrust available the less weight we are able to carry off a given runway. In airline terms less weight = less revenue. In most flight operations nowadays flight plans are created by computers automaticly gathering data online from various agencies and then reviewed by a flight dispatcher and the captain. If a 05 temp is used instead of a m05, the amount of weight we are able to lift will be affected. This error might or might not be noticed when we look at the maximum allowable take-off weight limitation, as the OAT (at least in my companies flight plans) is not presented in the same sections. Something I am going to ensure I compare from now on.
As for the METAR corrections overwriting the incorrect data, the incorrect METAR is still shown in most weather requests as usually (here in Canada at least) the corrected METAR is only a partial report changing the incorrect data. The reader of the corrected METAR must compare the two to determine where the mistake/correction is. Do the programs that collect the METAR information collect the corrected partial METAR issued anytime through out the hour or just the COMPLETE METAR issued at the start of each hour.
It only takes one METAR reporting error during the day to make a new massive maximum daily temp. The operators can do 23 correctly and one wrong one like this influences the daily max temp massively. Very good investigative analysis.
Sattelite data is oviously a more reliable data gathering technique.
why dont thosse who create the GHCN look for obvious anomalies in the data? Surely a one off hourly jump like that is obvious and could be filtered out? It would not get the habitual operator errors as reliably but still would be an improvement.
Nice write up. Although it does far as a case of compelling trollogic rather than just spelling human errors.
Anyone that knew of these errors afore hand, for instance, can always claim ignorance of the problem. Would such a person have been as ignorance of the errors if the error had been reversed, making positive numbers into negative numbers? Now, with M(ystery), the lack of QA and error control can just be used to reinforce AGW hypothesis wether knowingly or unwittingly. Talk about positive feedback. :p
This post explains a lot. I am speechless that these types of errors are as systematic and uncorrected as they apparently are. Where are the adults?
Anthony, were there not errors the other way,ie. wrongly encoding colder temps?
REPLY: I didn’t see any, but that does not mean they are not there. However, the only way to get colder temps, given the way the METAR decoder works, is to invert or mistype the data. For example 20/15 becomes 02/15, but in that case, the DP would be higher than the temp, and one would think they at least recognize that physical impossibility. – Anthony
Three points:
1. Subscriptions, please.
2. Proposal from real scientists and investigative journalists re how to establish an accurate data base of the raw temperature readings — historical and contemporary) that remain and how to evaluate them with as much accuracy as is humanly (with computer assistance) possible.
3. Congratulations, Anthony, for an excellent overview of your research and investigation. This is at the cutting edge of “climate-global temperature” science. I look forward to further “publications”.
1. Michael Larkin (20:56:10) 4/17: “PS – Anthony: $50 on its way to you via the tip jar – wish it could be more. Hope it’s a useful contribution to your upcoming trip.”
I am sending my 2nd quarter $10/mo subscription to WUWT because Anthony provides some of the best real science and investigative reporting on any blog. I challenge regular readers to do the same — we need sites like WUWT and we need their creators to understand our approval of their entrepreneurial choices. I have taken these funds from my former subscriptions to pseudo-science publications (Nature, etc) and political rags (like the LA Times and NY Times).
I wish Anthony would create a list of regular, voluntary subscribers so the reading public could see the mass media/sci-journal transformation occuring. Even a place where the numbers of subscribers is made public would do — it might encourage others to subscribe as well.
2. Anthony, Steve(s), EM, Bob, Willis, Leif, and many others: please imagine and articulate how this mess can be cleaned up (including who should be fired) and suggest how government money for (its-our community, using our taxes) responsibility in weather/climate accounting and dangers could be spent REASONABLY. Imagine the decline in national debts when real scientists take over. I offer three quotes from commenters:
2a. kim (20:31:45) 4/17 :
Judith Curry, in comments at Bishop Hill and at Pielke Fils, is saying that the whole temperature and paleo records need to be reworked. I think she’s tired of the foolishness, and I think there are a lot of other people who are tired, too.
2b. Davidmhoffer 20:39:49 4/17
“Given the gaping hole you just exposed, and the clear lack of professionnal standards when it came to not just writing and documenting code, but managing the integrity of the data itself, I can’t but help thinking that it is all junk and we need to start with the paper records themselves. again.”
2c. JRR Canada (21:03:36) :
“I agree with KIrk.The deeper we get to look the sadder this science seems.Is it because these people are all govt employees?I mean the errors are so pathetic its like they just do not care, no repercussions for poor work no consequences for mistakes.Lets save billions, their work can not be trusted? Stop funding.”
2d. GregO (21:46:07) 4/17:
“‘Bob (20:35:01) :
‘Anthony:
Very good find!
I am a professional (bio)statistician of more than 50 years experience… As an aside, in my early career I was part of a study to determine the sources of error in data that had been computerized. We found that more than 90 % of data errors arose during transcription of data from one form to another (e.g., writing down a reading from an instrument, copying data from one paper from to another, recoding data. etc.). Surprisingly few errors were made during key punch data entry.’
“I too in an earlier lifetime had the opportunity to deal with a similar type of problem in precise data transcription. What we found is that to assure better than 99% data integrity in data transcription the following guidelines needed to be understood and followed: 1) Limit the number of individuals making the transcriptions, make them responsible for error rates, reward accuracy. 2) Audit transcriptions in as close to real time as possible – provide immediate feedback to transcribers, make corrections as immediately as possible. (In this particular situation, to find an error it took 24 hours for it to show up, some delta T to fix it, another 24 hours for the “fix” to show up. 3) error, if left to accumulate, was logarithmic in growth and created a constant data offset.”
Bernd Felsche (23:14:08) wrote :
“What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO
??”
Yes, but, of course, it is common knowledge that human concentration deteriorates at higher temperatures, so I think it is more likely that we will see this headline:
INCREASING NUMBER OF TYPOS CAUSED BY GLOBAL WARMING SAY SCIENTISTS!
(Indeed, we may just have discovered here the most important positive feedback loop for notional AGW.)
It seems like this error could be quantified. Couldn’t GIStemp be run against all non-airport stations and that result compared to their standard reports? Sounds like a job for Superman, er um, Chiefio.
Hmm,
The minus sign problem is only a problem when temps are cold and the vast majority of the global warming in the Northern hemisphere has occurred in the Winter.
Don’t have time right now to read all the comments, so apologies if I’m repeating something already said. Before my promotion to clown I was one time a programmer for an analytical lab. It was a rule to have input data validation checks and either reject or flag any apparently (obviously?) spurious data delivered to the program. You’d have to be nuts to skip that first sanity check.
C the C
I have seen this issue alot at wunderground myself.
While we may disagree about the relative merits of METAR and SA-USA (deceased), it seems very reasonable that in an international business like aviation the whole world uses one and the same system.
It makes no sense to measure windspeed in knots and/or mph, reduced air pressure (QNH) in hPa or inch Hg, temperature in °C and °F and so on.
Another factor that probably amplifies the error further is the griding process because there are less stations in these remote northern areas, thus less data is used to generate a grid value – essentially less data averaging to generate the grid value, thus any error will have any even greater effect than it would in a region with more dense station spacing.
That being said, until we know how widespread this problem may be, it is hard to quantify what the effect is in the global temp record.
A thoroughly investigated and analyzed piece of work. Now the question is, will the legislators hell bent on taxing their citizenry to oblivion see or even care there’s a warming bias built into the temperature records through persistent entry error? The political desire for AGW theory to be true far surpasses the climate scientists who made proof of this theory their life’s work. We may need to grab them by the collar and make them look.
I think I actually understood this! Good job!
Brent in Calgary
After reading your post last night, with no comments, and now all of the comments by this morning, I think that you should try to find a sleep disorder clinic to advertise on this site. Great work.
As I noted in the earlier article about the missing minus sign in Finland I was more concerned about how many times this error occured in the past before automation took pencils out of the loop and before there was such intense scrutiny of the records. The surface station records are a bad joke. Spotty coverage, error prone, lack of enforced quality controls, and never designed to track long term changes in average temperature in tenths of degrees. The only reliable record we have with the required accuracy, reliability, and global coverage begins in 1979 with satellite data. Unfortunately 30 years of data is still more a record of weather than it is of climate.
By the way, it’s been twenty years since I was behind a yoke, but I was a private pilot flying in southern California in the distant past. I seldom worried about low temperatures there but high altitude destinations were popular where one needed to be concerned about higher temperatures adversely effecting takeoff distance and climb rate in normally aspirated piston singles. That said, aren’t low temperatures a concern when it makes icing possible? A missing minus sign in that case could spell the difference between deciding to stay safely on the ground or end up crashing into it inside a flying ice cube.
Wow. Thanks to Anthony for a very clear explanation of what’s going on. The potential explanation for why the GISS anomolies are often strongly positive at the poles is particularly interesting! How amazing, that such errors have been in the system for such a long time, clearly visible to anybody who bothered to look.
Wow, now Phil Jones and Hansen & Co. have a second reason for their Preferential siting of temperature stations at airports!:
http://hockeyschtick.blogspot.com/2010/03/another-awful-email-from-phil-jones.html
Neel Rodgers (19:34:22) :
And lastly, could this be more of a problem of relying too much on computers as opposed to human interaction? In the old code, a space could just as easily be placed in the temp, or the minus sign could be forgotten. Human observers would catch and fix this, computers would not. And thus we would be in the same position we are now.
Anthony’s entire point is that human error is causing this problem. That said, if I were in charge of the technology where this data is ultimately stored, I’d write a computer program with algorithms for sniffing out the simplest anomalies…the most obvious errors could be reviewed and corrected very quickly.
I say “more computers and less humans, please” realizing that humans create computers and the programs. It all comes down to good science.
I commented in another thread about the problem with Finland and supposed it to be the result of station deletion as done by GISS and NOAA, compounded by averaging algorithms from distant stations. I still think that is a factor. And to see this article, wherein non-competent transcribers are recording data and not in a rigidly defined format is creating screwy data sets. Of course, evidently, expecting scientific rigor and quality control from NASA/GISS and NOAA is akin to expecting congress to represent the people. Oops, I think I said that out loud and used my outside voice. As for non-competent, that is not an insult. I borrow it from OSHA. OSHA defines the competent person on a job as the one trained in the rigors of the safety processes. So, a competent data collector would be trained in the proper way to take measurements and encode them in a defined format that is instantly readable and reduces errors. Apparently, we don’t have that.
Sera (21:04:16) :
Sera (21:12:55) :
Can anybody get this data?
If it as bad as Sera illustrates it should be added to Anthony’s findings, especially as it is in the Arctic.
Not a single one of the miscoded METAR stations records at Weather Underground have bled through into the GHCN Daily station data.
See post here: Watts: Dogs barking, can’t fly without umbrella
REPLY: GISS uses GHCN plus other stations that are not part of GHCN. CRU also had their own list that chose selected stations plus GHCN. If the issue is not a problem, how then do you explain this statement from GISS?
The data shown between 4/13 and 4/15 were based on data downloaded on 4/12 and included some station reports from Finland in which the minus sign may have been dropped.
My point is that since GISS has shown it to be a problem in one case, there were probably others. I demonstrated that there were. GHCN might do quality control that solves the issue, OTOH GISS admits to a quality control problem involving missing minus signs. Can you say that GISS and CRU are both unaffected by the issue? We know for certain at least GISS is.
My whole point here is that the minus signs issue is real and observable over different locations and time frames, and that since GISS admits to exactly this type of error affecting the final climate product. The fact that it does bears further investigation. – Anthony
Ben Lankamp (02:42:01) :
Regarding the impact of this error, GISS says: ,,The March 2010 global mean temperature was affected by about 2/100 of a degree Celsius, well below the margin of error (about 15/100 of a degree for monthly global means).”
But of course that was the only error that was quite so obvious and found not by GISS but the home country who new it couldn’t be true.
How many other less noticable errors are there.
To have no error checking on something supposedly so important is a disgrace, but then again using Airport data is as well.
My previous comments should read: Not a single one of the miscoded METAR stations records identified by Anthony have bled through into the GHCN Daily station data.
I am not excluding the possibility that there has been some bleed-through of miscoding errors. Just not the ones listed in the OP.
Watts: My whole point here is that the minus signs issue is real and observable over different locations and time frames, and that since GISS admits to exactly this type of error affecting the final climate product. The fact that it does bears further investigation.
So if I’m reading you right, Anthony, when you subtitled this thread “its worse than we thought”, what you intended to say was something more along the lines of “I have identified an issue in METAR encoding that might be problem in GHCN and GISS data sets, but we don’t know that yet”
REPLY: It is worse than we thought. GISS Identified one instance of minus sign error, I showed there were many more. Hence, “worse than we thought”. Since you are harping on titles, would you care to explain yours?
“Watts: Dogs barking, can’t fly without umbrella”
Perhaps it was designed just to portray me negatively and that my points about the other instances of missing minus signs have no merit, even though GISS admits to the problem and some of the station used in my post appear in GISS, even though they are not part of GHCN?
If you want to look at data, fine. If you want to tell the world you don’t like me, fine too. But please don’t mix the two and then complain that I’m the one with a negativity (or minus sign if you prefer) problem. 😉
– Anthony
Ron Broberg (08:36:28) : Not a single one of the miscoded METAR stations records identified by Anthony have bled through into the GHCN Daily station data.
So you are saying Anthony is lying then!
Let’s take for example, a station in Mirnvy, Russia. It is in Siberia at 62.5° N 113.9° E and has an airport, is part of GHCN,
MMMM — Mighty interesting!
Anthony GREAT piece of detective work. My sense is that the warmist side looks at these errors like so:
Opps, lets fix this error here and move on.
They dont bother to investigate the CAUSE of the error and see how pervasive it may or may not be.
They also tend to assume that all such human errors balance out. This one is clearly biased toward heating.
You make a strong case for automating the temperature measurements. And we have the satellite data that shows a quite different picture than the ground based observations. Why then do NASA and NOAA continue to rely on airport weather reports?
You pointed out that pilots would understand what you’re talking about. As a pilot myself I know the temperature isn’t of much interest unless density altitude is a problem. But I can attest to some errors made by FAA tower and ground controllers. For instance, being cleared to turn to right base leg only to find I’m staring at another aircraft on left base for the same runway. Human error is everywhere.
If data can be obtained by automated means then the human error in data entry is simply eliminated.
Good detective work!
Very nice detective work Anthony. Although I also have to wonder why the mistakes always increase warming.
(Sing a song, brother)
If the sun refused to shine,
I don’t mind, I don’t mind.
(Yeah)
If the mountains fell in the sea,
Let it be, it ain’t me.
Got my own world to live through
And I ain’t gonna copy you.
Now, if 6 turned up to be 9,
I don’t mind, I don’t mind.
If all the hippies cut off their hair,
I don’t care, I don’t care.
Did, ‘cos I got my own world to live through
And I ain’t gonna copy you.
This is a very astute article Anthony.
However as a guy who ran weather editing sites for years even before METAR code became common, these and other errors are not new, nor generally in my day unknown to handlers/users. Maybe some of the old AWN editors will chime in here.
First I will remark that, as far as I know, all weather reports, in whatever code, have for decades only been transmitted with day-of-month and Zulu hours and minutes. The networks passing this “operational” data therefore have few (but not zero!) problems with date/time groups. It has always been the responsibility of the archiving agency, for example NCDC, to sort and archive this data in the right year and month bins. This is NOT a new problem. Do they do it with 100% accuracy? Your guess is as good (or better) than mine.
The RULE was always that an observation MUST have a decodable location and time as a minimum.
Since we MAILED 24 hours worth of data to Asheville daily, this is likely to be more of a problem for non-US archivers. I always thought that Asheville got the dates right, but that may not be correct.
I will comment that there are observation decoders, and there are observation decoders. Some errors are not detectable/correctable without reference to the last observation or analysis, and sometimes that may not even cause it to be either rejected or corrected.
The former USAF forecast center (AFGWC) ran “validators” which, among other thing compared current observations to recent observations. This (hopefully) caught the notorious problems in Siberian reports in coding very cold temperatures. (A long story — but in Synoptic, not METAR code!) There were also problems decoding dew-points from Mexico for a long time. I remember talking to our WMO rep about this. I believe this may have been in Airways code — which you identify as SA. Airways and AERO preceded METAR, which the US fought for years before its adoption. I have even seen library problems causing confusion in what units some reports were coded.
I would suspect that very few airports consider their primary function to be climatology rather than safety.
I would also suspect that most climatologists have essentially zero experience with observations beyond the archives. As a matter of fact, it would not surprise me if most code formats were unknown to these folks.
I might also make the point that when we first started automating weather relays (in the 60’s) we found that about 30% of the world wide traffic needed manual assistance.
No mater how it is sliced, you have just shown that real-world experience is invaluable.
Thank you, thank you, thank you.
JDN (19:02:42) :
Notice how the errors as shown ALWAYS result in higher temps.
————–
That seems to be the natural consequence of (i) the unusual (in the context of normal mathematical/scientific usage) format – is someone more likely to inadvertently drop an M where there should be one, or inadvertently include an M where there shouldn’t? I know where I’d put my money – and (ii) that dropping the M still leaves a valid data format. If the rules required a P prefix for positives as well as M for negatives, then there would presumably be equal chance of accidentally dropping either prefix, and the resultant number would show up as invalid format.
A C Osborn (09:07:40) : So you are saying Anthony is lying then!
I am not saying that Anthony is lying.
I will leave it up to Anthony to decide if he wants to further clarify the bound and scope of the claims that can accurately be made given the current state of his investigation.
REPLY: So no comment then on the questions I posed to you? – Anthony
The thing that jumps out at me is how could the map showing Finland burning while all around is freezing have been published without someone (besides Anthony) saying wait a minute and looking ito it further. No QC obviously. Any computer program worth anything should flag these sort of errors for investigation.
Also how many conversion errors thoughout the temperature record are there? Didn’t NASA crash a Billion dollar probe into the surface of Mars a few years ago due to a conversion error?
How do you decode the AAXX formatted lines? I was looking at Calgary and saw a strange temp reading for Feb 21, 2010 at 5:00 PM:
4:00 PM 6.0 °C
5:00 PM -4 °C
5:00 PM 4.0 °C
The first 5:00 PM reading is based on the AAXX printout while the second is based on the METAR printout. I found this to be very confusing.
http://www.wunderground.com/history/airport/CYYC/2010/2/21/DailyHistory.html
Class act, Anthony.
“Since you are harping on titles, would you care to explain yours?”
Heads up to the Tip Jar!
Incommmminnng!!
From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?
Anthony, is it possible for you to create a pdf of this post?—there is a lot to go through and being old school I’d like to read a hard copy and keep it in my files.
Richard M (07:15:46) : It seems like this error could be quantified. Couldn’t GIStemp be run against all non-airport stations and that result compared to their standard reports?
Um, the problem these days is finding NON-airport stations in GHCN to use.
For France, there was ONE station not flagged as an airport (and even that one had a commenter assert that it really IS at an airport and the meta-data were just failing to report that…
See this comment:
Demesure
“But at least Perpignan is not an Airport… ”
@chefio,
Err… untrue. It is at an airport : http://maps.google.fr/maps?ll=42.729468,2.8703895&z=13&t=h&hl=fr
Except Mont Aigoual, there is not a single French GHCN station with some decent data (continuous temperatures for more than 2 decades) which is NOT at airport,
Here is the list of the 6 (six !) “reference” stations for Meteo France, with precise coordinates, all are at airports (except Mont Aigoual).
From this article:
http://chiefio.wordpress.com/2010/03/30/france-hide-the-decline/
In the entire Pacific Ocean basin the Islands are headed for 100% Airports. The USA in GHCN was about 92%.
The bottom line is that looking for current non-airport sites in the “Global” data is like looking for a Ham Sandwich at a Kosher Deli… Sometimes you can find one, but it’s not at the best deli, it’s over to the side away from everything else, and close inspection will often show it’s really a turkey ham…
The other problem is that it’s a bit hard to run GIStemp on “variety data” with parts left out. It tends to suffer ‘brittle failure’ as it expects exact station matches at different points in the process (it matches files line by line on station ID from several different files. Yeah, 1970’s old school style…)
Sounds like a job for Superman, er um, Chiefio.
Wow! Er, um, stammer (Blush)… Golly….
I think it might be easier to run a report of the MIN temps and the MAX temps. What you ought to see is the MIN taking an upturn after 1996 while the MAX does not. Also, if done “by month”, there ought to be more “impact” in the “shoulder months” where temps could plausibly be either plus or minus C than in the dead of winter high latitude when the guys coding the reports ought to get used to putting an M on everything.
So get some Finish or Russian MIN / MAX data and look for differential behaviour with some kind of a ‘knee’ about the onset of the change to M flags. Ought to flag an “issue” pretty quickly…
Ron Broberg (08:36:28) :
You could be doing a better service if you found why GISS used bad Finland data instead of saying Anthony’s examples are not the data points at fault.
BTW, I am intrigued that several records you found reported no high temperature for the day. That causes the obvious and other problems (like the perpetual restatement of the monthly temp for the indefinite future).
The bottom line in all this our raw data is bad in more ways that we thought and the errors make it into the final products.
This is excellent analysis and I accept Anthony’s position regarding the airport temps being temporary and disposable data and therefore not subjected to the scrutiny a scientific data point should be.
However, when that is combined with what Sera discovered in the Argo buoy data, I have to agree with:
===============
jim karlock (03:27:07) :
Why are there sign errors in hourly buoy data – aren’t they automatically transmitting data electronically?
Thanks
JK
===============
Could it be possible that there are some renegade ghosts in the machine? Knowingly dropping M’s from time to time, and knowingly switching signs on data that is surely never transcribed by a human (buoy data)?
If they knew that daily averages would become monthly averages and the initial daily/hourly data was scrapped, it would be easy to nudge the numbers up.
I’m not a paranoid conspiracy theorists, but one or two zealots ‘doing the right thing – because you *know* it’s warming’ could do real harm to the temperature record, given the obvious lack of statistical quality control.
How can buoy data be acceptable that varies by ten degrees C back and forth 42 times?
kwik (03:39:32) :
Theo Goodwin (23:03:40) :
“I have a complaint about climate scientists’ use of the word “anomaly.””
Maybe using “Absolute” and “Relative” would be better?
Actually the correct “term” should be (is)- New Average.
Logically, scientifically, statistically you CAN NOT have an “anomaly” from/between averages. And this so called monthly global temperature is an average of averages of averages of averages. Even the satellite measurements as are an average AT the point in time of acquisition that are then averaged and summed and averaged.
When once again you “take” multiple measurements, average by area, then “fill” holes with averages, then average by geo region and “correct” by averaging, then sum and divide – – all you have is a new average, and the difference between it and the old average is NOT an anomaly.
You can ONLY have an anomaly from a “normal” or “fixed” or “set” condition. There is no such thing as global “normal” temperature.
While semantically we can collectively agree to “call” that difference anything we want, i.e., “anomaly”, you are correct kwik, for a scientist or statistician or for us here, for that matter, to use that term instead of calling it what it really is, A NEW Average, is technically wrong and I believe highly inappropriate AND plays into the argument from the AGW/CO2 side, which is of course why they use that term.
By identifying it as an entity on to its self and specifically calling it an anomaly, it gains stature and therefore meaning because people instinctively (even in other languages than English) think of it as a change from normal which implicitly means there IS a normal.
To simply call it what it is – an increase/decrease to the previous average would put it in the “almost meaningless” category.
We couldn’t have that now, could we…..
Anthony,
I found an example of the reported temp for Repulse Bay in Nunavut, Canada flipping the sign as well:
4:00 PM -22.0 °C -25.0 °C
METAR CYUT 112200Z 00000KT 25SM FEW014 FEW015 M22/M25
5:00 PM 25.0 °C 25.0 °C
METAR CYUT 112300Z CCA 00000KT 25SM FEW014 FEW015 25/25
http://www.wunderground.com/history/airport/YUT/2010/2/11/DailyHistory.html
I find the data trail from Nizhnevartovsk to be interesting in a minor way. Same consistent error at roughly the same time on the two different days, an addition of 14-15 degrees Celsius to the previous values. The timing suggests someone giving the regular reporter a dinner break (7:00 p.m. local time) which would be consistent with a 3 to 11 shift. So that would suggest some form of conversion process that the breaker did not know was done by the regular person.
If the addition had been 17-18 degrees, I would say that it was likely that they were converting from a Fahrenheit thermometer to the Celsius scale, but forgetting the 32 degree offset. But 14-15 degrees C is only 25-27 degrees F, and I cannot think of anyway that that could be the conversion problem. Anyone have some ideas that could be associated with that temperature difference?
Crusty the Clown (07:23:09) : It was a rule to have input data validation checks and either reject or flag any apparently (obviously?) spurious data delivered to the program. You’d have to be nuts to skip that first sanity check.
There are a bunch of “QA Checks”, but they are not (IMHO) as well thought out as they might be. They use absolute temperature values for limit tests (instead of variable by altitude and latitude) so have differential error bands as those things change (high lats and altitudes have wider dynamic ranges, the filter doesn’t…)
from:
http://chiefio.wordpress.com/2010/04/11/qa-or-tossing-data-you-decide/
2) Climate division consistency. Departures of a station’s data from the monthly average of the data are calculated for all stations within a climatic division [see Guttman and Quayle (1996) for a description and history of the 344 climatic divisions in the contiguous United States]. The average departure for each day is then calculated. A datum is flagged for further review if the departure for a given station and day differs from the divisional average for the day by more than ±10°F. For a given day, temperature means and variances are estimated from all the divisional data that have not been flagged for further review. Any flagged data that exceed ±3 standard deviations from the mean for the day are then flagged for replacement. Replacement values are calculated from data for up to six nearby stations by the following procedure.
There are a couple of “issues” here. First off, the 10 F fixed limit. Normal LOW excursions will be further than NORMAL high excursions (convection limits highs, not lows) so more valid lows will be dropped than valid highs.
Then the dropped values are replaced by an AVERAGE. So if you drop a valid low excursion, you replace it with an average that will never be as low as a low excursion (averages are like that…).
Also, the “reference” is the accepted data… so errors that ‘make it’ can help assure more errors are ‘acceptable’ and make it too… As you add more (warmer) airports, more warm data is ‘acceptable’ and less cool data is ‘acceptable’.
There’s a lot more, but this is just a comment and I don’t want to reprise the whole article. There are some reasonable checks that would tend to catch the worst of these excursions (such as rate of change checks – but those also toss some valid data). But even those “have issues”. Not the least of which is that the data which makes it into GHCN come from the various National Met. Services already computed. Those NMS use their own procedures that may not include the QA steps used by NOAA on the US data. So this problem will have high variability by country…
The bottom line is that by the time we’ve made this Mulligan Stew of bits found by the airport all over the globe, it’s a real mess.
A C Osborn (08:22:16) :
Sera (21:04:16) :
Sera (21:12:55) :
Can anybody get this data?
If it as bad as Sera illustrates it should be added to Anthony’s findings, especially as it is in the Arctic.
Well, you can get the METAR and CLIMAT reports from Ogimet here:
English: http://www.ogimet.com/index.phtml.en
En Español: http://www.ogimet.com/
there are also other sources anyone can use on the net.
Leif Svalgaard (19:15:23) :
A cultural/social effect may also be at work. When I was the US Representative to the Soviet Union on the Protection of the Environment [included weather forecasting] back in 1976 http://www.leif.org/EOS/Nature/261546a0.pdf I noticed, both from verbal interchange and from the evening news on TV that in the middle of winter when the temperature was typically -30C, the Russians never mentioned the minus sign; they would simply state “it’s 30 today”.
Leif you have it!
Hay we live at the 45th parallel and even here we DO NOT REPEAT over and over it’s bellow freezing! When you have 20 feet of snow and it is jan-feb when some one say’s it made it to 60 at night, You know what they mean ! know one repeats over and over the same thing, like, minus, below, under, see how this sounds;
well it is a minus, 10 degrees Fahrenheit below the point of water turns to ice.
what is the temperature? 22F? 10 below a point of scale? minus 10?
most people do not like the Kalvin scale but this would salve this problem.
TX A for the post
Tim L
10 degrees Fahrenheit below (the point of water turns to ice)
here we say most often 10 below 20 below 30 below, 20 below today and 40’s tonight!
As a scientist dealing with regulatory bodies, we have been forced to use redundancy in measurement and recording of data. This was a pain, but well justified. In USA, the CFR (Compendia of Federal Regulations) dictates that we must use both redundancy and witnessing of recorded data. It was educational to see how, even with a witness, data are recorded sometimes erroneously, especially with regard to introduction of systematic error. On top of redundancy and witnessing, a QA (Quality Assurance) Officer, reporting to the CEO not to QC, is employed. The reporting line function to the top, not an employee reporting to the QC (Quality Control) Officer is mandated so that there is no collusion to make the QC function look good, or pressure the subordinate to make the data look good. This insurance is also the basis for ISO (International Standard Operation) manufacturing guidelines, so that manufactured goods can be trusted for their quality. This methodology is costly, and a pain in the rear end for scientists, but with the billions being thrown around today in climate science, this should be the minimum standard insisted upon.
Another point – apart from inadvertent error, there is pressure to tow the company (AGW) line, that may lead to funny business, as we have seen occurring with Jones and Mann et.al. Who can rule out skulduggery in the current environment? Anthony is too much the gentleman to bring this up, but I don’t suffer from the same affliction!
Another problem, the practice of interpolation and extrapolation in the climate world of data processing has me very nervous. If an erroneous peak is introduced, and Anthony has shown how easy this is, how do the final masseuses massage that into the mean? If a final arbiter at CRU, GISS, or NASA sees a high peak, does he interpolate the peaks around that high to be intermediate and then does he/she employ a “correction”? IS a QC witness or a QA officer watching? (Not!) Does the data logger interpolate if that happens; if so, then the error becomes compounded as follows: How does that result affect the neighboring station that does not have a number for whatever the reason? Does the data handler extrapolate the number to the blank station? Interpolate with a distant station? What are the SOPs for these situations? Does this affect the Arctic more, since there are dismally few stations there?
Quite a piece of work, Anthony. Very good and thanks!
GaryPearse (06:38:40) :
Anthony, were there not errors the other way,ie. wrongly encoding colder temps?
REPLY: I didn’t see any, but that does not mean they are not there. However, the only way to get colder temps, given the way the METAR decoder works, is to invert or mistype the data. For example 20/15 becomes 02/15, but in that case, the DP would be higher than the temp, and one would think they at least recognize that physical impossibility. – Anthony
======
http://www.wunderground.com/history/airport/KGYH/2009/11/18/DailyHistory.html
Doing a little bit more digging around on my own, I went looking through the records of a nearby airport, of which I know their procedures for getting data to the NWS, and I found a negative-going error (I know, shocking). This actually seems to have been a misplaced decimal on the part of the NWS when they typed it into the system:
Time Temp Dewpt
1:47 PM 57.2 °F
2:47 PM 33.8 °F 39.2 °F
3:47 PM 57.2 °F
At this facility, during those hours, the controllers are reading data from an AWOS, and calling it into the NWS. I would seriously doubt that the controllers would botch the numbers that way, and by that much.
It would seem to me, that there is no quick check going on at the NWS at their data entry stations, or by the folks entering the data and that is a travesty. As Anthony pointed out, DP isn’t going to be higher than Temp, unless something has gone horribly wrong with our understanding of physics 🙂
I wouldn’t expect, that it would slow things down too excessively, for the data entry terminals to do a comparison with the last hour’s data to see if the data was seriously wrong, and give the operator the option of passing the data through if it has indeed changed that much (ie the volcano started erupting, etc).
Dear Sir,
In case you don’t have enough volunteers already, I have the technical knowledge and experience to organize a system to pull the METAR information across the internet, check for data issues and handle them appropriately. I could also provide statistical data summaries for each file processed. (I believe that those summaries would form the basis for doing a unsupervised learning in hopes of catching more than just a missing minus sign, the “black swan issue”.)
I am a dinosaur who has made a concerted effort to keep up with improvements programming in the areas computer science, and computational science. I have worked a lot with FORTRAN, C/C++, Python, and R (in both Linux/UNIX and WIN32). My first professional job included data collection and processing at a hydrodynamics lab.
I have done literate programming and have no issues with providing the code and supporting information for this project. I have always believed that I should be prepared to be challenged on my work and how it was done.
I have been between day jobs for a year. I would like to contribute something positive. Applying statistical learning techniques to my existing knowledge of reviewing and cleaning data would be a payback for me at this point. I also understand a new subject once I can get an interesting project going.
If you would like further information on my technical abilities, or a more detailed project design; please feel free to let me know. I have not read the entire thread yet because I wanted to get my offer into the queue while it might still be relevant.
I will understand if you already have all the resources that you can handle for now (the “Mythical Man-Month”, I have also been there and done that.)
Glenn McKee
I notice that the GISS diagram has the minus signs missing in the legend.
I commented earlier about por old Tasmania just below Australia being a nonsense “hotspot” in the GISS figure. I notice there is no such hotspot but normal temps in the satellite diagram for Tasmania. That makes much more sense.
Comparing the two diagrams I’d say GISS needs to do some urgent quality control of their data. Their grid resolution is also very poor.
“Eureka, NWT is a great example, going from -43°C to +43°C (-45.4°F to 109.4°F) with one missing “M”.”
Holy Crud !
Keep up the good work. Surely some of this must be penetrating somewhere !
Whoa! Judith Curry just now on Bishop Hill: ‘I am no longer substituting the IPCC’s judgment for my own judgment in this matter’. The world in a tiny little sentence.
=============
My daughter is a Russian linguist and I asked her about Leif Svalgaard’s comments about Russian media and others leaving out the “minus” or “negative” in weather reports during the winter. She said that yes, this would be likely – that it is common to leave out things that are “duh obvious” such as saying “30 degrees” when it was “-30 degrees”. She had seen that when she lived in Russian. While she has no specific expertise in meteorology she also said that she believes many of the reporting stations very likely continue to record data manually, not automatically. Consequently, simple manual recording error might be a factor in not coding as an “M”.
Rod Smith: April 17, 2010 at 6:42 pm
From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?
~~~~~~~~~~~~~~~~~
Yes: FIGURES SHIFT (FIGS)
http://nemesis.lonestar.org/reference/telecom/codes/baudot.html
observa (04:50:57) :
” Noticeably warmer nowadays with an airport that’s been privatised, an international terminal added and everything from a Harbourtown shopping centre with large bitumen carparks to logistics warehouses and even old folks home being built ever closer to the runways. That would have been unheard of for pilots in the past on safety grounds. Clearly better navigation and flight control has been the catalyst for a shrinking safety area around the runways and with it a rapidly increasing UHI effect over the last decade or so. Presumably Adelaide Airport is not alone due to such a technology factor?”
There’s an even simpler explanation than “better navigation and flight control”. Simple greed and the knowledge that nobody will be called to account when the inevitable instant urban renewal program occurs.
What Digsby accentuated with a good laugh in his response to Felsche seems important to me:
Digsby (06:56:49):
Bernd Felsche (23:14:08) wrote :
“What are the odds of seeing the headline
GLOBAL WARMING CAUSED BY TYPO ??”
… I think it is more likely that we will see this headline:
INCREASING NUMBER OF TYPOS CAUSED BY GLOBAL WARMING SAY SCIENTISTS!
For me, this echoes Methow Ken (04:49:09):
But I almost dispair at finding a practical solution to one huge problem; i.e.:
How do we ever get even a concise summary of important information like this considered by the MSM, let alone the average citizen; who has NO IDEA what is actually involved in gathering and adjusting all the data used to (at least theoretically) come up with statements like ”this was the warmest month on record in Oz”, etcetera ?? …
If even 1 out of 1000 people in North America have any idea that last two above are true let alone why and impact of same, I’d be surprised. But without at least a basic grasp of these and other key facts laid out in this post, citizens (and Members of Congress) are unable to make informed judgements on key issues; and remain vulnerable to self-appointed ”leaders” who contine to demagogue on the ”urgent need” to address CAGW regardless of cost and other priorities.
Is there anyone out there who gets it (Anthony’s terrific work) who can use their 6 degrees (or less) of separation to get a little quality time with a national morning television show weather man (or woman, if that’s the case) to present this? (I’d try to go through John Coleman in San Diego, but people have probably already been warned by their masters not to talk to him.)
With the Democratic leadership still experiencing inner gloat from “healthcare”, I think it may be approaching *urgent* that the Cap and Tax be shot dead in its tracks.
What with all these disappearing Ms, I bet Mr McIntyre and McKitrick are feeling mighty nervous right now. And Mr Watts, you be careful – a W is only an M upside down. And you know how good these guys are at flipping things upside down.
David A (05:09:59) :
Regarding Wren (20:46:49) :
There may be a tendency for recorders to forget to place minus signs in front of numbers. I believe this is called a systematic error. It could cause an upward bias in recorded levels but should not affect trend or change in levels unless there was an increasing tendency to forget to use the minus sign.
Wren, you may have demonstrated a classic case of confirmation bias. You wish to discount any possible effect on the trend, and so fail to note that this mistake is from “airport” stations, and there had been a dramatic move towards airport stations in the GISS data base; and therfore this would impact the trend, and even more so if these readings are extrapolated to dropped stations.
=====
There is a limit to how much a systematic bias can affect a trend. For example, I can gain weight faster if, in addition to eating more, I drink more and more bottles of beer each day immediately before weighing. But eventually I won’t be able to hold any more beers to give an upward bias to my weight gain trend.
If the recorders at airports are more likely to forget minus signs( whether indicated by “-” or “M”) than recorders not at airports, a move toward airport stations as a data base could introduce an upward bias in temperature records until there was no longer a move toward airport stations.
However, I doubt recorders at airports are any more likely to forget minus signs than other recorders. I think I am demonstrating a classic case of skepticism rather than a classic case of confirmation bias.
Re :”Rod Smith (10:48:05) :
From memory – which is none too reliable these days – I seem to remember that the old baudot teletype machines were not capable of transmitting a minus sign, or for that matter a dash.
Can anyone confirm that?”
You are touching on the basis for why such a problem in coding exists today.
A little history:
Until the implementation of high-speed data networks from about the late 1960s forward, most WX data was transmitted by teletype. In the US their were three basic circuits used by the NWS for the dissemination of WX data.
Service “A” for the aviation hourly data, Service “C” for the National and regional synoptic reports, upper air (RAOB) data and NWS text information, and Service “O” for international data content similar to the domestic Service “C” data. The major constraint on these systems was the time it took for data transmission. The characters were 5 bits each. This would allow for only 31 characters except that one character was a shift character which was equivalent to shifting the keyboard into one of two cases (letters or figures).
All of the letters were in upper case while the figures case consisted of numerals, punctuation and special characters and non-printing characters for functions like CR, LF, Space, FIGS, LTRS, etc. The standard teletype speed was originally 45.5 baud or about 60 words per minute, by the 1950s it increased to 57 baud, roughly equivalent to 75 wpm per in baudot code. Service “A” machines had special weather charcters for their hourly messages which included arrows for wind direction and transmitted in the SA format which used these symbols. The shift to the METAR codingeliminated these characters and replaced these wind direction symbols using either numbers for 10s of degrees (32 = NW) or one or more combinations of the cardinal points (WNW = west-northwest). In the standard keyboard, the minus sign was in the Figures mode where the upper case “A” lay in the Letters mode. In the older SA code, tyhe minus sign was the equivalent of an open circuit signal when in the fugures mode. The differences in these two formats led to the removal of the minus sign as a useable character since its buadot code was different for each model configuration. Thus the infamous letter “M” was introduced to replace it. Today that would be unnecessary as the 8-level ASCII code allows for many more characters to be used unambiguously.
The highly abbreviated nature of the messages were essential because of the constraints of time at these low baud rates. In fact, experienced map-plotters could manually plot the data fast enough to stay ahead of the machines including the Model-15 and Model-28 page printers generally in use. So blame the genesis of the minus sign problems on the need for the highly abbreviated coding used in the hourly and synoptic data transmissionsduringb the mid 20th Century.
By the time I stopped plotting data in real time manually and moved on to using the facsimile charts and then computerized plotting, the transmission speed of NWS data had increased in stages from 100 baud, to 300, then 1200 and finally 9600 baud for digital data transmission to printers.
BTW, When I loooked at the Finnish CLIMAT transmission, they had it coded correctly, and this is what GISS should have been ingesting. What might have been misleading was that in the color coded portion of their message which was produced for public consumption, the temperature departures from normal for each station did not have the minus sign, although the accompanying color coding of the entries for those stations clearly used the dark blue indicating much below normal!
Thanks Anthony 🙂 Haven’t had time to read all the comments, so this may be redundant. I was always intrigued by Warwick Hughes pointing out some anomalous temperatures years ago. +30C in Antarctica, close to +100C in Siberia… Now we know how it probably happened. Mind you, the Russian Vodka effect theory explanation remains true in tandem with yours 🙂
Uncertainty estimates in regional and global observed
temperature changes: a new dataset from 1850
P. Brohan, J. J. Kennedy, I. Harris, S. F. B. Tett & P. D. Jones
http://hadobs.metoffice.com/crutem3/HadCRUT3_accepted.pdf
” Calculation and reporting errors can be large
(changing the sign of a number and scaling it
by a factor of 10 are both typical transcrip-
tion errors; as are reporting errors of 10◦ C (e.g.
putting 29.1 for 19.1)) but almost all such er-
rors will be found during quality control of the
data. Those errors that remain after quality
control will be small, and because they are also
uncorrelated both in time and in space their
effect on any large scale average will be negli-
gible. For these reasons ϵRC is not considered
further. ”
Dr. Jones said so. It’s clear that this paper scientifically establishes that the errors will be caught and that those that are not caught will be small and it will all average out. What more proof do you need?
When people want to claim, as Jones does, that the errors THEY DO NOT CONSIDER, are small, the burden of PROOF is on them. The burden isnt on Anthony to prove that these errors matter. The burden is on JONES. Jones made the positive claim that.
1. these errors are caught
2. those that are not caught are small
3. That they are UNCORRELATED in time.
None of this was established in Jones paper. It was claimed.
How much uncertainity will this cause in the record? dunno. how much Bias?
dunno. But the claim made in the paper seems questionable.
Psst. it wont change the science, but the record should be stated with proper error bounds
I believe I read here that most reported warming has been in the NH, winter months, night time. This would be exactly where you would expect changes from positive to negative temperatures and where the minus sign mistake would show up most. (Sorry if someone else already made this point, just too many posts to track.)
I sure hope this arctic station isnt included in GISS map: http://www.wunderground.com/global/stations/01009.html
It has been showing 20-30c for months now and it is one of the official weather stations.
This was another top-notch quality presentation, Anthony. I really love it when you’re in top form.
I love facts. I love certainty.
Unfortunately it seems many skeptics has been self defeating by adopting the specific argument ‘no real scientists calls anything a ‘fact’ as truth can never be proven.’
This is doomed to fail. You can’t argue truth if you don’t recognize its existence. Poor Willis.
When skeptics become nihilists, they Kant do just like the university narcissists who rehearsed the same thing but more.
Honest to god, people need to sort out the fundamentals of epistemology before they hit age 5.
There is a type of person who just hates the concept of truth because they can’t tolerate anybody who is certain about anything.
They can’t tell you why this motivates them so powerfully to make missions out of attacking with any available rhetorical tool.
The reason is that reason, itself, is an existential threat to them because it enables their prey to defend against them.
it’s reason, truth and certainty of those things that they hate because they require the benefits of doubt to survive – your doubt- please note well that they are not at all uncertain about the truth of their argument that YOU must not be certain and that YOU can’t know truth – and therefore they shall deliver the gospel as revealed to them…lol
You provide the antidote, Anthony. You and Mr. Macintyre and some others – too few.
Perhaps we could avoid all this stuff by just using Kevin
Wren (20:46:49) : “Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value.”
That’s how it WAS done. AW made that point:- “when a minus temperature occurs, such as -10 degrees Fahrenheit, it was reported as “-10″………In the METAR report above, instead of using the normal way we treat and write negative numbers, some policy wonk decided that we’ll use the letter “M” to report a negative number. Only a bureaucrat could think like this.”
From my repost of this on digg.com
http://digg.com/environment/GISS_dial_M_for_missing_minus_signs
This is quite an interesting article, not just because of its implications on historical temperature records from airports, but because it demonstrates quite nicely how human factors interact with accuracy and how _not_ designing reporting systems to take that into account can result in simple errors have a dramatic impact.
For instance if the processing software had just simply rejected reports using a ‘-‘ instead of a ‘M’ then the error would have been stopped, instead it looks like the ‘-‘ got ignored and the temp got processed as positive – whoever designed that should be given a very direct lesson in what ‘data format validation’ means.
I’ve also been doing some digging of my own around the airport temperature records, and I kept finding the reported winter mean monthly values were always slightly higher then my hourly averaged values across the month. This might explain it as the daily min/max sum to monthly mean gives a false positive value in winter potentially 24 times the weight than just summing to mean all the hourly values taken in a month.. I’ll see if I can recreate this effect..
Pompous:
As per Jonesy: “Milady doth protest too much, methinks”
Next milestone, for the good professor: “Alas, poor Yonesy, I knew him well, Pompous”
Very good piece, Anthony. Ties together both ends of the process, from data producer through to consumer, explaining the human interpretation/conveyance stage, the changes in required data formats thru to the final data consumer/acchiver, GISS et al.
I am surprised, with the plethora of PhDs (and no doubt a goodly number of interns hired every summer to work at GISS) that GISS (or some other data aggregator) would not have implemented some means to simply check incoming data samples against previous data samples for the purposes of a rudimentary data consistency checks.
.
.
Sorry, I meant to address Steven. I owe a Shakespearean reference to Pompous Git.
Bruckner8 (08:00:23) :
Anthony’s entire point is that human error is causing this problem. That said, if I were in charge of the technology where this data is ultimately stored, I’d write a computer program with algorithms for sniffing out the simplest anomalies…the most obvious errors could be reviewed and corrected very quickly.
I say “more computers and less humans, please” realizing that humans create computers and the programs. It all comes down to good science.
Computers are not free of errors either. The computers that are ingesting this information and dissiminating it can be just as prone to error. How software is programmed can sometimes have this effect. I have seen it happen with winds on one of my sensors in the past. A thunderstorm collapsed and gave us 72 knots of wind (based off of a hand-held sensor). The main sensor registered 217 knots. There is a significant difference there but having a person do Quality Assurance could see this error and fix it.
The example given above of consistently hitting M27 and then jumping to M13 could have been one of those software glitches. The example where it is +20C with light snow (the -SN) would also be caught, because you will not have snow at 68F.
If one can forget to put in an “M” for Minus, they can just as easily forget to put the “-” which tells me it isnt a problem with the code so much as human nature.
So how do we fix it? The number crunching software should have a Q/A ability. Compare the observations on each side of the current one to look for significant inconsistencies such as these, and then alert a person to them so they could be adjusted/fixed accordingly. Also set it to be able to recognize variations with spaces in it.
I just think the wrong thing is being villified here. Is it the code, or is it inadequate Q/A?
page printers? The surplus Model 28’s we used in the late 70’s all used continuous roll paper … maybe ‘page printer’ refers to 8 1/2″ wide paper versus paper tape …
Model 28 info page:
http://www.baudot.net/teletype/M28.htm#M28-KSR
.
.
Here is a quote from proff Judith Curry in November 2009
” Judith Curry November 25 2009
J. Gavin Schmidt deserves credit for what he is doing. If something is broken, give credit when somebody tries to fix it. Why on earth anyone is hiding their data is beyond me. I find it hard to believe that Phil Jones has anything to hide in this data set. sure a few little glitches may come to light, but if this happens and they get fixed, that would be a good thing. When the auditing was done of the GISS temp data set, a few minor glitches were found and fixed and frankly my confidence in the GISS temp data set went up several notches.”
Hmmmm I wonder how many notches her confidence will slip in view of this excellent analysis?
As an accountant I have always been taught that even minus signs can be missed so to ensure that all negatives are bracketed.
So -20 becomes (20) which becomes hard to read as a positive; even when looking through a long set of numbers.
From the Teletype corporation archives, the CAA (forerunner to FAA) “coded hourly weather reports”; special symbols are noted in the format:
http://www.rtty.com/TTYSTORY/page25.jpg
.
.
If meteorology/climatology were handled by real scientists instead of glorified weather “soothsayers” this problem would not occur because temperatures would be recorded using the Kelvin (absolute temperature) scale rather than the Celsius and Fahrenheit scales with their arbitrary zero points. The only thing you can have less than nothing of is money.
the Kelvin scale is used for scientific calculations in part because of problems with the properties of numbers 1.0 and less.
the use of humans to record numbers introduces a potential error that is greater than 0.25% which is the percentage of temperature change that supposedly occurred during the 20th Century
Anthony believes that calling this incorrect temperature measurment fraud is false, I disagree, heres why.
It could be quite true that the people who report this temperature sincerely believe it is the correct temperature, while the temperature may be a total fraud, and that sincerity also fraud. Imagine, if you will, a pre alchoholic. he tells himself “one drink will do me no harm” and he may be right. Later, he tells himself “one more drink will do me no harm”, and as the number of drinks and frequency of drinking increases, he comes up with more and more rationalizations like this (“it’s just social drinking”). Eventually he can be drinking heavily while still being sincere in his belief that he is not an alchoholic.
For the AGW crowd, it is like this, one little rationalization piling on another, but with new twists to furthur cover up from themselves the fraud. One of the strongest is that the little rationalizations that pile up on each other (like rounding all temperatures up, or dropping stations in areas that are colder) are done by many different people, thus the frauds are so spread out and diffuse that you can ignore them if you know where not to look. They can also reinforce their belief in their own correctness by peer pressures of all kinds, after all, all their buddies believe this (meanwhile they conveniently forget that any buddy that does not is blackballed). They can also do the standard propaganda ploy of not necissarily lying, but simply not telling the whole truth, thus they can tell themselves that they are not lying even when what they put out may be a geat big whopper, since they didn’t actually include any deliberate lies. Thus, if they don’t bother to check if the station data they get is correct, they aren’t really lying, just not bothering to find out the truth. They of course will also not check on each others work, after all, that would show a lack of trust in their buddies, possibly reasulting in bad feelings, we wouldn’t want that! It’s all about working together as a team!
Pulling together is the aim of despots and tyrannies. Free men pull in all kinds of directions. paraphrased from Lord Vetinari
They are also told many rationalizations by those who want to use AGW to gain power, and tell others these same ones, like “Choosing message frames for climate change that are consistent with the values of target groups is one important way to make the recommended behaviors or policies easier to accept.” …“Conservation messages, for example, can use an economic frame (This is an excellent way to save money); an energy independence frame (This is a means for our country to free itself from dependence on foreign oil); a legacy frame (This is a way to protect our children’s future); a stewardship frame (This is how I honor my moral obligation to protect the abiding wonders and mystery of life); a religious frame (This is a way to serve God by protecting His creation); or a nationalist frame (Innovative technology will keep our nation’s economy strong). By these repeating these rationalizations to each other and themselves, they can drown out that nagging little voice that tells them that they acyually stand to gain or lose hugely if AGW is true or false. As Thucydides wrote, men will accept without argument conclusions they find agreeable; but will bring all the force of logic and reason against those they do not like.
Plus, admitting that they have been wrong and are now hugely wrong and have been for many years would make them feel like what, and would cause others to think of them how? Global warming has now been so strongly supported for so many years by so many different institutions (like the press) that admitting falsity now would simply be to costly. As Upton Sinclair once said: “It is difficult to get a man to understand something when his job depends on not understanding it.”
And finally, they are supported but that largest of rationalizations, the fact that if AGW is true they become the saviors of the world. Now who wouldn’t want to be known as that? It is, in essence, “the big lie”, and thus more likely to be beieved by many than a small one. “Have you noticed, once you have succeeded in convincing a man of something incredible, he believes it with an enthusiasm that he wouldn’t dream of showing for an obvious, simple fact?” – Sir Harry Flashman. Basically, for its proponents, AGW feels good, so they keep doing it, and non AGW feels very bad, so they do anything to avoid admitting it. They can cover up that basic fact with convoluted rationaizations like “post normal science” and such so that they don’t ever have to admit it to themselves.
In short, they believe it the same way an alchoholic sincerly beleives one more drink will do him no harm.
It is also supported, or at least not opposed, by the majority now who have become so apathetic that they don’t bother to check it out for themselves, after all, you only need fool some of the people some of the time. “The opinion of ten thousand men is of no value if none of them know anything about the subject.” —Marcus Aurelius. And of course, with sheep like that, it’s just too big of a temptation not to lead them about: “The whole aim of practical politics is to keep the populace alarmed (and hence clamorous to be led to safety) by menacing it with an endless series of hobgoblins, all of them imaginary” -H.L. Menchen. In short, the AGW crowd is now acting like the alchoholic, and the populace is now like the co-dependant.
So I believe this recent “warmest on record” month IS a fraud, it’s a very carefully disguised fraud, disguised so well even the ones doing it can tell themselves they are sincere. And the AGW crowd has SO much invested in this that they cannot stop now, I expect the next moves will be to cover themselves by silencing the opposition. Someone is going to wake up in a gutter, and they don’t want it to be themselves.
Perhaps I am wrong, but I didn’t get the impression plus signs formerly were used in front of the positive numbers.
If missing negative signs(“-” or M) and other errors tend to bias the GISSTEMP global temperature trend upwards, something else must be offsetting the effect, since this trend does not diverge from the RSS and UAH trends. Looking at the three over at woodfortrees, GISSTEMP appears slightly closer to RSS and UAH recently than it was in 1978 when the latter two began.
http://www.woodfortrees.org/plot/hadcrut3vgl/last:360/offset:-0.146/mean:12/plot/uah/last:360/mean:12/plot/rss/last:360/m
Graham Dick (17:21:43) :
Wren (20:46:49) : “Perhaps the answer to addressing this systematic error is simply to require a sign, plus or minus, in front of each recorded value.”
That’s how it WAS done. AW made that point:- “when a minus temperature occurs, such as -10 degrees Fahrenheit, it was reported as “-10″………In the METAR report above, instead of using the normal way we treat and write negative numbers, some policy wonk decided that we’ll use the letter “M” to report a negative number. Only a bureaucrat could think like this.”
====
I’m sorry Graham Dick. My previous post was a reply to your comments above, but I forgot to address it to you.
Haven’t we seen this story before …
A major problem with using airport reporting stations in snow prone areas is that airports have to clear runways so planes can take off and land without sliding all over the place. Snow covered areas reflect sunlight back into space. Pavement converts sunlight into heat. snow piles on cleared pavement will melt from the heated pavement even when the air temperature is below freezing and the water will remain liquid until the sun sets. Cleared pavement transfers some of its heat to adjacent snow covered areas melting the snow and allowing the ground to heat by absorbing sunlight. You can see this in your yard if you clean the snow off your sidewalk.
@ aurbo: Thank you — maybe the old brain is functioning after all. I remember those old low-baud rate days well, and service “A” too. The USAF once tried to hook up to a country in South America during the 60’s, and they wanted to operate at 15 baud!
Thanks again.
Reference the use of a teletype and a SECO box (Sequential Control) for generating weather reports:
http://docs.lib.noaa.gov/rescue/wb_circular_ltrs/Qc875u5u651952.pdf
The following is an excerpt from: Discussion of CAA/Weather font set (Note: A minus sign seems to exist in this character set.)
Interesting account of a weather plotter:
http://forum.keypublishing.com/showpost.php?s=d86e866457d920c9f83cb46ad962017f&p=1469942&postcount=7
@ FrankK (12:52:13) :
“I commented earlier about por [sic] old Tasmania just below Australia being a nonsense “hotspot” in the GISS figure. I notice there is no such hotspot but normal temps in the satellite diagram for Tasmania. That makes much more sense.”
Dunno about “poor old Tasmania”. We just had a ripper of a summer after managing to not notice the Global Financial Crisis. Best summer since the sixties for those who remember back that far. If this is Global Warming, bring it on 🙂
And while talking about globes, Tasmanian Tim Bowden relates the following aphorism:
“Tasmania is the testicle of the Nation. It infuses it with vim and vigour. What a pity there aren’t two of them.”
@ Neo (17:21:39) :
“Perhaps we could avoid all this stuff by just using Kevin”
Us Australian sceptics avoid all this stuff by perceiving it as just P Wong 😉
This document:
http://docs.lib.noaa.gov/rescue/wb_circular_ltrs/Qc875u5u651954.pdf
pdf page 30 Circular Letter #10-54 describes “Service ‘A’ Transmissions of Aviation Weather Forecasts” re: teletype circuit loading.
Plus signs and Minus signs (as well as a few other special symbols) seem to be in evidence.
Meticulous work, with disturbing implications.
Especially so given the low number of reporting stations in the Arctic, including northern Canada.
Thanks Anthony,
Aargh (17:20:20) :
“There is a type of person who just hates the concept of truth because they can’t tolerate anybody who is certain about anything.”
Enlighten me please as to the Nature of this Truth. Is it the Coherence Theory of Truth, the Correspondence Theory of Truth, the Pragmatic Theory of Truth, or one that I’ve not heard of?
Oh, and why would anyone “hate” your theory of truth without knowing which particular theory of truth you espouse?
Just curious…
I haven’t read all the comments, so excuse me if this is a duplication.
Why not intensively examine a recent representative sample of the data that hasn’t been discarded, calculate the error rate and error effect, and correct the records for previous years by that rate?
For those who keep asking about the buoy data: From the sailwx website-
“Most of the information presented comes via a single, fallible pipeline with no redundancy. Data stream is processed automatically, without any human oversight, automatic quality control, or ‘sanity checking’.”
Btw- I’m only using this data to fill in the ‘pole hole’. And this data is easily corrected.
Re: Geoff Sherrington (Apr 18 04:26),
Not to be picky but Oz converted to decimal CURRENCY in 1966, then to decimal MEASUREMENTS in 1972.
On a serious note, I don’t think it matters that much what effect these errors have on “global anomolies”. The fact remains that this is yet more errors in an error riddled dataset.
To rely on this dataset for geopolitic decision making is folly.
@ baahumbug (22:21:42) :
“Not to be picky but Oz converted to decimal CURRENCY in 1966, then to decimal MEASUREMENTS in 1972.
On a serious note, I don’t think it matters that much what effect these errors have on “global anomolies”. The fact remains that this is yet more errors in an error riddled dataset.
To rely on this dataset for geopolitic decision making is folly.”
Not to be picky, but I seem to remember currency and weights & measures going decimal at the same time. 1972 was when Imperial weights and measures became illegal.
_Jim (18:13:41) :
“aurbo (14:42:51) :
In fact … the machines including the Model-15 and Model-28 page printers
page printers? The surplus Model 28’s we used in the late 70’s all used continuous roll paper … maybe ‘page printer’ refers to 8 1/2″ wide paper versus paper tape …
Model 28 info page:
http://www.baudot.net/teletype”
You are right. The model-15 and model-28 teletype machines used paper rolls. I was descriminating between their 8½” wide 72 characters per line output and the WU type tape printers that were in current use back then for pasting up telegrams and for stock market tickers. In fact, the Western Electric manual describes the model-15 as a page printer see here.
The problem was that the models formated with standard character symbols had the – sign coded on the upper case “A” (Figs + “A”) key producing binary baudot code of 00010 while on those formated for wx symbols the lower case (Figures) 00010 coding was an up arrow used for one of the wind directions. To avoid confusion on networks where machines supporting both formats might have been used, the minus sign was dropped for the temperature group, the wind and cloud-cover symbols disappeared and when the METAR coding replaced the SA codes, the letter “M” replaced the minus sign for temps.
As stated in an earlier post, the whole problem began in an attempt to limit the number of characters per line bcecause of the slow transmission rate. This continues to cause problems today even though time is no longer an issue. For example, many errors occur in precipitation climatology because in the need to save transmission time, there is no positive zero. That is, if no precip has occurred in the past 6 or 12 or 24 hrs, no coded precip group is sent. If a precip group of all zeroes is sent, it connotes a trace! Thus you can’t tell the difference between no precip having occurred or the fact that there was precip but the precip group was missing or not observed. This not only occurs in the hourly observation transmissions but in the primary sequence in the synoptic code as well. I would say that precipitation data errors are much more frequent, mostly due to missing data, than you’ll find with temperatures. But, hey…that’s climate science.
Ok lets throw out the entire temperature record and instead just make geopolitical decision making based on the science. Which is that CO2 emissions are likely to significantly increase temperatures. All agreed? excellent.
One more comment of Leif’s observations.
As was indicated, Russians, like many other people in cold climes, usually leave out the “minus” or “plus” simply due to the fact that the difference between +15C and -15C is blindingly obvious to anyone around, and fairly easily deduced from a calendar.
HOWEVER, in the range around +/-5C, where the difference is not necessarily so immediately sensible, my experience was that people — at least in the part of Russia where I lived — would speak of, for example, 2 degrees мороза (of frost) or тепла (of warm).
At the same time, this was only a colloquialism. Anytime people were talking about temperatures AS DATA, they would use минус (minus).
Amazing how just a small oversight can turn -40°C into plus 40°C. You’d think this would be carefully monitored so as not to corrupt the results. Yet, no one seems to care. Now if the error worked the other way, i.e. exaggerated cooling, you can be sure they’d do everything to keep the error from happening.
As someone who has worked in shop floor data collection the metar ergonomics escape me. if it were designed to be morsed or sent in clear spoken using phonetic alphabet; i could understand the need to eliminate symbols, and a leading ‘M’ to denote sign would be reasonable, but other bits of the message use symbols..!!..???.
No-one reading a print out or a guage face marked as minus is unnerringly going to replace it with an ‘M’. Requires the operator to actively think about each reading rather than shunt it from eye to pen.
As an aside, when I did this for a living there were a few rules of thumb we used.
1) any manually entered data contains 1 error per 300 key presses. i.e even when the operator is putting the right bit of info into the correct entry field 1 in 300 characters will be incorrect (standard figure drawn from US DoD Generic Task Analysis studies). If you are lucky these will occur in non-key ,non-value, not-analysed fields.
2) Murphy’s Law and its principal corollary applies to data entry errors…..
3) The overall error rate on manual input is about 5% (Data entered in wrong fields, data not entered, wrong document type used etc etc etc includes errors due to point 1)
4) If a document isn’t entered within 15 minutes of its arrival you are on a not very protracted slope to it’s never being entered at all.
5)Errors not detected or trapped at or near to point of entry are never detected until they interfere with a critical process or are found under embarrassing circumstances by regulators and external auditors (Murphy’s Special theory for corporate record keeping……)
Hence the emphasis on Bar-Coded documents, EDI etc…
R. Gates (19:54:37) :
What difference or effect will this all make in the actual global temperature anomalies?
Inevitably, the Official-Answer ™ is going to be “The errors are insignifigant because of averaging and smoothing.”
Another log on the fire to do away with anomalies.
The use of Degrees Kelvin would have eliminated a whole host of errors, misinterpretations, and willful deceit regarding temperature data.
The use of anomalies is just sloppy practice.
Wren (14:26:11) :
If the recorders at airports are more likely to forget minus signs( whether indicated by “-” or “M”) than recorders not at airports, a move toward airport stations as a data base could introduce an upward bias in temperature records until there was no longer a move toward airport stations.
However, I doubt recorders at airports are any more likely to forget minus signs than other recorders. I think I am demonstrating a classic case of skepticism rather than a classic case of confirmation bias.
Seems you overlooked a detail there, Wren. METAR is the format for records from airports, nowhere else. It’s only airports that have the issue with using the letter M (or not, as the case may be) to signify a negative Celcius reading. For non-METAR formats, a ‘-‘ symbol may be retained to signify a negative reading; for METAR, it will definitely be dropped and show a positive Celcius reading.
Therefore, as there is a continuing trend towards using a higher proportion of airport data, so the issue becomes a growing one, creating an ongoing false trend.
Interesting stuff. There is of course no 100% adequate replacement for an experienced human brain in doing QA, but certainly the situation described above should be amenable to a computer algorithm to help identify the most obvious cases for further human review. Sure, the difference between -1C and 1C would be a toughie. But -10C and 10C, looking at historical records for which is more likely in that location on that date, ought to stick out nicely even to a computer to say “Hey, somebody take a look at this one. . . “.
A comment about methodology after a little further thought…
While the occurances of these errors will tend to increase the anomoly in the short term, the anomoly can’t really keep growing indefinitely due to these sorts of problems unless the rate of errors actually increases (or systemically starts getting applied only to colder and colder temp readings.)
So, while temperature may have risen in the short term as more and more data sets used METAR and introduced the potential problem, eventually it’s going to have to top out and the anomoly will stabilize at (Actual_Anomoly + Avg_Measurement_Error)
It makes one wonder if the recent stabilization and declines that the ClimateChange guys are lamenting is due to this levelling-off.
It also makes the conspiracy-minded wonder if this temperature-error-coding possibility was behind the decision to bias reporting stations largely toward airports… show a spike in temperature and then try to use that to scare the world fast before it levelled out. Someone could easily have run different data sets through the computer to see what the result would have been.
Theory of truth vs just outright lying–
http://alethonewsa.wordpress.com/2010/04/18/global-warming-fraud-in-worlds-most-highly-regarded-scientific-journal/
http://freestudents.blogspot.com/2007/05/grapes-of-math-global-warming-fraud.html
http://alethonewsa.wordpress.com/2010/04/18/global-warming-fraud-in-worlds-most-highly-regarded-scientific-journal/
On the issue of dropped zeroes:
some time back I questioned the data showing that January 2010 was the warmest January on record in the northern hemisphere.
I said at the time, could this be due to transcription error ? Because this finding sure goes against all reasonable expectations. A few dropped minus signs would go a long way to explaining this conclusion.
WOW !!
Even when there is an “honest” mistake, it is a warming bias !!!!!!!
Mr. Watts,
from your article I deduce that your “thesis” is : Minus sign errors in temp. data are responsible for the warming effect in the global trend.
For this to be valid you must have a positive answer to following 3 questions:
1. Is the percentage of the total local temp. data stations which have this error, statistically significant?
2. Is this error systematic, i.e, it always appears in the same stations at the same time?
3. This error is not presently taken into account in statistical processing of temp. data?
Unless you are able to answer to this questions, your investigation amounts to the discussion of a real but statistically insignificant error.
Cheers
Since these human errors are detectable in polar regions, it should be possible to use the polar sites to determine the human error rate, then project the bias effect at temperate latitudes.
The bias estimation problem is greatly simplified by the fact that the error should almost exclusively go in one direction. As an earlier commenter noted, people aren’t likely to be entering M’s by mistake:
Since these human errors are detectable in polar regions, it should be possible to use the polar sites to determine the human error rate, then project the bias effect at temperate latitudes.
The bias estimation problem is greatly simplified by the fact that the error should almost exclusively go in one direction. As an earlier commenter noted, people aren’t likely to be entering M’s by mistake:
If there are “superfluous M” errors being made then it should be possible to estimate the error rate by selecting data from temperate-region airports at times when the average temperature is high enough that a superfluous M would stand out. (It wouldn’t do to estimate the superfluous M error rate using equatorial airports, where they would always stand out, because the operators would not be sometimes called upon to enter “M”s, making the mistake less likely at the equator.)
As Anthony points out, it is easy to see how the missing-M mistake could be made. In contrast, the most likely reason a person would enter an M by mistake would be if he actually misinterprets the positive temperature reading as a negative one (possible if relaying a celsius reading perhaps, or reading a digital readout in a hurry). That is very different from omitting an M, or typing a minus sign instead, and should be much less likely.
It seems that the resulting warm bias could be quite large. Is this the explanation for the reported high Siberian temperatures for the winter of 08-09? If the error is substantial, developing an estimate of its size would be important.
ANTHONY or MODERATORS.
Did you see this offer of assistance at Glenn McKee (12:34:07) : as I did not see any response.
Wren,
You need to understand the way that this operates. Missing M’s in the data are not a systematic error, they are a random error, as shown by Anthony in his post.
Let’s say that a station in Siberia has a relatively steady reading of -35C (or M35) for the entire month of February. Now, let’s say that on a particularly cold day (about -43C for the entire day, just for example) for 8 hours the M is missing in the dataset, so for 8 hours the temperature is reported as 43C instead of -43C. This is a +86C error for 8 hours. Instead of a daily average of -43C for that day, the daily average becomes -14.3C for that day, for a difference of nearly 29C for the daily average due to this error. Even when this erroneous data gets averaged in to an entire month and we assume that the other 29 days had a correct average which was stipulated to be -35C, that monthly average suddenly becomes -34.3C, which is a +0.7C error in the monthly average (for this calculation I assumed a 30 day month, 29 days with an average of -35C, and one day with an erroneous average of -14.3C).
I made the math much simpler than I am sure it really is, but just using basic math it is possible to show that if 8 hours per month of data at a really cold station are missing the M, it could easily throw off the monthly average by anywhere from +0.5 to +1.0C.
Since we are talking about a “global temperature anomoly” on the order of +0.5C which is supposed to be a “major cause for alarm” and we “need” to spend trillions of dollars to “do something about it”, it is critically important to note that if cold/polar stations even have 8 corrupted hours of data over an entire month of 720 hours of data, the monthly average magnitude of that error might well be > +0.5C
This is why this finding is so critical.
Alec Rawls (05:55:48) :
Alec Rawls (05:56:38) :
> As Anthony points out, it is easy to see how the missing-M mistake could be made.
And note that Alec has just pointed out how easy it can be to leave the ‘/’ out of </blockquote> 🙂
@Skeptic
The error DOES NOT HAVE TO BE SYSTEMATIC in order to be statistically significant.
See my post above.
It only takes EIGHT HOURS of bad data from a Siberian or Polar site in an ENTIRE MONTH to throw off the monthly average by +0.7 C.
Also, no, this error would NOT be currently taken into account in the current statistical processing of the data. The whole reason for this post was real-life erroneous Finland data that stood out like a sore thumb. Only AFTER it was mentioned to the “powers that be” was it actually admitted to be in error and subsequently corrected.
If outside sources had not brought it to their attention, the erroneous data would have remained in the databases.
It does not matter if the error is random or systematic, what matters is the frequency and location of the error. All it takes is 8 hours of bad data out of 720 hours of monthly data to throw the average off by quite a lot at a cold station.
@MattB
Ok lets throw out the entire temperature record and instead just make geopolitical decision making based on the science. Which is that CO2 emissions are likely to INSIGNIFICANTLY increase temperatures. All agreed? excellent.
FTFY
When I click on the satellite map it gives me a 404 error, ie not found. Could this be fixed please as I want to see the map for Canada as March was incredibly warm where I live in Ottawa.
Mike Odin (05:17:40) :
“Theory of truth vs just outright lying–”
http://alethonewsa.wordpress.com/2010/04/18/global-warming-fraud-in-worlds-most-highly-regarded-scientific-journal/
Good grief!
It will be interesting to see how many errors will lean towards cooling instead of warming, when the next Ice Age scare comes up. It will come.
It’s also a critical error in that it demonstrates (again!) a pathological approach to lack of error investigation.
WHEN the bogus datum was found, the answer was of the form: “I took the fly out of your bowl of soup, what more do you want?”
Rather it ought to have been of the form: “We need to find out how flyS get into the soup and assure it does not happen again WHILE we make a new pot of soup”.
We saw the same thing in the “HarryReadMe” file where a bad datum caused havoc (squares of a number that were NEGATIVE) and the answer was to remove that ONE bad data item. I showed how to fix the code so it could never happen again in a posting at the time of the Climategate scandal hitting the news.
The “climate science” guys just don’t seem to understand root cause analysis and early defect removal. It’s all “stick a bandaid on it and move on”.
tommy (16:53:08) :
“I sure hope this arctic station isnt included in GISS map: ”
http://www.wunderground.com/global/stations/01009.html
“It has been showing 20-30c for months now and it is one of the official weather stations.”
Should be around -14 Deg Celcius;
http://www.yr.no/place/Norway/Svalbard/Phipps%C3%B8ya/
Aargh (17:20:20) :
Enlighten me please as to the Nature of this Truth. Is it the Coherence Theory of Truth, the Correspondence Theory of Truth, the Pragmatic Theory of Truth, or one that I’ve not heard of?
Oh, and why would anyone “hate” your theory of truth without knowing which particular theory of truth you espouse?
Just curious…
No, you are not ‘just curious’. You are spoiling to make an attempt to claim that there are varieties of truth in order to devalue the concept in its entirely.
You can’t do it because you must first steal the premise in order to deny it.
You are regurgtitating pickup lines from the cannibal handbook.
I see through you. Go find somebody innocent and naive to pervert.
PeterB in Indianapolis (10:08:11) :
Wren,
You need to understand the way that this operates. Missing M’s in the data are not a systematic error, they are a random error, as shown by Anthony in his post.
Let’s say that a station in Siberia has a relatively steady reading of -35C (or M35) for the entire month of February. Now, let’s say that on a particularly cold day (about -43C for the entire day, just for example) for 8 hours the M is missing in the dataset, so for 8 hours the temperature is reported as 43C instead of -43C. This is a +86C error for 8 hours. Instead of a daily average of -43C for that day, the daily average becomes -14.3C for that day, for a difference of nearly 29C for the daily average due to this error. Even when this erroneous data gets averaged in to an entire month and we assume that the other 29 days had a correct average which was stipulated to be -35C, that monthly average suddenly becomes -34.3C, which is a +0.7C error in the monthly average (for this calculation I assumed a 30 day month, 29 days with an average of -35C, and one day with an erroneous average of -14.3C).
I made the math much simpler than I am sure it really is, but just using basic math it is possible to show that if 8 hours per month of data at a really cold station are missing the M, it could easily throw off the monthly average by anywhere from +0.5 to +1.0C.
Since we are talking about a “global temperature anomoly” on the order of +0.5C which is supposed to be a “major cause for alarm” and we “need” to spend trillions of dollars to “do something about it”, it is critically important to note that if cold/polar stations even have 8 corrupted hours of data over an entire month of 720 hours of data, the monthly average magnitude of that error might well be > +0.5C
This is why this finding is so critical.
======
Sign errors may or may not have enough effect to be concerned about. As Anthony Watts said at the end of his post, ” In this post I’ve demonstrated the errors. In a later post, I hope to do some data analysis with the numbers provided to see how much effect these errors actually have.”
If missing negative signs(“-” or M) and other errors do tend to bias the GISSTEMP global temperature trend upwards, something else must be offsetting the effect, since this trend does not diverge from the RSS and UAH trends. Looking at the three over at woodfortrees, GISSTEMP appears slightly closer to RSS and UAH recently than it was in 1978 when the latter two began.
http://www.woodfortrees.org/plot/hadcrut3vgl/last:360/offset:-0.146/mean:12/plot/uah/last:360/mean:12/plot/
When errors offset, correcting only one can results in a net loss in accuracy.
Latest joe bastardi video also reports on this issue: http://www.accuweather.com/video/78827407001/more-fun-with-global-temperatures.asp?channel=vblog_bastardi
Looks like he might have also found another finland hotspot like anomaly.
I don’t want to apologizes for errors, but humans ARE error prone, especially when distracted or hurried. There are several thousand METAR stations reporting each hour and thus the chances of an error in one or more reports is not zero, but needs to be viewed in perspective.
COOP observations are used extensively in our so-called climatology and only a few figures are transcribed, high and low temperature plus precip amount (I think). I wonder what the known error rate is for this simple task?
And what might be the error rate of a “scientist” creating a temperature where one didn’t exist?
Anthony, you stated to Nick way above that “Surface project is coming along fine, a paper is being prepared. Thanks for asking.”
What is your definition of coming along fine? Do we have any hope of getting a look at it this year?
Seems like you would want to post it so you could counter the several analyzes that have shown that the sitting issue does not significantly alter the temperature trends.
Thanks again, Anthony.
Wonder what new light this casts on the infected blob of reddish-black that is Afghanistan, Pakistan, and Tajikistan (if I’m reading the map right)?
There are other problems in the Arctic region also.
One of these is “Zero” measurements. Found one instance of it, where temperatures were below -20ºC, but than 0!
Also, differences between METAR and AAXX frequently popup.
I’ve also noticed clear anomalous temperature rises.
Please see more at: http://ecotretas.blogspot.com/2010/04/arctic-heat.html
Ecotretas
The random omission of minus signs is a SYSTEMATIC error. It leads to an error always on one side of the correct value. In this case, obviously it biases the mean high.
For it to be a RANDOM error, there would have to be an equal probability of PUTTING IN a minus sign when there was either no sign there originally, or of substituting minus for plus.
I don’t know if these data are reported with “+” signs for positive temperature anomaly, or if a number without a sign is automatically assumed positive.
However, I believe the simple omission of “-” signs will be a systematic bias on the mean, not a random error.
Wren,
“If missing negative signs(“-” or M) and other errors do tend to bias the GISSTEMP global temperature trend upwards, something else must be offsetting the effect, since this trend does not diverge from the RSS and UAH trends. Looking at the three over at woodfortrees, GISSTEMP appears slightly closer to RSS and UAH recently than it was in 1978 when the latter two began.”
How well, precisely, do they agree? Do they agree withing 0.02C, within 0.1C, within 0.25C, or what? (I did look, so I know the answer, but I am not giving ti away since I want other people to look as well).
This is a key question, because when the global temperature “anomoly” is only 0.5C, even a few hundredths of a degree of “disagreement” could potentially be significant.
One further question: How well do the lower troposphere and surface temperatures actually match? I would not expect them to match, given that they are taken at significantly different altitudes, and yet they seem to match fairly well, and as you point out, the match seems to be improving over time.
This may be due to better surface measurements, or it may simply be due to an “improved” method of “homogenizing the data” (adding fudge factors to make it match).
baahumbug (22:21:42) : Re: Geoff Sherrington (Apr 18 04:26),
“Not to be picky but Oz converted to decimal CURRENCY in 1966, then to decimal MEASUREMENTS in 1972”.
It was officially encouraged to switch science to decimal in 1966 – much was already so. In Anthony’s case of the dropped minus sign, it makes a rather big difference because the incidence of minus signs is much greater in Celsius than Fahrenheit. When C is at zero and going negative, F is at +32 with a lot of positive numbers “as stored heat in the pipeline” to borrow a trendy phrase. The difference decreases as you cool to minus 40, which is the point at which C = F.
But have a look at the graph I plotted for Mawson – one can only guess at the mechanisms. There is one sequence which gives a higher number of really low values each year, compared to the earlier years.
BTW, I think there is a lot of value in using daily Tmax and Tmin data especially if you are going on to stats dissections. Should not really use lower resolution data for statistics. It’s like buying an expensive camera then disabling the lens focus mechanism.
Geoff Sherrington (04:26:08) :
Anthony, this is a most important subject, well explained and documented. Congratulations.
The big unknown, of course, is whether the dropping of a negative sign happens enough times to swing the country or global data, or whether some countries have QQ that detects it. The Antarctic would seem particularly at risk.
In looking at data from one of the Australian Antarctic stations, Mawson WMO 94997, there was not enough metadata available to me to address the same points that you raise. However, the first look shows in essence that the extreme temperatures are somewhat ragged at the edges. (For the record, Mawson has an airstrip. Australia changed to decimal in Feb 1966, but as a scientific base, the records were possibly in C from the start in 1954). The data are from the Australian Bureau of Meteorology, duly acknowledged.
An initial look is presented now in the form of three spreadsheet portions and a graph plus a comment on each. I stress that we might be looking at different mechanisms, but what I present does not preclude errors of the type that you describe with METAR.
Overall, the data set had 19,419 days, with Tmax and Tmin. There were 87 days of missing Tmax and 22 days of missing Tmin, reason for missing unknown.
The mean of all Tmax values is -8.38 deg C. The mean of all Tmin is -14.27. The mean difference is 5.87 deg. Condider the sign dropping exercise with these figures, first with the Tmax going from -8.38 to +8.38, causing the Tmean to change from -11.33 to -2.93. If the sign was dropped from Tmin, the two Tmean would change from -11.33 to +2.93. As this shows, the dropping of sign on the minimum is a bigger problem than dropping the sign on the maximum, as you noted.
Mawson1 is a small part of the Mawson daily maximum temperatures sorted from highest down, for the first few weeks.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson1.jpg?t=1271577750
If there was a sign error for Tmax, because its average is close to -8 Deg C, there are 2 consequences. Where Tmax is already above zero, in about 20% of the cases, we are not subjected to missing a negative since there are none. Where Tmax is below zero and the negative sign is dropped, this leads to an apparent warming that can report in 80% of the data. Tmax can be as low as -32 deg C, so dropping the sign gives a healthy wrong warming.
Mawson2 is for the minimum temperatures sorted from the highest down.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson2.jpg
This spreadsheet partially shows that only a few days had Tmin above zero. The vast majority have a negative sign and dropping the negative sign would cause a strong warming impression. It is also more probable an error than with Tmax. The average of all readings is -14.2, the coldest reading is -36 deg C with a potential for a daily Tmean error of 36 deg C.
Mawson3 shows the difference between Tmax and Tmin.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/Mawson3.jpg?t=1271577538
The largest difference is 32 degrees. If there were large mistakes made from occasional sign dropping, there would be occasional large values that did not seem to be part of a smooth distribution. However, it is plausible that such “unlikely” figures would be culled by the hand of man, making this detective work less relevant.
Then there is a daily graph showing the differences calculated in Mawson3.
http://i260.photobucket.com/albums/ii14/sherro_2008/2010/MAWSONGRAPH.jpg?t=1271577707
Mod, my apologies. Wanted to copy the last line URL but the whole last post got copied. Everything from Geoff Sherrington (04:26:08) : is an error of repetition, I just wanted the bottom line.
Anthony,
Amazing work! I look forward to the analysis post.
Anthony,
A great post, my compliments for your work. Another astonishing hole revealed in the “official” climate (sorry, weather) data collection procedures..
I spent years running central processing of data from 50 plus countries worldwide.
As well as workable unambiguous standardisation guidelines, we needed central validation programs to check all input because it was always changing! Why? – changes of local software and hardware giving wrong exchange formats, errors in programs, human error in input etc. etc. Kept a few central programmers busy updating the routines and five plus central users busy correcting the input.
The management never quite understood, they thought you could just rely on the deliveries from each country…
ANY IT professional knows how this stuff happens, and what to do about it. Where are the professionals?
@ Aargh (14:23:43) :
“No, you are not ‘just curious’. You are spoiling to make an attempt to claim that there are varieties of truth in order to devalue the concept in its entirely.
You can’t do it because you must first steal the premise in order to deny it.
You are regurgtitating pickup lines from the cannibal handbook.
I see through you. Go find somebody innocent and naive to pervert.”
Aargh, consider the following two statements:
(i) 1 + 1 = 2 [Pompous Git]
(ii) It is The Truth that “1 + 1 = 2” [Aargh]
This is not stealing premises, as if such a thing were possible. Nor is it any of your other fantasies. The theory of truth I prefer, though I am not absolutely wedded to, is called a Deflationary, or Disquotational theory. It’s not perfect, but it works for me.
BTW, I cannot recall ever using a pickup line in my whole life; I’ve not needed to. One of the effects of the 60s revolution was the discovery by women that they could make the sexual advances and there’s a certain kind of woman attracted to nerds. Happily, one of those women has looked after me since 1980. In “The Git’s Guide to Life, the Universe and Everything” it is written: “the secret to a long and happy marriage is finding a She Who Must Be Obeyed who is worth obeying.”
So far the only correct conclusion you have leaped to about me is my age 🙂
Pompous Git (13:03:06) :
Obedience is the negation of independent thought, which, for a human being, is tantamount to suicide.
I’m sorry if you haven’t sorted the simple things yet and I understand that the recruitment campaign is so intense and ubiquitous that you are quoting from the pamphlets without understanding. It’s never to late to find your inner child and make him a bride of Christ, eh?
it sounds like you had no room for the law of identity even in your manger.
Enjoy what you can, even if it isn’t yourself. I will hope that you may find closure promptly. For inspiration, visit http://www.darwinawards.com. All your friends can be expected to show up there- unless they get funded.
Good bye.
For those who may still be interested, US stations use a slightly different form of METAR code – call it a regional dialect – and append a temperature group beginning with a “T” at the end of each ob. This is followed by a sign indicator (0 for positive and 1 for negative) then the temperature to tenths of degrees C. The dew-point is then appended to this group in a similar fashion.
THe M04/M07 illustrated, if from a US station, might look like T10431076.
So it is possible to cross-check things, but if they are different, which one is correct may still not be obvious in some cases. But if you are holding in a racetrack pattern waiting for fog to clear, you might be a bit better informed.
Worse yet!