UPDATE: See the first ever CONUS Tavg value for the year from the NCDC State of the Art Climate Reference Network here and compare its value for July 2012. There’s another surprise.
Glaring inconsistencies found between State of the Climate (SOTC) reports sent to the press and public and the “official” climate database record for the United States. Using NCDC’s own data, July 2012 can no longer be claimed to be the “hottest month on record”. UPDATE: Click graph at right for a WSJ story on the record.
First, I should point out that I didn’t go looking for this problem, it was a serendipitous discovery that came from me looking up the month-to-month average temperature for the CONtiguous United States (CONUS) for another project which you’ll see a report on in a couple of days. What started as an oddity noted for a single month now seems clearly to be systemic over a two-year period. On the eve of what will likely be a pronouncement from NCDC on 2012 being the “hottest year ever”, and since what I found is systemic and very influential to the press and to the public, I thought I should make my findings widely known now. Everything I’ve found should be replicable independently using the links and examples I provide. I’m writing the article as a timeline of discovery.
At issue is the difference between temperature data claims in the NCDC State of the Climate reports issued monthly and at year-end and the official NCDC climate database made available to the public. Please read on for my full investigation.
You can see the most current SOTC for the USA here:
http://www.ncdc.noaa.gov/sotc/national/2012/11
In that SOTC report they state right at the top:
Highlighted in yellow is the CONUS average temperature, which is the data I was after. I simply worked backwards each month to get the CONUS Tavg value and copy/paste it into a spreadsheet.
In early 2011 and late 2010, I started to encounter problems. The CONUS Tavg wasn’t in the SOTC reports, and I started to look around for an alternate source. Thankfully NCDC provided a link to that alternate source right in one the SOTC reports, specifically the first one where I discovered the CONUS Tavg value was missing, February 2011:
http://www.ncdc.noaa.gov/sotc/national/2011/02
That highlighted in blue “United States” was a link for plotting the 3-month Dec-Feb average using the NCDC climate database. It was a simple matter to switch the plotter to a single month, and get the CONUS Tavg value for Feb 2011, as shown below. Note the CONUS Tavg value at bottom right in yellow:
All well and good, and I set off to continue to populate my spreadsheet by working backwards through time. Where SOTC didn’t have a value, I used the NCDC climate database plotter.
And then I discovered that prior to October 2010, there were no mentions of CONUS Tavg in the NCDC SOTC reports. Since I was recording the URL’s to source each piece of data as well, I realized that it wouldn’t look all that good to have sources from two different URL’s for the same data, and so for the sake of consistency, I decided to use only the CONUS Tavg value from the NCDC climate database plotter, since it seemed to be complete where the SOTC was not.
I set about the task of updating my spreadsheet with only the CONUS Tavg values from the NCDC climate database plotter, and that’s when I started noticing that temperatures between the SOTC and the NCDC climate database plotter didn’t match for the same month.
Compare for yourself:
NCDC’s SOTC July 2012:
http://www.ncdc.noaa.gov/sotc/national/2012/07
Screencap of the claim for CONUS Tavg temperature for July 2012 in the SOTC:
Note the 77.6°F highlighted in blue. That is a link to the NCDC climate database plotter which is:
Screencap of the output from the NCDC climate database, note the value in yellow in the bottom right:
Note the difference. In the July 2012 State of the Climate Report, where NCDC makes the claim of “hottest month ever” and cites July 1936 as then benchmark record that was beaten, they say the CONUS Tavg for July 2012 is: 77.6°F
But in the NCDC climate database plotter output, the value is listed as 76.93°F almost 0.7°F cooler! They don’t match.
I initially thought this was just some simple arithmetic error or reporting error, a one-off event, but then I began to find it in other months when I compared the output from the NCDC climate database plotter. Here is a table of the differences I found for the last two years between claims made in the SOTC report and the NCDC database output.
In almost every instance dating back to the inception of the CONUS Tavg value being reported in the SOTC report, there’s a difference. Some are quite significant. In most cases, the database value is cooler than the claim made in the SOTC report. Clearly, it is a systemic issue that spans over two years of reporting to the press and to the public.
It suggests that claims made by NCDC when they send out these SOTC reports aren’t credible because there are such differences between the data. Clearly, NCDC means for the plotter output they link to, to be an official representation to the public, so there cannot be a claim of me using some “not fit for purpose” method to get that data. Further, the issue reveals itself in the NCDC rankings report which they also link to in SOTC reports:
Note the 76.93°F I highlighted in yellow. Since it appears in two separate web output products, it seems highly unlikely this is a “calculation on demand” error, but more likely simply a database output and that is simply displayed data.
Note the claim made in the NCDC July 2012 SOTC for the July 1936 CONUS Tavg temperature which is:
The previous warmest July for the nation was July 1936, when the average U.S. temperature was 77.4°F.
But now in two places, NCDC is reporting that the CONUS Tavg for July 2012 is 76.93°F about 0.47°F cooler than 77.4°F claimed as the previous monthly record in 1936, meaning that July 2012 by that comparison WAS NOT THE HOTTEST MONTH ON RECORD.
The question for now is: why do we appear to have two different sets of data for the past two years between the official database and the SOTC reports and why have they let this claim they made stand if the data does not support it?
There’s another curiosity.
Curiously, the last two months in my table above, October and November 2012 have identical values between the database and the SOTC report for those months.
What’s going on? Well, the explanation is quite simple, it’s a technology gap.
You see, despite what some people think, the nation’s climate monitoring network used for the SOTC reports is not some state of the art system, but rather the old Cooperative Observer Network which came into being in the 1890’s after Congress formed the original US Weather Bureau. Back then, we didn’t have telephones, fax machines, radio, modems or the Internet. Everything was observed/measured manually and recorded by hand with pen and paper, and mailed into NCDC for transcription every month. That is still the case today for a good portion of the network. Here’s a handwritten B91 official reporting form from the observer at the station the New York Times claims is the “best in the nation”, the USHCN station in Mohonk, New York:
Source: http://www7.ncdc.noaa.gov/IPS/coop/coop.html
Note that in cases like this station, the observer sends the report in at the end of the month, and then NCDC transcribes it into digital data, runs that data through quality control to fix missing data and incorrectly recorded data, and all that takes time, often a month or two for all the stations to report. Some stations in the climate network, such as airports, report via radio links and the Internet in near real-time. They get there in time for the end of the month report where the old paper forms do not, hence the technology gap tends to favor more of a certain kind of station, such as airports, over other traditional stations.
NCDC knows this, and reported about it. Note my bolding.
NOAA’s National Climatic Data Center (NCDC) is the world’s largest active archive of weather data. Each month, observers that are part of the National Weather Service Cooperative Observer Program (COOP) send their land-based meteorological surface observations of temperature and precipitation to NCDC to be added to the U.S. data archives. The COOP network is the country’s oldest surface weather network and consists of more than 11,000 observers. At the end of each month, the data are transmitted to NCDC via telephone, computer, or mail.
Typically by the 3rd day of the following month, NCDC has received enough data to run processes which are used to calculate divisional averages within each of the 48 contiguous states. These climate divisions represent areas with similar temperature and precipitation characteristics (see Guttman and Quayle, 1996 for additional details). State values are then derived from the area-weighted divisional values. Regions are derived from the statewide values in the same manner. These results are then used in numerous climate applications and publications, such as the monthly U.S. State of the Climate Report.
NCDC is making plans to transition its U.S. operational suite of products from the traditional divisional dataset to the Global Historical Climatological Network (GHCN) dataset during in the summer of 2011. The GHCN dataset is the world’s largest collection of daily climatological data. The GHCN utilizes many of the same surface stations as the current divisional dataset, and the data are delivered to NCDC in the same fashion. Further details on the transition and how it will affect the customer will be made available in the near future.
See: http://www.ncdc.noaa.gov/sotc/national/2010/10
The State of the Climate reports typically are issued in the first week of the next month. They don’t actually bother to put a release date on those reports, so I can’t give a table of specific dates. The press usually follows suit immediately afterwards, and we see claims like “hottest month ever” or “3rd warmest spring ever” being bandied about worldwide in news reports and blogs by the next day.
So basically, NCDC is making public claims about the average temperature of the United States, its rank compared to other months and years, and its severity, based on incomplete data. As I have demonstrated, that data then tends to change about two months later when all of the B91’s come in and are transcribed and the data set becomes complete.
It typically cools the country when all the data is used.
But, does NCDC go back and correct those early claims based on the new data? No
While I’d like to think “never attribute to malice what can be explained by simple incompetence“, surely they know about this, and the fact that they never go back and correct SOTC claims (which drive all the news stories) suggests some possible malfeasance. If this happens like this in CONUS, it would seem it happens in Global Tavg also, though I don’t have supporting data at the moment.
Finally, here is where it gets really, really, wonky. Remember earlier when I showed that by the claims in the July 2012 SOTC report the new data showed July 2012 was no longer hotter than July 1936? Here’s the SOTC again.
Note the July 1936 words are a link, and they go to the NCDC climate database plotter output again. Note the data for July 1936 I’ve highlighted in yellow:
July 1936 from the NCDC database says 76.43°F Even it doesn’t match the July 2012 SOTC claim of 77.4°F for July 1936. That can’t be explained by some B91 forms late in the mail.
So what IS the correct temperature for July 2012? What is the correct temperature for July 1936? I have absolutely no idea, and it appears that the federal agency charged with knowing the temperature of the USA to a high degree of certainty doesn’t quite know either. Either the SOTC is wrong, or the NCDC database available to the public is wrong. For all I know they both could be wrong. On their web page, NCDC bills themselves as:
How can they be a “trusted authority” when it appears none of their numbers match and they change depending on what part of NCDC you look at?
It is mind-boggling that this national average temperature and ranking is presented to the public and to the press as factual information and claims each month in the SOTC, when in fact the numbers change later. I’m betting we’ll see those identical numbers for October and November 2012 in Table 1 change too, as more B91 forms come in from climate observers around the country.
The law on such reporting:
Wikipedia has an entry on the data quality act, to which NCDC is beholden. Here are parts of it:
=============================================================
The Data Quality Act (DQA) passed through the United States Congress in Section 515 of the Consolidated Appropriations Act, 2001 (Pub.L. 106-554). Because the Act was a two-sentence rider in a spending bill, it had no name given in the actual legislation. The Government Accountability Office calls it the Information Quality Act, while others call it the Data Quality Act.
The DQA directs the Office of Management and Budget (OMB) to issue government-wide guidelines that “provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies”.
…
Sec. 515 (a) In General — The Director of the Office of Management and Budget shall, by not later than September 30, 2001, and with public and Federal agency involvement, issue guidelines under sections 3504(d)(1) and 3516 of title 44, United States Code, that provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies in fulfillment of the purposes and provisions of chapter 35 of title 44, United States Code, commonly referred to as the Paperwork Reduction Act.
=============================================================
Here’s the final text of the DQA as reported in the Federal Register:
http://www.whitehouse.gov/sites/default/files/omb/fedreg/reproducible2.pdf
Based on my reading of it, with their SOTC reports that are based on preliminary data, and not corrected later, NCDC has violated these four key points:
In the guidelines, OMB defines ‘‘quality’’ as the encompassing term, of which ‘‘utility,’’ ‘‘objectivity,’’ and ‘‘integrity’’ are the constituents. ‘‘Utility’’ refers to the usefulness of the information to the intended users. ‘‘Objectivity’’ focuses on whether the disseminated information is being presented in an accurate, clear, complete, and unbiased manner, and as a matter of substance, is accurate, reliable, and unbiased. ‘‘Integrity’’ refers to security—the protection of information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification. OMB modeled the definitions of ‘‘information,’’ ‘‘government information,’’ ‘‘information dissemination product,’’ and ‘‘dissemination’’ on the longstanding definitions of those terms in OMB Circular A–130, but tailored them to fit into the context of these guidelines.
I’ll leave it to Congress and other Federal watchdogs to determine if a DQA violation has in fact occurred on a systemic basis. For now, I’d like to see NCDC explain why two publicly available avenues for “official” temperature data don’t match. I’d also like to see them justify their claims in the next SOTC due out any day.
I’ll have much more in the next couple of days on this issue, be sure to watch for the second part.
UPDATE: 1/7/2013 10AMPST
Jim Sefton writes on 2013/01/07 at 9:51 am
I just went to the Contiguous U.S. Temperature July 1895-2012 link you put up and now none of the temperatures are the same as either of your screen shots. Almost every year is different.
2012 is now 76.92 & 1936 is now 76.41 ?
Just in case it was some rounding / math issue with Javascript, I checked the source code & then checked the page in both IE & Chrome… the data for the comma-delimited data is distinct and matches those of the plot. So, in the 2 days since your post it has changed yet again… for all years apparently?
That’s verified, see screencap below made at the same time as the update:
This begs the question, how can the temperatures of the past be changing?
Here’s comment delimited data for all months of July in the record:
1895,71.04
1896,73.43
1897,72.97
1898,72.93
1899,72.68
1900,72.82
1901,75.93
1902,71.81
1903,71.58
1904,71.06
1905,71.60
1906,72.03
1907,72.20
1908,72.80
1909,72.24
1910,73.66
1911,72.28
1912,71.90
1913,72.66
1914,73.68
1915,70.53
1916,73.92
1917,74.19
1918,72.00
1919,73.95
1920,72.31
1921,74.24
1922,72.61
1923,73.37
1924,71.49
1925,73.72
1926,73.01
1927,72.28
1928,72.98
1929,73.24
1930,74.63
1931,75.30
1932,73.75
1933,74.73
1934,75.98
1935,74.76
1936,76.41
1937,74.19
1938,73.36
1939,74.44
1940,73.72
1941,73.62
1942,73.55
1943,73.89
1944,72.39
1945,72.53
1946,73.43
1947,72.43
1948,72.90
1949,73.85
1950,70.85
1951,73.26
1952,73.69
1953,73.75
1954,75.13
1955,74.10
1956,72.73
1957,73.98
1958,72.29
1959,73.27
1960,73.56
1961,72.92
1962,71.77
1963,73.39
1964,74.40
1965,72.37
1966,74.79
1967,72.28
1968,72.64
1969,73.86
1970,73.73
1971,72.18
1972,71.97
1973,73.08
1974,73.95
1975,73.39
1976,72.77
1977,74.30
1978,73.68
1979,73.03
1980,75.63
1981,73.79
1982,73.08
1983,73.92
1984,73.07
1985,73.94
1986,73.51
1987,73.26
1988,74.75
1989,74.15
1990,73.27
1991,73.93
1992,71.28
1993,72.25
1994,73.53
1995,73.61
1996,73.56
1997,73.24
1998,75.49
1999,74.44
2000,73.90
2001,74.61
2002,75.90
2003,75.50
2004,72.98
2005,75.34
2006,76.53
2007,74.77
2008,74.21
2009,72.74
2010,74.83
2011,76.28
2012,76.92
SUPPLEMENT:
For now, in case the SOTC reports should suddenly disappear or get changed without notice, I have all of those NCDC reports that form the basis of Table 1 archived below as PDF files.
State of the Climate _ National Overview _ October 2010
State of the Climate _ National Overview _ January 2011
State of the Climate _ National Overview _ February 2011
State of the Climate _ National Overview _ March 2011
State of the Climate _ National Overview _ April 2011
State of the Climate _ National Overview _ May 2011
State of the Climate _ National Overview _ June 2011
State of the Climate _ National Overview _ July 2011
State of the Climate _ National Overview _ August 2011
State of the Climate _ National Overview _ September 2011
State of the Climate _ National Overview _ October 2011
State of the Climate _ National Overview _ November 2011
State of the Climate _ National Overview _ December 2011
State of the Climate _ National Overview _ January 2012
State of the Climate _ National Overview _ February 2012
State of the Climate _ National Overview _ March 2012
State of the Climate _ National Overview _ April 2012
State of the Climate _ National Overview _ May 2012
State of the Climate _ National Overview _ June 2012
State of the Climate _ National Overview _ July 2012
State of the Climate _ National Overview _ August 2012
State of the Climate _ National Overview _ September 2012
State of the Climate _ National Overview _ October 2012
State of the Climate _ National Overview _ November 2012
Why do the math, when you know what answer you need?
Its the old joke about the accountants looking for a job.
And of course the new attitude of our bureaucrats, “what you want me to do my job? thats discrimination.”
Garbage in Garbage out. Answers are worked backwards not by NASA or NCDC but the media.
It appears that the warmest ever turns out to be average. Bummer.
Additionally it appears we’re headed into a prolonged Arctic visit for Europe and North America. It would be nice to have this data updated quickly so that the media has to report it. Fat chance on that.
Thanks Anthony and keep the facts and circumstances coming.
Excellent detective work Anthony … again. How many bureaucrats are required to provide inconsistent information? Clearly a lot more than just the one unfunded person who can find a systematic error in the output they produce.
Anthony, Maybe you found the Janus set, not the Conus..
Keep their incompetent/and/or/lying feet to the fire, Anthony! Bottom line: they’re playing fast and loose with these data, and our tax dollars!
Cheers,
A Avery
Now how the heck are they going to explain that?
The corruption in CAGW lies not in bad data, incomplete analyses or agenda-driven adjustment biases (I’d call these moral or ethical failures) but in the insistence that reported data, statements about conditions, are unequivocally correct, accurate and precise to a fine level. The corruption lies in the claim of certainty, of being above discussion, internal dissension and external reproach.
Very nice work. I have not run the numbers but the writing and thinking is very linear and easy to interpret. Nice work!
As to the numbers, I hope you have found the thread that will unwind Hansen and the others “data cooks” we have at NOAA, NASA and the NSIDC.
Later reports leading to cooling?
No that can’t be – they have corrected for all that stuff like location, heat islands etc.
Or so they claim.
Thanks
JK
Excellent work, as a mere humble reporter, Anthony. This is the kind of work the newspapers used to do, before they allowed themselves to become little more than rump swabs.
This sort of chaos in the sets of data seems to be increasing. I fear it is a symptomatic of a growing gap between fact and “policy.”
Once “policy-makers” stray from Truth, it is a bit like having one foot on a row boat and the other foot on the dock. The gap gets larger and larger, and they wind up all wet. If you are kind you can throw them a life line, but be careful not do get dragged down with them as they fall.
Frankly, I’m impressed that you found any information in the morass that is http://www.ncdc.noaa.gov/ . I wasted much of the morning trying to find the biggest one day snowfalls at NWS sites around New England. I had seen them before and discovered somewhat to my surprise that Concord NH’s big storms area smaller than those in Boston, Worcester, Hartford, and even Providence.
I was trying to find that information again to give to a TV met at necn.com and failed. I did find data for 1981-2010 at a NWS site, but that database engine crashed after lunch when I decided to add Portland, Maine.
Grr. While the data I found doesn’t have the major storms from 1978 in southern New England, the pattern still held. Only in March did Concord have a bigger snowfall than the other locations. See http://wermenh.com/wx/index.html . I’ll be adding more stuff to it tonight.
Of course, if that data is wrong, then I’ve wasted even more time!
Apologies for a mostly OT rant, I really hate visiting the NCDC web site.
As Artie Johnson said on Laugh In.
“Very Interesting.”
One of the uncertainties that is neglected, IMHO, in weather data is what I call “database error.” This is distinct from instrumentation error, read error (parallax, electronic drift, dead batteries, etc.), representativeness error (e.g. surface stations dot org and/or UHI), systematic error, and homogenization error (such as time of observation error (TOBS) and station moves). Database error would include transmission error, transcription error (i.e. data input of written reports), wrong station IDs, data lag (this case) and the data version of the game telephone (where data changes as it works its way through transmission from station to country to world databases and archives). In this case, the issue is data lag, as different packets of data move through the various transmission and collation systems at different speeds. I would be surprised if there were any publications specifically regarding U.S. and world weather databases that attempted to calculate the magnitude of this uncertainty. There may be publications regarding this issue that have studied other large database systems.
I still think this one is amazing… Look at all the 2012 records compared to 1934!!!
http://stevengoddard.files.wordpress.com/2012/12/screenhunter_320-dec-01-11-30.jpg?w=640&h=484&h=484
Hmm, in an effort to be charitable, is it possible we’ve two different people calculating the “average” in a different manner?
In a more realistic notion, it think it’s likely this outfit, much like all the other climate tracking outfits I’ve experience with, probably don’t know their azz from a hole in the ground and simply spew their brand of jibberish without bothering to check for accuracy.
Why am I not surprised?
Good catch Anthony, thanks.
Anthony, I am always amazed by your ability to find these needles in the haystacks. Great work… and I am looking forward to updates from you as this progresses. Maybe there’s a “reasonable” explanation, but at the moment I’m… “skeptical.”
I do believe they are trying to con us.
From the outside looking in, it is hard for me to know if this is a little thing that can be easily fixed, or the biggest fraud since (enter your favorite scandal here).
It so happens I am about a third into Senator Inhofe’s book “The Greatest Hoax”. In the book, he details numerous instances of “mis-representation” of data. Anthony, if this is a big deal, is it something Inhofe should be made aware of?
Mod: A few typos here that ought to be corrected….
“highlighted in blue. that is a link” –> blue. That
“plotter output, the value is list as” –> the value is listed as
“in cases like this station, the observer send the ” –> observer sends
“though don’t have supporting data at the moment.” –> though I don’t
“that the agency charged with know to ” –> charged with knowing
“I’d alos like to see them” —> also
[working on it. Thanks, mod]
The “Rate This” scale only gave me 5 stars to pick from. Not nearly enough relative to other “top” posts. Great work, A.
Well spotted.
Small error. You have written “alos” instead of “also”
[Fixed, thanks. — mod.]
Anthony: You wrote: “Continental United States (CONUS)”
But CONUS means “contiguous US”–i.e., the lower 48 states, excluding Alaska. “Continental US” includes Alaska. Could it be that NOAA isn’t being clear, or has confused itself, about these two measures?
(I’ve made a couple of comments on this distinction in past years here, in response to other commenters.)
REPLY: I’ve always heard it the other way. Either description gets the point across – Anthony
Where’s Waldo
PS: I wouldn’t put it past NOAA to use the warmer of the two measures (contiguous US or continental US) to affect public opinion when it issues its press releases.
Great work Sir, thank you very much.
NOAA Top Ten Excuse List.
These 0.7 degree errors:
1. are small and can be ignored.
2. are small compared with the all-time temperature record being broken by 0.1 F.
3. are expected, as they are within the error bounds of our world-class temperature measurement network.
4. have been caused by some Koch-funded denier hacking into our computers.
5. are irrelevant to the global trend, since the U.S. is only a few percent of the global surface area.
6. will be re-processed and re-adjusted on an annual basis going forward.
7. are worse than we thought.
8. are exactly the sorts of extreme events that result from CO2 emissions.
9. result from a careful balance between being effective and being honest.
10. are specious nonsense from the oil-funded network of climate denier flat-earthers.
I’m of the opinion that historical temp records were recorded by individuals with better things to do and took the data with a grain of salt. If they needed to be in another county they took the reading when they could and either extrapolated or entered the data as it was.
Today we are still humans prone to diversions and intrigue. After all in the scheme of things how important is a number on a sheet of paper, if a six was a nine I don’t mind.
REPLY: Since NCDC says “contiguous” on their plotter page, I’ve updated the text to reflect that for consistency, thanks – A
I think you miss rogerknights’ point that maybe different parts of NOAA have different meanings for CONUS, thus that may be why the temps are different.
You can see that NOAA does appear to be confused, in their glossary they say: “CONUS – Contnental U.S.”
http://www.srh.noaa.gov/mrx/glossary.php#c
But this page: “Graphical Forecasts – CONUS Area” shows a map of the contiguous US.
http://graphical.weather.gov/sectors/conus.php
rogerknights,
CONUS can mean either:
http://www.acronymfinder.com/CONUS.html
When I was in the military, CONUS specifically meant the continental U.S.
Mr. Watts’ find would be incriminating if any accountability would exist in our government.
As there’s none, they will take obamacare of it.
Mr Watts, please see to it that your tea does not contain any transuranic elements.
Someone must have noticed the flipping down of the figure when the paper records were interpolated (by up to 1.1F which is huge in the light of the decadal trend claims).
What’s more, although I wouldn’t put it past them to copycat the Met Office with their press-scare-statements-that-defy-the-data, it really is questionable to be furnishing the entire economy, agricultural, industrial and commercial with junk data (i.e. to every last person in the land who just wants the right data to help haul us out of a recession. But that’s the consequence of spinning to the press.
Scute
James Sexton says:
January 6, 2013 at 6:59 pm
Hmm, in an effort to be charitable, is it possible we’ve two different people calculating the “average” in a different manner?
There is only one way to calculate an average. But you may be right and someone has made a simple arithmetic error. Easily done in computer code that isn’t thoroughly tested. And I doubt any of the climate agencies test their code properly, and I am sure the climate models, etc. are riddled with errors.
I’ll put money on NOAA blaming a software problem.
BTW, there are no software problems. Only, failures to test properly.
Looking into it more, it seems that CONUS is usually defined as the “continental US” (e..g DoD and NOAA) and does exclude Alaska. Thus there should be no confusion by NOAA (apart from they can’t spell continental 🙂
A joke comes to mind: Three people are vying for an executive position. An engineer, a marketer and a lawyer. There’s one last interview by the CEO.
One question: “What is 2 plus 2?”
The engineer states that while untrained people would say 4, in real life these numbers would have noise, distributions and ranges, not simple integers, so the answer is 4 plus or minus the range on each number. “Very precise” the CEO says.
The marketer states that mere linear thinkers would say 4, but that more creative types would consider 22 to be just as valid. “Very insightful” says the CEO.
The lawyer enters, closes the door and curtains, unplugs the phone and intercom, turns on a radio loudly, approaches the CEO and says quietly “What do you NEED it to be?”
The marketer and the engineer were not called back.
RE the argument over contiguous/continental and maybe how this caused the problem… The July 2012 SOTC http://www.ncdc.noaa.gov/sotc/national/2012/07
says:
The link they have embedded in the text (duplicated in the quoted paragraph above) go to the database plotter, which also says right at the top “Contiguous”, which matches the text quoted above.
Lies, damn lies, and climate “science”.
Don’t ask me why this popped into my head, but when I read Anthony’s post, I immediately thought that some NOAA employee playing “Foggy Mountain Breakdown” on a banjo could help deflect attention from this issue… much like Steve Martin suggested that Richard Nixon should have done nearly 40 years ago
January 6, 2013 at 6:31 pm | Glenn Sliva says:
Garbage in Garbage out. Answers are worked backwards not by NASA or NCDC but the media.
—————————
This is subject to the well known GIGO forcing, so GI = GO^3
I compared charted the 60 month period ending Nov 2012 and was a little surprised to see it listed as the 11th warmest! In other words, not particularly noteworthy last few years.
http://www.ncdc.noaa.gov/temp-and-precip/ranks.php?periods%5B%5D=60¶meter=tmp&year=2012&month=11&state=110&div=0
Heads are going to roll this time Anthony you maniac. Good Job, get some well deserved rest, you deserve it.
Anthony,
Yet again you deserve a clap on the back for great work.
Im not surprised that you found this data inconsistency. The people who mined this data for their own purposes knew what they were doing, but never expected the ignorant skeptical public to find and understand their manipulation. You did it and Cudos to you!
Unless a particular person of Congress, with considerable standing, picks up on his issue I fear it will die on the vine. However, if someone in the Budget realm, begins a formal inquirey, there could be traction. Particularly if expenditures for wind/solar power etc. are challenged against measured data.
If I didn’t know better I would think James Hansen worked on these reports. Of course we know he doesn’t. Right?
i bet you’re right…contiguous, continental vs. US average as a likely source of confusion. Or different groups using different data, and no one has bothered to check or compare reports, data and graphs….?
Good catch, Anthony. impressive
.
thisisnotgoodtogo says: January 6, 2013 at 6:40 pm
Anthony, Maybe you found the Janus set, not the Conus..
===========================
ha ha – can’t let this one go by unremarked: Janus, the two faced Roman God
Re: Saving NCDC pages as PDFs (good idea) and @Ric Werme comment at 6:50pm regarding NCDC pages: I find the NOAA / NCDC websites very frustrating in terms of links to information constantly disappearing – I find interesting data / information and then soon after it’s gone…. it would be nice if they were more organized (or maybe they are – organized to disappear information that doesn’t meet the consensus AGW view and to prevent a systematic review.)
Very impressive Anthony; thanks so much for all your great work!
I think this is the likely source of the mixup, somewhere along the line. (Or maybe only SOME of the mixup?) It would be like the mixup on that Mars mission between metric and Imperial units.
This is a great discovery. Is this related to the systematic ‘adjustment, homogenisation and other adjustments to the climate record? Keeping two sets of books is standard practice in criminal organisations which are intent on defrauding the public, the tax-office and the government. It all begins to make real sense, very slowly.
Well done.
I wonder if the NCDC computer storage devices have something like a HARRY_READ_ME.txt file hidden in a deep mountain vault?
Some might need a ref.: http://wattsupwiththat.com/2009/11/25/climategate-hide-the-decline-codified/
Reading your investigation that’s what came to mind. Well, the contiguous versus continental aspect made me think of Alaskan statehood (1/3/59) but that shouldn’t involve this recent data.
@Philip Bradley
There is most definitely more than one way to calculate an average.
Arithmetic Mean
Subcontrary Mean (harmonic mean)
Geometric Mean
When calculating the average of metrics that are ratios, the harmonic mean should be used if the frame of reference is in the denominator. For example if you average the fuel economy readings of a vehicle expressed in litres per 100 km you will get the wrong answer. Calculating the harmonic mean gives the correct answer.
Oddly enough, I would not ascribe this to malice. My guess is the press and the pols were pressuring them for numbers and so they started reporting when the bulk of the information was in, on the assumption that the rest of it would have the same or similar average. What I’m guessing is that the reports that come in by mail are from locations with no electronic access at all (otherwise why mail them?). That being the case, the early data comes from sites that are urbanized (read UHI inflated) while the late data comes from sites with no urbanization (hence no inflation). They didn’t go back to check their assumption as to averages not being changed much by the later data….. and ooooops! I bet that if you did a match between late data and category 1/2 sites, you’d get a strong correlation.
There was a thunder clap not too long ago, the sound of thousands of bureaucratic sphincter valves snapping shut. Wow Anthony, what a find!
It seems the Russians are coming up with the right answer-
http://www.slattsnews.observationdeck.org/?p=7008
and they more than any of us should know.
Averaging numbers without knowing their inherent distribution or the scope of their inherent error or precision is futile. Since virtually all land based temperatures prior to the 1970s are based on non-random, non-replicated single daily observations, Why would you expect to get anything other than anecdotal information (garbage) from it ?
We do not even know if the recorded temperature of ANY one day during that period is the ACTUAL minimum or maximum, the variance, standard deviation, size or type of errors, or almost anything except we have this one worthless number for minimum and one for maximum of unknown utility.
So we use it. We adjust the number(s) to fit our agenda or to correct perceived errors. But don’t have any illusions about the value of the data thus obtained.
I find it especially concerning that the lag in data making via mail for the areas which are more rural, get used to artificially warm the results. This is good thinking to consider that. The urban heat island effect is real… and used to benefit alarmists with an agenda.
Great job! Well done. 🙂
Don’t remember the specifics but in one of the climategate emails “they” conspired to release an early statement so the latter official report would hopefully be overlooked by the media. Anyone recall this email?
Can we start calling you Dr. Wattson now? 😉
Why don’t you report the DQA violations for inflation and unemployment while you’re at it….would simply be lancing windmills. It’s not just the data that’s been corrupted.
And in 2008 (B91 from Mohonk) they are/were still only measuring temperature to whole degrees Fahrenheit !!! And how much guesswork (confirmation bias, anyone?) does that involve when the mercury is mid-way between gradations?
Another perfect example of why WUWT has become so popular. Unbiased facts, presented as they fall, with readers shown the evidence and encouraged to draw they’re own conclusions as that evidence suggests. Anthony Watts continues to embarrass those who would have us believe in the contrived fairy story that is AGW climate catastrophe.
Why is it, whilst reading the Data Quality Act, I couldn’t help but keep thinking of one James and one Gavin. Has anyone been cheeky enough to send the DQA to the aforementioned gents?
Good work Anthony.
Crispin in Waterloo says:
January 6, 2013 at 10:07 pm
I was referring to the arithmetic mean, and while you correctly point out there are other means, the statement that ‘there is only one way to calculate an average (that is arithmetic or any other kind of mean)
is still true. Although of course, different calculations are required for different types of mean.
BTW, your example is wrong. If I were to average (arithmetic average) the fuel consumption of vehicles each of which travelled 100 km, I would get the average amount of fuel required to travel 100 km.
Amazing post mr watchdog. This is why mainstream media is dying and is replaced by blog reporting. The internet is only 20 years old, an already changing the world.
The GHCN has Zombie Thermometers that suddenly have data show up long after they seemed dead… and start walking the earth again. This means that any “average” gives slightly different values based on when the data were looked at and not just what span of time is chosen…
https://chiefio.wordpress.com/2010/02/15/thermometer-zombie-walk/
There is no standard average temperature. Then again, since we don’t know if temperatures have a standard normal distribution the “mean” may well be statistically undefined anyway:
https://chiefio.wordpress.com/2012/12/10/do-temperatures-have-a-mean/
Then again, since temperature is an intrinsic property, and the average of an intrinsic property has no physical meaning, any average of a temperature is kind of meaningless:
https://chiefio.wordpress.com/2011/07/01/intrinsic-extrinsic-intensive-extensive/
(One really needs mass and specific heat / enthalpy to have an extrinsic property – heat – to average and have meaning.)
But “it’s what every one does” even if it is meaningless…
One thing it does do, though, is (IMHO) offer pretty darned good evidence that the move to automated equipment “warmed” the series. The newer automatic MMTS are reporting “now” while the older slower are reporting later…
@Michael D. Smith:
Wow! That’s some chart!
@BioBob:
Glad to see someone else who “gets it”. BTW, at one time I found and online copy of the directions to the folks reading the temperatures and put up a link to it. Shortly after it was scrubbed / removed…. This was before I learned to screen capture EVERYTHING so the Data Langoliers don’t disappear it…. For years (decades? centuries?) the official guideline for how to read the thermometers said basically “If you don’t know, guess.”
It was encouraged to just make up what you thought the temperature was and write that down.
What kind of error bars does that put on the record?
@SimonJ:
At least it isn’t in whole degrees of C!
https://chiefio.wordpress.com/2012/01/21/degrees-of-degrees/
But yes. That’s why I keep trying to point out the absurdity of saying anything about temperatures to more than 1 F of precision. (Yes, you can remove RANDOM error via an average, but not SYSTEMATIC error And what we have is lots of systematic error. UHI, wrong way ‘adjustment’ for MMTS rollout, Stevenson Screen paint aging, etc.)
@Philip Bradley:
Except in computer programming the exact order of processing can change the result due to various number limitations and underflow / overflow / bit precision artifacts. Do you add all the max-min then divide? Or do each one one at a time? It matters…
An example here that warms GIStemp:
http://chiefio.wordpress.com/2009/07/30/gistemp-f-to-c-convert-issues/
100 billion dollars in US taxpayers money spent on climate science, and not a single climate scientists spotted this problem in the data? How many other data problems have they missed?
It is clearly a data lag problem, as per davidmhoffer January 6, 2013 at 10:12 pm The early data is what gets reported, and the later data adjusts the figures downward. Thus the most recent months show no problem.
This would not have gone unnoticed at NOAA – which does suggest malice in that they have not taken steps to correct the problem. Most likely out of fear of running afoul of the politically correct line and the threat to continued employment.
I understood that three sets of books were normally kept. The ones for the tax man, the ones for the accountant and the real ones. No soup for you until you find the real ones!
“Continuous” and “contiguous” are not interchangeable. Continuous is a time term. Contiguous is a space term. At present, I am writing a reply. “I am writing” is the present continuous tense of the verb “to write”. At present, Alaska and Hawaii continue to be part of the USA. They are part of the continuous USA. They are not part of the contiguous USA.
Time and Space are two different things.
Great work Anthony. Keep at them, and belated best wishes for the New Year from the old world.
So temps from the faster-reporting sites (airports, cities, etc) are higher than those from the slower sites (more rural). And when they’re all in, the temp goes down. Doesn’t that suggest UHI?
Typical of a large government bureaucracy. Probably nothing deliberately corrupt, although I won’t rule it out, just civil service incompetence and confusion.
Mr Watts, you are a machine!!!!
Definition of CONUS? I couldn’t give a continental.
But seriously… is there anywhere on the internet where I can get global historical temperature data in table form going back at least 20 years? Charts available on several sites but transcribing is laborious work and subject to error.
Anthony
This is an interesting story you have written about which strangely is related to a couple of articles I wrote. Firstly, I wrote about ‘Mohonk best in the Nation.’ Well, if that’s the best the US has got problems. The article starts;
“The Little Ice Age thermometers http://climatereason.com/LittleIceAgeThermometers/
which predate Giss and Hadley/Cru, provide an interesting insight into the longer term climatic cycles that the shorter records often seem to miss. This was demonstrated with Uppsala/Stockholm and Hohenpeissenberg in my article; http://noconsensus.wordpress.com/2009/11/05/invisible-elephants/
Today we examine another temperature triplet linked by the Hudson river, drop in to see James Hansen and Gavin, visit a shanty town and pay our respects to John Lennon. In other words my usual eclectic mix of history, trivia, science and serious investigation.
The first of our records comes from Mohonk, which in the world of climate science is a bit of a hero. It is a most interesting station, as this link demonstrates; ”
http://noconsensus.wordpress.com/2009/11/25/triplets-on-the-hudson-river/#comment-13064
Last year I followed this up with an article on the unreliability of the temperature record right from the days it became big business rather than a scientific endeavour-in the States that was around 1880 when many current stations were set up. The manner in which data was collected was roundly condemned at the time, including by a leading climatologist writing in 1900. That story was carried here
http://wattsupwiththat.com/2011/05/23/little-ice-age-thermometers-%e2%80%93-history-and-reliability-2/
—- ——
After some 5 years of writing historically based climate articles, the two things I have learnt is that the temperature record is a moveable feast that depends on who is creating it and what purposes they want to use it for, (there is usually an ‘agenda’ but not deliberate malpractice) combined with astonishment at the sheer unreliability of portions of the data.
I think a lot of the problem is that since the advent of the computer number crunchers like to play about with data and create models and scenarios and sometimes dubious data is then further manipulated. Some of the basic material on which far reaching studies are being based is frankly bizarre, there is no better example than SST’s which we believe we have a global knowledge of back to 1860 to fractions of a degree.
It will be interesting to see how your current investigation plays out, but Hubert Lamb-first director of CRU – had it sussed out by saying that when examining temperatures ‘we can know the tendancy but not the precision.’
tonyb
If you wany something done properly, ask a busy man. Well done Anthony!
If it is the case that that, as in most Govt. statistics (think employment numbers, GDP, etc) there is a preliminary estimate (which is hawked around and moves the media and markets) and then a set of revisions in subsequent months (which are ignored) then NOAA is guilty of not making it clear that the headline figures are provisional, and not making any announcement of the revisions. This is b-a-a-d.
It’s difficult to believe, after the billions spent on climate science, that the network still relies on pony express technology to communicate the figures. Even some domestic electric and gas meters can radio in their updates automatically.
Make you wonder what sort of mess 3rd world monitoring must be if the world’s high tech leader can’t get it right.
CONUS is the “lower 48” excluding Hawaii and Alaska and territories. This was, at least, the definition when I was in the military. Alaska and Hawaii were considered overseas assignments.
There is a notable lack of troll comments in response to this posting, I wonder why?
For me, this article was one of the most damning ever produced on WUWT. It shows either gross incompetence and/or a deliberate attempt to deceive by government bureaucrats and ‘researchers’.
However, the really scary things are:
1. The Global Warming Industry will refuse to acknowledge the existence of these serious ‘discrepancies’, likewise so will most of the media.
2. NOAA will not provide feel itself obliged to offer any explanation for this, as Anthony is obviously not one of their recognised ‘climate scientists’.
3. Much of the general public will remain blissfully ignorant on how bad the data sets (both raw and manipulated/homogenised) are which supposedly support the alarmist headlines of imminent climate disaster.
4. These revelations will be no more than a squeeky wheel for the global warming gravy train.
http://english.pravda.ru/opinion/columnists/04-01-2013/123380-global_warming-0/ someones really pissed off 🙂
As part of your “full investigation” did you happen to contact NOAA and ask?
Although incompetence and confusion typical of bureaucracies is usually the first excuse, how many working there have Green Agendas- are Greenaucrats and True Believers in The Cause? Plausible deniability only goes so far. Inquiring folks would like to know.
Be careful Anthony.
I build IT systems and I was tasked with putting together an application that reported on bestsellers across the globe. One of the bits of data I needed was ‘Selling Price’
Most of the fields had obscure names but luckily for me, there was a lovely field called selling price that had currency values in it.
Of course I got into deep water for not using the correct field – Selling Price is obviously not held in the field ‘Selling Price’, it’s held in the field ‘intLandedAdj’
Please make certain the NCDC db is not similarly confusing
Looks like the data is being “FIXED” (in the sense of fixing a fight). The article (and screen capture) has “July 1936 from the NCDC database says 76.43°F”, but now the database has it as 76.41 – someone at NCDC must have become suddenly aware of a problem.
It blows my mind that the same organization(s) that try and claim global warming is an issue and the numbers they talk about are quoted to high degrees of precision and accurac, can’t even get a months data correct for one set of stations across one country. And I’m expected to believe some trend of world temperature averages to less than a tenth of a degree Fahrenheit over centuries ? GMFB! ( Give me a ….break)!
I deal with data every day in my career. Even in the most controlled settings of data collection you get 1-2% RSD on a good day. Expand that collection over various sites, various people, transcribed forms etc. etc. and these types of discrepecies are not surprising. That’s why you can’t trust temperature measurements to better than a few degrees. This tenths of a degree analysis crap these guys engage in is total garbage science.
I can here it now; “…a simple rounding error.”
Floating NOAA’s Boat.
Dear Anthony, didn’t you know that NOAA was directed to bring 2 of each species ?
Although July was the “hottest” month evuh, only one state actually broke its record, Virginia with 79.0F.
At the time, I did an analysis of all the USHCN stations there, which showed that July 1934 had been much warmer.
http://notalotofpeopleknowthat.wordpress.com/2012/08/29/hottest-july-in-virginia-or-maybe-not/
I’ve now gone back to check the latest NCDC numbers, and lo and behold, the 79.0F has gone down to 78.8!
Unfortunately I did not keep a screenshot of the graph at the time, but the latest NCDC version is here.
http://www.ncdc.noaa.gov/oa/climate/research/cag3/va.html
Nigel S says:
January 7, 2013 at 1:03 am
The real ones have probably been shredded!
Thanks Anthony for the find and posting. I am going to reserve judgment until we hear what NOAA has to say about the disparity, but this in no way way trivializes your find. A little off topic and a bit naive, I found this GAO report on temperature records that was initiated by Sen. Inhofe and sounds like your station siting work was the catalyst for.
http://www.gao.gov/products/GAO-11-800
So, it appears that the NOAA knows they have accuracy problems but still they seem intent on making “hottest event ever” statements in lieu of this knowledge. Seems a tad bit disingenuous, but again I will reserve judgment as I don’t have the complete story.
Also of note, I just completed my first year statistics course which I chose to sample the temperature records for a single station in my home town of K. Falls, OR for my final project. I downloaded the daily TOBS record from the NOAA database for the decade of 1990-1999 and 2000-2009 for my populations. Wow, I was quite surprised at what I found. No data from 1990 till 1996, many days missing temperature records, and many readings of “0” which seemed to be inaccurate for the season, like somebody just put zeros in place of missing data. Now this is one station in an obscure part of nowhere but how can you get an accurate CONUS average from this kind of record? I am assuming this isn’t an isolated example…
Does this sum it up?
* NCDC records preliminary (incomplete) data and final (complete) data for each month.
* SOTC reports are based on preliminary data, both for current and historical temperatures.
* The NCDC climate monitoring web site uses final data, except for the most recent month (or 2 moths?) where final data is not yet available.
* NCDC never compares preliminary (warmer) data to final (cooler) data.
The only issue I see is that comparing the most recent month on the climate monitoring web site to historic data is misleading, because you compare recent preliminary/incomplete/warmer data to historic final/complete/cooler data.
July 2012 is still the warmest month, both in the preliminary data set (from the SOTC report) and in the final data set (from the climate monitoring web site).
It would of course still be a good idea to have an early preliminary SOTC report and a final SOTC report later.
mbw says:
January 7, 2013 at 2:51 am
As part of your “full investigation” did you happen to contact NOAA and ask?
___________________________
Howdy mbw,
If past is prologue, then Anthony has given NCDC a chance to respond to this article before publication. We’ve seen this sort of thing before.
Notice please, that more information is forthcoming. From the article:
“I’ll have much more in the next couple of days on this issue, be sure to watch for the second part.“
anthony –
in australia, we now have this claim to ponder:
8 Jan: News Ltd: Patrick Lion: National heat record expected by Bureau of Meterology
WEATHER analysis to be released today is expected to show Australia is sweltering through its hottest days in history…
The national temperature is calculated from about 700 weather stations across the country, but is processed as a mathematical interpolation instead of an average…
http://www.news.com.au/national/national-heat-record-expected-by-bureau-of-meterology/story-fncynjr2-1226549102215
Anthony, I suspect the discrepancy is due to “adjustments”. At one time, the “database” probably had 77.4 as the 1936 CONUSA value & the SOTC report used it. As you found, it no longer does.
Paul Homewood’s Aug. 29, 2012 blog noted the July 1934 Virginia temp “79.0F has gone down to 78.8!”. Check it now. The July 1934 Virginia temp is now down to 78.6! http://climvis.ncdc.noaa.gov/cgi-bin/cag3/hr-display3.pl
I don’t know what date you did your screen capture of the the July 2012 number highlight above as “76.93”, but it is now 76.91.
I’ve looked at data from many individual stations from the original observer sheets & it is very common for the earlier data to be warmer by up to a few degrees F. The historic temp data seems to be under constant “adjustment”. Perhaps due to some type of blanket algorithms.
Steve Goddard has observed adjustment issues in US temps that I have summarized elsewhere:
“Hansen99 ‘GISS analysis of Surf T change’ Plate A2(last page)(1 usa gov/MqZUhC) shows cooling across most of continental US of up to 1°C from 1950-1998.
Current GISS data shows not cooling, but warming up to 0.5°C. (1 usa gov/SAqNlN)
That was entirely due to adjustments since ’99 after the warming stopped!”
The historic temp record is a travesty. Who knows what the real temps were.
Philip Bradley said ‘There is only one way to calculate an average’. Alright, but you can do it with or without removal of outliers. Because the deviations Anthony found were positive and because temperature distributions are skewed with a long left tail, some extreme lows may have been removed. Are the reported means based on the same data?
I cannot count the number of times I’ve gotten onto my soapbox at work and stated “to be considered a document a written communication must at a minimum have three properties: a title, a date, and an author.” Date is important. I have seen plenty of time wasted in meetings because not everyone has the same version of the document being discussed.
It’s fine if the NCDC wants to issue a preliminary report early each month, but the report should be subtitled “preliminary — not all stations reporting — final report to be released …”.
While it’s tempting to infer an intent to mislead, I see this kind of negligence regarding dates on documents all the time — including online “news” stories. All you have to do is add the well-established laziness of the press and it’s easy to see why this happens. However you’d think someone at NCDC would have the integrity to warn the press that the report is only preliminary is is likely to be adjusted down when all the stations reports are factored in.
Good work catching this.
Nice catch Anthony, congratulations. Only diligent, hard work turns up this sort of thing, with or without the serendipity.
Ric Werme says: January 6, 2013 at 6:50 pm
There is a new, few years old, cooperative network for recording daily precipitation. Unlike the old mail-in paper system we did in the 80s, this is updated by computer entry. You can view maps or pull down the data for a state. If you pull down a data list, note the time of observation, then go to the last entry for that observation time. That last entry for a given time is the highest ranked, most, precipitation reported for that time. Then one can check the few different times to find the highest value amoung the different times for a date. That is the easiest manual search for the most precipitation that I know of. I don’t know how they are integrating this information into ‘the’ national database, the project, CoCoRAHS, is funded by NOAA and NSF. I think there are 30K+ observers.
http://www.cocorahs.org
OT
Today is 70th anniversary of death of Nikola Tesla, one of the world’s greatest inventors http://www.teslasociety.ch/info/ton/Fiorello_La_Guardia_uber_Tesla.wma
DQA? hahahahahahahahah….whew! You forget, this is gummit at work here. still using paper forms! The same gummit that can’t find my mail after my vacation and can’t fire bad cops or teachers.
It seems to me that the more quickly reported data is likely to come from more modern sites that are more likely to show UHI effects. This could be wrong. I haven’t gone through the sites and i can’t swear to it. But it seems intuitively likely and it would explain why the preliminary report is usually warmer.
A political system in which “we the people” are losing faith … a financial system that “we the people” now know is diseased with debt and derivatives, no thanks to the greedy “international money changers” … a Federal Reserve that “we the people” now know is unconstitutional and more powerful than Congress and doing the bidding of its banking cartel owners … a fractional reserve banking system that “we the people” know is fraudulent. If that’s not bad enough, now “we the people” cannot even have faith in the scientific integrity of a government agency, NOAA. What is happening to the United States of America?
Clicking on the link above for July 2012 I see slight differences in the reported data than in your screen capture. In all instances from 2002 to 2012, the current plot shows the data is 0.01 to 0.02 degrees cooler than your screen capture. That suggests to me that, in addition to the technology gap issue you pointed out, there is also estimation going on in the NCDC algorithm that is dependent on the data as it comes in from the field. I’m guessing this estimation process is similar to the one employed by GISS, and will cause the historical averages to always exhibit dynamic behavior.
Simple GISS example: Suppose the record for June 1948 for East Foobar, Iowa was never recorded. In order for a CONUS average to be calculated, that missing record would be estimated from the existing records and trends for East Foobar. As data is added to the East Foobar record over time, the estimate for June 1948 will change – not necessarily by a lot. If you have enough missing records in play (and hence a lot of estimation going on), the addition of records to the database at the end of each month will noticeably change the averages for prior years and months.
Is this incompetence or malfeasance?
It’s irrelevant. Both are grounds for dismissal, which is what should happen immediately to Tom Karl, head of the NCDC.
They’re reading thermometers for God’s sake, not tea leaves! If they can’t read a set of thermometers and average the numbers, it’s time to fire them all! Competent third graders could do a better job! And this is not a ‘living data set’ (as Mosher would have us believe), these are observed, fixed measurements that shouldn’t be diddled with ad nauseum. There have been too many wonky, adjustments and ‘errors’, all with the effect of cooling the past and warming the present. This smacks of hindcasting, where climate modelers twiddle the variables until the desired result is achieved. This is inexcusable!
My suggestion is that all the WUWT readers in the USA send an e-mail comment on this to the House Committee on Science, Space and Technology
The e-mail form is at:
http://science.house.gov/contact-us/email-us
The website: http://science.house.gov/press-releases
So are there three potential average temperatures; contiguous, continental, and including Hawaii? Perhaps the database is continental, and therefore lower because of Alaska, and SOTC is contiguous. Not sure where Hawaii would fit in.
The last two numbers are correct because of the timing of the arrival of data. Unless their is a convincing explanation from NCDC, it looks fiddled just to ensure a new record over that pesky 1934 which is the bugbear of the CliSci fiddlers.
How can surface station temperature measurements made by a ‘cooperative network’ possibly be trusted to be accurate to within 1/10th of a degree? It seems to me that there should be error bars of at least 1 full degree whenever temperature is measured by a surface station.
Well spotted Anthony.
To put it into something like the language of Yes Minister...
Confusion about the exact meaning of words (like CONUS, continental and contiguous) aside;
and presuming the most innocent explanation
i.e. that the difference in the two sets of figures stems from:
1. the late reporting of older type stations; and
2. failure to update the record after the late reporting;
and presuming that stations that report electronically also have newer designs of thermometer etc.;
and as one set of figures is consistently higher than the other;
… does this suggest that the newer-design stations (either because of their design or their typical locations e.g. at airports, which have the money for the latest kit or some other UHI issue) have some sort of bias for higher readings (relative to older ones)?
———————–
I would be interested to see if they issued any caveats (e.g. “this is preliminary data, subject to change”) anywhere on their website or in their press releases or similar. If that were the case, they could try and blame the sensationalism of the press for any misreporting. But it wouldn’t let them off the hook for not updating all their records with the correct information.
Excellent detective work Anthony,seems the SOTC report if for the media and media, only to then shout it around the global news rooms with the desired effect!
Lets see how they wiggle out of this..”perhaps they could get Richard Muller to look in to it and declare all is well” sarc off
Taken a copy of the comma-delimited data into a spreadsheet for subsequent analysis – select previous twelve months (plotted) but clicking on the comma delimited data below the graph gives the full data set from 1895 01. Presumably there will be an update to the data over the next few days, and the differences throughout the whole record can then be examined for any pattern or systematic adjustments.
P.S. The “Display comma-delimited data.” option is much easier than cutting and pasting individual values – which Anthony probably just did as a consequence of starting with the SOTC reports, but if the whole dataset had been his starting point, this issue would not have come to light.
It’s become clear to me that the temperature records (I have copies of NCDC’s GSoD, CRU’s, Bests and recently I downloaded a copy of the work V Venema has done) are not all that good, lacking good coverage prior to ~1972, Adjusted to remove real UHI, linearized non-linear temps over area, and that no one cares about any of it (except us).
Maybe in 5-10 years if temps keep falling someone will take notice, but even that leaves me flat.
I think the only way to make progress is to change the rules of the “game”, that average temps increasing are not caused by increases in CO2.
I’ve been extracting a daily difference between day time rise to daily max, and the following nights drop to daily Min. This shows no loss of nightly cooling.
I’ve also just picked up a non-contact IR thermometer to see if I can find any evidence of CO2 impacting surface temps.
We have to counter the meme that CO2 is the cause of rising temps, or we’ll have to wait until nature gives us uncontroversial evidence.
davidmhoffer says: @ January 6, 2013 at 10:12 pm
……What I’m guessing is that the reports that come in by mail are from locations with no electronic access at all (otherwise why mail them?). That being the case, the early data comes from sites that are urbanized (read UHI inflated) while the late data comes from sites with no urbanization (hence no inflation)…..
>>>>>>>>>>>>>>>>>>>>>>>>
Exactly what I was thinking.
Now take it a step further.
We all know and so do they, that the early sites are “UHI inflated’ because there has been a whole lot of ruckus about UHI and papers written. Therefore ignorance of that bias can not be considered any excuse especially after the first couple of times the final information lowers the reporting number. Heck even a CYA statement that all the data is not in and the number is likely too high would be acceptable but they did not do it.
A quicky refresher on UHI:
James Hansen investigated the UHI effect using the most recent data and methods as of 2010, (see Page 4). http://pubs.giss.nasa.gov/docs/2010/2010_Hansen_etal.pdf
Of course the ‘Rural Stations” are often airports…..
Dr. Roy W. Spencer did a study a couple of months before Hansen’s (Surprise!) Global Urban Heat Island Effect Study: An Update, March 10th, 2010 And made further updates WUWT 2012
2011 – 12 was the only one that was lower (by 0.3). 35.3 instead of 35. The rest are all higher. I too seriously doubt it is an automated error. There is no plus or times “x” type pattern that I can see.
Good sleuthing as always. It will be interesting to hear their reply.
REPLY: I believe I can demonstrate a physical mechanism for this flip. That will be in part 2. – Anthony
Hans H says:
January 7, 2013 at 2:32 am
http://english.pravda.ru/opinion/columnists/04-01-2013/123380-global_warming-0/ someones really pissed off 🙂
________________________________
Some ’round here found that link through other sources and along with other links in Pravda, alarms have been raised… can the leopard change it’s spots?
The conspiracy theorists believe that Pravda is just one player in a huge agitprop operation, designed to drive world citizens (read: armed US citizens) into just the sort of rebellious state which would offer the controlling masterminds the opportunity to seize ultimate control under the auspices of emergency powers.
Some would argue that the US gov’t foments enough rebellion all by itself.
Many rational people take conspiracy theories with the same grain of salt that they take with Pravda, or any other modern journal.
Confusing…but if you stick to the NCDC climate database plotter and compare the two numbers isn’t 2012 hotter than 1936?
REPLY: Yes, but I also demonstrated that NCDC is showing two sets of numbers for July 1936. So which one is real? by the 77.4 number they cite in the SOTC in July, the current temperature out of the plotter is cooler. The issue is “what is the real temperature”? – Anthony
I just tweeted the following to Matt McGrath, environment correspondent for the BBC and famed here at WUWT for reporting so breathlessly on the warming of the West Antarctic Peninsula a couple of weeks back:
MattMcGrathBBC Matt, can’t wait for your piece on Anthony Watt’s discovery that NCDC have touted inflated temps by up to 1.1 degF for years
zz says:
January 7, 2013 at 3:11 am
Looks like the data is being “FIXED” (in the sense of fixing a fight). The article (and screen capture) has “July 1936 from the NCDC database says 76.43°F”, but now the database has it as 76.41 – someone at NCDC must have become suddenly aware of a problem.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
WUWT is watched like a hawk by the other side. The best way to embarrass skeptics is to ‘Disappear the evidence’ and then say Anthony is lying and they know it. That is why screen capture is so important.
REPLY:
This is probably simply due to new data being adding this morning. One COOP station, which may have sent B91 in mail very late, combined with delays due to holiday, could do that.My mistake, I thought you were discussing 2012, not 1936. Obviously no B91’s in the mail for 1936. – AnthonyAfter a bit of digging, perhaps the temporal discrepancies also involve the switching from USHCN v1 to USHCNv2 to USHCNv2.5 (the latter apparently being reprocessed every day, and thus continually in flux). See here:
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/
It seems a difficult task to proclaim records when data re-processing tomorrow (or anytime in the future) could make such a proclamation false. Perhaps NCDC ought to simply refrain from making such statements considering the nature of their (continually changing) data product.
-Chip
Bad Apple says:
January 7, 2013 at 6:38 am
How can surface station temperature measurements made by a ‘cooperative network’ possibly be trusted to be accurate to within 1/10th of a degree? It seems to me that there should be error bars of at least 1 full degree whenever temperature is measured by a surface station.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
SEE: A. J. Strata’s error analysis
As stated in my previous comment, I don’t think the change to July 1936 of 0.02 degrees is due necessarily to the addition of a single B91. Rather, I think it is due to the method in which NCDC estimates missing data. As many B91s are added to the record each month, the estimates are updated, and the effect ripples back through the record. The averages being reported, therefore, are never static.
As an aside, I’ve always found it odd that the averages are reported to two decimal places when nearly all of the measurements taken are reported as integers. At the very least the rules of arithmetic significant figures would limit the reporting of averages to integers, correct?
REPLY: I mistook what year was being discussed, my error. As for integers, the “law of large numbers” is claimed for the precision – Anthony
I noticed on the form that temps only had two digits of precision, but the final averages were reported with 4 digits of precision. Having a few brain cells remaining that were trained with slide rules, my BS alarm is ringing!
pat says:
January 7, 2013 at 5:13 am
BwaaHaha – that’s quite funny! Why don’t we take a temperature at the north pole and one at the southg pole and interpolate between the two? This would, of course, show the average earth temp to be -20degC or something equally stupid! Mathematical interpolation my ar$e! It amazes me how they can find different ways to hype up anything AGW related!
The figure Gail is talking about is the average for 1936. Data for that should not be arriving now!
The fact that it is decreasing seems to be some sort of auto adjustment to ensure that it doesn’t exceed the current “record”.
Continuously adjusting history is really not on!
John Goetz says:
January 7, 2013 at 7:49 am
re: reporting to integers;
Hmm.. Yes and No – I totally agree with your frustration at the temps being reported to 2 decimal places, but this is perhaps a combination of the old and new data measurements (e.g integer readings and readings (digital) to say 1/10th of a degree) – but also a facet of the data processing. Sum a few hundred integers and divide by the few hundred and you will get a xx.xx result! That result is entirely valid in itself. What is not valid is to fail to report the +/- figure associated with that result – in this case, say at least +/- 1degF ? (if we were talking about the whole degree F of readability of a manual thermometer – and I’m ignoring any instrument error, which should be added on top!)
just as an aside – re my earlier comment to John Goetz – and that to pat – I am surprised the warmista don’t actually use the error margins in their reporting. For example, if we have a global temp of 15.5 degC +/-1.2 degC – they could report that as a value of ”14.2 to 16.7 degC” and then subsequently use the higher figure to alarm us all further! Oh dear, I hope none of them are reading this! Mind you – as I said, nothing they do to hype AGW amazes me anymore. I’m still waiting for an official (and reasonable) explanation as to the change from 15degC baseline average earth temperature to 14degC (and indeed, who, what, why and where they decided to pick the baseline in anycase!)
Steve Keohane says:
January 7, 2013 at 5:29 am
Ric Werme says: January 6, 2013 at 6:50 pm
I’m a member of CoCoRaHS, station NH-MR-33. They haven’t been around long enough (Colorado was first in 1998, New England states were added in 2009) to get the record storms I wanted (especially 1978).
They track new snow between morning obs, but you can only search on liquid equivalent. I wanted the 24 hour snowfall, something the NWS maintains, but mainly derived from a not too regular observing schedule.
CoCoRaHS data is not being added to NCDC records, it is used primarily for studying drought and heavy rain events. While they get some funding from NOAA and NSF, they are a non-profit corporation with few, if any, fulltime employees.
I am considering using it for some of my Snow Depth Day reports at http://wermenh.com/sdd/index.html but won’t have time for a while.
If a few late reporting stations can change the average that much, how can you possibly compare temperatures reported in 1936 to temperatures reported in 2012? How many of the 1936 stations have been dropped? How many new stations have been added since then?
They are just comparing the average apples to average oranges.
Back in the old days, once the polls closed on election day, it took time for ballot boxes to arrive at the county courthouse for tabulation. The “early boxes” were always urban, the courthouse being in the center of town, and so were liberal. The later boxes were always suburban, and more conservative. They took longer to arrive, physically, for counting. So, election after election, the results from early boxes would make it look like we liberals were doing better than expected.
Then, the suburban boxes would temper the emerging results.
No one was ever foolish enough to make any sort of “Dewey Beats Truman” press announcement based on early boxes.
It is also weird that the red Feburary trend line has an actual slope (measured from the screen) from 32.4F in 2002 to 34.9 F in 2012 which is 2.5 F per DECADE yet it is marked as 2.77 F per CENTURY. Or am I missing something which is obvious to all the others commenting here? Or is the trend line not derived from the 10 years of data displayed on the graph?
Gary D. says:
January 7, 2013 at 6:22 am
“Not sure where . . .”
Keeping track of the States is difficult. Look at this map:
http://mappery.com/map-of/United-States-Map-2
See, that’s why you can’t drive to Alaska – it’s an island! They also send their B91 forms to the mainland by ship.
Okay, you laughed or not, but some years ago a young man on TV expressed surprise upon leaning that Alaska was not an island and that folks actually drove there. He probably works for NCDC.
Physics Major says:
January 7, 2013 at 8:43 am
….. How many of the 1936 stations have been dropped?
>>>>>>>>>>>>>>>>>>>>>>>>
On the station drop out issue see:
Graph dT vs Station count (global)
The ‘Station drop out’ problem This looks at the station count by region. Posts on either side of this one go into further analysis
The DQA does include a statement: “…establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated by the agency that does not comply with the guidelines…”
While I understand that this may be tilting at windmills, have you considered using the DQA to rub their noses in the fact that they have two different values reported for what should be the same temperature and they both can’t be correct (i.e., they don’t comply with the guidelines)?
thelastdemocrat says:
January 7, 2013 at 8:45 am
But now, a mere 35 years later, with all inner cities and urban counties under the firm control of liberal (immoral, corrupt) governments protected by liberal immoral corrupt mass media cronies and corporations with a monopoly on what they choose to report and who they choose to investigate and how they choose to report (or hide!) the results of those investigations…
It is the corrupt inner city and urban county precincts that wait until the honest suburbs have “counted” their votes. THEN, when the needed total is determined, the inner city corrupt precincts report. Dead last. But enough to win. In Houston, vans carrying the urban vote were found circling the freeway on election evening, waiting until the surrounding polls reported, waiting to know how many votes were needed to cover the (more conservative) suburban votes that had been already submitted. It is the corrupt urban precincts who fight voter identification laws. Who demand on-place, same day registration so people can be vanned from one precinct to another voting as many times as needed. Who refuse audits. Who intimidate voters and threaten poll watchers.
True, they cannot influence every election. But they can affect any election closer than 1% difference between candidates. And have. Deliberate gerrymandering – adjusting precinct and representative boundaries to get “safe” districts of racially proper voters who can be assured of voting liberally each time makes it even easier to skew the results – to the point where several Philadelphia precincts cast tens of thousands of votes – all for ONE single candidate of the favored party. (More seriously, several districts are reported to have submitted more votes cast for that one candidate than they had registered voters. But our ruling party’s monopoly media of ABBCNNBCBS refuses to even investigate that issue to determine whether the reports are true. Sobering.) Seriously, of tens of thousands of “voters” in scattered across several precincts in a city not even ONE person voted against their party bosses in these urban districts?
Peter Miller says:
January 7, 2013 at 2:09 am
There is a notable lack of troll comments in response to this posting, I wonder why?
=============================
Exactly what I was thinking.
;- )
I saw a comment above where the commenter stated that while he was in the military CONUS meant Continental United States. I beg to differ. CONUS, at least in the Air Force meant Contiguous United States. Alaska has always been considered to be “overseas”. Hawaii, too.
“How can surface station temperature measurements made by a ‘cooperative network’ possibly be trusted to be accurate to within 1/10th of a degree? ”
This is one of the common misunderstandings of what it means to report that the “average” temperature is 14.546327, when in fact your data points are only measured to a degree accuracy.
When we say the “average” is 14.52 what are we really saying? technically, the ‘average’ is not a thing, does not exist. Its math we do. What it means is this. ..
lets imagine that we have a barrel of big apples. And you have a scale that can weigh things to a ounce. You pull out 100 apples randomly and weigh them. 50 of them weigh 9 ounces and 50 of them weigh 10 ounces. what is the average? well, we say the average is 9.5. whats that mean?
heck my scale was only good to the ounce..
Here is what that means. Suppose god came along. And god said. I measure perfectly. I want each of you to give me an estimate of the exact weight of the next apple to be pulled from the barrell. closests to the truth will win.
Simply put the “average” temperatures reported are the best estimate ( smallest error ) of the temperature taken at a random location. That average is created by applying math to measures taken at known locations.
The “average” is an estimate that minimizes the error.
Anthony,
Good and persistant detective style follow up on your initial happenstance finding of a discrepancy.
Some observations:
– I will assume that this site is well monitored by virtually all: environment media departments, IPCC endorsing science blogs, university climate departments. Then I would expect, wrt your post, to see in a response pattern from them all that would provide indicators about their relationship with the NCDC (NOAA) leadership. Look forward to their comments and detecting any patterns.
– you need more of everything, have you considered getting associates together to organize and manage for you an association or institute or foundation or academy or etc? You could do much more by that concept with your current energy level.
– have some associates arrange and manage for you some co-authors to guide / support you on a book on the history of the climate science dialog.
Thank you for all you do.
John
I just went to the Contiguous U.S. Temperature July 1895-2012 link you put up and now none of the temperatures are the same as either of your screen shots. Almost every year is different.
2012 is now 76.92 & 1936 is now 76.41 ?
Just in case it was some rounding / math issue with Javascript, I checked the source code & then checked the page in both IE & Chrome… the data for the comma-delimited data is distinct and matches those of the plot. So, in the 2 days since your post it has changed yet again… for all years apparently?
Doug Proctor says:
January 6, 2013 at 6:40 pm
The corruption in CAGW lies not in bad data, incomplete analyses or agenda-driven adjustment biases (I’d call these moral or ethical failures) but in the insistence that reported data, statements about conditions, are unequivocally correct, accurate and precise to a fine level. The corruption lies in the claim of certainty, of being above discussion, internal dissension and external reproach.
————————-
Also known as employer review by the citizenry whose lives are spent making money to throw at real research, not thinly veiled crime
Jim Sefton says:
January 7, 2013 at 9:51 am
If that’s true – I hope Anthony ‘retraces’ his steps and takes further screencaps!
Anthony,
The answer was clearly before you on the NCDC “About Us” page as seen on the screencap you provided:
NCDC is well positioned to respond to this need by building upon sixty-one years of data and customer-focused science, service, and stewardship.
That 1930’s stuff is clearly far too old to be data. At best it’s an educated guess. Freely ignore it as unscientific hearsay as NCDC has no data older than 1951, when as it says on the About Us page:
Got it? There is no meteorological past older than 1951, when the NWRC first created data.
We now return you to your regularly scheduled reality.
Jim Sefton (Jan 7, 2012 at 9:51 am),
As I mentioned in my previous comment, apparently the NCDC national temperature data product (the complete history which is built from the USHCNv2.5 data) is ever-changing (even on a day to day basis). See here for the NCDC explanation/description/admission of this: http://www.ncdc.noaa.gov/oa/climate/research/ushcn/
Specifically, here is NCDC’s description of the USHCNv2.5 “Re-processing frequency”:
“The raw database is routinely reconstructed using the latest version of GHCN-Daily, usually each day. The full period of record monthly values are re-homogenized whenever the raw database is re-constructed (usually once per day).”
-Chip
Keep your eyes on the way they measure the tempatures.
They will find a way to rig the instruments that mesaure the tempatures.
Great lies to cover up ever larger lies.
Is it any surprise that the reported data is systemically skewed to higher values? I think not.
Perhaps a double post. My first one locked up. Please delete the extra.
So much for the CAGW climate guru grubers being able to calculate temperature trends to the millijoule…
A favorite topic here, often discussed in portions and shown as crucial in Anthoney’s station project is DATA QUALITY! As in; what should we ask interested legislators to enforce as NOAA data quality!?
No data base should have adjusted data overwriting original data. EVER!
Data should be shown as originally collected. ALL adjustments along with criteria for adjustment are maintained separately. This meta data repository allows researchers to better understand any datum’s history.
Database keepers should be alert to sudden changes to data that signal problems with the data collection, changes to environment, equipment malfunction, etc. And like all data managers, they should squawk loudly when those miniscule unknown changes creep in… Not that the unlawful JH would allow it; but seeing “JH mandated change” in the meta data would simplify everything.
Yeah yeah, piss whine moan and all that; but any governmental investigation should be guided by the best data management rules extant.
*–*Can you tell I once worked with data keepers? I can’t even add up the times they showed up in my cube demanding to know, at once, who they should talk to about missing or incorrect metadata… Maybe if I hadn’t tried to forget every interruption every day with the aid of tequila or Vodka & tonics (lime wedge of course). Data keepers work is crucial, even if they did squawk very loudly. ;->*–*
I last downloaded the Monthly US data in September (finalized August numbers).
At that time, July 1936 was 77.43 F
But, EVERY month has now changed considerably – down that is. It looks like they implemented some new algorithm that has gone wrong. There are huge changes which vary by the season.
Have a look. Changes from the August 2012 version to the December 2012 version.
http://s9.postimage.org/juyx0pjzz/NCDC_Changes_in_US_Conus_Augto_Dec2012.png
.
Dear Anthony,
I am probably being a bit thick here, but:
In your third graph, the average temp for 2011 is 76.29 and for 2012 76.92. How can the average be above 76 degrees, when none of the data plots go above 74 degrees?
Thanks.
NEWS – See this important related story:
http://wattsupwiththat.com/2013/01/07/announcing-the-first-ever-conus-yearly-average-from-the-climkate-reference-network/
[snip. Read the site Policy. — mod.]
Anthony, neither you nor the other commenters have mentioned that the hottest July, as listed above, prior to 2012 was 2006 @ 76.53. Curious, no?
Bill Illis says:
January 7, 2013 at 10:43 am
Bill – That’s astounding! If what you are observing in the plot can be verified, I think you need to post something here at WUWT ASAP!
I received this snippet a short while ago, for EU Cabbage insert “To measure the average US temperature.”
Pythagoras’ theorem – 24 words.
Lord’s Prayer – 66 words.
Archimedes’ Principle – 67 words.
10 Commandments – 179 words.
Gettysburg address – 286 words.
US Declaration of Independence – 1,300 words.
US Constitution with all 27 Amendments – 7,818 words.
EU regulations on the sale of cabbage – 26,911 words
Re: NCDC People must also understand that NCDC re-adjusts their database each month. With every passing month, temperatures older than 1945 or so are adjusted slightly cooler and temperatures after that are adjusted slightly warmer. So if you go and look at the database values for, say, 1933 and 1998 and compare them, then go back and check them again in a year’s time, you will notice they are different than they were a year ago. 1933 will have grown cooler over the past year and 1998 warmer. This graph shows the accumulated total monthly change in NCDC values since 2008:
Months in red have been adjusted warmer since they initially appeared in the database, months in blue have been adjusted cooler.
http://climate4you.com/GlobalTemperatures.htm#NCDC MaturityDiagram
Its frankly amazing how by ‘lucky chance’ these mistakes always end up favouring ‘the cause ‘ , makes me wonder seeign how ‘lucky’ they are why they never do the lottery .
If you follow the link I gave in my last comment and scroll down, an example is given for how temperatures for two dates have changed in the database over time. These are for January 2000 and January 1915. In may 2008 the difference between these years was 0.39C. In November 2012 the difference had grown 0.51C. There was more than 0.1 degree of additional “warming” applied to the database AFTER the data were initially entered.
http://climate4you.com/images/NCDC%20Jan1915%20and%20Jan2000.gif
“This begs the question, how can the temperatures of the past be changing?”
Hi Anthony,
I believe the answer to this question is that the historical QFLAG values in GHCN daily dataset are being modified over time for “quality assurance purposes” which results in temperature summaries for a month being modified based on when the monthly/yearly summaries are done.
I recently did a record by record comparison of a copy of ghcnd from a copy of ghcnd all vs. the most current version vs a copy I’d downloaded last july to check for changes to the historical record. I haven’t gotten through the entire dataset yet, but a sample shows that about 1/4 of the stations I’d compared had modifications to their records. This involved both changing good records to bad and bad records to good. On occasion the values actually do change as well.
Interestingly, there is still an effort underway to add in historical data to ghcn. The record for Benton AR was added to between this July and January (records all the way back to the 19th century) .
I think great liberties are taken in the name of “quality assurance” that truly modifies the historical record. It would be nice if NOAA documented this better and the reasons were were provided with the changes.
Steven Mosher says:
January 7, 2013 at 9:30 am
“How can surface station temperature measurements made by a ‘cooperative network’ possibly be trusted to be accurate to within 1/10th of a degree? ”
Simply put the “average” temperatures reported are the best estimate ( smallest error ) of the temperature taken at a random location. That average is created by applying math to measures taken at known locations.
The “average” is an estimate that minimizes the error.
++++++++++++++++++++++++++++++++++++++++++++++++++
Really ? Sometimes I wonder about you Steven.
You know very well that generally the average or mean of daily NCDC temperatures has absolutely no relationship to the actual mean of any day’s temperature. The values averaged are typically one minimum and one maximum observation. This does NOT estimate the actual daily average temperature because the population of daily temperature virtually NEVER forms a symmetrical normal distribution, which would be the ONLY population distribution yielding an actual arithmetic mean that matched the average of the two observations = min+max/2.
However, we can never know the actual daily mean temperature from the vast majority of surface stations over the vast majority of their observations because:
1) the daily sample size of one min and one max (N=1) means that there is not enough data for anything requiring statistics since the size, type, or magnitude of errors is unknown and effectively infinite
2) the data is drawn from non-random, non-replicated observations so the magnitude of errors is unknown and effectively infinite
3) any one particular day does NOT have the same population of temperatures as ANY other day and therefore parametric statistics CAN NOT be employed despite standard pseudoscience usage.
4) the limit of observation of the liquid in glass instruments (+-.5C) exceeds your stated precision
Furthermore, the central limit theorem has absolutely no business being employed on these data since it REQUIRES a mean of a sufficiently large number of independent random variables, each with finite mean and variance.
Probably need to change that first link I gave to:
http://climate4you.com/GlobalTemperatures.htm#NCDC%20MaturityDiagram
Warmers apply Finagles Theorem liberally.
Here is the posting I created last night on Anthony Watts’ important announcement:
http://tinyurl.com/bxdr67j
Tom Harris, B. Eng., M. Eng. (Mech.)
Executive Director,
International Climate Science Coalition (ICSC)
P.O. Box 23013
Ottawa, Ontario K2A 4E2
Canada
http://www.climatescienceinternational.org
613-728-9200
Do they use a version control system and a configuration management plan for dealing with the data? One should be able to get previous versions very easily with a nearly perfectly traceable history.
RACookPE1978 says:
“It is the corrupt urban precincts who fight voter identification laws. Who demand on-place, same day registration so people can be vanned from one precinct to another voting as many times as needed. Who refuse audits. Who intimidate voters and threaten poll watchers.”
Thanks for the cred! Our efforts and ingenuity are so under-appreciated.
But seriously, when I discuss this stuff with my liberal friends from party politics of the last few decades, it is like talking to cult members. Whether it is voter fraud or global warming. There is some serious Kool-Aid being drunk.
Heartfelt thanks, Anthony.
John Whitman says: January 7, 2013 at 9:37 am
Anthony,
Good and persistant detective style follow up on your initial happenstance finding of a discrepancy.
Some observations:
– I will assume that this site is well monitored by virtually all: environment media departments, IPCC endorsing science blogs, university climate departments. Then I would expect, wrt your post, to see in a response pattern from them all that would provide indicators about their relationship with the NCDC (NOAA) leadership. Look forward to their comments and detecting any patterns.
=====================================
A litmus test. I enthusiastically endorse this idea.
TomR,Worc,MA,USA says: January 7, 2013 at 9:13 am
Peter Miller says:
January 7, 2013 at 2:09 am
There is a notable lack of troll comments in response to this posting, I wonder why?
=============================
Exactly what I was thinking. ;- )
===============================
Shall we send out some invitations?
If this is true that they adjust down later why does Hansen always adjust up
cn
Please send this to Washington ASAP! We want answers
Congratulations, Anthony. Very cool!
@Pete Miller:
I use “the sound of silence” to indicate when they have been ‘caught out’ on a topic and just want it to die quietly. Works well…
@Oldfossil
Don’t know if anyone else has answered, but you can download the USHCN and GHCN data sets directly. I have directions up at my site, plus a lot of tabular reports up to about 2009 or 2010. Start here: https://chiefio.wordpress.com/gistemp/
and get data directions are here: http://chiefio.wordpress.com/2009/02/24/ghcn-global-historical-climate-network/
but it’s a bit dated as they have a Version 3 data set now.
Same places, though. So, for USHCN: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly
was it, now it’s: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/
which GHCN has gone to: http://www.ncdc.noaa.gov/ghcnm/v3.php
as they play their endless game of walnut shells and the data…
@Mosher:
“The ‘average’ is an estimate that minimizes the random error but does nothing about systematic error.”
There, fixed it for ya’…
(For example, if you ‘adjust’ for the MMTS change by a wrong number, that will be ‘systematic’ and no amount of averaging can remove it.)
@Jaye Bass:
No. There is some evidence that GHCN might be moving toward that direction in v3. V2 changed day by day with no history…
@Chuck Nolan:
The past is adjusted down, the recent goes up…
@BioBob:
I’m so glad that someone else “gets it”. Thanks just for being there…
https://chiefio.wordpress.com/2012/12/10/do-temperatures-have-a-mean/
There are so many wrong assumptions about what can be done with an ‘average’ that it is just sickening. Basic science and math skills sorely lacking in the warmers. An average can never remove systematic error, and most of the error in the data looks systematic to me. A mean has no meaning on some distributions, and the temperature distributions are not shown to be ‘standard normal’. It is an intrinsic property, so averaging has no meaning anyway. Sigh…
https://chiefio.wordpress.com/2011/07/01/intrinsic-extrinsic-intensive-extensive/
But I’m now happy knowing at least one other person was awake during stats and chemistry / physics classes 😉
BTW, I’ve also heard CONUS = Conterminous United States. Similar to contiguous, but a bit more sweeping. (and that’s my annoying post for the day 🙂
Good to see the number of people following the GHCN data changes on Humlum’s site and others.
It’s important that people may freely investigate claims made by their government. Another part of the story is besides NOAA/GHCN is GISTemp/GISS. I’ve been downloading some station data from them for years and have found frequent changes to “data” from decades ago. For example, all but the most recent Yakutat GIStemps have changed by large amounts in the last six months, and some years (1970, 1971, 1985 and 1986) with missing data have been filled in. There are no notes explaining such changes. The web page data files are completely replaced monthly. Only by saving the text “data” every month can a track of changes be recorded. I’ve found many other such changes, mostly to convert decades old temps to lower values. In some cases there are sudden “step” changes, but most are gradual. Some stations show no changes at all, such as Amundsen-Scott.
Or he’s a member of the House of Representatives.
(Remember Will Guam tip over?)
I didn’t read all the posts but would this lowering of the temps be a result of the new USHCN 2.5 dataset? They brought the new dataset online in Oct 2012. It changed everything! Does anyone know what the real temps are anymore. GEEESH!
http://www.ncdc.noaa.gov/oa/climate/research/ushcn/
Regarding Steven Mosher’s apple example: He’s wrong.
Suppose the apples’ weights are produced by distribution with mean, m, such that
P(x = m + 1/2) = 0.
In other words, all the apples weigh between m – 1/2 and m + 1/2.
Suppose the scale weighs everything between 8.5 and 9.5 as 9, and everything between 9.5 and 10.5 as 10. That is, the measured weights are the actual weights rounded to the nearest integer.
If m is between 9 and 10, all the apples will weigh as 9 or 10.
What’s the expected value for measured weight?
It’s just EV = 9 * P(8.5 < x < 9.5) + 10 * P(9.5 <= x < 10.5).
Since all the apples weigh between 8.5 and 10.5,
EV = 9 * [1 – P(9.5 <= x < 10.5)] + 10 * P(9.5 <= x < 10.5)
EV = 9 + P(9.5 <= x < 10.5)
Clearly the average weight isn't the best estimator of the true mean weight (even if the distribution is symmetric).
If the measured values span a wide range of integer values, the average of the actual vales should be near the average of the rounded values, but there's no law of large numbers that shows they converge as the number of measurements increases.
My previous comment was distorted by the fact that some angle brackets used as less-than and greater-than symbols were processed as HTML tags.Most of it got through, though, and I think the point (such as it was) is pretty clear.
Hey, folks, dig the ENSO meter!
GIGO = CON US
Don’t know if this has been mentioned in earlier comments, but there is likely a systematic, cool bias in the data from the COOP sites which arrive a few weeks after the end of the month. Let me explain: In some areas, COOP observers report their daily data sometime in the morning, say between 6 and 9 AM. They report 24-hour high and low temperatures once a day at those times. Let’s say, for example, that on Monday at 7 AM they report a high of 65, low of 45 and current temperature of 46. It can be assumed that the low temperature reported occurred on Monday morning shortly before the reporting time. Later on Monday, a warming trend began. The next day (Tuesday), the 7 AM observation is a high of 75, low of 46 and a current temperature of 55. From the data here, it can be implied that the actual Tuesday low was closer to 55 than 46 because that was the temperature reported at the observation time. The reason the low is 46 is because that was the temperature the instrument was reset to on MONDAY morning after that observation was taken. When listed as a daily temperature report, the Monday morning low is 45 and the Tuesday morning low is 46. A more realistic daily temperature report would have Tuesday’s low closer to 55. Therefore, the COOP data is “double-reporting” the Monday low because of the time and manner in which the data is reported, not because of an intentional skewing of the data. Data from NWS ASOS stations is reported from a midnight-midnight, which leads to far less of this “double-reporting” of morning lows.
“The question for now is: why do we appear to have two different sets of data for the past two years between the official database and the SOTC reports and why have they let this claim they made stand if the data does not support it?”
Without exploring motives, though they should be clear at this point, the facts remain — The SOTC reports are released first, which is of particular importance to those personally invested recalling that It is a widely accepted belief that you never forget ‘your first’!
Jaye Bass says:
January 7, 2013 at 12:32 pm
Do they use a version control system and a configuration management plan for dealing with the data? One should be able to get previous versions very easily with a nearly perfectly traceable history.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
That is what the US government demands us peons do. The thousand little gods help you if the FDA or FAA finds you messed up you version control system. (I have been through audits by both)
Since my last post I noted and read the link to the About Us page which as far as I could see did nothing to allay my fears of the historical data changing… I concur that current plots can change as data is added from stations still using snail mail and the like, but I cannot see from anything on their site why the 1936 temperature should have changed in the 2 days since the WUWT post?
The only explanation is they are playing with either the Math behind arriving at a plot, or creating new data that was missing from a station’s reports using… dare I say it… a computer model to fill in the vacant holes… and as noted in other posts, the 1936 temperature is dropping… conveniently…?
Today’s NOAA 1936 plot is 76.41 and 2012’s is 76.92, which is what they were when I posted… 76.43 & 76.93 being the temperatures on the WUWT screenshots. 0.01 actual difference may not be much, but as also mentioned… it does seem convenient that the modifications favour the AGW argument… or at least maintains the half a degree increase. It is almost as if the 1936 temperature plot is linked to the 2012 plot… i.e. when they ‘had’ to adjust the 2012 plot ‘down’ because of late data turning up, which I accept as OK, some algorithm then kicked in and adjusted 1936 and other plots down to compensate?
It will be interesting to watch this over the following weeks and if the plot for 2012 ‘needs’ further adjustment as late data is added and therefore changes the plot value, what happens to the 1936 plot? Will an increase in 2012 spur an increase in 1936 and conversely a downward 2012 make 1936 go down?
Will watch for a while and report back if it does.
The headlines about 2012 being the hottest year are everywhere. According to the NY Times, it was not even close.
Don’t let up Anthony.
This is yet another example of the preposterous nature of the anthropogenic climate change debate. Everyone learned in grade school that the precision of a result can only be as precise as the least precise variable in the calculation. We also no by experience, than any deviation of inaccuracy is exaggerated the further out you expand that error. As such, the climate change crowd wants it both ways, they want to be able to project their models out decades and centuries into the future, while dismissing errors and gaps in the precision of the data used in those models. They can’t have it both ways……Bottom line is that it is absolutely impossible for anyone to prove or disprove anthropogenic climate change because our understanding of the non-linear interconnected systems is so poor, and the data is so imprecise, that any result is garbage.
We are spending billions, if not trillions, on trying to prove something that cannot be proven either way. As such, all we have is a multibillion/trillion dollar marketing, fundraising, propaganda, and scientific welfare program.
Mark H says:
January 9, 2013 at 8:08 am
“Bottom line is that it is absolutely impossible for anyone to prove or disprove anthropogenic climate change because our understanding of the non-linear interconnected systems is so poor, and the data is so imprecise, that any result is garbage.”
If we base our understanding on average temperature, I agree 100%.
But I think there are ways to disprove AGW. I’ve been analyzing the nightly drop in temps in the temperature record, which shows no trend.
I’ve also just purchased a handheld IR thermometer, and pointing it skyward on a clear 35F day, and it was colder than the minimum temp it reads -40F. Which looks to eliminate any CO2 signal out to ~12.5um. I think this is proof any heating from CO2 is minimal, if it’s even measurable.
Trying to argue based on temperatures is futile, the warmest own that dialog, but I think there’s plenty of evidence that makes that argument meaningless. We have to change the rules of the games.
E.M.Smith says:
January 7, 2013 at 12:33 am
========
Thanks for the links to your temp processing example (if that would be a correct term).
That, along with what Anthony has presented here suggests that something as elementary as maintaining temp data and processing it seems to be in the hands of the incompetent.
One other problem could be an inconsistent method of calculating the average. Let’s say you have 100 sites reporting for a month.
1. Do you take each site, add up their temperatures to get a monthly average, then add each sites average and divide by 100 to get a total average;
or
2. Do you take all 100 sites, add them up together and divide by the total number of days used, in this case approximately 3000 (30 days x 100 sites)?
I know that I would use the second process but in all likelihood they are probably taking the average of an average which could change the output by +- 0.5 degrees or so. Especially considering that sites will at times have missing data so you might compare one site with 25 days to another with 30 days which is no longer apples to apples if following the first process.
The NOAA web pages do prominently list contact information for concerns like this. Before accusing someone of incompetence and/or malice, I would think you might ask them about their side of the story. Quite possibly you are right, and by bringing the discrepancies to their attention you would actually help improve government services. Or possibly their methodology is correct and you are making something out of nothing.
In any case, July 1936 and July 2012 were undoubtedly hot in the lower 48. This doesn’t strike me as warranting a lot of self-congratulation and backslapping.
I have to admit, I was deeply troubled by what you found. There seemed to be no logical explanation. However, I also would have hoped and expected that you would take the next step and contact NOAA to alert them to this issue or at the very least see if they can explain it. To me it seems doing so would be the bare minimum research to call this truly “objective” work. Now I am not discounting your thorough research, nor the concerning confusion this data presented, but your story is incomplete.
I read your article on 1/8/12 and immediately contacted NOAA via email (http://www.ncdc.noaa.gov/climate-monitoring/contact.php). I described the questionable data that is presented on their website as your research highlighted (including a link to this story) and asked for an explanation. Honestly, I did not expect a response. Yet, within 24 hours I received an email back. Turns out someone else had the same concerns, but chose to contact NOAA before publishing a potentially misleading article…perhaps something to take note of in the future. The response is pasted below in its entirety.
I think this neatly clarifies the discrepancies found.
“The differences seen in the CONUS temperature for July 2012 are not due to late arriving data. They are due to our change from version 2.0 to version 2.5 of the US Historical Climatology Network (USHCN), from which we calculate the CONUS temperature index.
First, as part of the changeover, we recalculated the baseline temperature upon which “anomalies” are compared, which were first computed nearly 20 years ago in an era with less available data and less computer power. In other words, we now have a better estimate of what is the average July temperature (and the average August, and September, and so on). This resulted in a cooler baseline value for all of the July values throughout the record. This component was applied equally throughout the record and does not affect the trend, nor does it change the characterization of 2012 temperatures as the warmest, nor does it change the relationship between one year and any other.
The other factor that affected the July 1936 temperature was the improvements to the methodology in v2.5. The basics of the changes and their impacts are below. But please visit the more detailed information at http://www.ncdc.noaa.gov/oa/climate/research/ushcn/, where the differences are detailed thoroughly, and even the code used to run the two versions is available for download.
The short version: v2.5 improved the efficiency of the algorithm that confirms inhomogeneities (“breaks in the record” due to siting or instrumentation – documented or undocumented). This means that more of the previously undetected break points are now confirmed and corrections can be applied. The impact varies from station to station. Because, on average, the newer instrumentation reads “cooler” than the historical instrumentation (please see attached Menne et al., 2009 paper), the adjustments, on average, cool the deeper past data to better conform to the observing practices of today.
Speaking strictly to the July values that you are interested in: the July 2012 temperature decreased about two-thirds of a degree from v2.0 to v2.5, entirely due to the new baseline temperature for July. The July 1936 temperature cooled mostly because of this same baseline change (about two-thirds of a degree), but also an additional one-third of a degree due to the improved breakpoint detection.
It is erroneous to compare the v2.0 version from one year to the v2.5 version from another, as they refer to different baselines. In version 2.0, the CONUS temperature for July 2012 was approximately 0.2F warmer than that for July 1936. In version 2.5, the difference is approximately 0.5F. This is the result of the improved inhomogeneity detection algorithms. In both data sets July of 2012 is warmest ever measured.
We announced the change prominently within the September 2012 report itself (http://www.ncdc.noaa.gov/sotc/national/2012/9), just below the “Significant Events” map, and before any results are presented. The announcement contained and still contains links to more detailed information, including a technical paper describing exactly the differences between the two versions, and links to the code for each.
Thanks again for contacting us before writing your article. We appreciate the consideration.”
One part of this article that I don’t get. In one place they have 77.6 in 2012 vs. 77.4 in 1936, and in the other it’s 76.92 in 2012 vs. 76.41 in 1936. Do they then not agree, at least, that it was warmer in 2012 in 1936?
GW is a religion, just like BO.
This comment is probably the funniest thing I have ever read.
As one who believes global warming to be the greatest challenge facing humanity, I find your work meticulous and your conclusions troubling.
Doug Broome says:
=============================
What global warming, pray tell
NoCherryPicking says:
January 9, 2013 at 1:17 pm
=================================
And so, did you swallow their justification for adulterating the data?
Also:
“Turns out someone else had the same concerns, but chose to contact NOAA before publishing a potentially misleading article”
And who was that? Pray tell.
Looks like a good response by NOAA. They should, however, go back and caveat their monthly summaries to avoid this type of confusion in the future. It kind of reminds of the unemployment statistics that make headlines and then are readjusted later, but the new numbers get less press.
I suggest everyone read this story that has the responses from both sides. (Yeah it is Fox but at least they presented both sides) http://www.foxnews.com/science/2013/01/10/hottest-year-ever-skeptics-question-revisions-to-climate-data/
A NOAA rep. Peter Thorne gives NOAA’s reason for changing the data and you also get Dr. Roy Spencer’s comments.
I would note that the NCDC website has held back on finalization of all Class 1 LCD data for December 2012, by leaving Missing data in the final day of the month… well past it’s normal time. January 2013 data is already available through the 10th.
When I called NCDC on Monday the 7th, I was told by staff that the final runs and upload would take place later that day. Here it is Friday the 11th, and my call today elicited exactly the same response… that the final runs and upload would take place later today.
Could LCDC be holding back on that final day’s data so that the “official” monthly temperatures for December 2012, and by implication the “official” Annual temperatures for 2012, will not be available until they are finished with manipulations?
As for the response to explain why monthly historic temperatures are changing, the link to an explanation of differences between Version 2.5 and Version 2.0 was
“Last Updated Thursday, 4-Oct-2012 08:25:07 EDT by ron.ray@noaa.gov “, based on the articles date stamp.
If the V2.5 has been available since last October, why are the changes just showing up this week? Did they not turn the switch on for V2.5 until now? Did some influx of new data create a seismic change in the temperatures that propagated back to the 30s?
Also, it now appears that we will no longer see “raw” data [Version 1] from any stations in the NCDC database. Everything after January 2006 is now “QCLCD”, Quality Controlled Local Climatological Data, in Version 2 [automatic QC] and Version 3 [Final QC, subsequent to the end of each month]. http://cdo.ncdc.noaa.gov/qclcd/qclcdimprovements.pdf
“Pay no attention to the man behind the curtain.” – WOO