Guest Essay By Walter Dnes
There have been a number of posts on USHCN temperature adjustments, including 1, 2, 3, 4, 5 and 6. They have focused primarily on annual adjustments. Whilst looking into the USHCN adjustments, I noticed that each of the 12 months is adjusted differently. Here is a plot of average USHCN temperature adjustments, for each month plus the annual average, by year for 1970-2013:
Here is the full plot for 1872-2013.
The calculation of average monthly adjustment consists of:
- Calculating the total accumulated values of (final-temp – raw_temp) where the raw and final values of USHCN monthly temperatures were both non-missing
- Count the number of occurrences where the raw and final values of monthly temperatures were both non-missing
- Divide item 1 by item 2.
As you can see, there are marked differences in adjustments since 1970 for each month. To analyze in more detail, we need to look at some numbers. In the table below…
- The columns “2013” and “2014” list the average adjustment in Celsius degrees for the corresponding months in the years 2013 and 2014 (where available).
- “Slope” means the slope attributable to USHCN adjustments, in Celsius degrees per century, during the period from 1970 to 2013.
| Month | 2013 | 2014 | Slope |
| January | 0.1206 | 0.0991 | 1.432 |
| February | 0.1735 | 0.1687 | 1.519 |
| March | 0.1357 | 0.1313 | 1.455 |
| April | 0.0089 | 0.0101 | 1.281 |
| May | -0.0785 | -0.0828 | 0.955 |
| June | -0.0842 | -0.0856 | 0.775 |
| July | -0.0922 | -0.0881 | 0.676 |
| August | -0.1344 | 0.783 | |
| September | -0.1298 | 0.949 | |
| October | -0.0751 | 1.046 | |
| November | -0.0244 | 1.206 | |
| December | 0.0283 | 1.220 | |
| Annual | -0.0126 | 1.108 |
What are the implications of the USHCN adjustments?
- The talk about winters in the USA getting warmer may be an artifact of the adjustments. The adjustments for January/February/March are the highest of the 12 months. In 2013, they combined to average +0.1433 Celsius degree, while the overall annual adjustment for 2013 was -0.0126 Celsius degrees.
- This is a booby-trap for the unwary. When you see articles in February/March/April about HUGE upward adjustments by USHCN so far during the year, you’ll know why. By the following January, the adjustment will cover the entire calendar year and look more reasonable. Mind you, this is still over half a Celsius degree above the adjustments for the 1930s.
- Speaking of the 1930s, one wonders if this an attempt to disappear the heat waves and droughts of “The Dirty Thirties” in a manner similar to attempts to disappear the Medieval Warm Period. It’s hard to talk about “the hottest ever”, when there’s “inconvenient data” around, showing that the 1930s were hotter. The 2nd graph shows the adjustments from the 1870’s onwards. Compare 2013’s -0.0126 annual adjustment with annual adjustments for the 1930s…
- 1930 -0.5586
- 1931 -0.5628
- 1932 -0.5639
- 1933 -0.5770
- 1934 -0.5877
- 1935 -0.5851
- 1936 -0.5846
- 1937 -0.5907
- 1938 -0.5852
- 1939 -0.5810
Data Sources
USHCN monthly data is available on the web in the ftp directory ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ The specific files used for my analysis were…
- readme file (data formats and basic instructions): ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/readme.txt
- station metadata: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn-v2.5-stations.txt
- raw monthly mean temperature data: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tavg.latest.raw.tar.gz
- final monthly mean temperature data: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2.5/ushcn.tavg.latest.FLs.52i.tar.gz
Odds and Ends
Interesting stuff I stumbled across whilst working on this article…
- Station USH00381310, i.e. “CAMDEN 3 W”, South Carolina has raw data for August 1853. The next piece of raw data for that station is August 1906. The first piece of any final data for any station is 1866. I wonder if the date is a typo.
-
- There is some USHCN raw data for the years 1853, 1868, 1869, and 1871 onwards.
- There is USHCN final data from 1866 onwards
Wait a minute. Where does the USHCN final data for 1866, 1867, and 1870 come from, if there is no USHCN raw data for those years? A closer look shows that the small amount of data for those 3 years is all from station USH00303033 “FREDONIA”, New York. It’s located at 42.4497 -79.3120 which translates to 42° 26′ 59″ North 79° 18′ 43″ West. This is near the shore of Lake Erie, not that far from Canada. Let’s check what nearby Canadian data is available for that time span.
- Point your web browser to http://climate.weather.gc.ca/advanceSearch/searchHistoricData_e.html
- Select the “Search by Proximity” tab
- Select 200 in the “kilometres away from” menu
- Click on the “location coordinates:” radio button and enter latitude 42 26 59 and longitude 79 18 43
- Click on the “for years from” radio button, and select 1866 to 1870
- Click on the “Search” button.
The 2 closest sites are Simcoe at 90 km, and Hamilton at just over 100 km. There are another 6 sites within 165 km of Fredonia.
It sort of makes sense that the Fredonia data was created from these sites. I can’t think of any other semi-reasonable explanation. I’ll leave it to professional meteorologists like Anthony to comment on the validity of using data from sites located northwest of Lake Erie to generate estimated data for a site southeast of Lake Erie.
- “The Rise and Fall of USHCN Raw Data” is of interest, in that the less raw data available, the more estimation has to be done to fill out the data set. The theoretical full annual complement of data is 1218 stations with 12 months of data per year, meaning 1,218 * 12 = 14,616 station-months each year. The following graph shows the number of raw and final station-months in the USHCN data over the years. The graph ends at 2013.
- Plots of adjustments over period of record first alerted me to the fact that adjustments were 12 separate data sets, 1 for each month. I ran a script to crank out adjustment plots for all 1218 stations in USHCN, “to see what I could see”. The plot below is an example. Note the period from 1904 to 1911. The adjustments for all 8 of those years were…
- January -2.91 degrees each year
- February -2.94 or -2.95 degrees each year
- March -3.00 degrees each year
- April -3.03 degrees each year
- May -2.93 degrees each year
- June -2.84 degrees each year
- July -2.86 or -2.87 degrees each year
- August -2.87 or -2.88 degrees each year
- September -2.90 or -2.91 degrees each year
- October -2.88 or -2.89 degrees each year
- November -2.95 or -2.96 degrees each year
- December -2.94 or -2.95 degrees each year
Other portions of the data have their own stretches of the same adjustments 12 months apart.
Walter Dnes – Click the pic to view at source
There is no global temperature it does not exist,it is not a description of reality.It is possible to describe the earth and that description is clear,there is no single temperature on the earth.The average temperature is meaningless just statistics and nothing else,it is just another invention like the la nina ,el nino thing which was never discovered by observation but which we are told is real but only seems to be defined by statistics.What happened in 1998/1999 ? Nothing that I can remember it was just another year like any other but if you look at the global temperatures for these years there is a massive spike.How can this describe reality?
And as nobody else has mentioned it, I will.
UHI
In his paper “A closer look at United States and global surface temperature change”, back in 2001, Hansen included a table showing that just 17% of USHCN sites were truly rural, i.e “unlit”.
The percentage is almost certainly less now.
Urban sites dominate the US record.
to Steven Mosher –
“Can you explain why is it that current state of the art automated stations still can not take temperature measurements at proper times and need to be adjusted?”
and add to that why a station has been adjusted multiple times for a given year in the past. Didn’t get it right the first time.. or the second time.. or the third time.. or the fourth time.. or the fifth time…..
Steven Mosher says: August 23, 2014 at 10:32 pm
“The tobs adjustment is a function of the month, latitude, longitude, and other factors.
Hmm, in a little bit Zeke will have a post showing folks
1. why tobs is required
2. what kind of error you get if you dont apply it.
The tobs adjustment is required. this is Provable.
The adjustment made is correct. this was proved in both papers about it.”
Will be interesting to see another explanation from Zeke.
Pardon my ignorance, but how do people like Mosher, Zeke and the other “adjusters” know that the old temperature readings were too low and as we now see, the warm thirties were not as warm now as they were then? I would like Zeke or one of his ilk to explain exactly how we now “know” that the thirties were not as warm as they were even just a few years ago before the latest batches of adjustments. Where did the dustbowl heat go? the deep ocean below 2000 metres without a trace? I cannot imagine this issue will go away. Someone will need to re-write Steinbeck and destroy all old issues.
Steven Mosher says:
August 23, 2014 at 10:32 pm
The tobs adjustment is a function of the month, latitude, longitude, and other factors.
Hmm, in a little bit Zeke will have a post showing folks
1. why tobs is required
2. what kind of error you get if you dont apply it.
The tobs adjustment is required. this is Provable.
The adjustment made is correct. this was proved in both papers about it.
understand.. when you continue to play the fraud card about adjustments you undermine the credibility
of the people who want to make a legit case about micro site.
_________________
So basically you are saying “don’t call us on our bullsh@t so you can call us on our other bulsh@t” That is how I am reading this.
When you calculate a multiyear trend with multiple stations, it is necessary to trend each month separately and then average the months (preferably weighting for days per month, but that doesn’t change things much). Doing it by year leads to spurious results because sometimes months are missing, and if that month is a winter month, the year will have a warm bias and vice-versa.
So I would assume that they also mangle the results by month, as well.
The tobs adjustment is required. this is Provable.
Something is required, for sure. TOBS bias is as real as a punch in the nose.
If it occurs at the very beginning or end of a study period or if it flips back and forth leaving roughly equal phases at each end (important!), then it matters little, at least to the linear trend. But if it comes in the middle, it is a walloping big trend bias.
The adjustment made is correct. this was proved in both papers about it.
But why even go there? It’s a step change. So why not simply split the trend and let that act as the “adjustment”? I have looked at a couple of stations so far (and will do more). When I split the trend, the (very real) step change is removed and the combined trend is increased. BUT the final results are still lower trend than the TOBS-adjusted data. This suggests they may be overcorrecting. I won’t know, of course, until I’ve done up a few more stations.
Urban sites dominate the US record.
No. they do not. ~9% of USHCN sites are urban. It is true that only 2% of land area is classified as urban, so there is overrepresentation. But it does not dominate.
Furthermore, UHI is not the main problem with trends (sic!). The problem is not mesosite. The problem is microsite. Well sited urban stations have much lower trends than poorly sited rural stations. Yes, yes, I know it is warmer in cities, but not so much for the trend.
When it comes to trend (which is what’s important), Microsite is the “new UHI”. If you lean on UHI, you are barking up the wrong tree. But bad microsite (Anthony’s premise) utterly dominates trend.
Don’t confuse or conflate the two.
Walter Dnes says:
“Of interest is the fact that the CAGW crowd swears up-and-down that tree-ring data is valid when it doesn’t show a warm Medieval Warm Period, but totally invalid, when it shows falling temperatures after 1960.”
It’s a matter of science — a lot of work has been done to validate proxies.
You never answered my other questions:
1) Where is the data for Lamb’s graph? I’d like to calculate some trends.
2) What is the scale on its vertical axis? 0.1 C? 1 C? 10 C?
I can’t put any trust into a graph that doesn’t have any data, or that doesn’t label its axis. How can you?
Thanks.
Steven Mosher says:
August 23, 2014 at 10:32 pm
The tobs adjustment is a function of the month, latitude, longitude, and other factors.
Hmm, in a little bit Zeke will have a post showing folks
1. why tobs is required
2. what kind of error you get if you dont apply it.
The tobs adjustment is required. this is Provable.
The adjustment made is correct. this was proved in both papers about it.
Steven,
It doesn’t need to be provable it needs to be validated.
So as there are such a very small number of stations in the USHCN, the TOBS adjustment (and any other messing with the data) needs to be validated by taking out a known _correct_ TOBS station and then creating its new data from the ‘homogenization’ algorithm; and, take a known correct automated station reporting hourly and from its noon temperature observation create its correct TOBS temperature. The largest of the errors that you will get will become the error bars for the algorithms in use.
From a QMS standpoint, every single station should have a history recorded of actual data and correction, why the correction was made, how the correction was made and a sign off by a forecaster or better qualified meteorologist NOT a computer scientist or a treemometer specialist. Repeated corrections to old data (which we appear to be seeing) need to be formally documented and signed off by senior members of NOAA stating why the repeated adjustment of old data is necessary after the first correction, specifying what the errors were in the old correction algorithm and the impact on each station’s reporting. I know that documentation, quality management, and validation are not what academics do – preferring instead getting agreement from a set of like-minded academics (called peer review), but this creation of a climate record is no longer an academic game it is directly responsible for energy poverty worldwide.
So climate science must move from academia to the engineering world of QMS. validation and detailed documentation of each site – there are startlingy few to do after all. In particular the creation and use of changed data that then has to be revisited and changed again indicates a gross error has been made and that cannot be ignored it must be documented: what the error was, why it was made and who incorrectly accepted the work, and who is going to fix it and how. If an engineer builds a bridge and its subsequently collapses and say 100 people die then all hell is let loose. If climate scientists mislead politicians with incorrect data and thousands of people die of cold in energy poverty _in a single month_ oh that’s alright {shrug} they’ll create another unvalidated algorithm and agree it between them, That approach is no longer acceptable. It is time that climate scientists took full responsibility for their provision of (mis)information – this is not an academic argument.
SonicsGuy says: August 23, 2014 at 7:30 pm (Edit)
By the way, can you show us where is the data for Lamb’s hand-drawn graph? .
http://wattsupwiththat.com/2009/11/29/the-medieval-warm-period-a-global-phenonmena-unprecedented-warming-or-unprecedented-data-manipulation/
That Lamb graph is from IPCC AR1 Chapter 7, page 202:
http://ftp.ipcc.ch/ipccreports/far/wg_I/ipcc_far_wg_I_chapter_07.pdf
Steve McEntyre has done much research on it;
http://climateaudit.org/2012/10/09/the-afterlife-of-ipcc-1990-figure-7-1/
it’s origins stretch back to Lamb, 1965;
http://www.climateaudit.info/pdf/others/lamb.ppp.1965.pdf
and a good description of Lamb’s approach to temperature reconstruction can be found in chapter 5 of Lamb’s 1982 book Climate History and The Modern World:
http://books.google.com/books/about/Climate_History_and_the_Modern_World.html?id=MZIOAAAAQAAJ
Jones et al. 2009, analysed the provenance of IPCC AR1 Fig 7.1c, see page 34;
http://pubs.giss.nasa.gov/docs/2009/2009_Jones_etal_2.pdf
and a reading of some of Lamb’s prodigious writing of the time;
http://www.cru.uea.ac.uk/cru/pubs/byauthor/lamb_hh.htm
should help you to collect the data you are seeking. Alternatively, I’d suggest reaching out to the IPCC to see if they can provide you the data, as I am sure that they must request, validate and maintain the data on any anything that appears within their tomes…
Or provide a scale for the vertical axis, on the graph shown here:
According to Jones et al. 2009;
“The Lamb (1982) time series does have an explicit temperature scale, and the best-fit scaling between this curve and the IPCC curve indicates that one tick-mark interval on the IPCC figure
corresponds almost exactly with 1°C. The degree of smoothing for both these curves is unknown, but Lamb (1982) states that the red curve is based on 50-yr means (supported by earlier publications).”
Interesting post about BOM adjustments in Australia to create warming by making past cooler.
http://joannenova.com.au/2014/08/the-heat-is-on-bureau-of-meteorology-altering-climate-figures-the-australian/
Steven Mosher
I am just as convinced that these adjustments are based upon sound science as I am that the IRS treatment of conservative groups was warranted and/or a small local mistake, that the Benghazi murders were due to a movie, that inflation is negligible (when if calculated using the 1980’s techniques it would be 9.5%) and that GDP is presently positive given all the changes made to that calculation recently. Figures lie and liars figure.
> angech says:
> August 24, 2014 at 2:25 am
> Walter,
> one of the conundrum’s with the USHCN is that infilling is
> routinely and widely used as you know.I have had a long
> running battle with Steve Mosher, Zeke and Nick Stokes
> on several sites over the number of real stations in the 1218
> listed which they refuse to answer, presumably because
> the number of made up [infilled] stations is extremely high.
Assuming you can trust the raw data file, the monthly raw data counts I get from my early August download are
Year 2013 2014
Jan 921 895
Feb 915 891
Mar 913 893
Apr 910 896
May 915 885
Jun 897 767
Jul 912 638
Aug 899
Sep 899
Oct 898
Nov 899
Dec 890
It looks like some data may be slow coming in and being processed. Hopefully, within the next few months, the late June and July data will come in. This does raise a question though…
Item… NOAA puts out monthly summaries soon after the end of the month.
Item… late data comes in over the next 2 or 3 months
Item… I believe I’ve seen people stating that NOAA’s summary numbers are changed a few months later
This actually “makes sense” in a wierd way. They’re putting out a quickie summary based on incomplete data, and adjusting as more data comes in.
[Mosher said ““angech The TOBS adjustment would be done once. the adjustment that would/could change on a daily basis is PHA. Man are you dense”]
To the best of my memory, I haven’t heard of PHA before. What does it mean, and how can it be vald?
It may be worse
. Mosher says “Monthly values calculated from GHCN-Daily are merged with the USHCN version 1 monthly data to form a more comprehensive dataset of serial monthly temperature and precipitation values for each HCN station”
How many months later do they merge The GHCN data with the USHCN? Does this mean they make amendments on amendments.
Walter, the figures are very interesting, both Nick Stokes and Zeke who would know better have insisted 4 months ago that the number of presumed actual stations is around 918 in replies to me when they both would have known that it has been under 900 for a year on your figures.
The ease with which they hide the truth when they know it is hard to find makes their arguments to be open and honest difficult to swallow.
USHCN originally only used 138 stations out of the 1218/9 for the purpose of making up their historical graph and persisted with it for a long time.
Is it your understanding that the current graph represents all 1218 stations or are they still using a selected subset.
The increase from 638 listed July 2014 to the 896 range April may not be due to missing stations just modified infilling from the GCHN that comes in later to make it look as if they are using more real stations.
Zeke has already implied that there are only 650 working real stations in an offhand comment.
Pairwise Homogeneity Adjustment Software The automated pairwise bias algorithm (PHA) software (Menne and Williams 2009) version 52i.
Steve Mosher says “The tobs adjustment is a function of the month, latitude, longitude, and other factors.”
Not really, It should be an adjustment made to a temperature record due to it not being recorded at an inconvenient time. The function of the month, latitude, longitude is basically irrelevant if you have a complete set of trustworthy records as they are all incorporated in it.
On the other hand if you want to make up a record for a non existent or closed station or a station in a site that you would like instead of where it really is, then you need all those bits of data to construct such a station and then pretend that the temperature it has was recorded at the wrong time and needs an additional TOBS adjustment made to it.
> angech says
> August 24, 2014 at 4:07 pm
>
> USHCN originally only used 138 stations out of the 1218/9 for the
> purpose of making up their historical graph and persisted with it
> for a long time.Is it your understanding that the current graph
> represents all 1218 stations or are they still using a selected subset.
To be honest, I don’t know.
> The increase from 638 listed July 2014 to the 896 range April
> may not be due to missing stations just modified infilling from the
> GCHN that comes in later to make it look as if they are using more
> real stations.
Again, I don’t know.
> Zeke has already implied that there are only 650 working real
> stations in an offhand comment.
What was the context of that comment? My August 5th download had only 638 stations with July raw data. If the comment was in the context of getting the montly summary out right away, that looks correct. A later official summary can use late data that trickles in during the next month or so.
justthefactswuwt says:
“http://www.climateaudit.info/pdf/others/lamb.ppp.1965.pdf”
Thanks for the link. Lamb’s much quoted result is Figure 3. In the caption it says those data have been “adjusted,” and some of it is “opinion.”
Hardly sounds very scientific, does it?
Jim G says:
“I am just as convinced that these adjustments are based upon sound science as I am that the IRS treatment of conservative groups was warranted and…”
Richard Muller formed BEST specifically to take a new look at the data. They did, thoroughly, and essentially found the same results as everyone else.
So why not accept it? That was the close look at the data from a self-identified “skeptic” that so many people like you had been calling for.
But now it’s clear that since it didn’t give the result you want, you won’t accept it. So what will it take? Clearly it will take a dataset that matches your preconceived notions. And tha’s not science.
justthefactswuwt says:
“According to Jones et al. 2009;
“The Lamb (1982) time series does have an explicit temperature scale, and the best-fit scaling between this curve and the IPCC curve indicates that one tick-mark interval on the IPCC figure
corresponds almost exactly with 1°C. The degree of smoothing for both these curves is unknown, but Lamb (1982) states that the red curve is based on 50-yr means (supported by earlier publications).”
I can easily imagine the howling that would take place here if the IPCC today offered up such a graph. I’m sure you can too.
This won’t get resolved until someone from NOAA is forced to explain the adjustments on the witness stand while under oath.
SonicsGuy says:
What a coincidence that the adjustments look like a hockey stick! I guess that I am skeptical of your self described skeptic’s motivations. The adjustments are so obviously biased in a certain direction. As far as the result ” I want “, just stick to the observed data.
Walter, took an hour to find
What was the context of that comment? My August 5th download had only 638 stations with July raw data. If the comment was in the context of getting the monthly summary out right away, that looks correct. A later official summary can use late data that trickles in during the next month or so.
Zeke Hausfather says: at Real Climate A Different Approach To The USHCN Code
Posted on May 11, 2014
May 12, 2014 at 3:00 pm
“The difference is straighforward enough. Even if you use monthly rather than annual averages of absolute temperatures, you will still run into issues related to underlying climatologies when you are comparing, say, 650 raw stations to 1218 adjusted stations. You can get around this issue either by using anomalies OR by comparing the 650 raw stations to the adjusted values of those same 650 stations.
The reason why the 1218 to 650 comparison leads you astray is that NCDC’s infilling approach doesn’t just assign the 1218 stations a distance-weighted average of the reporting 650 stations; rather, it adds the distance-weighted average anomaly to the monthly climate normals for the missing stations. This means that when you compare the raw and adjusted stations, differences in elevation and other climatological factors between the 1218 stations and the 650 stations will swamp any effects of actual adjustments (e.g. those for station moves, instrument changes, etc.). It also gives you an inconsistant record for raw stations, as the changing composition of the station network will introduce large biases into your estimate of absolute raw station records over time. Using anomalies avoids this problem, of course”.
As you can see Zeke implicitly admits there are only around 650 real [raw] stations in this slip of the tongue at Steve Goddard’s