I’ve been away from WUWT this weekend for recovery from a cold plus family time as we have visitors, so I’m just now getting back to regular posting. Recently on the web there has been a lot of activity and discussions around the issue of the dropping of climatic weather stations aka “the march of the thermometers” as Joe D’Aleo and I reported in this compendium report on issues with surface temperature records.
Most of the station dropout issue covered in that report is based on the hard work of E. M. Smith, aka “chiefio“, who has been aggressively working through the data bias issues that develop when thermometers have been dropped from the Global Historical Climate Network. My contribution to the study of the dropout issue was essentially zero, as I focused on contributing what I’ve been studying for the past three years, the USHCN. USHCN has had a few station dropout issues, mostly due to closure, but nothing compared to the magnitude of what has happened in the GHCN.
That said, the GHCN station dropout Smith has been working on is a significant event, going from an inventory of 7000 stations worldwide to about 1000 now, and with lopsided spatial coverage of the globe. According to Smith, there’s also been an affinity for retaining airport stations over other kinds of stations. His count shows 92% of GHCN stations in the USA are sited at airports, with about 41% worldwide.
The dropout issue has been known for quite some time. Here’s a video that WUWT contributor John Goetz made in March 2008 that shows the global station dropout issue over time. You might want to hit the pause button at time 1:06 to see what recent global inventory looks like.
The question that is being debated is how that dropout affects the outcome of absolutes, averages, and trends. Some say that while the data bias issues show up in absolutes and averaging, it doesn’t effect trends at all when anomaly methods are applied.
Over at Lucia’s Blackboard blog there have been a couple of posts on the issue that raise some questions on methods. I’d like to thank both Lucia Liljegren and Zeke Hausfather for exploring the issue in an “open source” way. All the methods and code used have been posted there at Lucia’s blog which enables a number of people to have a look at and replicate the issue independently. That’s good.
E.M Smith at “chiefio” has completed a very detailed response to the issues raised there and elsewhere. You can read his essay here.
His essay is lengthy, I recommend giving yourself more than a few minutes to take it all in.
Joe D’Aleo and I will have more to say on this issue also.
steven mosher (23:24:23) :
perhaps I should have put a smiley somewhere there. I’m pretty sure he didn’t mean it quite the way it came out.
One of the most interesting figures in Hansen et al 2001 (http://pubs.giss.nasa.gov/docs/2001/2001_Hansen_etal.pdf) is the one titled “1900-1999 U.S. Temperature Change (deg C) – (A) USHCN Data”. It shows a matrix of images for unlit (rural), peri-urban and urban stations across the top and down the side the raw data, time of observation adjustement, Max/Min & SHAP data adjustment and data fill and UHI adjustment.
Anyway, the point is that the raw station data for rural stations shows a cooling of -0.05 deg C over the century. It is only when you add in the adjustments that you get a warming of +0.39 deg C for rural stations. I am not saying that the raw data is correct in itself, but they are placing a lot of confidence in the accuracy of these adjustments.
It is my opinion that, if you wish to try to measure changes in the heat content of the atmosphere (which is what global warming actually is!) then you should use only data that is not locally contaminated by man’s influence. Consequently, you should ONLY use rural stations. Pretending that you can adjust for UHI and use contaminated stations is just fooling yourself.
Alex Heyworth (22:38:10) :
“Carrot eater and Steve Mosher, thanks for your responses. CE, I was aware of the basics of the RSM from reading (so far) the first half or so of Hansen 1997. If all subgrids only had two stations to worry about, the method would be uncontroversial. It is what happens as a result of reiterating the process dozens of times that is more of an issue to my mind.”
Try it out, and see what it does. If you repeat my exercise and give station B a trend of some sort, you’ll see some odd things happen; of course you lose information about that trend after the station drop, but you also get a weird little jump at that point. In that case, having multiple stations helps water down these problems.
The other unsatisfying thing about the RSM is that the order in which you add the stations matters to some extent, when records are of different lengths. That’s what Tamino tried to address with his modified RSM.
Re: steven mosher (Mar 8 13:55),
Steve,
Sorry I missed your query at the end there . Jan brought it to my attention 😉
First a comment on non-problem 5
5. The notion that “averaging” results in the loss of data.
Well, of course it does diminish information content. The neat thing about the GISS approach to anomaly is that they take advantage of this. Getting a correct anomaly for all stations individually is hard, because of missing values in the agreed period. But on gridding, you lose the information of the individual stations anyway. Therefore might as well calculate an anomaly for the smallest unit that remains, the grid cell (or its centre point). Then with no loss you get better coverage in the base period.
So OK, the Open questions ( not problems, but open questions)
1. What is the provenance of the data being used.
Yes, an important question.
2. What adjustments are made and what are the exact calculations.
Yes, though fortunately with GISS this is not now open. You can see
3. Is there UHI contamination in the signal
And how much? And is it changing the trend (as opposed to a static bias)?
4. Does microsite bias matter? how much
Yes, but relate it to the trend. Stationary bias doesn’t matter much.
5. How should the uncertainty due to spatial coverage be computed?
I would have said estimated. But yes.
6. What is the optimal method for station combining and area averaging
( see romanM)
Yes – I haven’t got into that discussion, except to point out that no-one seems to refer to what GISS actually does. I’m not sure how much it differs from Tamino’s, but it does incorporate the monthly basis of Roman’s.
Nick Stokes (04:23:06) :
GISS uses monthly offsets, but calculates them one by one, as each station is added to the mean. So the ordering of the station can matter.
Tamino got rid of that ordering by calculating all the offsets simultaneously, but he used single offsets, not monthly. It’s unclear to me whether he knew he was diverging from GISS in that respect, but he hadn’t seemed to have studied that particular issue much yet.
So Roman completed the loop by telling him to put the monthly offsets back in again. So now it preserves that feature of GISS, while having the innovation that Tamino was looking for in the first place.
All that said, we don’t actually know the simultaneously calculated offsets actually works any better, yet. In principle it should, but the improvement has not been illustrated yet.
E.M.Smith (22:43:21) :
Anthony Watts (23:08:26) :
Here’s a csv paste of year, high,low,(high+low/2) for Ashland that I produced
from the raw data:
1888,67.35753425,40.42,53.88876712
1889,69.00821918,41.91780822,55.4630137
1890,64.56438356,38.24383562,51.40410959
1891,63.8739726,39.97534247,51.92465753
1892,64.1369863,39.6,51.86849315
1893,61.39726027,34.59452055,47.99589041
1894,64.05753425,39.19452055,51.6260274
1895,65.64931507,37.68493151,51.66712329
1896,65.10410959,40.26849315,52.68630137
1897,64.87123288,39.42739726,52.14931507
1898,65.3890411,38.9109589,52.15
1899,63.97260274,39.02465753,51.49863014
1900,64.70136986,39.81917808,52.26027397
1901,64.70136986,39.81917808,52.26027397
1902,64.09863014,39.58630137,51.84246575
1903,64.47671233,39.07123288,51.7739726
1904,65.57260274,40.6739726,53.12328767
1905,65.58356164,42.05205479,53.81780822
1906,65.21917808,43.34246575,54.28082192
1907,63.96438356,43.04383562,53.50410959
1908,64.4630137,42.02876712,53.24589041
1909,63.52876712,41.77808219,52.65342466
1910,65.04109589,42.6739726,53.85753425
1911,62.72876712,40.00821918,51.36849315
1912,61.80958904,41.4,51.60479452
1913,62.62465753,39.44383562,51.03424658
1914,65.44109589,39.19726027,52.31917808
1915,65.52054795,40.1260274,52.82328767
1916,64.02739726,37.49041096,50.75890411
1917,64.90410959,38.65479452,51.77945205
1918,66.6109589,39.2,52.90547945
1919,65.7369863,37.78082192,51.75890411
1920,65.77534247,38.24109589,52.00821918
1921,66.23013699,39.38356164,52.80684932
1922,65.36438356,37.96164384,51.6630137
1923,66.42191781,38.56438356,52.49315068
1924,67.93972603,38.99452055,53.46712329
1925,65.87123288,40.51506849,53.19315068
1926,68.87123288,41.64109589,55.25616438
1927,64.72328767,40.27671233,52.5
1928,66.25479452,40.05479452,53.15479452
1929,66.11506849,38.06575342,52.09041096
1930,65.7369863,39.29589041,52.51643836
1931,67.40821918,40.76986301,54.0890411
1932,64.47671233,39.77808219,52.12739726
1933,64.52054795,39.43835616,51.97945205
1934,68.48493151,42.67123288,55.57808219
1935,64.90958904,39.6630137,52.28630137
1936,66.44931507,40.63561644,53.54246575
1937,63.75068493,40.43150685,52.09109589
1938,64.54794521,41.61917808,53.08356164
1939,66.88493151,41.28767123,54.08630137
1940,67.24931507,42.75890411,55.00410959
1941,66.6109589,41.69041096,54.15068493
1942,66.85753425,40.70410959,53.78082192
1943,67.68219178,40.15342466,53.91780822
1944,67.24931507,39.90136986,53.57534247
1945,67.02465753,40.35342466,53.6890411
1946,62.96712329,41.21917808,52.09315068
1947,63.15616438,42.33150685,52.74383562
1948,60.07671233,39.63835616,49.85753425
1949,63.70958904,38.31780822,51.01369863
1950,64.03835616,39.0630137,51.55068493
1951,64.8109589,38.3260274,51.56849315
1952,63.97260274,38.20273973,51.08767123
1953,63.00547945,39.1369863,51.07123288
1954,63.98082192,38.40547945,51.19315068
1955,62.68219178,38.18082192,50.43150685
1956,62.90136986,37.93424658,50.41780822
1957,63.16438356,38.62739726,50.89589041
1958,66.51232877,41.28219178,53.89726027
1959,65.01369863,37.90958904,51.46164384
1960,65.48493151,38.90136986,52.19315068
1961,64.67945205,38.65205479,51.66575342
1962,63.85479452,38.12054795,50.98767123
1963,63.4630137,39.60547945,51.53424658
1964,63.38630137,37.89863014,50.64246575
1965,64.8,39.11506849,51.95753425
1966,66.21917808,39.9260274,53.07260274
1967,65.79452055,39.74246575,52.76849315
1968,66.1890411,40.59178082,53.39041096
1969,65.1260274,39.82191781,52.4739726
1970,66.67123288,40.77260274,53.72191781
1971,62.74520548,39.24383562,50.99452055
1972,65.22739726,39.5260274,52.37671233
1973,66.20547945,40.83835616,53.52191781
1974,65.90410959,39.91232877,52.90821918
1975,62.83287671,39.19863014,51.01575342
1976,64.07260274,38.68767123,51.38013699
1977,65.56164384,40.63013699,53.09589041
1978,64.50136986,40.76849315,52.63493151
1979,65.60273973,41.56164384,53.58219178
1980,65.10136986,40.27671233,52.6890411
1981,65.89726027,41.63150685,53.76438356
1982,63.81643836,39.98630137,51.90136986
1983,64.01643836,40.24383562,52.13013699
1984,63.8369863,38.34109589,51.0890411
1985,65.48767123,36.78630137,51.1369863
1986,67.23561644,40.03561644,53.63561644
1987,68.13150685,38.88493151,53.50821918
1988,67.52876712,38.06849315,52.79863014
1989,65.2,37.35342466,51.27671233
1990,65.98630137,38.1369863,52.06164384
1991,66.22465753,38.79178082,52.50821918
1992,69.70410959,40.58356164,55.14383562
1993,66.2,38.57534247,52.38767123
1994,67.00821918,38.01369863,52.5109589
1995,67.52876712,40.0109589,53.76986301
1996,67.24931507,39.65753425,53.45342466
1997,66.0739726,40.30410959,53.1890411
1998,66.04383562,40.01369863,53.02876712
1999,65.92876712,37.68767123,51.80821918
2000,66.10684932,38.15342466,52.13013699
2001,67.9369863,37.92054795,52.92876712
2002,67.99178082,36.2630137,52.12739726
2003,68.49041096,38.74794521,53.61917808
2004,68.02191781,38.67945205,53.35068493
2005,67.04383562,38.30958904,52.67671233
2006,67.57534247,38.35890411,52.96712329
2007,66.59452055,36.67945205,51.6369863
2008,66.57534247,36.60547945,51.59041096
2009,66.56438356,36.60547945,51.58493151
And I kept a record of any holes plugged from B-91 forms, nearest neighbor (Grant’s Pass), AMS Journal Monthly Weather Review or preceeding/following reading extrapolation.
What I would like to produce to go with my efforts is how to assign a confidence level. Ex – I fill in 2 days out of 365 by using other than an actual reading. How do I figure that?
E.M.Smith (22:24:26) :
“A station with ‘too few rural neighbors’ will not get a UHI correction at all. So as the rural stations are dropped, there are an ever greater number of urban stations that have their temperatures simply ‘passed through’ unchanged.
vjones (01:02:47) :
“As I understand it, an urban station with no rural neighbours will not “pass though” unchanged” but will simply not be used. There are many instances of trunkation of urban stations also where the urban record is long but the rural record adjusting it is short. The urban record is truncated to the date where it can be corrected by the rural neighbour(s).”
Where is you verification that this is actually what is happening? Even the guys who do the programming find that what they THINK is happening in the program is not necessarily what IS happening. Also WHO dropped the stations the computer program or the human inputting the information?
There is also the problem with the definition of “rural” The “UHI effect” can be caused “by any replacement of natural vegetation by man-made surfaces, structures and active sources of heat.” Dr. Spencer performed an analysis comparing International Hourly Surface data to population density. The study seems to indicate the effect of UHI on the temperature anomalies is the greatest during the growth of rural to a population density of 1000 people per sq. km.
http://wattsupwiththat.com/2010/03/04/spencers-uhi-vs-population-project-an-update/
My other problem with station drop out is that not all stations are created equal. As Chiefio showed the Pacific without Australia and New Zealand is basically flat as a pancake.
As is Central Park in NYC
http://www.john-daly.com/stations/WestPoint-NY.gif
Washington DC as of 2000 was cooling slightly
http://tinypic.com/view.php?pic=k2ekh5&s=6
The temperature pattern in many cases is cyclical and 30 yrs or even 100 yrs is much too short to show that cycle. So the biggest issues I have is first the short time period for the base line data and the observed data. And second choosing the baseline at the bottom of the cycle and then comparing that to the data gathered during the upswing of the cycle.
http://www.climate-movie.com/wordpress/wp-content/uploads/2008/12/temperature_adjustments1.gif
Longest running Siberian station
http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=222202920005&data_set=1&num_neighbors=1
I am sure the political pressure to DO SOMETHING NOW is because “they” are afraid we are going to head into a temperature down swing. That is why the chant changed from global warming to climate change. If a math student could back out the 60 yr data from the HadCRU3 time series, I am sure it is not a deep dark secret except to the unwashed masses, no matter how often we hear “hottest …. ever”
Math students analysis: http://dev-null.chu.cam.ac.uk/htm/soundandfury/220709-analysing_temps.htm
I’ll do Medford next from raw data, just to see what GISS has been up to.
Go look at Roman M’s blog.
==============
Roman M’s blog is statpad.wordpress.com
You can also click to it through Jeff Id’s ‘The Air Vent’ in the sidebar.
===============
steven mosher (23:10:55) :
Here is a nice link:
http://clearclimatecode.org/the-1990s-station-dropout-does-not-have-a-warming-effect/
Is that the graph that you are using to debate this point?
If so, IMHO then we may be debating apples and oranges (or I may be misinterpreting the debate myself).
I think E.M.’s position is;
1. the station drop-out is affecting the recent trend. The dropout station data in your graph is not displayed post ~1992. Complete the comparative analysis by continuing the data for the dropped out stations to date.
2. the dropped out station data is still being included in the determination of to-date anomalies even after being dropped out. In other words, current station anomalies are being averaged against a base period that includes the dropped out stations.
Tim Clark (11:15:24) :
“the station drop-out is affecting the recent trend. The dropout station data in your graph is not displayed post ~1992.”
If it showed no signs of affecting the trend before 1992, then you have no strong reason to think it would greatly affect the trend after 1992. It’s possible, but you can’t say it’s likely or certain.
“Complete the comparative analysis by continuing the data for the dropped out stations to date.”
Until somebody goes and gets all that data, this will be difficult. Roy Spencer avoided this difficulty by using a different data set altogether; his entirely different data set with no station drops shows similar trends to anybody elses’s. It sounds like the NOAA is collecting some of the missing data from the individual countries as we speak, so some of it may show up this year.
“the dropped out station data is still being included in the determination of to-date anomalies even after being dropped out. In other words, current station anomalies are being averaged against a base period that includes the dropped out stations.”
The graph linked here http://clearclimatecode.org/the-1990s-station-dropout-does-not-have-a-warming-effect/
shows that this objection does not matter. On the global scale, the dropped out stations weren’t doing anything different from the current stations.
Comparing Medford, Ore to Ashland, Ore
http://www.robertb.darkhorizons.org/TempGr/Med_AshAv.GIF
Raw data.
It’s obvious that while Ashland cools at night, Medford holds in UHI.
But the real news is that the daytime highs don’t really change that much.
So much for UHI Barbecue Days in Medford.
Next to determine what time of the year is most affected, if it’s not universal across the year.
Re: “Popping a Quiff” (Mar 8 22:07),
is the anomaly for the dropped stations that same as those retained?
Steven’s link shows that well. The answer is yes. Zeke has a new post looking at various breakdowns re associated factors.
And Tim , I think the CCC plot is the right one here. EMS’s claim is that the 1990’s dropout was selective. It chose stations known to perform differently. But that plot shows they weren’t performing differently.
Nick Stokes (13:51:54) :
To be fair to EMS, I don’t know if he’s outright and explicitly claimed the dropout was selective. That claim is more clearly in the SPPI report.
carrot eater (13:12:19) :
Tim Clark (11:15:24) :
“the station drop-out is affecting the recent trend. The dropout station data in your graph is not displayed post ~1992.”
If it showed no signs of affecting the trend before 1992, then you have no strong reason to think it would greatly affect the trend after 1992. It’s possible, but you can’t say it’s likely or certain.
I’m not taking a position either way. But please don’t be condescending. Look at the adjustments on the GISSTEMP page and you’ll notice they keep going up, up, up. So the effect of the adjustment is increasing. If the remaining stations have a greater adjustment value versus the dropped stations, then yes, I do have strong reason.
“Complete the comparative analysis by continuing the data for the dropped out stations to date.”
Until somebody goes and gets all that data, this will be difficult. Roy Spencer avoided this difficulty by using a different data set altogether; his entirely different data set with no station drops shows similar trends to anybody elses’s. It sounds like the NOAA is collecting some of the missing data from the individual countries as we speak, so some of it may show up this year.
So you are saying the data for the dropped stations post dropout is unavailable. Fair enough.
“the dropped out station data is still being included in the determination of to-date anomalies even after being dropped out. In other words, current station anomalies are being averaged against a base period that includes the dropped out stations.”
The graph linked here http://clearclimatecode.org/the-1990s-station-dropout-does-not-have-a-warming-effect/
shows that this objection does not matter. On the global scale, the dropped out stations weren’t doing anything different from the current stations.
That’s the same graph I looked at and linked to in my original posting. It does nothing to validate your contention post-dropout.
Tim Clark (14:59:56) :
GISS only makes one adjustment, and it’s for UHI, and this is what it does:
http://clearclimatecode.org/gistemp-urban-adjustment/
“That’s the same graph I looked at and linked to in my original posting. It does nothing to validate your contention post-dropout.”
It rather puts a damper on the idea that the missing stations were intentionally made to be missing in order to introduce some sort of spurious warming trend. And again, it gives you no particular reason to think that finding those missing data and putting them back in would have a major effect.
Tim Clark (14:59:56) :
And to clarify: if the two subsets add up to the same trends during the past, then there is nothing unfair about keeping the missing stations in the past. This is directly puts away the concern expressed in
“In other words, current station anomalies are being averaged against a base period that includes the dropped out stations.”
What about attaching weather stations to vehicles? I am assuming a fully automated system, possibly solar powered, and including GPS data, which either automatically transmits the data via wi-fi when they return to base, or via text or satellite links, depending on availability.
If, say, they were attached to mail vans then the urban deliveries would build up a good model of the UHI effect, and the more rural deliveries would provide an interesting picture of variability due to location.
In some of the remote areas of the world then attaching them to long distance lorries, buses or trains would provide information which could not possibly be delivered from a few scattered surface stations.
Just a thought.
carrot eater (15:38:12) :
Tim Clark (14:59:56) :
And to clarify: if the two subsets add up to the same trends during the past,
_________________________________
They most certainly do not add up to the same trends.
I just showed you a UHI influenced Medford, Ore vs a no-trend Ashland, Ore less than 20 miles apart on the map.
rbateman (16:34:36) :
You’re telling me about two stations someplace.
Zeke, Clear climate, Tamino and now Ron Broberg are showing what happens when you use all of them. All four find that if you calculate the global mean anomaly history using only the stations that continue past 1992, you get the same result (pre-1992) as when you use only the stations that dropped off by 1992.
And on top of that, it looks like both Medford and Ashland have data through to the present. Medford has data through at least 2009 in the GHCN. Ashland has data through 2009 in the USHCN. So what have they got to do with station drops, anyway? Neither were among the dropped. Maybe you’re making some other sort of point, I don’t know, but the topic here is whether the drop-off in stations around 1990 had any effect on the calculated global trends.
A couple of points:
1) carrot eater: EMS made his claim about “colder” high latitude or high altitude stations in one of his earliest posts on this subject. I don’t feel like wading through the morass of junk on his site to get you an exact link, but you know where the site is. This does seem to be the root of the SPPI claims.
2) Tamino partitioned the two sets of sites, post 1992 and pre 1992, into two datasets and ran the analysis. They both showed essentially the same anomalies except in the earliest periods.
Nick Stokes (13:51:54) :
I’m sure it’s not exactly the same. But it doesn’t matter since the stations retained could have been selected so that the anomaly of them would match the anomaly as if none had been dropped. There is room for manipulation to no end in deciding which stations to retain and which to drop. You should know this. And if you wanted to be unbiased it should matter to you. You also should be pointing out all the potential problems of dropping stations.
But the question that is vital is this:
is the temperature reading of the retained stations the same as those dropped?
Because I see only GIStemp is making claims about 2006 and 2009 that other data centers are not.
Rattus Norvegicus (18:30:21) :
So you are saying rural and mountain stations have not been dropped from GISS use?
Re: Rattus Norvegicus (Mar 9 18:30),
I like the “except in the earliest periods”.