Posted by Jeff Id on January 5, 2010
So I’ve learned a great deal playing around with the GHCN data, I think this is a reasonably significant post. Ya know, it’s hard to know anything until you try it yourself and I hope more of the readers here will. Again, there was a problem in my last CRU post, however, the more I look for it, the more avenues there are to explore. The issues have been corrected by avoiding the remaining possibilities in this post.
Of all the details worked out over the last two days, one is a decent gridded average of temperature data. Unfortunately for us skeptics it looks like Figure 1 which is pretty similar to the CRU plot.
Figure 1 – all data gridded
Yes, there is warming according to our temp stations, but I don’t think comrade Phil Climategate Jones would like this curve, because the warming in this curve happens entirely after 1975.
It’s nice to see a good quality CRU similar curve after the previous effort, but that’s how things happen when you do your work in public. The plot above uses all the data with each 5 digit temp station code averaged together individually, as my first post did. Anomaly is calculated over the entire series length.
The concern which was explored in some detail, regarded the hypothesis that the loss of stations in recent years created or biased the trend. It came about since so many stations are lost in recent years as Ken Fritsch pointed out in the recent CRU #3 thread.
Figure 2 – Stations per year GHCN per Ken Fritsch from KNMI database
I’ve run dozens of plots over the last several days, some of which contained an error in them created from data selection or a code problem in my previous post. Using the algorithm which averages together individual station ID numbers, I get very consistent CRUesque patterns. the warming is common to a variety of data sorting processes. This methods avoids the issues of data selection or code problems in the other methods and I’m confident in the accuracy of these results, but you should check them.
Several methods were employed to test the consistencey of result, including sorting for Rural and Urban, and sorting for several different time lengths of station data. All varieties so far produced very similar same results. There are, however, interesting revelations from examination of the slight differences.
Figure 3 – 100% Urban Data
Figure 3 is a plot is the urban data only. Of note is that the warming starts at 1978 with only slight warming beforehand and launches up about 1.2 C with no end in sight. Also, 1982 isn’t much reduced from around 1940 which is different from the global average in Figure 1. So the next thing I did was to plot the rural data.
Figure 4 – Rural station data.
That looks a great deal more like the satellite data. The temp rose and fell again prior to 1978 and rose again since 1978 is maybe 0.5C total. I tend to ignore data prior to 1900 due to the very small number of stations. I don’t think the drop in temps to 1900 levels in the early 70’s is the kind of curve that supports the high CO2 sensitivity claimed by climate science. Does anyone remember the snow storms of the early 70’s? Yeah, yeah just weather, I know.
One of the other avenues explored at great length , yet still isn’t finished, was how station starts and stops affect the trend in recent years. To explore that, one of the several methods I used was to sort data according to number of available data points. Below, I presented the gridded global average for all stations with at least 100 years (1200 points) of available data, since many of the stations in Figure 2 were started in 1950.
Figure 5 – Urban gridded data from stations with at least 1200 points
The urban data only in Figure 5 has an even steeper curve, you would expect this from longer series in this type of analyis. The temp rise since 1978 is about 1.2C. The rural 100 year curve is below.
Figure 6 Rural temperature stations with at least 1200 points
So the Rural stations show about 0.7C of warming since 1978. Visibly less warming than the urban stations by themselves. Also note the slight downtrend in recent years. Since the industrial revolution occurred a hundred years ago, it’s hard to imagine this curve is created by CO2. Still I’m not denying the heat capturing ability of CO2, just that the curves here don’t show a continuous warming but rather a short term recent spike.
So of course we should look at the difference between urban and rural stations.
Figure 7 – Difference between urban and rural data as identified by GHCN
Figure 8 – Difference between urban and rural data from GHCN stations with at least 100 yrs of data (1200 monthly points)
Look at that curve! Despite the crudeness of the categorization of thermometers, there is a clear warming bias for big city data. The curve in Figure 8 ends at 0.6C difference. What’s more, the trend between the two looks statistically significant. If Phil Climategate Jones and Michael Marx Mann can choose which data they want to show and hide the rest, I think it’s only fair to choose to look at trends only from Figure 8 since 1978 (even though it won’t make much difference). After all, one hundred percent of global warming has apparently occurred since that time. Let’s do a simple significance test.
Figure 9 – Statistical significance between rural and urban stations
Woah, it’s not even close, a trend of 0.12 and a no trend null hypothesis limit of +0.04. The difference between urban and rural warming is as great as the entire trend in UAH data over the same timeperiod.
Just how much trend do the ground stations show.
Figure 10 – All urban stations different Y scale from 11
Figure 11 – All rural stations different Y scale from 10
Even Figure 11 is still greater than UAH and RSS satellite data but it’s one heck of a lot less than the urban stations. Of course we would be remiss to not mention that WUWT has taught us what rural stations often look like.
What could go wrong with sophisticated technology like that?
The R code for this post is here.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Richard Wakefield (06:23:29) ….I have also been looking at Environment Canada’s data for the town of Smithers in central BC…..I have found that from 1943 to 2009 the summer and the winter have a different trends…a bit odd
For summer it is 0.013 deg-c/year
for winter it is 0.026 deg c/year
In other words summers have not really warmed up much but the winters have…???…the airport at Smithers is out of town (google earth)
Jeff ID: “I’m confident in the accuracy of these results, but you should check them.”
Now there is something you will never read in any Team literature.
Jeff,
I have trawled throught your site and sites such as climate audit and real climate etc and cannot find any explanation for the methods used in actually measuring the temperatures. Can anyone point me to where the specifications for the surface thermometers are kept for all this data? What I do not understand is how thermometers with a visible resolution of 2 deg F and and accuracy of +/- 4 deg F (at best!) can be used to provide monthly average data to 0.01 deg F? In my world, this is just not acceptable, and I have been measuring pressures and temperatures for over 25 years as part of my job. In my business, if a thermometer (talking about mercury or alcohol glass thermometers) has graduations of 2 deg C on the scale, the temperature is recorded to the nearest 1 deg C. If you average 1000 readings and get something like 12.77 Degrees, you have to round it up to 12C. Why are Climatologists carrying ‘false’ accurate temperatures to 2 decimal places?
Tom P (07:05:07) :
Hmmm. No. I think that you are missing the point.
Very interesting what you say about Clean Air Acts. Does anyone have dates for when these came into force around the world? The UK act was passed in 1956 and it marked the end of the London “pea-souper” smogs. Not 1970s as postulated above, at least in this industrialised country.
Also surely particulate emissions are growing in developing countries with all the industry and forest clearing that is going on now. Wouldn’t that cancel out the clean air acts in developed countries?
Great example of UHI here in Denver this AM
16 below zero at the airport (pretty much rural), 1 deg above at City Park
17 degress difference!
You can see this consistently in the hourly obs:
http://www.crh.noaa.gov/product.php?site=bou&product=rwr&issuedby=CO
Latest hourlys aas I post (9AM):
OZ035-036-038>040-071700-
…NORTHERN COLORADO FRONT RANGE…
CITY SKY/WX TMP DP RH WIND PRES REMARKS
DENVER INTL AP MOSUNNY -7 -11 83 SE7 30.32R WCI -21
DMNS CITY PARK N/A 10 6 82 CALM 30.33R
AURORA MOSUNNY 1 -8 66 W1 30.29R
FRONT RANGE AP SUNNY -6 -9 84 E6 30.32 WCI -18
CENTENNIAL MOSUNNY 2 -5 72 CALM 30.27R
BROOMFIELD MOSUNNY 1 -4 78 CALM 30.30R
LOVELAND MOSUNNY -8 -18 59 S5 30.37R WCI -19
$$
Still 17 degrees warmer in the city than the airport (-7 vs +10)
I see this all the time here. The odds UHI is undercompensated for is very very high.
“Veronica (08:29:31) :
[…]
Also surely particulate emissions are growing in developing countries with all the industry and forest clearing that is going on now. Wouldn’t that cancel out the clean air acts in developed countries?”
Not necessarily. A lot of that industrialization happens with very modern technology, for instance german or american companies building car production lines in say Mexico or India. China produces solar cells, they use the most modern equipment from germany (Roth+Rau machines for instance). Often developing countries can skip the bad solutions we had in the past.
This may be OT but I think it’s worth mentioning. It’s the irony that the first thing the (Green) Brits turn to when the weather gets a bit nippy is: fossil fuels.
This from the (Green) British Broadcasting Company.
“The mercury fell to -18C overnight in places and temperatures were typically between -8C and 0C at lunchtime.
The Arctic conditions are expected to continue for up to a week.
The National Grid has issued its second gas alert in three days, with demand expected to hit a new record of 454 million cubic metres on Thursday.”
As an oil industry geologist, and not a climate modeler, am I wrong in feeling a little smug?
May I also state that there’s no way I’m going to shave of what’s left of my hair and grow a goatee beard.
lws (06:46:41) :
That information on Dallas is a lovely find and makes a very good point.
There are three stations on the GISS site:
Dallas-Fort W; 32.9 N/97.0 W
Dallas/Faa Ap; 32.9 N/96.8 W
Dallas/Hensley Fld Nas; 32.7 N/97.0 W
All have the same WMO code 72259 and are classified as Urban (>4,000,000 popn). They are therefore adjusted which makes a substantial difference to the trends.
Your local knowledge suggests adjustment for Hensley field should not ‘require’ adjustment. That could make a big difference to local anomaly values. That is why this is so flawed. I am tempted to look into this station set further.
Richard Wakefield (06:23:29) : I’m currently downloading the daily temp data from 1900-2009 of all 1300 Canadian stations from Envrionment Canada’s website… I have found something interesting… We are seeing less cold and shorter winters, with virtually no change in the summer temperatures…
Ah, but there is more, Richard. Just, just, just have a look at this plot from Salehard, edge of Yamal, Russia. I think it is FordPrefect’s work but, arghh, I forget… What is stunning to me is that the seasonal difference shouts recent UHI from the rooftops. So I wonder if what you observe is undetected UHI.
It would sure be nice to have a surface temperature data roadmap that describes the process of taking actual thermometer data and ending up with global temperature graphs.
It would answer questions like:
1. How are daily averages for each station calculated?
2. What processes are applied to raw data to create the GHCN dataset?
3. What processes are applied to GHCN data to get CRU and GIStemp?
4. How is missing data handled in each dataset?
5. How are missing land and sea areas handled in the various datasets?
6. What raw information and process descriptions are missing that would be required to duplicate the work of GISS, CRU, etc.?
I envision an open-source “World Temperature Project” with some chapters missing, but at least identified as areas that need to be filled in. It would include the most recent versions of software, as available. Such a thing could also include information from surfacestations.org.
This would enable the web community to duplicate and verify the work that is being relied upon for so many policy decisions. And perhaps, using the power of the web, to create even better temperature datasets.
Richard Wakefield (06:23:29) :
Thus what we are seeing in the average temperatures is not a physical measurment of an increase in real temperatures, but a narrowing of the variation within the years over time.
Very nice analysis. As an outsider, I have trouble understanding the desire to come up with daily then monthly averages for each station, then to group stations geographically and average further to finally to come up with some average “global” temperature for a month. There’s way too much interesting data that’s tossed in that sort of strategy. Arithmetic averages between a daily high and a daily low miss so much variation when there are hourly readings available. Analyses such as yours provide ample justification for making full use of all the [raw!] data.
But then I’m a splitter, not a lumper.
Strange then that young Peters plot on Youtube of US rural sites show no warming?
Unadjusted raw data, is it now lost for ever?, did they hide it somewhere?, if so, could we try some enhanced interrogation techniques-no statistical of course- in order to make them confess where it is? ☺☺☺
ah, raw … I assumed as much … thanks for the answer though 🙂
Since almost all of met stations are on the Northern hemisphere, the result is skewed to warmer trend. Tropics, Antarctic and Southern globe have not definite trends as N extratropics. If every station occupies 10km2 on the whole surface, then plain average would do fine.
Until Arctic (where the CO2 rise in dry cold air should manifest mostly) shows 40ties as warm as present and falling again, there is no much sense in the whole CO2 warming theory.
http://climexp.knmi.nl/data/icrutem3_hadsst2_0-360E_70-90N_na.png
Thanks for the careful analysis of temperature data.
Here is a Christmas gift for all WUWT readers that I received from my good friend, Kirt Griffin of Climate Topics.
http://tinyurl.com/yfbgsjw
With kind regards,
Oliver K. Manuel
Nice effort but since we know the data has been grossly corrupted, there is really know what is going on – maybe we can get some reliability from the sat data.
When doing a station survey it is extremely important to note any light colored walls to the NORTH of the device.
Walls or landscape features can reflect and refract enough of the sun to increase temps by 15-20 degrees (or more) in an enclosed space (room). {I have used this ‘trick’ to make Senior housing warmer and more marketable}
Until someone tests the effects of this type of heating of a Stevenson screen – we need to remove these stations from the record.
Sorry but it should read:
Nice effort but since we know the data has been grossly corrupted, there is really “No Way To” know what is going on – maybe we can get some reliability from the sat data.
Jeff Id (08:14:45) :
“Identical? You sure?”
Yes, the same y-axis scale.
“This post calls into question the station data itself so claiming a trend of one magnitude or another is beside the point.”
Well, the fact that the recent rural trend from your analysis is greater than CRUs trend is hardly evidence of UHI bias in their plot – that’s a very relevant point to make.
“This would say CO2 had basically limited warming effect prior to 1978…”
Agreed in that no strong effect is discernible and consistent with the CRU data: their warming trend after 1978 is more than six times the slope of the trend before that year. The CO2 warming is understood to have been offset by cooling from particulate emissions during the previous decades.
“…it appears that urban bias is a real effect which has significance…”
Also agreed, and why UHI is compensated for by both CRU and GISS in deriving their gridded temperatures. This might explain why the CRU and GISS recent trends are below even your rural-station trend, which itself would not be immune to UHI effects. The UHI compensation is validated during the period of greatest warming through agreement with the satellite data.
“Scientifically we cannot ignore the reality of the problem…” and nor has it been by climate scientists in producing their results, as I think you might now be appreciating. I’d be interested to see any proof in the emails that UHI has been deliberately ignored – Jones has actually published papers on this subject!
But I appreciate your efforts in producing these plots. Those with contrarian beliefs about global warning, whether you with surface temperatures or Roy Spencer with satellite temperatures, are producing consistent results in agreement with others. This should give some pause to those who claim bias, or worse, in these temperature plots.
Jeff Id and Lucy Skywalker
Temperature information only means something if we are comparing like for like. In this respect we should always bear in mind that Thermometers are designed purely to measure the micro climate immediately around them.
I have been tracking a number of historic temperature records and there are two main reasons for recent temperature increases.
The first is UHI- cities have grown enormously since most weather stations were set up in open fields or parkland at the edge of what were then small towns. The thermometer is recording the change in its immediate microclimate as buildings grow around it.
The second factor is that many stations have physically moved. As an example I have been tracking the temp data for Bologna Italy which has been recording since 1814. In recent years it has moved some 20 miles and is now here.
http://server.gladstonefamily.net/site/LIPE
So the Bologna temperature record now resides at airport marconi –also known as borgo panigale. This is the Bologna temp record from 1814 to current. http://climatereason.com/LittleIceAgeThermometers/Europe.html
This study demonstrates the Bologna UHI effect is up to 6 degrees C.
http://www.springerlink.com/content/7lt5g728v38k0538/
The rural areas on the Adriatic side of the Appennine mountain chain have been cooling in recent decades so the recent uptick in Bologna is a combination of UHI/station change
Each record needs to be looked at individually otherwise we are in danger of comparing apples with oranges. There is simply no comparison between the temperature record of the site in parkland at Bologna University in 1814 and that now being taken at an airport 20 miles away in 2009.
No amount of ‘adjustment’ will disguise that we are talking about fundamentally different temperature records from very different locations which just happen to have the same name, but other than that bear no comparison to each other.
Tonyb
” Jimmy Haigh (08:42:12) :
This may be OT but I think it’s worth mentioning. It’s the irony that the first thing the (Green) Brits turn to when the weather gets a bit nippy is: fossil fuels. ”
There’s no wind blowing so we could have a 200 million wind turbines all producing nothing and solar’s a bit lacking in sun this time of year. Nuclear is not popular but could do the business. We’ve got lots of coal under the ground…not popular since the 80’s. Hobson’s choice methinks.
Jimmy Haigh (08:42:12) :
This may be OT but I think it’s worth mentioning. It’s the irony that the first thing the (Green) Brits turn to when the weather gets a bit nippy is: fossil fuels.
And 4×4’s, 4×4 drivers have been volunteering to undertake recovery duties of 2WD cars and deliveries for OAPs (old age pensioners)
How about a refund of the hugh road tax hit for them Gordon ?
Huge not Hugh, spelling is not my strong point!!!
Jeff ID:
Of course that is what the climate science says: According to the GCMs, prior to the mid-1970s, anthropogenic CO2 and sulfates more or balanced each other.
While I agree on the need to be cautious, urban islands are real sources of heat. Eventually, that isn’t a bias in temperature, but a source of actual heating of the local climate.
What we want is the mean value of the surface temperature field. If you could do it with an infinite budget (this is a gedanken experiment_ you’d do it e.g., with a resolution of 1-m.
It wouldn’t matter if a particular thermometer is sitting on an air conditioner or not. That’s the real surface temperature for that site. And the average over that 510 trillion or so thermometers (sure hope I did the math right) is the real mean temperature field.
No let’s do another thought experiment: ManBearPig launches an assault at all the power stations, and all of those urban heat islands shut down. Would there be a net difference in surface temperature in the next year? The short answer is “yes”, though for an average over the globe it’d be pretty small (most land surface is not urban, and 3/4 of the Earth is ocean, so…)
In practice, we want to do is replace that array of 510 trillion or so thermometers with a much more course grained array. Is this doable in principle?
Well the answer is yes, if all you care about is daily measurements, because of wind, which advects hot and warm turbulent cells over a given microphone. With a 3 m/s wind, you end up over a day with a column of air about 250-km long that you are averaging over.
Even if a site is in an urban location (say in a park so it’s well away from local heat sources), you should be able to recapture the average temperature over that hypothetical grid of 1-m spaced thermometers. But my point here is, what you don’t want is the urban site replaced by what it would have been the urban site was not there, you just want the “true” value for the urban site, and one biased by very localized heat sources,
This same argument applies mutatis mutandis to changes in environment from environmental effects due to land-usage changes, such as deforestation for agriculture or even (as we are seeing where I live) reforestration of old farms to provide for the timber market.