Posted by Jeff Id on January 5, 2010
So I’ve learned a great deal playing around with the GHCN data, I think this is a reasonably significant post. Ya know, it’s hard to know anything until you try it yourself and I hope more of the readers here will. Again, there was a problem in my last CRU post, however, the more I look for it, the more avenues there are to explore. The issues have been corrected by avoiding the remaining possibilities in this post.
Of all the details worked out over the last two days, one is a decent gridded average of temperature data. Unfortunately for us skeptics it looks like Figure 1 which is pretty similar to the CRU plot.
Figure 1 – all data gridded
Yes, there is warming according to our temp stations, but I don’t think comrade Phil Climategate Jones would like this curve, because the warming in this curve happens entirely after 1975.
It’s nice to see a good quality CRU similar curve after the previous effort, but that’s how things happen when you do your work in public. The plot above uses all the data with each 5 digit temp station code averaged together individually, as my first post did. Anomaly is calculated over the entire series length.
The concern which was explored in some detail, regarded the hypothesis that the loss of stations in recent years created or biased the trend. It came about since so many stations are lost in recent years as Ken Fritsch pointed out in the recent CRU #3 thread.
Figure 2 – Stations per year GHCN per Ken Fritsch from KNMI database
I’ve run dozens of plots over the last several days, some of which contained an error in them created from data selection or a code problem in my previous post. Using the algorithm which averages together individual station ID numbers, I get very consistent CRUesque patterns. the warming is common to a variety of data sorting processes. This methods avoids the issues of data selection or code problems in the other methods and I’m confident in the accuracy of these results, but you should check them.
Several methods were employed to test the consistencey of result, including sorting for Rural and Urban, and sorting for several different time lengths of station data. All varieties so far produced very similar same results. There are, however, interesting revelations from examination of the slight differences.
Figure 3 – 100% Urban Data
Figure 3 is a plot is the urban data only. Of note is that the warming starts at 1978 with only slight warming beforehand and launches up about 1.2 C with no end in sight. Also, 1982 isn’t much reduced from around 1940 which is different from the global average in Figure 1. So the next thing I did was to plot the rural data.
Figure 4 – Rural station data.
That looks a great deal more like the satellite data. The temp rose and fell again prior to 1978 and rose again since 1978 is maybe 0.5C total. I tend to ignore data prior to 1900 due to the very small number of stations. I don’t think the drop in temps to 1900 levels in the early 70’s is the kind of curve that supports the high CO2 sensitivity claimed by climate science. Does anyone remember the snow storms of the early 70’s? Yeah, yeah just weather, I know.
One of the other avenues explored at great length , yet still isn’t finished, was how station starts and stops affect the trend in recent years. To explore that, one of the several methods I used was to sort data according to number of available data points. Below, I presented the gridded global average for all stations with at least 100 years (1200 points) of available data, since many of the stations in Figure 2 were started in 1950.
Figure 5 – Urban gridded data from stations with at least 1200 points
The urban data only in Figure 5 has an even steeper curve, you would expect this from longer series in this type of analyis. The temp rise since 1978 is about 1.2C. The rural 100 year curve is below.
Figure 6 Rural temperature stations with at least 1200 points
So the Rural stations show about 0.7C of warming since 1978. Visibly less warming than the urban stations by themselves. Also note the slight downtrend in recent years. Since the industrial revolution occurred a hundred years ago, it’s hard to imagine this curve is created by CO2. Still I’m not denying the heat capturing ability of CO2, just that the curves here don’t show a continuous warming but rather a short term recent spike.
So of course we should look at the difference between urban and rural stations.
Figure 7 – Difference between urban and rural data as identified by GHCN
Figure 8 – Difference between urban and rural data from GHCN stations with at least 100 yrs of data (1200 monthly points)
Look at that curve! Despite the crudeness of the categorization of thermometers, there is a clear warming bias for big city data. The curve in Figure 8 ends at 0.6C difference. What’s more, the trend between the two looks statistically significant. If Phil Climategate Jones and Michael Marx Mann can choose which data they want to show and hide the rest, I think it’s only fair to choose to look at trends only from Figure 8 since 1978 (even though it won’t make much difference). After all, one hundred percent of global warming has apparently occurred since that time. Let’s do a simple significance test.
Figure 9 – Statistical significance between rural and urban stations
Woah, it’s not even close, a trend of 0.12 and a no trend null hypothesis limit of +0.04. The difference between urban and rural warming is as great as the entire trend in UAH data over the same timeperiod.
Just how much trend do the ground stations show.
Figure 10 – All urban stations different Y scale from 11
Figure 11 – All rural stations different Y scale from 10
Even Figure 11 is still greater than UAH and RSS satellite data but it’s one heck of a lot less than the urban stations. Of course we would be remiss to not mention that WUWT has taught us what rural stations often look like.
What could go wrong with sophisticated technology like that?
The R code for this post is here.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
The rural temps are also affected by the heat island influence. It is part of a degree, but it takes a flat line into an upward slope.
Jeff’s analysis is spot on – the conjunction of trend, adjustment and location requires more in depth investigation.
I’ve been looking at trends and adjustments in raw and adjusted data for over a week now. They are mapped here (by KevinS):
http://82.42.138.62/MapsNOAA.asp
http://82.42.138.62/MapsGISS.asp
The analysis is not quite ready to post yet, but it is clear there is a lot of warming in the raw data – even the rural data. It is far from uniform.
There is more information on the database that allowed production of these maps here: http://diggingintheclay.blogspot.com/2010/01/climate-database-development.html
Some people have reported difficulties with registration and apologies if that is still a problem. Fixed soon I hope.
I’m currently downloading the daily temp data from 1900-2009 of all 1300 Canadian stations from Envrionment Canada’s website. It’s about half done after 2 weeks. However, I have started to do some evaluations of this daily temp data, and have found something interesting. Indeed, for the one location, Belleville, Ont, the yearly average temperature has increased and followed the basic anomaly graphs we have seen, with an increase up until 1945, followed by a slight decrease until about 1980, followed by an increase until today.
But what does that actually mean? I have discovered that the over all temperature ranges for each year has been narrowing. That is, in the 1940s they had very hot summers, and lots of them (4.1% of the days above 30C), and very cold winters with more than twice the number of days below -20 than today. The number of hot days (over 30) dropped to just 1.7% between 1955 and 1983, then increased to 3.7% and remained there since 1984. The number of days below -20 has consistently dropped since the 1930’s to half.
Thus what we are seeing in the average temperatures is not a physical measurment of an increase in real temperatures, but a narrowing of the variation within the years over time.
We are seeing less cold and shorter winters, with virtually no change in the summer temperatures.
If this methodology can be applied to other areas of Canada, which I will do when I get the rest of the stations, and then applied to other global stations, and the same trend emerges, then AGW is dead in the water. The planet is not heating up at all. Just the variation between extreme temperatures is narrowing. Hardly scary.
I was checking out Charlotte, NC tabular data on the GISS website
Here’s the data
http://data.giss.nasa.gov/work/gistemp/STATIONS//tmp.425723140001.1.1/station.txt
Somehow they are missing July 2009 for Charlotte.
Well they gave a June-July-August temperature average of 25.7 for Charlotte.
That means they gave July a temperature of 26.5
I checked Charlotte out day by day and the temp there for July was 25.0
^^^ GISS somehow has the temperature there a full 1.5 degrees Celsius warmer than it really was for July.
What is really amazing is that the Charlotte data goes from 1881 to present day. Only 3 months are missing (May 1985), (Feb 2004), and (July 2009).
I checked July 2009 and GISS mad some “man-made” global warming to that month. Wonder what their great formula’s are doing to other sites where they “somehow” don’t have the data.
Excellent post Jeff. The basis of AGW is erroneous temperature analysis. The analysis is either mishandled, or intentionally biased. In either case, errors are made in assumptions/treatments of that data.
This post will be highlighted on RealClimate within the hour. The hockey stick returns to life. At least this time it is balanced, nuanced and believable. Great post. Thanks
Jeff, I very much like your blog and visit there often. As you note there seems to be great siting problems with now the majority of GHCN stations being really urban even though classified as rural. Then there are problems with the rural stations with the “new remote sensors” that can’t be located an adequate distance from the “base atation” due to cabling problems. It would be a real treat to read the reasoning for the closing of so many stations (I believe that most of these are still in operation). It would seem that most of the closures coinside with the supposed catastrophic rise in “global warming”.
Soon there will be a major change in our perspectives as it gets cooler and more and more people are drasticly effected by the “climate.”. With the shenagians that have beem noted by the :climate” experts it is no wonder that we know very little about our climate system even with Millions and millions being spent on computer power. When the controling authority over the research is more interested in political correctness and maintaining the cash flow than in finding the truth and knowlege our scientific system has a true problem. We were warned about this in the fare well address of Dwight Esienhouer. I believe he was well aware of what the future could be and rried to give us an insight.
Great Job, I just wish that a collection of data carried forwared from many of those “closed” stations could be collected and if they were mostly of what type and locations.
Thanks for your good work.
Bill Derryberry
(spell checker appears to not be working this morning my appologies.)
Question:How well is the Urban Heat Island Effect compensated for ?
I went to the GISS website and downloaded the temperature record for two locations I know a little about.
Here is the link
http://data.giss.nasa.gov/gistemp/station_data/
The two data sets I downloaded were
Hensley Field [ a Naval air station which started in 1923 and is substantially unchanged]
DFW airport was a cow pasture 1/2 way between Dallas and FW] prior to 1977 when a massive airport was built.
Since they are about 15 miles apart they should have the same temperature differences.
No raw data is available but the least “corrected” data had “USHCN corrections” applied.
DFW airport pattern makes a big “U” witgh the upturn starting in …..You guessed it 1977.
Who would think Jets and runways etc would have an effect on temperature ?
The Hensley fields pattern was an “L”. there was no uptick after 1977.
The “Fully Adjusted data for both don’t change the pattern at all.
Conclusion THE ADJUSTMENTS DON’T REMOVE THE PHONY WARMING FROM THE DFW RECORD..
Since this is the main station for the DFW area the whole area is mis-represented.
This is in the USA with the most accurate surface station records in the world ?
What must Asia be like ?
This is a non peer reviewed study although I have given enough data it can easily be replicated in 20 minutes.
I hate taking even the skeptics assertions at face value.
If you look into bias introduced by stations that have been dropped, you might consider looking into elevation. Since temperature in the atmosphere falls about 6.5 degC per kilometer, there is a significant opportunity for a warming bias to be introduced into the land record when higher elevation stations drop out. (All 20th century warming is roughly equivalent to a 100 m drop in each elevation.)
I don’t know if the algorithms used to extract “higher value” data from the raw temperature record includes elevation. If you have a grid cell with uneven terrain and one of a small number of stations dropped out, there should be a major discontinuity in the record. Step discontinuities in individual station records are often assumed to be due to relocation or equipment change and a given a step-function correction.
what about that wonderful graph from noaa that can be found here:
http://cdiac.ornl.gov/epubs/ndp/ushcn/ts.ushcn_anom25_diffs_urb-raw_pg.gif
Supposedly there is an explanation for all the adjustments but are even half of them capable of meeting common sense validation?
What are the fudge factors tossed into the mix for Jeff’s starting data?
The mid-1970s is when the Clean Air Acts started the progressive reduction in urban particulate pollution (smoke and haze). This produced the observed increasing minimum temperature trend due to increased early morning sunlight reducing minimum temperatures.
Which BTW accounts for well over 100% of the increase in surface temperature since that time. Something even Jones and Karl agree is significant.
http://www.ncdc.noaa.gov/oa/climate/mxmntr/mxmntr.html
Having the most powerful computer ever made means silch, really, crap data fed into crap mathematical models using whatever code will still produce crap.
Is there a definition ready at hand for RURAL and URBAN as far as classification of data points goes? Is the definition universally accepted and applied, or do different groups have different definitions?
Certainly, the notion of “rural” regarding US Census data is NOT always what you would expect.
Has anyone tried to classify the stations into an independent Rural-Urban grouping, on the basis of a GIS analysis of the surrounding land use or population density, rather than a zipcode or county-based classification. (I’m guessing that’s how it’s done now.)
Here’s Jeff Id’s rural temperature plot compared to the CRU data in red. I’ve offset the CRU data to match by eye, but the scales, and hence trends, are identical:
http://img99.imageshack.us/img99/8886/cruvsidrural.png
So the unlikely conclusion: Climate sceptic identifies a higher recent warming trend than CRU!
(Off topic) Not sure it this has been noted before, but I just read on the UK Met Office website the explanation by their chief scientist, Prof. Julia Slingo, of the proof that CO2 causes global warming.
The question put to this “scientist” is:
“How do you know CO2 is responsible for the change in climate and can you prove it? And how do we know that CO2 released is from human activities?”
Her response comes down to this:
“We are now rapidly approaching 390 parts per million which means it’s been a 40% increase. Most of that increase has happened in the last 50 years. And if we know that carbon dioxide is a greenhouse gas, it’s hard to believe that if you increase it by 40% you’re not going to do something to the temperature of the planet.”
So, it’s “hard to believe” that it won’t “do something” – there’s scientific rigor for you!
http://www.metoffice.gov.uk/corporate/pressoffice/in-depth/ask/julia-slingo.pdf
Try riding a motorbike in winter time.Even the warmth from a squirrel fart is perceptable when you are cold enough.
The Location of the above ‘Rural’ station so near to a badly insulated shiplap building with facing windows would certainly skew the figures.
Looks pretty similar to the PDO index with an added high level of TSI to give it a boost.
This is still a good post, to me it is the best real evidence for warming I’ve seen in years.
Of course the warming is only half of what has been claimed and the bias corrections are lacking so that the half a degree that remains is almost certainly a large overstatement but at least this work doesn’t have deliberate fudging of the data. If Philip B is right, as I suspect he is, there may indeed be no evidence for AGW or any warming in the CHCN data.
But, at least it’s possible that we may now learn the truth and it is possible there has been some warming. Prior to this I was starting to think it was all pure baloney.
Which data was used, raw or adjusted ? Because in a high percentage of station records I have checked in the GISS dataset there I have found a negative stair step adjustment with later years being adjusted DOWN significant amounts and gradually stepping up to no or little adjustments in the last decade/s.
I’m still not convinced the raw records, even with the UHI, show anything like the slope published by the warmers. Given that most adjustments I have seen are NOT UHI adjustments (which would grow over time, not shrink) I still think the data is being gamed.
Jeff is your analysis based on raw raw data or adjusted raw data? If it’s really raw raw data may I ask where you got it? I’ve been unable to find it. Even what they call raw is not really raw.
thanks
dave
So, we have diddled data in the code, we have Mann’s 0.5 degree y2k increase, we have the UHIE on both urban and ‘rural’ stations, we have the result of reducing particulates in the atmosphere in the 1970s.
If all of these effects are removed from the data, would would be the result? A hockey stick pointing -down-?
As a non-scienist but someone with a background in engineering I’ve picked up on what I see as two significant comments posted so far.
1) Since the mid 1970’s the winters were warmer but the summers were not hotter.
2) Polution in urban areas was reduced considerably since the mid 1970’s which would help explain much of the urba/rural warmth difference.
Wouldn’t these two facts pretty much account for the warming trend that most of us experienced as “weather” between 1975 and 1998?
Anthony,
I commented on Jeff’s site that I wondered what the relationship of the blade was to the mmts installation rate. Specifically the change in distance from the sensor to the building due to the cable run. Didn’t you have a graph of that data at one time?
REPLY: Not a graph, just an observation about the issue – A
Was the rural data here raw or adjusted? I’m asking because of FILNET. If FILNET was used to fill in missing rural station data, then the data is contaminated by the well known urban station problems.
Is the UAH the best data?
Jeff (07:28:04) :
Raw data was used.
Tom P (07:05:07) :
Identical? You sure?
Before I start, this post has nothing to do with trend but rather a problem in the data potentially created by local heating influence. This post calls into question the station data itself so claiming a trend of one magnitude or another is beside the point.
As you note in the rural plots, all of the warming happened in recent years. This would say CO2 had basically limited warming effect prior to 1978 – unless you use urban data of course. If you need me to explain the point further, it appears that urban bias is a real effect which has significance. It would be foolish to simply assume (eyes closed) that the same bias doesn’t exist in the rural stations as you can see from the last picture.
Scientifically we cannot ignore the reality of the problem, or the recent proof in climategate emails that those in power like Jones are happy to ignore it. In one email a scientist discussed removing points from the temperature curve in his presentations so people wouldn’t see the decline. What we need is good surfacestations.org style QC of the global station data, nothing more, and definitely nothing less.