Posted by Jeff Id on January 5, 2010
So I’ve learned a great deal playing around with the GHCN data, I think this is a reasonably significant post. Ya know, it’s hard to know anything until you try it yourself and I hope more of the readers here will. Again, there was a problem in my last CRU post, however, the more I look for it, the more avenues there are to explore. The issues have been corrected by avoiding the remaining possibilities in this post.
Of all the details worked out over the last two days, one is a decent gridded average of temperature data. Unfortunately for us skeptics it looks like Figure 1 which is pretty similar to the CRU plot.
Figure 1 – all data gridded
Yes, there is warming according to our temp stations, but I don’t think comrade Phil Climategate Jones would like this curve, because the warming in this curve happens entirely after 1975.
It’s nice to see a good quality CRU similar curve after the previous effort, but that’s how things happen when you do your work in public. The plot above uses all the data with each 5 digit temp station code averaged together individually, as my first post did. Anomaly is calculated over the entire series length.
The concern which was explored in some detail, regarded the hypothesis that the loss of stations in recent years created or biased the trend. It came about since so many stations are lost in recent years as Ken Fritsch pointed out in the recent CRU #3 thread.
Figure 2 – Stations per year GHCN per Ken Fritsch from KNMI database
I’ve run dozens of plots over the last several days, some of which contained an error in them created from data selection or a code problem in my previous post. Using the algorithm which averages together individual station ID numbers, I get very consistent CRUesque patterns. the warming is common to a variety of data sorting processes. This methods avoids the issues of data selection or code problems in the other methods and I’m confident in the accuracy of these results, but you should check them.
Several methods were employed to test the consistencey of result, including sorting for Rural and Urban, and sorting for several different time lengths of station data. All varieties so far produced very similar same results. There are, however, interesting revelations from examination of the slight differences.
Figure 3 – 100% Urban Data
Figure 3 is a plot is the urban data only. Of note is that the warming starts at 1978 with only slight warming beforehand and launches up about 1.2 C with no end in sight. Also, 1982 isn’t much reduced from around 1940 which is different from the global average in Figure 1. So the next thing I did was to plot the rural data.
Figure 4 – Rural station data.
That looks a great deal more like the satellite data. The temp rose and fell again prior to 1978 and rose again since 1978 is maybe 0.5C total. I tend to ignore data prior to 1900 due to the very small number of stations. I don’t think the drop in temps to 1900 levels in the early 70’s is the kind of curve that supports the high CO2 sensitivity claimed by climate science. Does anyone remember the snow storms of the early 70’s? Yeah, yeah just weather, I know.
One of the other avenues explored at great length , yet still isn’t finished, was how station starts and stops affect the trend in recent years. To explore that, one of the several methods I used was to sort data according to number of available data points. Below, I presented the gridded global average for all stations with at least 100 years (1200 points) of available data, since many of the stations in Figure 2 were started in 1950.
Figure 5 – Urban gridded data from stations with at least 1200 points
The urban data only in Figure 5 has an even steeper curve, you would expect this from longer series in this type of analyis. The temp rise since 1978 is about 1.2C. The rural 100 year curve is below.
Figure 6 Rural temperature stations with at least 1200 points
So the Rural stations show about 0.7C of warming since 1978. Visibly less warming than the urban stations by themselves. Also note the slight downtrend in recent years. Since the industrial revolution occurred a hundred years ago, it’s hard to imagine this curve is created by CO2. Still I’m not denying the heat capturing ability of CO2, just that the curves here don’t show a continuous warming but rather a short term recent spike.
So of course we should look at the difference between urban and rural stations.
Figure 7 – Difference between urban and rural data as identified by GHCN
Figure 8 – Difference between urban and rural data from GHCN stations with at least 100 yrs of data (1200 monthly points)
Look at that curve! Despite the crudeness of the categorization of thermometers, there is a clear warming bias for big city data. The curve in Figure 8 ends at 0.6C difference. What’s more, the trend between the two looks statistically significant. If Phil Climategate Jones and Michael Marx Mann can choose which data they want to show and hide the rest, I think it’s only fair to choose to look at trends only from Figure 8 since 1978 (even though it won’t make much difference). After all, one hundred percent of global warming has apparently occurred since that time. Let’s do a simple significance test.
Figure 9 – Statistical significance between rural and urban stations
Woah, it’s not even close, a trend of 0.12 and a no trend null hypothesis limit of +0.04. The difference between urban and rural warming is as great as the entire trend in UAH data over the same timeperiod.
Just how much trend do the ground stations show.
Figure 10 – All urban stations different Y scale from 11
Figure 11 – All rural stations different Y scale from 10
Even Figure 11 is still greater than UAH and RSS satellite data but it’s one heck of a lot less than the urban stations. Of course we would be remiss to not mention that WUWT has taught us what rural stations often look like.
What could go wrong with sophisticated technology like that?
The R code for this post is here.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Tom P (14:39:24) :
Regrettably, for your purposes, HADCRU figures must be considered toxic and self serving. There was no altruism at the University of East Anglia. Only the funding mattered, AFAIK.
OTOH, http://www.surfacestations.org/ has the chinks to show for it.
Moderator, please be advised, chinks are coins. Reference “Shakespeare in love”. I am suggesting that Surfacestations’ findings are more precious than gold.
“Shakespeare in love”
Apothecary. Tell me, in your own words.
Shakespeare. l-lt’s as if, my quill is broken. As if the organ of my imagination has dried up. As if the proud tower of my genius has collapsed.
A- lnteresting!
S- Nothing comes!
A-Most interesting.
S-lt’s like trying to pick a lock with a wet herring.
A-Tell me, are you lately humbled in the act of love ? How has it been ?
S- A goodly length in times past, but lately–
A- No, no! You have a wife, children ?
S-Aye. I was a lad of 18. Ann Hathaway was a woman, half as old again.
A- A woman of property ?
S- She had a cottage! One day she was three monthsgone with child, so–
A- And your relations ?
S- On my mother’s side, the Ardens.
A- No, your marriage bed?
S- Four years and a hundred miles away in Stratford. A cold bed, too, since the twins were born. Banishment was a blessing!
A- So, now you are free to love–
S- Yet cannot love, nor write it.
A-Here is a– a bangle… found in Psyche’s temple on Olympus. Cheap at fourpence. Write your name on a paper and feed it into the snake.
S- Will it restore my gift ?
A-The woman who wears the snake will dream of you, and your gift will return. Words will flow like a river. See you next week.
TomP:
I didn’t just ignore serial correlation. Had I done so, the result would have been significant at the 15 sigma level. When you subtract the two series before computing the trend in the residual, much of the serial correlation is removed, though I admit some still remains.
My best guess is p < 0.1, but I admit an improved statistical model is needed here, and it may increase the confidence intervals more.
This still is a substantially improved result over your blithe claim that “There’s no statistical difference in the warming trends.”
If you can reliably measure the difference between the two paradigms (which almost has to exist given they really aren’t measuring the same physical quantity using the exact same methodology), then of course it’s “scientifically interesting”, your cherry-picked quote of John Christy aside.
Or as is equally plausible, RSS is as flawed as people have been saying, and shouldn’t be agreeing so well with CRU. In other words the apparently strong agreement could be due to confirmation bias on the part of the RSS group.
If one focuses on the differences between UAH and CRU, which is a valid thing to do, one explanation for the differences has to do with an oversampling of urban versus rural sites in the surface temperature reconstruction using the surface temperature instrumentation.
So, no, you can’t say it’s “incorrect”, just that it is one of several explanations, and that includes errors in the UAH methodology.
As I’ve said above, the real problem here is one wouldn’t expect two methodologies to agree, even if both did everything “right”. There are differences to be expected between measurements in the boundary layers and measurements above it, for starts.
You seem to have some odd need to gloss over scientifically interesting questions, and have a great willingness to make statements of certainty where none exists. Why is that, do you suppose?
After 1975 or so, the coating on the temperature collection boxes in the USA was changed from whitewash (calcium carbonate) to latex paint. This has added a portion of degree of warming during the course of the day and warmth retention at night.
If the rural temps have increased by a fraction of a degree, the gradual implementation of this coating change could explain some of it.
Tom P,
The bias toward the northern hemisphere is the reason the trend is a bit higher in recent years. Don’t worry, when the whole globe is used together the truth of the matter is a little different. As I said above, you are premature in your judgment and reading too much into the trend. If you want, I’ve provided the code so you can do it yourself and see the answer before I write it up.
Carrick (09:42:49) :
“You seem to have some odd need to gloss over scientifically interesting questions…”
Not at all. The difference between UAH and RSS is worth looking at. But it’s curious that you discount the straightforward explanation of the difference given by John Christy and instead invoke confirmation bias by RSS without offering any evidence.
And why the bizarre accusation that I “cherry-picked” John Christy’s statement? Are there many more quotes where he offers alternative explanations?
Jeff Id (09:56:10):
“The bias toward the northern hemisphere is the reason the trend is a bit higher in recent year.”
OK, if we take nearly double the warming trend of your plot compared to HadCRUT to be “a bit”. But I look forward to your full writeup on this.
To Richard Wakefield
I am not at all a scientist and have measured nothing but I have noticed that over the past three years in Sydney (Australia) there has been an evening out of temperatures. The winters simply have not been cold for more than a few days.
I have not bought the children parkas for three years because it has been too warm for them in Sydney in winter.
Go back a five years and they had parkas. I hope someone with the ability to check this will do so. Is it simply my perception or is the Sydney winter disappearing? The last few summers have not been scorchers either.
I wish I had time to look at the data too. Here’s a thought. It seems that winters are warming since 1975 but summers are not, with the effect stronger in urban areas. This could be an urban heat island effect even in the rural stations— am I right in thinking that the UHI effect would warm winters more than summers, because air conditioning 20 degrees has less extreme effect than heating 50 degrees? (90 to 70 vs. 20 to 70).
Here are a couple more UHI implications to check for:
1. The warming should rise with the coldness of the winters at the station.
2. The warming should be greater at night than during the day — a bigger effect on the winter minimum than on the winter maximum daily temperature. (This is because more heat is needed at night, and this test holds holds time of year constant.)
Tom P:
The fact that their time series “happens” to line up so perfectly with CRU, when CRU doesn’t agree nearly as well with GISTemp (purportedly measuring the same quantity) is itself the evidence for confirmation bias. Of course the history of measurements of fundamental constants is replete with examples of confirmation bias, so it’s not like I’m suggesting the RSS guys are doing something people haven’t done 100 times before.
As I see it, CRU and GISTemp should agree, RSS and UAH should agree, but the two methods shouldn’t agree with each other, even if each measured what it purports to measure “perfectly”.
For what it’s worth, I actually like the job GISTemp does better than HadCRU, even though it shows more warming (how much warming is exhibited shouldn’t be a selection criterion, I like how GISTemp handles missing station data better, and the fact that their algorithms are online and fully replicable increases my confidence in what they are doing).
The fact GISTemp and UAH don’t disagree doesn’t disturb me particularly, with the one thing we should all agree on being that this disagreement doesn’t necessarily prove UHI contamination in the surface temperature record.
On the other hand, you no doubt selected CRU and RSS precisely because of the four temperature series, they were the only ones that agreed.
John Christy clearly understands that he’s measuring a different quantity than the surface temperature record. What he’s obviously saying is “given how they aren’t measuring the same quantities”, it’s interesting how close they are to each other, nonetheless.
Beyond that, bringing up a qualitative statement in an attempt to refute a statistical inference is just another version of appeal to authority on your part, which amounts to a form of logical fallacy on your part. (If the quantities vary by a statistically measurable amount, what Christy’s opinion of the degree of agreement is, is irrelevant.)
I simply MUST comment on that photo of the rural station.
I see urban heat island happening, even in this still photo. How?
Look at the flag, blowing left to right. Any heat from the building will be carried by the prevailing wind toward the met station.
Look at the time of year (the trees and field). At least at this time of year the heated building is affecting temp readings to some degree.
There are fully six windows exposed on that side of the building, allowing more heat to leach out of the building than a solid wall would. Not only that, but because the six are spread out changes in the wind direction will still likely allow the heat escaping one of the windows to blow directly past the met station.
Look at the closeness of the station to the building (the most obvious error in met station placement and probably the LOL as to why this image was chosen here). Because it is so close, there is less probability of the building’s escaping heat to disperse with the wind. In addition, it is so close as to be in the “draft” of the building, sheltered from the winds (by intent?) – but that puts it within the cocoon of the building’s warmth. Unless the wind comes from about a 350 heading around eastward to a 190 heading the station is sheltered by a building, not exposed. That is the exact opposite of what a rural station (actually ANY station) should be.
And finally, look at the height of the station. It is exactly the same height as the vertical center of the windows, meaning the escaping heat does not blow over or under it, but right TOWARD it.
If you built a FUNNEL from the building to the met station you could hardly direct the escaping building heat – and then cuddle it there – to the met station any better. Even if the wind shifts, it is most likely still going to be readings some of the escaping building heat.
ALL of this would be moot, if the flag was blowing the opposite direction.
Since prevailing winds do just that – prevail – the odds of this photo being taken while the winds came from some anomalous direction does exist, but is not altogether likely. In my own location, winds come from the west about 75-80% of the time. I actually saw a map of this about 7 years ago, and I still recall the general info on that map, but only generally. As I recall, winds from the NE/E/SE came less than 10% of the time, and from the N or S about 15% of the time. So, there if that one is similar to my area (the features of the photo show it could have been taken in my area as well as any other), the wind probably blows as shown about 75% of the time.
The photo screams urban heat island effect. Even as the fields tell us it is rural, not urban.
And one final, final point:
If that building is air conditioned in the summertime, the air conditioner is extracting heat from the inside and pumping it outside (that is what air conditioners do, after all). And no matter what side of the building the exhaust is on, the prevailing wind and the turbulence around the building will direct some of that toward that met station.
Air conditioners is one of my own pet suspects – specifically because of the rearranging of heat out of buildings and into the external atmosphere where temps are measured. I notice that the timing of the urban rise somewhat matches the beginning of the widespread use of air conditioners. Is it a coincidence? Possibly, but I hypothesize that there is a link. I understand that the increased curve (whatever that actually is – I don’t trust the HadCRU adjusted figures any more than anyone here) is higher for the topics – exactly where air conditioners make life bearable and make urbanization much more possible – this would tend to support such a speculation that in measuring the temps in places like Malaysia we are not measuring the overall heat level (which includes the temp indoors, too), but only the external heat level, which is biased because of the extra heat added from within buildings and then pumped out of doors.
TomP:
Nearly double?
HadCrut is 1.57C/century over that period, Jeff’s rural land stations is 1.90C/century.
That’s just about a 25% difference. But you’re comparing to the wrong data set:
HadCRUT3 is land+sea
As Manfred points out above, GHCN is land stations only.
crutem3vgl (just land stations) gives 2.11C/century.
GISS land temperature by comparison gives 1.83 C/century.
Carrick:
Your arguments become ever more contrived. Now you see the excellent agreement between data as being evidence of bias in RSS! And you characterise of my citing of Christy’s quotation as “bringing up a qualitative statement in an attempt to refute a statistical inference” when you admitted your “statistical inference” was a guess, and the “qualitative statement” put in numbers for differences between the trends!
But as to comparisons between data, I agree it makes sense to compare like with like. So we have from 1978 to present for just the land stations:
Jeff Id’s rural stations: 1.90 C/century
crutem3vgl: 2.11 C/century
GISS 1.83 C/century.
So the spread is just 15%, with Jeff’s value in between.
I would be very surprised if either the GISS or crutem trends were statistically different from Jeff’s value. Certainly there is no scientific significance in the differences. From all three datasets it looks like for the last thirty years we’ve had a land warming trend of 1.95±0.15 C/century.
If such warming were to continue it would certainly be a cause for concern.
There’s nothing contrived about the fact that satellite measurements don’t measure the same physical quantity as the surface measurements, and the default is they shouldn’t agree as well as they do (as I mentioned there are specific quantifiable differences that can and should be modeled).
It is “very surprising” that the agreement between RSS and HadCRUT are so tight compared to any other time measurements especially given what we know about things that are left out of HadCRUT, because of the way it grids temperature, it underestimates the SST trend for one thing.
When I see two quantities agree too well that I believe shouldn’t, of course my first reaction is “prove to me that you didn’t unintentionally distort your result to match.” I think taht should be your instinct too, and I’m sorry that it isn’t, because in my opinion it demonstrates your lack of objectivity.
Again.
…
You are glossing over the science in pursuit of your political agenda. Not interested.
There are corrections left out of CRUTemp (and Jeff ID’s analysis) that GISTemp has included, and one would expect CRUTemp and Jeff’s analysis to both run high, and they do. Looks to me like the trend analysis has enough fidelity to pick this out to me. I’d like to see them agree to at least 5% after everything is said and done and I believe that should be very doable.
Also, if you have the fidelity to measure, you have the fidelity to discover anomalies in the measurements. Whether those anomalies that arise are “scientifically interesting” is left up to observer. I’m involved in physical measurement, so I find it intrinsically interesting, I don’t know your background that well, so I don’t find it interesting that you don’t.
After all, yesterday’s systematic errors are the basis for today’s measurement methods of choice.
While I’m not saying I think GISTemp is the “end of the game” in surface measurements, I do think it is a pretty decent piece of work. Fully open software approach, full access to the data they use, and people are replicating it now right and left. Yes one can pick on it for how they categorize urban versus rural, for example, but that is where the surfacestations survey will help.
From my point of view, the satellite measurements match more closely the physical quantities simulated in the GCMs, so there is work to be done to reconcile the two approaches.
Of course, I agree.
But we also need to make sure the measured values are accurate. It’s going to be hard to reconcile climate models with measurement if for example you have a 25% error in your temperature time series.
Carrick (11:29:42) :
“I’d like to see them agree to at least 5% after everything is said and done and I believe that should be very doable.”
I’d be happy to see measurements of the same quantity agree to within their errors, rather than some arbitrary percentage. To wish for anyhting better is not really scientific.
“But we also need to make sure the measured values are accurate. It’s going to be hard to reconcile climate models with measurement if for example you have a 25% error in your temperature time series.”
Actually the spread in the projections is wider than the spread in the surface temperature measurements:
http://www.realclimate.org/index.php/archives/2009/12/updates-to-model-data-comparisons/
It’s not the accuracy in the land surface temperatures which is mainly holding back the modelling here, but rather better observations of temperatures and energy flow in the oceans and upper atmosphere.
TomP:
For somebody who isn’t a practitioner you sure have your share of advise for those of us who are! Seriously, to “agree” in science implies within the uncertainty of the measurement.
Nah. It’s the inability of the models to capture short term variability, and maybe long term too.