This new paper by Dr. Ross McKitrick of the University of Guelph is a comprehensive review of the GHCN surface and sea temperature data set. Unlike many papers (such as the phytoplankton paper in Nature, complete code is made available right from the start, and the data is freely available.
There is a lot here that goes hand in hand with what we have been saying on WUWT and other climate science blogs for months, and this is just a preview of the entire paper.This graph below caught my eye, because it tells one part of the GHCN the story well.
1.2.3. Growing bias toward lower latitudes
The decline in sample has not been spatially uniform. GHCN has progressively lost more and more high latitude sites (e.g. towards the poles) in favour of lower-latitude sites. Other things being equal, this implies less and less data are drawn from remote, cold regions and more from inhabited, warmer regions. As shown in Figure 1-7, mean laititude declined as more stations were added during the 20th century.
Here’s another interesting paragraph:
2.4. Conclusion re. dependence on GHCN
All three major gridded global temperature anomaly products rely exclusively or nearly exclusively on the GHCN archive. Several conclusions follow.
- They are not independent as regards their input data.
- Only if their data processing methods are fundamentally independent can the three series be considered to have any independence at all. Section 4 will show that the data processing methods do not appear to change the end results by much, given the input data.
- Problems with GHCN, such as sampling discontinuities and contamination from urbanization and other forms of land use change, will therefore affect CRU, GISS, and NOAA. Decreasing quality of GHCN data over time implies decreasing quality of CRU, GISS and NOAA data products, and increased reliance on estimated adjustments to rectify climate observations.
From the summary: The quality of data over land, namely the raw temperature data in GHCN, depends on the validity of adjustments for known problems due to urbanization and land-use change. The adequacy of these adjustments has been tested in three different ways, with two of the three finding evidence that they do not suffice to remove warming biases.
The overall conclusion of this report is that there are serious quality problems in the surface temperature data sets that call into question whether the global temperature history, especially over land, can be considered both continuous and precise. Users should be aware of these limitations, especially in policy sensitive applications.
Read the entire preview paper here (PDF), it is well worth your time.
h/t to E.M. Smith
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
“sure looks to me like they are stand alone versions. Also the v2.mean.Z size is 12016 KB while the v1.mean.adj.Z size is 8672 KB which implies to me a near parity”
Near parity?
you have a file with 12MB of “raw” data. That data is adjusted. the adjusted lines are written out. 8.6MB. Otherwise you end up with like 12 stations in 2009.
Thats when the lightbulb went on. The majority of adjustments happen early on.
late in the record you have fewer adjustments. fewer lines to replace.
A first I thought it was stand alone, but when you actually plot out area maps of every year you can see that past 2005 there is almost NO DATA. Why? cuase its a change log.
THAT is why Ross chart goes bonkers.
Its a change file.
I believe carrot confirmed this with his contacts inside Noaa, or ron did.
http://rankexploits.com/musings/2010/the-ghcn-adustment-puzzle/#comments
EM Chad appears on that thread which is why I was asking Ross.
Good to see that climate science is finally progressing after years of sabotage and blockade by the hijacked peer review process.
This article should become standard literature for 10% of the climate scientists, while the other 90% will start do something useful, after the global warming swindle is over.
In before the lock as they say, or should that be the crys of the uninformed, ‘But it’s not peer reviewed’.
Saying that if i was a true scientist i would welcome such comment on my work as it can only make it better, whats that flipper it’s not about science but money, power and changing the way every single human being live’s, stop talking nonsense your just a dolphin.
And the ususal pat on the back, bag of peanuts and pint of beer for all the hard and excellent work carried out by Dr McKitrick.
Ross, there are a few areas with typos and expression could be clearer here and there, but who am I to comment?
I seek to raise an additional, possibly major source of error, on the topic of anomaly methods. It’s not novel.
On p.33 of your draft we have “CRU combines obviously duplicated (identical) records, then computes anomalies, then grids the resulting series (Brohan et al. 2006). Anomalies are computed by subtracting the 1961-1990 mean (the so-called “normal”). ”
Elsewhere as in fig 1.1, you point to 1961-1990 as a period when the greatest numbers of observing stations were included in the GHCN data. What, then, is the “normal”? Is it the constant historic number derived from the 1961-1990 stations in fig 1.1, or is it an ever-changing number that changes because stations were later dropped from the 1960-1991 period in great volume? The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
So, how does one treat anomaly calculations before 1961 if the normal is not a constant? OTOH, if the normal is a constant, how does one treat data after 1960? Presumably data have been dropped after 1960 because they have problems, so these problems will affect the normal from 1960-1991 in that it will include defective, rejected data.
Some have commented that the absolute values matter less than trends and that trends are less affected by the anomaly method, a solution with so many problems that it deserves a section of its own in your paper. For a start, should a trend be linear of polynomial and why? What is wrong with using physical real numbers?
There is great value in your paper, because it shows the artificiality of the adjustment methods, spiced up by admissions from climategate (and there are more than you quote). I’m reminded of John Cleese in Fawlty Towers, who is a master at digging deeper and deeper holes by successive slight adjustments to his stories. In the case of these global temperatures, the hole cannot be filled in again because no single person knows who made which adjustment of what size to whatever station data, before or after another adjuster had a go. The reconstruction back to truth is now irreversible.
Unless, of course you have the splendid ability of the Sir Muir Russell group who knocked over the prioblems in a couple of days. What a shallow, pathetic exercise that was.
If you seek another section for your paper, try an examination of the country data adjustments performed before GHCN again adusted their “raw” data.
Still, a double shot cappucino is a good way to start a boring day.
John Finn “Do airports have a greater warming trend than nearby rural locations? There doesn’t appear to be much evidence that they do.”
Do you have any quantitative data?
Why not model an airport in terms of the energy needed to taxi, lift and lower a 100 tonne object to 100m altitude, n times a day, spread the energy over the airport volume and see how many watt per sq m you get at ground level. This can then be converted to a temperature change. I can’t find anyone who has done this simple exercise.
Kevin:
It’s not my data either, and none of us are trying to “guide policy using it”, so this argument is just a distraction.
In the end, I would suggest that the adjustments don’t matter that much, because the two main effects seen in the data is the global mean trend (which is about 10x the corrections to it that people worry about) and secondly the south-to-north variability in the trend.
All data have warts, the $20 question is whether they matter. I would claim for the dominant questions (is it warming? is the northern climes warming faster than the southern ones?), the real effects are big enough that the problems with the data don’t affect those conclusions.
Dr. McKitrick,
Thanks for the clarification. And thank you for the paper. It was easy to read and understand. While I do have a background in statistics (my degree is in Economics), it has been a long time since I have had to use the science as my current profession lies in computer networks. But it was good to exercise some rusty brain cells!
Geoff Sherrington says:
August 4, 2010 at 5:20 am
Do you have any quantitative data?
Why not model an airport in terms of the energy needed to taxi, lift and lower a 100 tonne object to 100m altitude, n times a day, spread the energy over the airport volume and see how many watt per sq m you get at ground level. This can then be converted to a temperature change. I can’t find anyone who has done this simple exercise.
I’m not sure how that would influence the trend. I don’t think we need to go to that much trouble. If the trends at airport stations are broadly similar to trends at nearby rural stations then we can assume UH has little or no influence on the overall warming (or cooling) trend.
Thie issue is not whether airports are warmer than rural locations (the tropics are warmer than the poles) but whether airports have warmed at a faster rate. There appears to be no evidence to suggest there is a significant difference between the two.
Carrick says:August 4, 2010 at 6:15 am
In the end, I would suggest that the adjustments don’t matter that much, because the two main effects seen in the data is the global mean trend (which is about 10x the corrections to it that people worry about) and secondly the south-to-north variability in the trend.
A little bit of an poofy blather to make your point don’t you think. Considering the warming since 1970 is ~.7 F and the adjustments are ~.45 F., (TOBS, Station relocation, and instrumental change, none of which were made by side to side comparison with actual empirical data; but not including the “quality” changes made by GHCN prior to distributing the “raw” data) just doesn’t seem to add up to 10X.
Tim Clark.
TOBS is an empirical model. Validated with real observations. Its applied, as far as we know, in the US. Anyway, TOBS is REQUIRED. this can be proved with observational data. Go look at the CA thread on TOBS. Next, the method was developed by studying hundreds of stations across the US. Stations that had HOURLY data. This allowed researchers to calculate the effect from CHANGING the observer time. That study resulted in a prediction method. Based on the lat/lon time of year, position of the sun, etc, the prediction code would take the recorded min/max at say 7AM and predict the min/max at the prior midnight. The code was then verified against a portioon of the sample that was “held out” during model creation. Very well done. There is a SE ( standard error of prediction) BUT if you did not correct for TOBS you would have a biased record. TOBS removes the bias, but introduces additional uncertainty.
Changes in instrumentation are likewise based on observational studies. Station RELOCATIONS. The station moves that need attention are moves in lat/lon and moves in elevation. Elevation moves are corrected for by looking at lapse rates.
Its a one time adjustment. Same for lat/lon moves.
Geoff.
WRT your desire to model the “energy” required at an airport.
The difficultly is that to do this properly you need to understand the local wind conditions and things like the turbulant mixing that occur when a plane lands.
Also, You need to know the time of day. There are two critical periods.
The Tmin period ( say late night ) and the Tmax period ( say mid day).
If you dont have planes landing or taking off in the Tmin window then the chances of infecting that record are slim. Tmax? I would bet your chances are better looking at that.
Mitigating factors: The wind, clouds, rain. Since airports tend to be located in areas with a long fetch, long open fetch, with consistent wind patterns, you are going to have a lot of heat moved by that nice laminar flow over the undisturbed surface.
Again, People always look at the picture and forget the dynamics of the situation.
One data point. In the CRN study a small warming bias at the airport was found. .25C
Modulated by clouds, winds and rain. Those three factors kill UHI.
So, if you had a 2C warm bias on Tmax.. Then Tave would get a 1C bias. Tave=(tmax+tmin)/2. One BENEFIT of not integrating over time.
Next, if you see a 1C bias on sunny days and half your year is cloudy. that puts the bias down to .5C. Windy days? same thing. rainy days? same thing.
Is there a bias? probably, phsyics seems to indicate as much. How big? on a few days it could be large. Over the course of years and years? We can measure that my looking at lots of airports. Its on the list of things to do.
BUT to do it right, you want that wind data.
Geoff:
Elsewhere as in fig 1.1, you point to 1961-1990 as a period when the greatest numbers of observing stations were included in the GHCN data. What, then, is the “normal”? Is it the constant historic number derived from the 1961-1990 stations in fig 1.1, or is it an ever-changing number that changes because stations were later dropped from the 1960-1991 period in great volume? The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
Some clarity on how anomalies are calculated.
For EVERY station you do the following.
1. See if the station has a minimum number of “years” in the period. I use 15 FULL years. I can change this and make it anything I want. Doesnt change answers much.
I have also used other periods. The 1953-1982 is the “richest” period WRT total station months. ( months of data)
So: now for that station you compute a MEAN Jan, mean feb, mean march etc.
So, now you have a mean FOR THAT STATION.
jan =14C
feb =15.5C
mar=16.2C
And so on.
Now you look at the entire time series.
if Jan 1926 was 13.5C your anomaly is 13.5-14C. or .5C
If the station drops out in 1995.. you get NA
To calculate the global average you:
Take a gridcell: 120-123;45-48: (for example)
Average all the stations within that grid. You got a gridcell average.
do that for all grids with data.
Then: look at all the grids: SUM the area (on the sphere) that those grids cover. If they dont have data that month, then NA.
Then you look at the land area for each grid, on the sphere. Say 100sqkm
You then create a weight: cellarea/total area.
Multiply your grid anomaly by the weight. for every MONTH. Then you sum the weighted grid anomalies.
Thanks Steve (Mosher) for the query about the GHCN adj graph. I don’t have an answer at the moment. I have been tied up today reading a dissertation, and tidying up loose ends for a new paper with Steve and Chad, and tomorrow I am offline most of the day. I’ve asked Chad for help on this.
Personally I find the chimney brush very strange and I wouldn’t be surprised if there’s something we missed in the definition of delta that at least partly explains its bizarre appearance.
Also I hope that one of the points that emerges from my paper is that a lot of bloggers like Chiefio, Chad, Bob Tisdale, Jeff Id, RomanM etc. have done impressive work on explaining these temperature products, and their contributions deserve to be read. Not to say I was convinced by everything I came across, but I did often find myself wondering, why am I learning about this from a blogger, instead of from the IPCC (or NOAA)? They’re the ones that rely on the data to make their points, so they should also show the background information to gives readers a sense of the potential problems with continuity, sampling, measurement quality, etc., instead of always waving them away or making people guess about what’s going on.
When I saw the graph and term “chimney-brush”, I must confess I thought of a more common utensil nowadays – a toilet brush. Or in British parlance, a bog brush.
As E.M. Smith correctly points out, the essence of the bias built into the GHCN data set derives from the splicing together of anomalies from a set of stations that DIFFERS from the beginning to the end of the reconstructed record. If stations little affected by UHI are used in constructing the “norm” in any grid-cell, then virtually any apparent trend can be obtained by varying the ratio of corrupted-to-pristine records near the beginning and end of the reconstruction period. That something akin to this has been done is quite apparent from the steep drop-out of “rural” stations since 1990, especially in Asia, where the reputed “global warming” is most intense. There are also other issues, but this lies at the very core.
Paul Deacon, Christchurch, New Zealand says:
August 4, 2010 at 5:28 pm
When I saw the graph and term “chimney-brush”, I must confess I thought of a more common utensil nowadays – a toilet brush. Or in British parlance, a bog brush.
__________________________________-
Darn it now I have to clean my computer screen again. I think Dr McKitrick was just being polite but your name for it will probably be the one that catches on.
Steven mosher says:
August 4, 2010 at 11:10 am “WRT your desire to model the “energy” required at an airport.”
I agree with what you write, but I’m looking at ballpark figures. I have a suspicion that busy airports do not create enough fuel-derived heat even on a windless day to change the temperature much. If this is the case, it cuts short a lot of speculation about the desirability of airport versus non-airport station siting.
That would cut out quite a lot of verbiage. The present discussion seems to assume significant fuel heat.
The work I have done is limited, but I could pick up no significant difference from comparison sites where there is a genuine rural site close to an airport site. I did not use big hub airports.
WRT time of day, do you think that daily temperature max-mins are those recorded electronically these days – or is there curve fitting and rejection of transients? If the latter, it would be hard to splice mercury records to thermistor/thermocouple records as they were successively replaced. Have you ever sen an instrumental overlap comparisson or any code to cope with the transition?
Steven Mosher – where I speculate “The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
I am I right or wrong? I know the procedure you patiently describe again, but I am unsure of the fate of the “NA” cases or what effect they have on the global anomaly on a given day like today.
Steve, you are right, it’s a change file. The chimney brush uses a delta series constructed such that v2.mean is computed on its spatial basis, and v2.mean.adj is computed only using the gridcells with adjusted data, rather than filling the remaining cells with unadjusted data. That conflates two sources of difference, the adjustments themselves and the drop in sample size between raw and adj. To isolate the adjustment effect we’re putting together a new version where we only compare based on cells where both raw and adj are present. It looks like a chimney brush after being run over by a truck. The alternative is to assign a zero adjustment to all non-adj gridcells, which we are checking as well. But that assumes that the adjustment is estimated as zero, rather than being set to zero due to a lack of data. It is not reasonable to assume the adjustments would be zero everywhere the data happen to be missing for computing the actual adjustments.
Thanks for the assistance.
This may have already been answered (I know, I know. But it’s past 3 AM in the morning, and I don’t have the time to trawl through all the posts looking to see if this has been covered), but figure 1.6…puzzled me. How, exactly, was ~30% of GHCN data taken from airports in the 1890s? Is it just that stations were situated at sites that were later to become airports (which seems peculiar; why would that be so?) If that’s the case, anyone happen to know why airports were, apparently, often built where old weather-monitoring stations had been constructed?
Another question I had was concerning graph 1.7; were these mean values calculated by giving northern latitudes positive values and southern latitudes negative values, or was it all absolute value (I ask because if it positive for northern and negative for southern, the shift towards the equator could be considered good news, as it would mean that more of the southern hemisphere was being accounted for than before. This supposition doesn’t seem to be born out by the listings of station coverage for northern and southern hemispheres, though. If absolute value…well, less so, since an ideal coverage would (area-wise) yield an average latitude of 30 degrees, assuming I calculated things correctly)? This may have been covered in the appendix, but I’m not particularly computer literate and I don’t really know what the code given there means. Many thanks if anyone is willing to spare the time to answer these questions.
FWIW, I’ve put up a “Volatility” posting that explains the effect I mentioned above:
http://chiefio.wordpress.com/2010/08/04/smiths-volatility-surmise/
And per the issue of airport change / warming trend over time. It think it’s pretty clear that you can look at places like, oh, most any major international airport with dozens (hundreds?) of acres of concrete and tarmac and figure it was a grass field transpiring water to maintain a low leaf temperature 100 years ago. So go from grass field to black solar collector with constant growth over time for most of them. Yeah, I’d call that a ‘warming trend’. FWIW, in my lifetime I’ve watch much of the growth. Populations grow, airports grow. Little grass shack tropical fields are now 10000 foot jet airports with A/C and giant terminals and non-stop traffic of both cars and planes. Far more ‘trend’ than just about any other place and more than the nearby cities (that were often in existence prior to the first airplanes). I’ve got many examples in the links posted above, but one of my favorite is the Seaplane harbor at Milano that became tarmac as intercontinental travel moved from Pan Am Clippers to 707s. Water to concrete and tarmac does kind of count as ‘warming trend’. If you don’t believe me, go stand in the parking lot as long as your bare feet can stand it in August, then run and jump into the lake…
Sam Yates says: August 5, 2010 at 12:45 am
“Another question I had was concerning graph 1.7; were these mean values calculated by giving northern latitudes positive values and southern latitudes negative values, or was it all absolute value”
I think Sam’s suspicion is right. The mean latitude is signed:
lat_mean_unweighted[i] = sum(temp*lat)/sum(temp)
That gives wrong results. It means, for example, that the big dive in mean latitude in 1990 came when a big batch of China/Canada/Turkey stations was eliminated, and the sharp recovery came in 1992 when an even bigger batch of Australian stations was removed.
Is there any reason why this paper or a version of it is not being published in a peer reviewed journal?