This new paper by Dr. Ross McKitrick of the University of Guelph is a comprehensive review of the GHCN surface and sea temperature data set. Unlike many papers (such as the phytoplankton paper in Nature, complete code is made available right from the start, and the data is freely available.
There is a lot here that goes hand in hand with what we have been saying on WUWT and other climate science blogs for months, and this is just a preview of the entire paper.This graph below caught my eye, because it tells one part of the GHCN the story well.
1.2.3. Growing bias toward lower latitudes
The decline in sample has not been spatially uniform. GHCN has progressively lost more and more high latitude sites (e.g. towards the poles) in favour of lower-latitude sites. Other things being equal, this implies less and less data are drawn from remote, cold regions and more from inhabited, warmer regions. As shown in Figure 1-7, mean laititude declined as more stations were added during the 20th century.
Here’s another interesting paragraph:
2.4. Conclusion re. dependence on GHCN
All three major gridded global temperature anomaly products rely exclusively or nearly exclusively on the GHCN archive. Several conclusions follow.
- They are not independent as regards their input data.
- Only if their data processing methods are fundamentally independent can the three series be considered to have any independence at all. Section 4 will show that the data processing methods do not appear to change the end results by much, given the input data.
- Problems with GHCN, such as sampling discontinuities and contamination from urbanization and other forms of land use change, will therefore affect CRU, GISS, and NOAA. Decreasing quality of GHCN data over time implies decreasing quality of CRU, GISS and NOAA data products, and increased reliance on estimated adjustments to rectify climate observations.
From the summary: The quality of data over land, namely the raw temperature data in GHCN, depends on the validity of adjustments for known problems due to urbanization and land-use change. The adequacy of these adjustments has been tested in three different ways, with two of the three finding evidence that they do not suffice to remove warming biases.
The overall conclusion of this report is that there are serious quality problems in the surface temperature data sets that call into question whether the global temperature history, especially over land, can be considered both continuous and precise. Users should be aware of these limitations, especially in policy sensitive applications.
Read the entire preview paper here (PDF), it is well worth your time.
h/t to E.M. Smith
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
From Dr. Ross McKitrick’s paper section “1.3 Increasing magnitude of adjustments employed to try and fix the problems of sampling discontinuities”:
——————-
Dr. McKitrick,
I think you have made blog history with the term “the chimney-brush” in describing the ‘GHCN Delta = Adjusted GHCN – Unadjusted GHCN’ graph (your Figure 1-10).
I had not heard the term before. I like it.
John
The loss of temperature stations to the global temperature record is well known, even though the stations continue to exist and collect the data. The loss of higher elevation/cooler area stations noted here is even more alarming. Here in Canada we have 35 stations going into the record (if I got the number right) at present, though thousands are being recorded. Has Jones/Hansen/Schmidt, NOAA/HadCrut/GISS put out reasons for this data loss?
David Ball says:
August 3, 2010 at 7:48 am
You haven’t been paying attention in class, have you.
Well – yes I have actually – for several years as it happens and I can’t recall anyone showing that the trend at airport stations is significantly different to the trend at rural stations. Nick Stokes, in this post, suggests that airport and rural trends were similar
Nick Stokes says:
August 3, 2010 at 7:25 am
Zeke looked at this in detail, and found that a reconstruction using airports vs non-airports gave very similar results. I looked less thoroughly, but found the same.
My own observations are almost certainly less thorough than Zeke’s and Nick’s, but one in particular may be of interest. The Armagh Observatory in Northern Ireland is cited as an example of the perfect site for temperature measurments. David Archibald refers to it in his own research. The observatory is in a rural setting which is relatively unchanged in the past 2 centuries. Aldergove airport in Belfast is just a few miles away. Temperatures have been measured at Aldergrove since 1880.
Between 1975 and 2004 the Armagh (rural) trend is actually greater than the Aldergrove (airport) trend. The difference is even more pronounced if you go back to 1880. The Armagh trend is ~1.5 times the Aldergrove trend. Aldergrove data is part of the GISS station database. I’m not sure if this data is already adjusted for UH but if it is, it looks as though UH effect has been over-estimated.
Peter Stroud says: An excellent paper that should be read by all IPCC sponsored scientists. One thing puzzles me though. On page 11 we see the percentage GHCN stations located at airports from 1890 – 2009. But according to WikiAnswers the first airport was built in 1909, College Park Maryland.
That is an artifact of how GHCN stores the Metadata. (Explained in the link in the paper). Basically, GHCN has a broken metadata structure. You can only assign a “present status” to a location, not a status-by-year. So if a bit of dirt becomes an airport, it will be marked as an airport for all time. Similarly, an “urban” station will be “urban” even if in 1800 it was a cow field. The airport at Templehof Germany (you know, Berlin Air Lift…) is being converted to a shopping mall. Whenever the metadata are changed, it will suddenly evaporate from ever having been an airport and for all time in GHCN. Just another stupidity in the data…
So you can make a report of AirStation FLAGS, but not of actual airports, by year. This means it’s an approximation (that will be conservative) in that you must choose to “start time” a bit after 1900 as zero airports with the knowledge that the slope of airport increase is actually much stronger than shown in the metadata.
Ackos says:
Are similar bias present between high and low altitude? Rural v city?
Yes.
http://chiefio.wordpress.com/2009/11/16/ghcn-south-america-andes-what-andes/
http://chiefio.wordpress.com/2009/11/17/ghcn-the-under-mountain-western-usa/
http://chiefio.wordpress.com/2009/12/01/ncdc-ghcn-africa-by-altitude/
and a whole lot more at:
http://chiefio.wordpress.com/category/ncdc-ghcn-issues/page/3/
and other pages in that category.
BTW, Sinan Unur has great stuff well worth watching. The animations are very informative.
Thanks for posting, Anthony! I just opened the paper for a moment, and this popped out immediately:
——-
The GHCN website does not report which sites are included in the monthly updates, but recent tabulations using GHCN records shows that:
• coverage has fallen everywhere (including the US)
• the sample has become increasingly skewed towards airport sites;
• the sample has migrated from colder latitudes to warmer latitudes;
• the sample has migrated to lower altitudes.
—-
…as you’ve been reporting for many months!
Yeah, I can’t wait to see this pop up on CNBC, CBS, BBC etc. They are too busy reporting about plunging phytoplankton populations due to, what, maybe a 0.5 C increase in temperature? IF there is even a temperature increase!
JamesS says:
If it gets warmer because of increased urbanization, well then, it gets warmer. Are they trying to make the numbers look like what they might be if nothing had changed?
I found that a particularly interesting question, as the first step ought to be looking for warming in the base data ( I can’t call what they produce “raw” – it isn’t.) If there is no warming even WITH UHI, then there really is no warming.
I wonder what these graphs would look like if only the raw data were used? When I look at the raw data for site in out-of-the-way places in West Virginia, or other places that haven’t seen any major development over the years, there is no AGW signature visible in those records. What’s being done to the raw data for the sake of “accuracy” is a crime against data.
Also, if you look at long lived stations, there is no ‘warming signal’. It only shows up in the constantly changing kaleidoscope of station change.
Here’s what it looks like in the “base data”. The “unadjusted” GHCN (that has been adjusted during various “Quality Control” steps…) with full artifact inclusion using a very simple “self to self” anomaly where each thermometer is compared only to itself and only inside the same month. A very clean comparison. (graph in link). It’s pretty much dead flat in summer months, but the winters warm from the horridly cold ones of the Little Ice Age ( “year without a summer”..) to a flat recent trend.
http://chiefio.wordpress.com/2010/07/31/agdataw-begins-in-1990/
Then 1990 hits and it all goes very crazy as the instruments are changed in odd ways. The onset of the “new” methods (new “duplicate numbers” for old stations) is in 1986, then the old ones are dropped in 1990 (ish) so you get a ‘feathering’ of the change to blend the hockey blade onto the shaft…
http://chiefio.wordpress.com/2010/08/02/agw-jumping-sharks-since-1986/
John Finn says:
There is much criticism of the use of weather stations at airports. Is there actually any evidence that the temperature trends at airport stations are signiicantly greeater than the trend at nearby rural stations. I’m sure there must be some that are, but equally I’ve noticed some that quite definitley aren’t.
The effect varies a bit with wind speed, but one of the problems is that large airports are near large urban centers, so you often can simply have the city UHI blowing over the airport, so it’s both wind speed AND direction (and what’s nearby…) that matters most. But overall, yes, they are hot places. You can do A/B/ compares via Wunderground and see it for yourself some times.
from comments here:
http://chiefio.wordpress.com/2009/08/26/agw-gistemp-measure-jet-age-airport-growth/
“An interesting paper (from 1989) discussing heat island issues at airports: (see pg 18 of 45, re the Akron-Canton airport)
https://kb.osu.edu/dspace/bitstream/1811/23307/1/V089N2_001.pdf
It also uses the phrase “airport heat island.”
and
Schrodinger’s Cat was the initial condition but apparently it was a female and pregnant when put in the experimental box . . . . now it is Schrodinger’s Cats in the experimental box.
John
oops, wrong thread for Schrodinger’s Cat post,
John
JamesS you might be interested in this series of posts by Dr. William M. Briggs on the whole Homogenization business.
http://wmbriggs.com/blog/?p=1459
There is links to the other 4 parts at the top of Part 1, but here is the kicker line for you:
@Kevin Kilty
I think it is necessary to watch that paint dry when so as to be able to put in context the absurdity of statements such as:
previous centuries! Give me a break. See Idiocrats at work.
I appreciate the comments and feedback. A few people have emailed with a request that I draw conclusions about how all the discontinuities affect the trends. However I am not sure that it is possible to do such a thing. The main task at this point is to give people a better understanding of where these famous graphs come from. The implicit standard I have in the back of my mind is that of building official national statistics. If a sampling frame changes dramatically, the statistical agency is supposed to terminate one series and start another, and advise users that they are not necessarily continuous.
I also hope to emphasize the difference between ex ante and ex post tests of data adjustments. Thus far people have focused on getting a clear list of the adjustments. But this just gives you, in effect, the list of ingredients on the bottle. You still need to test whether the medicine cures the disease. The Muir Russell team got the distinction correct (p. 155):
FWIW, I do use all the GHCN stations.
Also, per the folks saying that colder vs warmer stations don’t mater because it’s the trend that matters: That would be true for a pure “self to self” thermometer anomaly process (such as the ones I use) if you did not then have to splice the series of anomalies together.
IMHO, it’s “all about the splice” (once “adjustments” are disposed of…)
Now in many of the codes, like GIStemp, they have two issues. One is “the splice” of patching all these series together (and there are splice artifacts to be found). The other is that they do their “grid/box” anomalies with one box of thermometers in the baseline and a different set in the present (and an even different set out of the baseline in the past historical settings). So you not only get “the splice” but you get the anomaly calculated by comparing your Jaguar to an old Chevy and saying the car is hotter now than it was then.
And yes, if you all do your “reconstructions” using similar techniques, you all find the same errors.
JamesS says:
I’ve been in the software and database development field for 27 years, so I know a little bit about data and analyzing same. Perhaps the problem here is a more basic one than climate scientists will admit: there isn’t enough data to derive a global average temp.
It’s worse than that. Temperature is an intensive variable. Calculating a Global Average Temperature will always be a bogus number as you need other things (like, oh, specific heat, heat of fusion, all that water cycle stuff…) to have meaning. I have more on the math of it in this link, but unless you are a math geek, it will just cause you to glaze. (Even if I think it is foundationally important):
http://chiefio.wordpress.com/2010/07/17/derivative-of-integral-chaos-is-agw/
But yes, there is simply not enough data both spacially and temporally to do the deed, so they must make stuff up. And with predictable consequences.
Vorlath says:
I’m just saying it’s amazing how often the selection, monitoring and reporting of temperature (but NOT the temperature itself) seems to match the warming trend (temperature). After a while, one has to wonder if this is just coincidence or if there is actually correlation and causation.
Another interesting graph is this one:
http://chiefio.files.wordpress.com/2009/12/jetfuel_globaltemp.png
from this posting:
http://chiefio.wordpress.com/2009/12/15/of-jet-exhaust-and-airport-thermometers-feed-the-heat/
(with reference to original source in the article).
Direct correspondence of jet fuel with ‘warming’…
Oh, and on the issue of “by altitude” or “by latitude” station dropout not mattering, this ignores two very important points:
1) It means you take another SPLICE (see above).
2) It means you are taking another Apples vs Oranges anomaly.
and there is a more subtle point:
It means you have one class of station (colder and volatile) in the record in one period of time and replace it with a different class of station (warmer and less volatile) in another. This, btw, it the issue I’d hinted at before but not posted on. If you let me swap a volatile series for a non-volatile AND let me pick when, I can tune that to things like the PDO and give you any “shape” of data warming you want via judicious splicing. If the trend is going against you, move to non-volatile, if it’s in your favor, go volatile. This, btw, is a standard technique in stock trading. That the issue of site volatility with temperature regimes is ignored is a lethal failure of GHCN. (If it is shown to not be ignored, but to be a deliberate move to non-volatile stations near water and at low altitude, then it’s worse…)
So you put volatile stations in during 1950 to 1980 while the PDO is cold and “lock in” very cold excursions due to the higher volatility. Then, when things are in the warm phase of the PDO, move to low volatility stations. It’s now impossible for them to find as much cold as the old stations did during the next cold PDO. They lack the volatility range to get there.
So the key elements of the “magic” are:
1) The Splice.
2) The “Jag vs Chevy” comparison.
3) High volatility in the past cold PDO with low volatility substitution recently.
There is also a ‘process change’ that happens with the change of “Duplicate Number” that happens between 1986 and 1990 which clips cold going excursions rather dramatically (either that, or we’ve never had a cold winter since, and I think 2009 pretty much showed we have… see the “hair graph” in the “jump the shark” link); but I’ve not yet finished finding out exactly what that process change is.
But don’t worry, I’ll get to it. I never stop.
I’ve taken to calling this approach of a bunch of distributed biases “The Distributed Salami Technique”. Similar to the Salami Technique used in financial shenanigans, but with the method spread over many steps and many individually minor “issues”. Sort of a steganographic approach to fudge. Though the question still exists if this is deliberate or if Hansen is just a “Clever Hans”en… 😉
Climate science is a game where one side is funded with tens of billion Dollars, creating data, models and publications. The other side is unfunded and it’s their job to find out where the well-funded side has pulled which trick. And we know in advance that after one trick has been exposed they’ll use another one. Maybe they have entire departments with the sole purpose of brainstorming new techniques of deception. And they’ll never be held responsible.
In Ross’s paper he makes generous statements about and includes some of the contributions of well known temp mavens – what I termed Serious Amateurs with Strong Data Analysis Skills or more neutrally Independent Researchers. But he refers to them as bloggers. I feel “bloggers” diminishes the substance of their contributions and suggested that he come up with a less pejorative label. However, this is only my opinion and I would be interested in how others feel.
Ross McKitrick says:
August 3, 2010 at 12:24 pm
I appreciate the comments and feedback. A few people have emailed with a request that I draw conclusions about how all the discontinuities affect the trends. However I am not sure that it is possible to do such a thing. The main task at this point is to give people a better understanding of where these famous graphs come from. The implicit standard I have in the back of my mind is that of building official national statistics. If a sampling frame changes dramatically, the statistical agency is supposed to terminate one series and start another, and advise users that they are not necessarily continuous.
I also hope to emphasize the difference between ex ante and ex post tests of data adjustments.
_________________________________________________________-
A very simple explanation of what you mean by ex ante and ex post tests, would go a long way to making this report completely understandable to the lay people this report needs to reach. Perhaps including a glossary of terms would help.
Again thanks for a very good well written report.
First, I admit I haven’t read the paper yet.
One question however – when I’ve mentioned the changes in locations to a preponderance of lower latitude locations to AGW believers, one rebuttal comes up at times – that since temps are measured in terms of anomalies, and supposedly the higher latitudes are warming faster than the lower latitudes, removing higher latitude sites would actually result in lower anomalies rather than higher ones… comments please?
Mosher,
So you are saying that stations that have ONLY 15 years of full data during the 1961- 90 period are an adequate basis upon which to determine temperature trend? No matter what your, and Zeke’s and others, reconstructions purport to show surely you need more data than this to produce a sound result?
A question on Dr. McKitrick’s paper (I have not read all the comments so it may have been addressed). On page 4, at the bottom, he writes (quoting Folland and Parker 1995):
However on page 5, first paragraph, he writes:
I was curious as to when the change actually occurred. At the start or end of WWII?
Wow! I’ve spent 1/2 hour reading through this work. Amazing! I’m just surmizing that the data used for input to the processing code is readily available and clearly referenced in the work.
THE WAY REAL SCIENCE AN ANALYSIS SHOULD NOW BE DONE. No excuses. NO processed data HIDING behind computer codes NOT available…
No data sets “dissappearing”, “We’ll, we had it when we did the work. Whoops, lost it, so sorry…but you can trust our work, here’s the conclusion(s).” Dang, as the old saying goes, “Adults get PAID to do this?”
Ross McKitrick:
1. In the paper you wrote, “The so-called Optimal Interpolation (OI) method used by GISS employs satellite measures to interpolate SST data for complete global coverage.”
GISS uses the Reynolds OI.v2 SST data, but the dataset originates at NCDC. So you may want to change GISS to NCDC.
2. A note regarding the “1945 Discontinuity” discussed in Thompson et al: The discontinuity also appears in numerous other datasets. Refer to:
http://bobtisdale.blogspot.com/2009/03/large-1945-sst-discontinuity-also.html
And to:
http://bobtisdale.blogspot.com/2009/03/part-2-of-large-sst-discontinuity-also.html
3. Thanks for the reference to my post on page 28.
4. You wrote, “GISS uses another NOAA product, the Reynolds et al. (2008) Optimal Interpolation version 2 (OI.v2) data base. This is based on ICOADS up to 1998. Thereafter, like Hadley, they switch to a subset that are continuously updated. The updated subset is weighted towards buoy data since many shipping records are provided in hard copy. OI.v2 also uses AVHRR satellite retrievals to improve the interpolation for unsampled regions. Unlike the ERSST data set the satellite input is still used in OI.v2.”
Should “Reynolds et al (2008)” be Reynolds et al (2002)? Als s far as I know, the satellite data has always been a part of the NCDC Optimum Interpolation dataset and I believe it is the primary data.
The combined use of satellite and in situ SST data was first discussed in Reynolds (1988):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/reynolds_1988.pdf
Reynolds and Marsico (1993) mention Optimum Interpolation but don’t describe it:
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/blend_w_ice.pdf
Optimum Interpolation is discussed in detail in Reynolds and Smith (1994):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/oiv1.pdf
The OI.v2 version is introduced in Reynolds et al (2002):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/oiv2.pdf
5. During the discussion of ERSST.v3 & .v3b you wrote, “This edition was called ERSST v3. However they noted that it reduced the trend slightly and deemed this effect a cold bias, so the satellite data were removed for version v3b.”
The trend wasn’t the problem. The satellite data changed which year and month had the record high temperatures, which brings nitpicky to an extreme. Refer to their webpage here:
http://www.ncdc.noaa.gov/oa/climate/research/sst/ersst_version.php
There they write, “While this small difference did not strongly impact the long term trend, it was sufficient to change rankings of warmest month in the time series, etc. Therefore, the use of satellite SST data was discontinued.”
The changes in rankings were illustrated in Table 6 of Smith et al (2008):
http://www.ncdc.noaa.gov/oa/climate/research/sst/papers/SEA.temps08.pdf
The irony in that is, in Smith and Reynolds (2004), the note how they perceive the satellite-based data. They write, “Although the NOAA OI analysis contains some noise due to its use of different data types and bias corrections for satellite data, it is dominated by satellite data and gives a good estimate of the truth.”
So if the “truth” changes which year has the second or third highest SST anomaly, they delete the truth. Link to Smith and Reynolds (2004):
http://www.ncdc.noaa.gov/oa/climate/research/sst/papers/ersst-v2.pdf
Thanks for the preview of the paper, Ross, and again, thanks for referring to my work.
Regards
Phil – good catch; my typo. The bucket transition was assumed to occur abruptly at 1941 in the Folland-Parker adjustment analysis. The abrupt SST change in 1945 was a different issue discussed in the Thompson et al. Nature paper.
I agree about measurements needing best value plus uncertainty. I don’t know all the details in the weighting given to sparse stations. I can imagine in the worst case that sparse stations might have too much weight because they are allowed to represent too large an area. I might point out that a number of skeptics attempt to do many things with this data, including seeing if it amounts to a hill of beans, but are sometimes stymied.
The operative word here is “large”. I agree with you that the shift to lower latitudes ought to produce a more moderate signal, but this I say without having thought about every influence that may occur. There was a time I would have thought that homogenization would not have had a systematic effect too, but after reading how NCDC does homogenization I now have serious doubts.
Whoa there, friend. First, you are implying that changing station distribution is the only “fly in the ointment.” It is not. Second, how come the onus is on me as critic? Maybe I should have tried that line on my dissertation committee?
It’s not my data. I’m not paid to do this work. I’m not one trying to guide policy using it. And I am certainly not one claiming these temperature series represent some sort of gold standard. Using sparse data to extrapolate trends over large regions deserves comment.
This is an interesting comment from p36:
‘(IPPC AR4) Summary for Policymakers stated:
“Urban heat island effects are real but local, and have a negligible influence (less than 0.006°C per decade over land and zero over the oceans) on these values.”
The 0.006°C is referenced back to Brohan et al. (2006), where it is merely an assumption about the standard error, not the size of the trend bias itself. ‘
Warmists often quote the UHI effect as being negligible … theeir source is yet another IPCC error !