This new paper by Dr. Ross McKitrick of the University of Guelph is a comprehensive review of the GHCN surface and sea temperature data set. Unlike many papers (such as the phytoplankton paper in Nature, complete code is made available right from the start, and the data is freely available.
There is a lot here that goes hand in hand with what we have been saying on WUWT and other climate science blogs for months, and this is just a preview of the entire paper.This graph below caught my eye, because it tells one part of the GHCN the story well.
1.2.3. Growing bias toward lower latitudes
The decline in sample has not been spatially uniform. GHCN has progressively lost more and more high latitude sites (e.g. towards the poles) in favour of lower-latitude sites. Other things being equal, this implies less and less data are drawn from remote, cold regions and more from inhabited, warmer regions. As shown in Figure 1-7, mean laititude declined as more stations were added during the 20th century.
Here’s another interesting paragraph:
2.4. Conclusion re. dependence on GHCN
All three major gridded global temperature anomaly products rely exclusively or nearly exclusively on the GHCN archive. Several conclusions follow.
- They are not independent as regards their input data.
- Only if their data processing methods are fundamentally independent can the three series be considered to have any independence at all. Section 4 will show that the data processing methods do not appear to change the end results by much, given the input data.
- Problems with GHCN, such as sampling discontinuities and contamination from urbanization and other forms of land use change, will therefore affect CRU, GISS, and NOAA. Decreasing quality of GHCN data over time implies decreasing quality of CRU, GISS and NOAA data products, and increased reliance on estimated adjustments to rectify climate observations.
From the summary: The quality of data over land, namely the raw temperature data in GHCN, depends on the validity of adjustments for known problems due to urbanization and land-use change. The adequacy of these adjustments has been tested in three different ways, with two of the three finding evidence that they do not suffice to remove warming biases.
The overall conclusion of this report is that there are serious quality problems in the surface temperature data sets that call into question whether the global temperature history, especially over land, can be considered both continuous and precise. Users should be aware of these limitations, especially in policy sensitive applications.
Read the entire preview paper here (PDF), it is well worth your time.
h/t to E.M. Smith
—
It should be noted that “GHCN” stands for “Global Historical Climatology Network,” which is managed by the National Climatic Data Center, Arizona State University and the Carbon Dioxide Information Analysis Center of the U.S. federal Department of Energy.
I confess to having flashed on “growth hormone” and “cyanide,” but that shows you how limited is the perspective of somebody skilled in sigmoidoscopy.
—
You’re right, very interesting:
“Section 4 will show that the data processing methods do not appear to change the end results by much, given the input data. ”
That’s the funniest thing I’ve read for a while. Thank god we have professors of economics to explain science to us.
I look forward to the blanket coverage in tomorrow’s media.
Even more persuasive is Fig 1-10 showing “Changes (“delta”) in the global average temperature resulting from GHCN adjustments”.
Quoting from the report:
“There are two notable features of the graph. The first is that the adjustments are mainly negative prior to 1940 and positive up to about 1990, effectively “cooling” the early part of the record and “warming” the later record. In other words, a portion of the warming trend shown in global records derived from the GHCN-adj archive results from the adjustments, not from the underlying data. Our calculations show that this adds about 0.12 degrees to the 20th century average over land.
“The second, and more obvious feature is the chimney-brush appearance of the graph, indicating a massive increase in the volatility of the adjustments after 1990. The instability in the record dwarfs the size of the century-scale global warming signal, with fluctuations routinely going to ±0.5 degrees C from one year to the next. The southern hemisphere (bottom panel) is particularly noisy.
“On substantive grounds I therefore conclude that after 1990 the GHCN archive became very problematic as a basis for computing precise global average temperatures over land and comparing them to earlier decades.”
… and I thought I knew something about the subject …
Clear, detailed, understandable, authoritative, extensive references, code and data provided, very readable, my congratulations to Ross. This one will be a classic.
w.
Let me know when you want to reactivate the GHCN gallery. I’ve got a few stations saved up for you….
jws
This post returns to the theme of the questionable value of areally averaged global temperature in understanding climatic processes. It must always hugely bias the input of temperature oscillation in favour of the tropics at the expense of the high latitudes, since the global area reduces exponentially toward the poles.
The final figure in the following commentary referenced below, credited to George Kukla, demonstrates the bias
http://calderup.wordpress.com/2010/05/14/next-ice-age/
Willis Eschenbach says:
August 3, 2010 at 1:32 am
I think that this is well worth repeating, again and again.
This is indeed, praise of the highest quality.
“On substantive grounds I therefore conclude that after 1990 the GHCN archive became very problematic as a basis for computing precise global average temperatures over land and comparing them to earlier decades.”
This is very much in tune with what E.M. Smith has been saying for a while, most recently yesterday: http://chiefio.wordpress.com/2010/08/02/agw-jumping-sharks-since-1986/
Is it just me, or do the two sudden falls in average latitude c. 1985 and c. 2000 on figure 1.7 correspond rather nicely with the instrumental changes that Chiefio has been blaming for the changing temprature measurements? It looks very like to me that there has been two major relocations/rebalancings of the entire GHCN within the last 30 years, at the same time as instrumental and methodological changes were introduced
Vince Whirlwind, you obviously didn’t download and read the paper which states that the problem is not so much with the variations in post-processing by GISS, HadCRU and NOAA (though it is somewhat) but in the input data itself from GHCN. Foot, mouth, insert.
A great read although I was relieved to find that half the pages were references.
I found the references to the CRU emails were very effective since they showed how corrosive they were in context. The UHI chapter was particularly shocking. How can research without any supporting data be accepted as true whilst research backed by data be dismissed, simply because someone say it is implausible?
So almost all the data from the various climate groups are based on one source. This source has significant biases and uncertainties, similar in magnitude to the purported underlying trend. The science is very doubtful but they get away with it because the police are in the pocket of the criminals – an arrangement that they call peer review!
thank you, Rich Matarese, I was just going to ask. I don’t understand all these abbreviated initial wotsits. I thought it was usual good practice to spell it out first time with the initials in brackets to be used thereafter. Picky, perhaps, but it helps people who are new to this stuff, like me, and it is the ‘proper’ thing to do.
And those deliberate changes culminating in a warming bias just happen to be chance? In politics and in turn, policy, nothing happens by chance; everything is planned and executed to the best of their abilities. The only hiccup is when meddlesome individuals interfere, requiring plans B) C) and so on to be adapted and implemented. This observation will only add to the “to do” list in order for the planned agenda to be realized.
A detailed overview like this is just what was needed. I’d had a go at looking at the adjustments in NOAA/GHCN data back in January*, but had looked at the effect on overall trend. What Ross McKittrick is showing – that the adjustments tend to cool the older records and warm the more recent analysis is something I have seen too but wondered how widespread it was. Now we know.
*Links that look at overall trends:
http://diggingintheclay.wordpress.com/2010/01/21/the-station-drop-out-problem/
http://diggingintheclay.wordpress.com/2010/01/28/adjustment-effects-on-temperature-trends-part-2-magnitudes-of-adjustment/
http://diggingintheclay.wordpress.com/2010/01/29/adjustment-effects-on-temperature-trends-part-3-effects-by-latitude/
It would be interesting to repeat these but segmenting by the periods suggested in the paper.
Paul Clark says:
August 3, 2010 at 3:12 am
“Vince Whirlwind, you obviously didn’t download and read the paper which states that the problem is not so much with the variations in post-processing by GISS, HadCRU and NOAA (though it is somewhat) but in the input data itself from GHCN. Foot, mouth, insert.”
He doesn’t have to read it to make his appeal to authority. He thinks there is some magic trick that only scientists can do when one adds two numbers together and divide by two. Obviously, this paper isn’t the only reading material he’s skipped out of reading.
An excellent paper that should be read by all IPCC sponsored scientists. One thing puzzles me though. On page 11 we see the percentage GHCN stations located at airports from 1890 – 2009. But according to WikiAnswers the first airport was built in 1909, College Park Maryland.
I’ve made a set of KMZ files which you can feed into Google Earth to visualize just what GHCN station changes were made in the 90’s and in 2005.. You can download them as a zipfile; just click on a filename in a file browser or use Ctrl-O in GE, and you will see in GE a set of markers indicating the stations removed, labelled and colored by whether they are urban/rural, sized by years of record, etc. There are other posts with files to show the state of GHCN in each “decade year”.
The post with more description and a download pointer, is here.
Climate science are regional events!
Far too much science is missing to generate an overall Global model.
Are similar bias present between high and low altitude? Rural v city?
Shameless self-promotion: Animation of locations with data Dude, where is my thermometer which visualizes the spatial distribution of locations in the GHCN by year and graphs of station counts by country by year.
So now we see the new iconic image of climate science is not the Hockey Stick – but the Chimney Brush. I look forward to seeing this image on the front page of the next IPCC report.
Thanks to Ross.
Vince Whirlwind at 1:08 AM and Paul Clark at 3:12 AM.
Thank you two for this critical point. This tends to absolve the post processors of chicanery with the data, but how was the change in stations used by GHCN made? And, dare I ask, why?
========================
Conclusion to be drawn:
There is no systematic requirement for a uniformity of data collection sites in terms of longitude and latitude, land against ocean, low altitude against high altitude.
Ergo: no reliable conclusions about global climate can be drawn.
Proposal: start again with a global consortium, operating openly, scientifically, robustly, skeptically, rigorously, with appropriate redundancy and funded by a partnership of stakeholders to include Administrations, corporations, farmers groups, tourism groups, shipping groups, fishing interests and interested members of the public.
And as the usefulness of this would only be anything other than zero if funding were continuous for at least a century and probably 500 years, it is not something to be undertaken lightly or wastefully………….
This is regarding an area representing 30% of the globe and these records have been known to be corrupt for some time.
The sea surface temperatures are in worse shape and that has also been known for some time but people continue to discuss “Climate” based on known corrupted data.
If someone claims the globe has warmed the response would be: There is no reliable evidence to make that claim!
While this is a good and extensive look at the surface temperature records it should have not been needed. It will be either ignored when possible or trashed by the faithful as not being from an insider. Even though NASA GISS admits that Surface Air Temperature is guess work at best.
Quote:
Q. If SATs cannot be measured, how are SAT maps created ?
A. This can only be done with the help of computer models, the same models that are used to create the daily weather forecasts. We may start out the model with the few observed data that are available and fill in the rest with guesses (also called extrapolations) and then let the model run long enough so that the initial guesses no longer matter, but not too long in order to avoid that the inaccuracies of the model become relevant. This may be done starting from conditions from many years, so that the average (called a ‘climatology’) hopefully represents a typical map for the particular month or day of the year.
From here:
http://data.giss.nasa.gov/gistemp/abs_temp.html
Put the high one in
And you take the low one out.
What’s it all about?
===============
David, UK (August 3, 2010 at 4:56 am) “the new iconic image of climate science is not the Hockey Stick – but the Chimney Brush.”
LOL – I love it. That has really made my day!
Sounds like this paper should be sent to every member of Congress, with a letter explaining its implications: forget about AGW—there isn’t any.
No point in sending to the White House.
/Mr Lynn
OT, Solar Magnetic Burst
I wonder if these may be normal part of longer term solar cycles which are related to volcanic activity.
How about an animation that shows station location changes? And one that shows regional weighting changes?
Thank you Ross and Anthony.
This is why they don’t freely allow people to look at the code and data. I’ve been saying for ages that this is the real man-made global warming. :o)
I wonder if their instruments are calibrated, and if they are what is the traceability of their calibrating standards and procedures. And how are their instruments maintained. As Jagger says there is no systematic requirement for uniformity In data collection in terms of locations and conditions.
I wish someone would explain why “adjustments for known problems due to urbanization and land-use change” are made to the raw data from a station. If the point of the exercise is to measure the temperature at a given spot, then why would one change the values based on the above parameters? The temperature there is the temperature there — period. If it gets warmer because of increased urbanization, well then, it gets warmer. Are they trying to make the numbers look like what they might be if nothing had changed?
I can see making adjustments if the instruments are changed and the new one reads half a degree higher than the old one. I can see moving the site if a new asphalt interstate is built 100 feet away. But if a site gets moved, then that data point should be terminated and a new one instituted. What they’re trying to do is to make the data appear as though there is an uninterrupted series of measurements at each location, and that means changing the data with a SWAG fudge factor.
I wonder what these graphs would look like if only the raw data were used? When I look at the raw data for site in out-of-the-way places in West Virginia, or other places that haven’t seen any major development over the years, there is no AGW signature visible in those records. What’s being done to the raw data for the sake of “accuracy” is a crime against data.
Please help me!!
Didn’t the World spent billions of dollars on new weather satellites in the last decade.
1) Wasn’t one of the purposes of the satellites to measure land and sea temperatures????
2) If those temperature are accurate, why do we need ground (water) based stations???
3) Where are the satellite temperature records?? Are there any??? Who keeps them??
4) If we are so advanced in measurement technology, why don’t we have “complete” global temperature records via space???
WUWT????
Re: airport v rural stations
There is much criticism of the use of weather stations at airports. Is there actually any evidence that the temperature trends at airport stations are signiicantly greeater than the trend at nearby rural stations. I’m sure there must be some that are, but equally I’ve noticed some that quite definitley aren’t.
Excellent summary. Now it is time that someone will try to fix the mess and recalculate everything all over.
RE:
John Finn says:
August 3, 2010 at 6:36 am
“greeater” should be “greater” and “definitley” should be “definitely”
Verity Jones says: August 3, 2010 at 3:41 am
“What Ross McKittrick is showing – that the adjustments tend to cool the older records and warm the more recent analysis is something I have seen too but wondered how widespread it was. Now we know.”
But what isn’t said is who uses the adjusted GHCN data. Not GISS, not CRU. I believe NCDC does. But Zeke showed that a reconstruction using the adjusted file gave very similar results to the GHCN unadjusted.
kim says:
August 3, 2010 at 5:11 am
“Put the high one in
And you take the low one out.
What’s it all about?”
===============
That’s called jiggery-pokey not the hokey-pokey. That’s what it’s all about.
What is of interest here is temperature trend, not absolute temperature. If you look at latitudinal effect on temperature, you see this:
Temperature trend land & ocean.
This figure shows the well-known increase in temperature trend (land only) with increasing latitude. Decreasing the mean latitude of the stations over time has the paradoxical effect of decreasing the net global warming over that period.
The magnitude of this latitudinal effect is significantly reduced or even removed if one does a more careful “area weighted” average. In that case, you have to look at how the particular analysis performs its area weighting. The only one I’ve looked at in any detail is CRUTEMP, for which there is still a residual bias in latitude even after gridding:
CRUTEMPmean latitude
Again this has the effect of understating early 20th century warming compared to late 20th century warming. However even the IPCC admits that prior to circa-1980, anthropogenic activity played little role in global warming.
So again, somewhat surprisingly, this latitudinal bias has the effect of deemphasizing the period of natural warming in comparison to the one for which it is thought (in the mainstream climate community at least) that anthropogenic activity played a large role.
In the past accurate measurement of important variables like time, length, mass and temperature occupied the thoughts of the greatest minds of the age. They were attracted by the knowledge that accurate measurement was fundamental to science and engineering and their whole future way of life. Accuracy still is still fundamental to science but it is not fundamental to business and politics. In fact it tends to get in the way of policy. Scientists are now paid to get results and the results are defined by companies and politicians. Truth and accuracy are no longer where the money is.
For example the quality of automotive science thoughout the world is astounding as are the scientists involved but would you expect a scientist from GM to say that Honda made a better car even if he knew that to be true. He is effectively paid to lie. I would even guess that most people would find that reasonable given that his continued employment would be at risk if he told the truth. And yet the general public find it hard to believe that climate scientists would lie about a political “vehicle” which they are being paid to engineer.
I think the reality is that 30 to 40 years ago when the current cabal of IPCC climate scientists started their careers it was a cinderella science which attracted mainly third rate minds (relative to the big money fields like finance and biosciences and the big prestiege fields such as high energy physics). They have often done third rate jobs and are trying to hide their mistakes with second rate subtifuge and first class marketing. It is a sad time for science.
There is good climate research going on now and I have read some really well written papers. However this research tends to be very specific and not to extrapolate from random noise to future catastrophy so never makes it into the news. Doing good research is not sexy so funding dries up. And then a well funded project comes along, no questions asked…. a man’s got to live!
Very useful video, except the first hundred years or so is like watching paint dry. However the video from 1985 on shows much more clearly for the mathematically innumerate the point of McKitrick’s graph.
aaron says: August 3, 2010 at 5:48 am
“How about an animation that shows station location changes?”
If you mean station distribution, Ron Broberg has it.
John Finn says: August 3, 2010 at 6:36 am
Re: airport v rural stations
There is much criticism of the use of weather stations at airports. Is there actually any evidence that the temperature trends at airport stations are signiicantly greeater than the trend at nearby rural stations.
Zeke looked at this in detail, and found that a reconstruction using airports vs non-airports gave very similar results. I looked less thoroughly, but found the same.
Peter Shroud says at 4:09 “On page 11 we see the percentage GHCN stations located at airports from 1890 – 2009. But according to WikiAnswers the first airport was built in 1909, College Park Maryland.”
Those figures are taken from an analysis at http://chiefio.wordpress.com/2009/12/08/ncdc-ghcn-airports-by-year-by-latitude/ The author, E. M. Smith, says there:
“This is a bit hobbled by the primitive data structure of the “station inventory” file. It only stores an “Airstation” flag for the current state. Because of this, any given location that was an open field in 1890 but became an airport in 1970 will show up as an airport in 1890. Basically, any trend to “more airports” is understated. Many of the early “airports” are likely old military army fields that eventually got an airport added in later years.
With that caveat, the charts are rather interesting. ”
The important point is not the absolute percentage of airports, but the warming imputed because the percentage has changed.
JamesS says: August 3, 2010 at 6:27 am
“I wish someone would explain why “adjustments for known problems due to urbanization and land-use change” are made to the raw data from a station. If the point of the exercise is to measure the temperature at a given spot, then why would one change the values based on the above parameters?”
GHCN publishes unadjusted figures, and that’s what the main indices use (though they may themselves adjust). But you’re right – adjusted figures do not give a better result for a particular location. The thing is, when you use temperatures to compile a global estimate, then the station temp is taken to be representative of an area (or more correctly, its anomaly is representative). So the corrections are to respond to known issues which would make it less representative.
Page 13 of the McKitrick report.
I believe most people here understand what you say, but the point is also that larger portions of the Earth, especially those considered to be most important to the detection of warming, are doing with fewer actual measurements. This does make the results more dependent on errors in measurement and adjustment, wouldn’t you say?
I suspect that all scientists, good and bad, consider themselves to be experts at using computers to analyse data. I also suspect that this is seldom the case. Thus, all scientific papers that rely extensively on computers to manipulate and interpret data should have the analysis reviewed by professional computer experts, such as Professor Ross McKitrick, not by other scientists.
Readers of WUWT have been familiar with a lot of this information for quite a while now. Thanks to Ross McKitrick for a clear concise academic paper on the subject. One should always examine the data source for problems before jumping to erroneous conclusion that the planet is catastrophically warming, unless it is catastrophic warming that you want to show. Anthony, your instincts on this have been correct. E.M. Smith as well. Now on to the public if we can be heard above the alarmists shouting and an agenda driven media. A herculean task.
I haven’t read the preview paper (will later), but the point about lack of continuity has always bothered me about surface temperature data. How does one realistically compare temperature readings in London in 1890 vis a vis 2010? There must be a huge number of adjustments that would have to be made in both up and down directions and how does one ever know if you’ve made all the adjustments required? Human changes to the environment are dramatic in a hundred plus year span, and that includes the outlying areas. I can see no way to properly compute and/or account accurately for these changes.
I suppose I’m stated the obvious, but every time I see the inevitable headlines recounting how the present decade is the warmest ever, I laugh at the certitude and the arrogance of so-called “climate science.”
John Finn says:
August 3, 2010 at 6:36 am
You haven’t been paying attention in class, have you.
For those doing climate studies and environmental assessments, McKitrick is sayng: “Stop! Look! Listen! Never Assume Anything!” And for everyone else, he is saying the same thing. For some strange reason mankind has been assuming more and more, indeed, so much more than ever before. Not only is Our Giant Civilization of Cards growing by leaps and bounds, but the cards are getting thinner and thinner; some of them we can even see through.
Page 16, Figure 1-10: Wow….
Great paper and resource. At one point, Dr. McKitrick is referencing the anticipated changes in HadSST2 and ocean SSTs to take into account the 1946 blip.
There is a draft paper in preparation which is already being cited – [Reassessing Biases and Other Uncertainties in Sea Surface Temperature Observations since 1850, 2009, Kennedy, J.J. et al – in preparation]. I’m not sure of the status of this paper but it seems that Climategate and Tom Wigley’s email has delayed the effort.
The draft HadSST3 changes are outlined/referenced in the following paper which is itself not peer-reviewed I believe (but it already has 8 citations) and is produced by the Met Office Hadley Centre, the WMO and a long list of other ocean SST experts. Two versions available.
https://abstracts.congrex.com/scripts/jmevent/abstracts/FCXNL-09A02a-1662927-1-Rayneretal_OceanObs09_draft4.pdf
http://rainbow.ldeo.columbia.edu/~alexeyk/Papers/Rayner_etal2009ip.pdf
The new HadSST3 would reduce the pre-1941 data by -0.1C, increase the post-1946 data by +0.2C and increase the post-2001 data by about +0.1C (the post-2001 increase is hard to explain unless …). This is shown in the following chart – top panel – HadSST2 is red – proposed HadSST3 is green.
There are two versions produced in the two different versions of the paper but I think the second one below just artificially reduced the base period so that the changes are more/less? visible. [The base period is supposed to be 1961 to 1990 so the second chart below is too low].
http://img16.imageshack.us/img16/1205/hadsst3.png
http://a.imageshack.us/img832/3174/newesthadsst3.png
As Dr. McKitrick expected, the new line would be mostly flat from 1940 to 1990.
Having pointed out the problems in an authoritative and comprehensive way it remains to see that something is done about it. There in lay the rub. I predict far more effort will be put into defense then corrective action.
I’m disappointed it’s being published by the GWPF. Will lessen credibility and impact I fear.
Ditto on Carrick comments.
Ross’s comments on the changes in sampling are but a first step.
Nobody who uses GHNC data uses all 7280 stations. So for starters you can’t look at the entire sample of GHCN and draw any substantive conclusion. For example, Zeke and I take in all 7280 stations and then we do a preliminary Screen. The first screen is to reduce the total number of stations to those stations that have at least 15 years of full data within the 1961-1990 period. That screen drops a couple of thousand stations, so of the original 7280 stations you end up using about 4900 of them. So you really have to study THAT distribution and how it changes over time.
( thats pretty easy I can probably whip something up or Ron B can.)
WRT altitude. Thats a non issue. When you transform by anomaly you are correcting for this effect. Its NOT the temperature at a station, its the CHANGE WRT its mean.
WRT latitude. to test for the sensitivity to station loss I’ve done re sampling tests.
Basically each station gets assigned to a grid ( 3 degrees by 3 degrees) When you do that with 4900 stations you get some grids with one station and other grids with as many as 36. For example, you might get 36 stations in the grid 120-123, 60-57. (lon/lat) What I did was randomly pick only one station per grid. did that repeatedly. The result? no difference. Basically the same answer comes up if you use fewer stations. Again what you have to look at is the distribution of grids covered, not stations. And to further Carricks point, losing coverage at high latitude (per grid) DECREASES the trend. High latitude is COLDER in temp, but the CHANGE in anomaly is higher. So, for example,if the equator changes at zero degrees per century, the poles will be increasing. Reducing your sample at high latitudes depresses the trend. To see this you merely have to look at the long record trends by latitude. The highest trending grids are poleward.
WRT GHCN adjustments. i dont use the GHCN adjustments. The file in question has a few minor issues that Zeke has noted, and some other issues that I’ve identified but havent made public.
Also, the metadata in the ghcn inventory is stale. Ron’s got some updates for that.
Anyways, maybe if I get some time I’ll update in a post. new code drop as well
Dr Lurtz
I think the short answer is that satellites don’t measure temperatures at ground level. Indeed, exactly where they are measuring temperature is something of a variable, as it depends on wavelengths, cloud cover, stratification and so on. The upshot is that a properly sited thermometer on the ground is still the standard reference, or would be if climatologists didn’t keep trying to compensate for the ones that ended up in car parks, aircraft runways or next to air-con outlets…
Ditto on Nick Stokes.
GHCN Adjusted is NOT used by CRU or GISS. Given the little quirks I’ve found in the file I wouldnt use it ( records failing to match, duplicate records, no clear provenance for the adjustments).
Kevin:
I agree with you, and actually this is my biggest gripe with the temperature reconstructions: Very few of them make any attempt at all to generate realistic error bars. This is experimental data, and the mean (and metrics derived from that) is meaningless without a statement of the uncertainty in the measurement. Of course the opposite is also true: Skeptics are wont to point out enumerable warts in the surface data sets, without ever sitting down and demonstrating whether they would amount to a hill of beans, numerically.
Few “actual measurements” may increase the error bars somewhat (not so much, as you note I have included error bars in my figure), but it wouldn’t explain a large systematic effect with latitude. If you think it does, the onus is on you as the critic to demonstrate how a sparsity of stations could explain this temperature trend.
In any case, you need to consider that a large land-surface latitudinal effect is expected from any sort of warming (at least one component of this is glacial retreat, which amplifies the warming through the reduced high-latitude albedo). So we have in this case, agreement of data with basic physics.
If you wanted to argue that the data don’t support an increase trend with latitude, you have a pretty steep stairs to climb. Simply suggesting that “it might matter” doesn’t hold any weight. I’m out of pocket the rest of day, so please don’t get offended if I can’t respond to any comments.
Possible typo in the paper:
Page 36 mentions an error of 0.006 degrees per decade, page 37 mentions 0.006 degrees per century. Two different references, so they may very well both be accurate reporting, but this is the sort of thing that might easily be a typo.
I’ve been in the software and database development field for 27 years, so I know a little bit about data and analyzing same. Perhaps the problem here is a more basic one than climate scientists will admit: there isn’t enough data to derive a global average temp.
Where’s the shame in admitting as much? Well, we’d like to come up with an average global temperature, but we just don’t have enough stations in enough places to really do that, so until we do, we’ll just publish data for the regions we do know about. Even a 250-km smoothing is too much when one considers that temperatures can vary from, say, Frederick, Maryland, USA to Annapolis, Maryland, USA, by three or four degrees C on any given day.
Climate science has fallen in love with crunching numbers and has lost sight of what those numbers are supposed to represent: reality. If you’ve only got five or six stations apiece in Siberia/Africa/South America/wherever, there is no way you know — or can derive — a decent average temp. You’d just be making up the numbers.
Roddy Campbell says:
August 3, 2010 at 8:19 am
I’m disappointed it’s being published by the GWPF. Will lessen credibility and impact I fear.
I will say again, where the f**k else are they allowed to publish if they do not tow the party line?!?! The skeptics have been handcuffed, so acquire a series of linked vertebrae and promote the paper as though it had been published in a “respectable” (using the term very loosely) journal. C’mon people!!
I’m going through the paper and it is extremely well written.
The airport percentage graph is very striking. It’s amazing how much the trend matches the warming trend. Not saying this is where the warming comes from or anything. I’m just saying it’s amazing how often the selection, monitoring and reporting of temperature (but NOT the temperature itself) seems to match the warming trend (temperature). After a while, one has to wonder if this is just coincidence or if there is actually correlation and causation.
I’m confused by Figure 1-6: Percent GHCN stations located at airports, 1890-2009, page 12. It’s showing that in 1890, 25% of temperature data came from airports. I’m just wondering… why there were airports in 1890, 13 years before airplanes were invented?
For those that say the altitude doesn’t matter the anomaly takes care of it are dead wrong, they miss the point entirely. What is that point? Simple: According to the models the mountains (higher altitude) is suppose to warm at a faster rate then low lying areas (lower altitude) [Snyder 2002].
If the Mountains do not warm at a rate higher then the valley’s then the models are wrong. In repeated studies it has been found that the Mountains do not warm at a rate faster then the Valleys, matter of fact the Valleys warm faster then the Mountains. The most famous (IMO) paper on this is the Christy et al 2006 paper published in the Journal of Climate (http://www.openmarket.org/wp-content/uploads/2009/08/2006_christynrg_ca.pdf).
Interestingly enough this paper came out in 2006 and the 4 stations in the GHCN dataset that are also in the Christy paper all of sudden have no data after two months into 2006, but at the same time still have data in the USHCN dataset up into 2009. What’s the excuse this time, that NCDC couldn’t give the report to themselves?
So yes Altitude does matter and the anomaly doesn’t save you because the trends they show are wrong according to the models.
Here is the list of papers that Dr. Christy references in a presentation on this in 2010
(4:17 mark http://www.youtube.com/watch?v=UcGgLoPpbBw )
Christy 2002; Christy et al 2006, 2007, 2009; Pielke Sr. et al 2008 and Walters 2007
Matter of fact these studies show that Tmean is a poor representation of how much “warming” may or may not be due to CO2:
From his 2009 EPA submission http://icecap.us/images/uploads/EPA_ChristyJR_Response_2.pdf
So what do you do when data goes against your narrative?
Well as shown by Climatologists for Paleoclimate studies we chop it off so it doesn’t cause Policymakers to ask akward questions. In the Surface data sets we stop using the high altitude ones.
Dr. Lurtz says:
August 3, 2010 at 6:27 am
Please help me!!
Didn’t the World spent billions of dollars on new weather satellites in the last decade.
1) Wasn’t one of the purposes of the satellites to measure land and sea temperatures????
2) If those temperature are accurate, why do we need ground (water) based stations???
3) Where are the satellite temperature records?? Are there any??? Who keeps them??
4) If we are so advanced in measurement technology, why don’t we have “complete” global temperature records via space???
WUWT????
I’m not an expert in the Sat. temp readings, but I’ll try to help.
Yes, we’ve spent billions on sat. temp measurements. They appear to be accurate, but I’ve questions about them. The satellites don’t measure the ground temperatures, the post from James P says: August 3, 2010 at 8:26 am is about of good explanation as I could give. Currently, RSS-MSU and UAH are the two most prominent sat temp collecting groups that I know of. Their data is updated on a regular basis. The way I get to the raw data for the various climate places, is first I go to the interactive graphs section at the woodfortrees web site. For example, here, http://www.woodfortrees.org/plot/rss . At the bottom of the page, there is a link to the raw data. You’ll find a text file with web addresses at the top of the page, go to them for more detailed analysis of the raw data.
The reasons we don’t simply use sat data is because the data is different than thermometer reading on the ground and sea. It would be an apples to oranges comparison. Even if we did use them, we have relatively no historical data from sat temps. We can’t compare a mercury reading from 1900 to at sat reading in 2010. I believe 1979 was the first year we start collecting sat. temp readings, but even then we’d have continuity issues on the way the satellites were calibrated.
At any rate, there has been a few posts relating to a couple of your questions. You should search the past postings here. For additional information, Dr.s Spencer and Christy maintain their own websites, http://www.drroyspencer.com/ and http://www.atmos.uah.edu/atmos/christy.html respectively. Both seem very open to communication.
Hope that helps.
a dood says:
August 3, 2010 at 9:03 am
“I’m confused by Figure 1-6: Percent GHCN stations located at airports, 1890-2009, page 12. It’s showing that in 1890, 25% of temperature data came from airports. I’m just wondering… why there were airports in 1890, 13 years before airplanes were invented?”
Or, maybe they built airports where they were measuring the temps. I’m not sure, hopefully Ross will answer. But it seems entirely plausible that this would be the explanation. With out airplanes and concrete ect. those would be ideal places to measure temps.
Vince Whirlwind says:
August 3, 2010 at 1:08 am
You’re right, very interesting:
“Section 4 will show that the data processing methods do not appear to change the end results by much, given the input data. ”
That’s the funniest thing I’ve read for a while. Thank god we have professors of economics to explain science to us.
====================
I was laughing too ( reading your post). It is going to be even funnier when instead of professor of economics, the plumbers (professionals) will start explaining the science to you in a very plain language
Heh, love my instincts! For those wondering about airports in 1890, click on the source link. http://chiefio.wordpress.com/2009/12/08/ncdc-ghcn-airports-by-year-by-latitude/ There, you’ll find this explanation:
“..This is a bit hobbled by the primitive data structure of the “station inventory” file. It only stores an “Airstation” flag for the current state. Because of this, any given location that was an open field in 1890 but became an airport in 1970 will show up as an airport in 1890. Basically, any trend to “more airports” is understated. Many of the early “airports” are likely old military army fields that eventually got an airport added in later years.”
Ross has left an email address if you have found typos, etc. He might find it more helpful than going through all the above comments – as good as they are. 😉
Huth says:
August 3, 2010 at 3:35 am
thank you, Rich Matarese, I was just going to ask. I don’t understand all these abbreviated initial wotsits….
_________________________________________________________
Dr. McKitrick may want to add a glossary. Otherwise use the WUWT glossary (on to tool bar) here:
http://wattsupwiththat.com/glossary/
Otherwise I found the paper very readable except perhaps the last few pages that was getting into more technical detail. Non scientists using the glossary should not have too much trouble understanding this paper.
Congratulations Dr. McKitrick on a job well done.
Bob(Sceptical Redcoat) says:
August 3, 2010 at 7:39 am
I suspect that all scientists, good and bad, consider themselves to be experts at using computers to analyse data. I also suspect that this is seldom the case. Thus, all scientific papers that rely extensively on computers to manipulate and interpret data should have the analysis reviewed by professional computer experts, such as Professor Ross McKitrick, not by other scientists.
________________________________________________________________
AND a statistician.
From Dr. Ross McKitrick’s paper section “1.3 Increasing magnitude of adjustments employed to try and fix the problems of sampling discontinuities”:
——————-
Dr. McKitrick,
I think you have made blog history with the term “the chimney-brush” in describing the ‘GHCN Delta = Adjusted GHCN – Unadjusted GHCN’ graph (your Figure 1-10).
I had not heard the term before. I like it.
John
The loss of temperature stations to the global temperature record is well known, even though the stations continue to exist and collect the data. The loss of higher elevation/cooler area stations noted here is even more alarming. Here in Canada we have 35 stations going into the record (if I got the number right) at present, though thousands are being recorded. Has Jones/Hansen/Schmidt, NOAA/HadCrut/GISS put out reasons for this data loss?
David Ball says:
August 3, 2010 at 7:48 am
You haven’t been paying attention in class, have you.
Well – yes I have actually – for several years as it happens and I can’t recall anyone showing that the trend at airport stations is significantly different to the trend at rural stations. Nick Stokes, in this post, suggests that airport and rural trends were similar
Nick Stokes says:
August 3, 2010 at 7:25 am
Zeke looked at this in detail, and found that a reconstruction using airports vs non-airports gave very similar results. I looked less thoroughly, but found the same.
My own observations are almost certainly less thorough than Zeke’s and Nick’s, but one in particular may be of interest. The Armagh Observatory in Northern Ireland is cited as an example of the perfect site for temperature measurments. David Archibald refers to it in his own research. The observatory is in a rural setting which is relatively unchanged in the past 2 centuries. Aldergove airport in Belfast is just a few miles away. Temperatures have been measured at Aldergrove since 1880.
Between 1975 and 2004 the Armagh (rural) trend is actually greater than the Aldergrove (airport) trend. The difference is even more pronounced if you go back to 1880. The Armagh trend is ~1.5 times the Aldergrove trend. Aldergrove data is part of the GISS station database. I’m not sure if this data is already adjusted for UH but if it is, it looks as though UH effect has been over-estimated.
Peter Stroud says: An excellent paper that should be read by all IPCC sponsored scientists. One thing puzzles me though. On page 11 we see the percentage GHCN stations located at airports from 1890 – 2009. But according to WikiAnswers the first airport was built in 1909, College Park Maryland.
That is an artifact of how GHCN stores the Metadata. (Explained in the link in the paper). Basically, GHCN has a broken metadata structure. You can only assign a “present status” to a location, not a status-by-year. So if a bit of dirt becomes an airport, it will be marked as an airport for all time. Similarly, an “urban” station will be “urban” even if in 1800 it was a cow field. The airport at Templehof Germany (you know, Berlin Air Lift…) is being converted to a shopping mall. Whenever the metadata are changed, it will suddenly evaporate from ever having been an airport and for all time in GHCN. Just another stupidity in the data…
So you can make a report of AirStation FLAGS, but not of actual airports, by year. This means it’s an approximation (that will be conservative) in that you must choose to “start time” a bit after 1900 as zero airports with the knowledge that the slope of airport increase is actually much stronger than shown in the metadata.
Ackos says:
Are similar bias present between high and low altitude? Rural v city?
Yes.
http://chiefio.wordpress.com/2009/11/16/ghcn-south-america-andes-what-andes/
http://chiefio.wordpress.com/2009/11/17/ghcn-the-under-mountain-western-usa/
http://chiefio.wordpress.com/2009/12/01/ncdc-ghcn-africa-by-altitude/
and a whole lot more at:
http://chiefio.wordpress.com/category/ncdc-ghcn-issues/page/3/
and other pages in that category.
BTW, Sinan Unur has great stuff well worth watching. The animations are very informative.
Thanks for posting, Anthony! I just opened the paper for a moment, and this popped out immediately:
——-
The GHCN website does not report which sites are included in the monthly updates, but recent tabulations using GHCN records shows that:
• coverage has fallen everywhere (including the US)
• the sample has become increasingly skewed towards airport sites;
• the sample has migrated from colder latitudes to warmer latitudes;
• the sample has migrated to lower altitudes.
—-
…as you’ve been reporting for many months!
Yeah, I can’t wait to see this pop up on CNBC, CBS, BBC etc. They are too busy reporting about plunging phytoplankton populations due to, what, maybe a 0.5 C increase in temperature? IF there is even a temperature increase!
JamesS says:
If it gets warmer because of increased urbanization, well then, it gets warmer. Are they trying to make the numbers look like what they might be if nothing had changed?
I found that a particularly interesting question, as the first step ought to be looking for warming in the base data ( I can’t call what they produce “raw” – it isn’t.) If there is no warming even WITH UHI, then there really is no warming.
I wonder what these graphs would look like if only the raw data were used? When I look at the raw data for site in out-of-the-way places in West Virginia, or other places that haven’t seen any major development over the years, there is no AGW signature visible in those records. What’s being done to the raw data for the sake of “accuracy” is a crime against data.
Also, if you look at long lived stations, there is no ‘warming signal’. It only shows up in the constantly changing kaleidoscope of station change.
Here’s what it looks like in the “base data”. The “unadjusted” GHCN (that has been adjusted during various “Quality Control” steps…) with full artifact inclusion using a very simple “self to self” anomaly where each thermometer is compared only to itself and only inside the same month. A very clean comparison. (graph in link). It’s pretty much dead flat in summer months, but the winters warm from the horridly cold ones of the Little Ice Age ( “year without a summer”..) to a flat recent trend.
http://chiefio.wordpress.com/2010/07/31/agdataw-begins-in-1990/
Then 1990 hits and it all goes very crazy as the instruments are changed in odd ways. The onset of the “new” methods (new “duplicate numbers” for old stations) is in 1986, then the old ones are dropped in 1990 (ish) so you get a ‘feathering’ of the change to blend the hockey blade onto the shaft…
http://chiefio.wordpress.com/2010/08/02/agw-jumping-sharks-since-1986/
John Finn says:
There is much criticism of the use of weather stations at airports. Is there actually any evidence that the temperature trends at airport stations are signiicantly greeater than the trend at nearby rural stations. I’m sure there must be some that are, but equally I’ve noticed some that quite definitley aren’t.
The effect varies a bit with wind speed, but one of the problems is that large airports are near large urban centers, so you often can simply have the city UHI blowing over the airport, so it’s both wind speed AND direction (and what’s nearby…) that matters most. But overall, yes, they are hot places. You can do A/B/ compares via Wunderground and see it for yourself some times.
from comments here:
http://chiefio.wordpress.com/2009/08/26/agw-gistemp-measure-jet-age-airport-growth/
“An interesting paper (from 1989) discussing heat island issues at airports: (see pg 18 of 45, re the Akron-Canton airport)
https://kb.osu.edu/dspace/bitstream/1811/23307/1/V089N2_001.pdf
It also uses the phrase “airport heat island.”
and
Schrodinger’s Cat was the initial condition but apparently it was a female and pregnant when put in the experimental box . . . . now it is Schrodinger’s Cats in the experimental box.
John
oops, wrong thread for Schrodinger’s Cat post,
John
JamesS you might be interested in this series of posts by Dr. William M. Briggs on the whole Homogenization business.
http://wmbriggs.com/blog/?p=1459
There is links to the other 4 parts at the top of Part 1, but here is the kicker line for you:
@Kevin Kilty
I think it is necessary to watch that paint dry when so as to be able to put in context the absurdity of statements such as:
previous centuries! Give me a break. See Idiocrats at work.
I appreciate the comments and feedback. A few people have emailed with a request that I draw conclusions about how all the discontinuities affect the trends. However I am not sure that it is possible to do such a thing. The main task at this point is to give people a better understanding of where these famous graphs come from. The implicit standard I have in the back of my mind is that of building official national statistics. If a sampling frame changes dramatically, the statistical agency is supposed to terminate one series and start another, and advise users that they are not necessarily continuous.
I also hope to emphasize the difference between ex ante and ex post tests of data adjustments. Thus far people have focused on getting a clear list of the adjustments. But this just gives you, in effect, the list of ingredients on the bottle. You still need to test whether the medicine cures the disease. The Muir Russell team got the distinction correct (p. 155):
FWIW, I do use all the GHCN stations.
Also, per the folks saying that colder vs warmer stations don’t mater because it’s the trend that matters: That would be true for a pure “self to self” thermometer anomaly process (such as the ones I use) if you did not then have to splice the series of anomalies together.
IMHO, it’s “all about the splice” (once “adjustments” are disposed of…)
Now in many of the codes, like GIStemp, they have two issues. One is “the splice” of patching all these series together (and there are splice artifacts to be found). The other is that they do their “grid/box” anomalies with one box of thermometers in the baseline and a different set in the present (and an even different set out of the baseline in the past historical settings). So you not only get “the splice” but you get the anomaly calculated by comparing your Jaguar to an old Chevy and saying the car is hotter now than it was then.
And yes, if you all do your “reconstructions” using similar techniques, you all find the same errors.
JamesS says:
I’ve been in the software and database development field for 27 years, so I know a little bit about data and analyzing same. Perhaps the problem here is a more basic one than climate scientists will admit: there isn’t enough data to derive a global average temp.
It’s worse than that. Temperature is an intensive variable. Calculating a Global Average Temperature will always be a bogus number as you need other things (like, oh, specific heat, heat of fusion, all that water cycle stuff…) to have meaning. I have more on the math of it in this link, but unless you are a math geek, it will just cause you to glaze. (Even if I think it is foundationally important):
http://chiefio.wordpress.com/2010/07/17/derivative-of-integral-chaos-is-agw/
But yes, there is simply not enough data both spacially and temporally to do the deed, so they must make stuff up. And with predictable consequences.
Vorlath says:
I’m just saying it’s amazing how often the selection, monitoring and reporting of temperature (but NOT the temperature itself) seems to match the warming trend (temperature). After a while, one has to wonder if this is just coincidence or if there is actually correlation and causation.
Another interesting graph is this one:
http://chiefio.files.wordpress.com/2009/12/jetfuel_globaltemp.png
from this posting:
http://chiefio.wordpress.com/2009/12/15/of-jet-exhaust-and-airport-thermometers-feed-the-heat/
(with reference to original source in the article).
Direct correspondence of jet fuel with ‘warming’…
Oh, and on the issue of “by altitude” or “by latitude” station dropout not mattering, this ignores two very important points:
1) It means you take another SPLICE (see above).
2) It means you are taking another Apples vs Oranges anomaly.
and there is a more subtle point:
It means you have one class of station (colder and volatile) in the record in one period of time and replace it with a different class of station (warmer and less volatile) in another. This, btw, it the issue I’d hinted at before but not posted on. If you let me swap a volatile series for a non-volatile AND let me pick when, I can tune that to things like the PDO and give you any “shape” of data warming you want via judicious splicing. If the trend is going against you, move to non-volatile, if it’s in your favor, go volatile. This, btw, is a standard technique in stock trading. That the issue of site volatility with temperature regimes is ignored is a lethal failure of GHCN. (If it is shown to not be ignored, but to be a deliberate move to non-volatile stations near water and at low altitude, then it’s worse…)
So you put volatile stations in during 1950 to 1980 while the PDO is cold and “lock in” very cold excursions due to the higher volatility. Then, when things are in the warm phase of the PDO, move to low volatility stations. It’s now impossible for them to find as much cold as the old stations did during the next cold PDO. They lack the volatility range to get there.
So the key elements of the “magic” are:
1) The Splice.
2) The “Jag vs Chevy” comparison.
3) High volatility in the past cold PDO with low volatility substitution recently.
There is also a ‘process change’ that happens with the change of “Duplicate Number” that happens between 1986 and 1990 which clips cold going excursions rather dramatically (either that, or we’ve never had a cold winter since, and I think 2009 pretty much showed we have… see the “hair graph” in the “jump the shark” link); but I’ve not yet finished finding out exactly what that process change is.
But don’t worry, I’ll get to it. I never stop.
I’ve taken to calling this approach of a bunch of distributed biases “The Distributed Salami Technique”. Similar to the Salami Technique used in financial shenanigans, but with the method spread over many steps and many individually minor “issues”. Sort of a steganographic approach to fudge. Though the question still exists if this is deliberate or if Hansen is just a “Clever Hans”en… 😉
Climate science is a game where one side is funded with tens of billion Dollars, creating data, models and publications. The other side is unfunded and it’s their job to find out where the well-funded side has pulled which trick. And we know in advance that after one trick has been exposed they’ll use another one. Maybe they have entire departments with the sole purpose of brainstorming new techniques of deception. And they’ll never be held responsible.
In Ross’s paper he makes generous statements about and includes some of the contributions of well known temp mavens – what I termed Serious Amateurs with Strong Data Analysis Skills or more neutrally Independent Researchers. But he refers to them as bloggers. I feel “bloggers” diminishes the substance of their contributions and suggested that he come up with a less pejorative label. However, this is only my opinion and I would be interested in how others feel.
Ross McKitrick says:
August 3, 2010 at 12:24 pm
I appreciate the comments and feedback. A few people have emailed with a request that I draw conclusions about how all the discontinuities affect the trends. However I am not sure that it is possible to do such a thing. The main task at this point is to give people a better understanding of where these famous graphs come from. The implicit standard I have in the back of my mind is that of building official national statistics. If a sampling frame changes dramatically, the statistical agency is supposed to terminate one series and start another, and advise users that they are not necessarily continuous.
I also hope to emphasize the difference between ex ante and ex post tests of data adjustments.
_________________________________________________________-
A very simple explanation of what you mean by ex ante and ex post tests, would go a long way to making this report completely understandable to the lay people this report needs to reach. Perhaps including a glossary of terms would help.
Again thanks for a very good well written report.
First, I admit I haven’t read the paper yet.
One question however – when I’ve mentioned the changes in locations to a preponderance of lower latitude locations to AGW believers, one rebuttal comes up at times – that since temps are measured in terms of anomalies, and supposedly the higher latitudes are warming faster than the lower latitudes, removing higher latitude sites would actually result in lower anomalies rather than higher ones… comments please?
Mosher,
So you are saying that stations that have ONLY 15 years of full data during the 1961- 90 period are an adequate basis upon which to determine temperature trend? No matter what your, and Zeke’s and others, reconstructions purport to show surely you need more data than this to produce a sound result?
A question on Dr. McKitrick’s paper (I have not read all the comments so it may have been addressed). On page 4, at the bottom, he writes (quoting Folland and Parker 1995):
However on page 5, first paragraph, he writes:
I was curious as to when the change actually occurred. At the start or end of WWII?
Wow! I’ve spent 1/2 hour reading through this work. Amazing! I’m just surmizing that the data used for input to the processing code is readily available and clearly referenced in the work.
THE WAY REAL SCIENCE AN ANALYSIS SHOULD NOW BE DONE. No excuses. NO processed data HIDING behind computer codes NOT available…
No data sets “dissappearing”, “We’ll, we had it when we did the work. Whoops, lost it, so sorry…but you can trust our work, here’s the conclusion(s).” Dang, as the old saying goes, “Adults get PAID to do this?”
Ross McKitrick:
1. In the paper you wrote, “The so-called Optimal Interpolation (OI) method used by GISS employs satellite measures to interpolate SST data for complete global coverage.”
GISS uses the Reynolds OI.v2 SST data, but the dataset originates at NCDC. So you may want to change GISS to NCDC.
2. A note regarding the “1945 Discontinuity” discussed in Thompson et al: The discontinuity also appears in numerous other datasets. Refer to:
http://bobtisdale.blogspot.com/2009/03/large-1945-sst-discontinuity-also.html
And to:
http://bobtisdale.blogspot.com/2009/03/part-2-of-large-sst-discontinuity-also.html
3. Thanks for the reference to my post on page 28.
4. You wrote, “GISS uses another NOAA product, the Reynolds et al. (2008) Optimal Interpolation version 2 (OI.v2) data base. This is based on ICOADS up to 1998. Thereafter, like Hadley, they switch to a subset that are continuously updated. The updated subset is weighted towards buoy data since many shipping records are provided in hard copy. OI.v2 also uses AVHRR satellite retrievals to improve the interpolation for unsampled regions. Unlike the ERSST data set the satellite input is still used in OI.v2.”
Should “Reynolds et al (2008)” be Reynolds et al (2002)? Als s far as I know, the satellite data has always been a part of the NCDC Optimum Interpolation dataset and I believe it is the primary data.
The combined use of satellite and in situ SST data was first discussed in Reynolds (1988):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/reynolds_1988.pdf
Reynolds and Marsico (1993) mention Optimum Interpolation but don’t describe it:
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/blend_w_ice.pdf
Optimum Interpolation is discussed in detail in Reynolds and Smith (1994):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/oiv1.pdf
The OI.v2 version is introduced in Reynolds et al (2002):
ftp://ftp.emc.ncep.noaa.gov/cmb/sst/papers/oiv2.pdf
5. During the discussion of ERSST.v3 & .v3b you wrote, “This edition was called ERSST v3. However they noted that it reduced the trend slightly and deemed this effect a cold bias, so the satellite data were removed for version v3b.”
The trend wasn’t the problem. The satellite data changed which year and month had the record high temperatures, which brings nitpicky to an extreme. Refer to their webpage here:
http://www.ncdc.noaa.gov/oa/climate/research/sst/ersst_version.php
There they write, “While this small difference did not strongly impact the long term trend, it was sufficient to change rankings of warmest month in the time series, etc. Therefore, the use of satellite SST data was discontinued.”
The changes in rankings were illustrated in Table 6 of Smith et al (2008):
http://www.ncdc.noaa.gov/oa/climate/research/sst/papers/SEA.temps08.pdf
The irony in that is, in Smith and Reynolds (2004), the note how they perceive the satellite-based data. They write, “Although the NOAA OI analysis contains some noise due to its use of different data types and bias corrections for satellite data, it is dominated by satellite data and gives a good estimate of the truth.”
So if the “truth” changes which year has the second or third highest SST anomaly, they delete the truth. Link to Smith and Reynolds (2004):
http://www.ncdc.noaa.gov/oa/climate/research/sst/papers/ersst-v2.pdf
Thanks for the preview of the paper, Ross, and again, thanks for referring to my work.
Regards
Phil – good catch; my typo. The bucket transition was assumed to occur abruptly at 1941 in the Folland-Parker adjustment analysis. The abrupt SST change in 1945 was a different issue discussed in the Thompson et al. Nature paper.
I agree about measurements needing best value plus uncertainty. I don’t know all the details in the weighting given to sparse stations. I can imagine in the worst case that sparse stations might have too much weight because they are allowed to represent too large an area. I might point out that a number of skeptics attempt to do many things with this data, including seeing if it amounts to a hill of beans, but are sometimes stymied.
The operative word here is “large”. I agree with you that the shift to lower latitudes ought to produce a more moderate signal, but this I say without having thought about every influence that may occur. There was a time I would have thought that homogenization would not have had a systematic effect too, but after reading how NCDC does homogenization I now have serious doubts.
Whoa there, friend. First, you are implying that changing station distribution is the only “fly in the ointment.” It is not. Second, how come the onus is on me as critic? Maybe I should have tried that line on my dissertation committee?
It’s not my data. I’m not paid to do this work. I’m not one trying to guide policy using it. And I am certainly not one claiming these temperature series represent some sort of gold standard. Using sparse data to extrapolate trends over large regions deserves comment.
This is an interesting comment from p36:
‘(IPPC AR4) Summary for Policymakers stated:
“Urban heat island effects are real but local, and have a negligible influence (less than 0.006°C per decade over land and zero over the oceans) on these values.”
The 0.006°C is referenced back to Brohan et al. (2006), where it is merely an assumption about the standard error, not the size of the trend bias itself. ‘
Warmists often quote the UHI effect as being negligible … theeir source is yet another IPCC error !
Rational Debate (August 3, 2010 at 1:25 pm)
“…since temps are measured in terms of anomalies, and supposedly the higher latitudes are warming faster than the lower latitudes, removing higher latitude sites would actually result in lower anomalies rather than higher ones…”
High latitudes show greater extremes of temperature and show greater warming or cooling (anomalies) than lower latitudes. +/-5C is usual at stations in the Arctic, but much less at lower latitudes. This shows up well in this graph of GIStemp latitude bands: http://diggingintheclay.files.wordpress.com/2010/07/latitude-bands.png
Basically in answer to this question:
http://diggingintheclay.wordpress.com/2010/07/14/are-all-anomalies-created-equal/ – no they are not – analysis and posting are still a work in progress 😉
Rational Debate (August 3, 2010 at 1:25 pm)
It is late here, I am tired and I misread your comment – yes to what you are saying, but…. and it is a big ‘but’.
Imagine that the high latitudes are present during colder times and the move to warmer times – that means large cold then warm anomalies, then they are progressively lost as we enter a period which may be starting to cool off, so the less extreme (cooler) anomalies at the lower latitudes then predominate. It is a bit tenuous, but it could matter, particularly if the lower latitude stations are biased by UHI (which ‘is worse than we thought’).
I mentioned this on a thread a few days ago and didn’t get any response. It was a bit off-topic for that thread. But, more on topic for this thread.
Has anybody seen any discussion on how the response time of modern electronic temperature sensors might affect the comparison of the modern temperature record with the past record? I seems a faster responding instrument would record higher peaks when a puff of hot air moved passed the station from some artificial source, like pavement, plowed field, jet engine exhaust, etc., compared to a slower responding instrument, like a mercury thermometer. This could be a source of significant upwards bias.
John Finn says:
August 3, 2010 at 10:38 am
Have you ever actually been to Aldergrove airport?
I have & for an airport site it’s not too bad, it’s actually quite rural
54.656422,-6.217589
DaveE.
Forgot to mention…
The measurement site is in the RAF portion of the airport which is well separated from the main airport.
DaveE.
Mike,
I feel the situation is much worse than you suggest. Read page 11 of this document
http://www.srh.noaa.gov/ohx/dad/coop/EQUIPMENT.pdf
Temperatures are now measured to 0.1 deg but recorded to 1.0 deg. What nonsense to suggest fractions of a degree in global temp changes !
[REPLY – “Oversampling” will do it. For example, a million die rolls should yield just about a 3.5 average. (Having said that, I don’t much trust the current adjustments) ~ Evan.]
Peter Stroud says:
August 3, 2010 at 4:09 am (Edit)
“An excellent paper that should be read by all IPCC sponsored scientists. One thing puzzles me though. On page 11 we see the percentage GHCN stations located at airports from 1890 – 2009. But according to WikiAnswers the first airport was built in 1909, College Park Maryland.”
Wikianswers only considers airports as those servicing heavier than air aircraft, and only considers those locations specifically built for powered aircraft operations, rather than pastureland with weatherstations that was used for aircraft prior to 1909 (since the powered aircraft was first flown in 1902, and individuals had been attempting for years prior to that). GHCN includes a number of airfields which originally belonged to the Army Signal Corps for their balloon operations, and other balloon operations run in other countries by their military organizations. A balloon is an aircraft, even if Wikianswers doesn’t think so.
Oh…
Anther point is, it’s quite close to the moderating influence of the leoch. (Lake).
DaveE.
[i]Vince Whirlwind says: ……
That’s the funniest thing I’ve read for a while. Thank god we have professors of economics to explain science to us.[/i]
Where were you when the wolf-in-sheep-coat “scientists” from CRU and Goddard tried to explain science to us?
Verity Jones says:
August 3, 2010 at 4:37 pm
……Basically in answer to this question:
http://diggingintheclay.wordpress.com/2010/07/14/are-all-anomalies-created-equal/ – no they are not – analysis and posting are still a work in progress 😉
____________________________________________________________________
Verity,
If I click on the (thread here) in this sentence it gives a no access are you lost? statement
“This was a response to a blog comment (thread here) about GIStemp and the calculation of global average temperatures, er, anomalies. “
Otherwise you make some very good points. I have looked at the GISS data for areas in my state and the shape of the trends are all over the place: straight line up trend, straight line down trend, sine waves with no trend,up trend, & down trend and two flat trend lines at different levels. So how the heck do you use the anomaly information from those different types of trends to “adjust ” anything??
—
At 10:25 AM on 3 August, Gail Combs had written:
“Dr. McKitrick may want to add a glossary. Otherwise use the WUWT glossary (on to tool bar) here.”
Not having been aware of the glossary maintained by Mr. Watts and his associates on this site, I had used a search engine to seek the meaning of “GHCN” on the Web at large.
Naturally, “Wiki-bloody-pedia” came up, and strictly to find out what the Flim-Flam squad has been doing over there, I checked out their page for the Global Historical Climatology Network.
Has anyone an opinion of when that page should be edited to incorporate the results of Dr. McKitrick’s paper? In order to give the warmists a bit more agita and to help the millions of search engine users gain a more accurate appreciation of the validity of the information provided by the GHCN (because the search engines seem very reliably to steer preferentially to that most malevolently prejudiced site), the findings of Dr. McKitrick’s study need to show up there as balancing information.
Certainly, we can’t count on the cabal of Cargo Cult Science peddlers dominating Wikipedia to provide such balance.
—
re: Gavin (not Schimidt) says: August 3, 2010 at 6:45 pm
[i]Vince Whirlwind says: ……
That’s the funniest thing I’ve read for a while. Thank god we have professors of economics to explain science to us.[/i]
:::::::::::::::::::::::::::::::::::::
Hi Gavin,
Just an aside, but you’ve got to use carets (e.g, greater than and less than symbols), not brackets to get the html code to work.
Ross,
Looking at your chart for GHCN adjustments
V2mean.adj is a Change File. It must be added to V2.mean and then you have to
replace the lines that are duplicated.
Looking through the code and comparing your results to other results it appears
you may have just assumed that GHCNV2.adj was a stand alone file. It’s not.
you basically have to search through v2mean. find a matching line in v2.mean.adj
and replace the line in v2.mean with the line from v2.mean.adj. Not clear that you did that. if you did, my bad
Further v2.mean.adj also has some anomalous records, at least by my accounting.
Some of the changes in v2.mean/adj do not have corresponding lines in v2.mean
and there are some( a handful) of duplicate records ( not the same as duplicate stations)
Just checking. I’ll double check chad’s code, but you might want to ask him how he combined v2.mean and v2.mean.adj
Bernie says:
In Ross’s paper he makes generous statements about and includes some of the contributions of well known temp mavens – what I termed Serious Amateurs with Strong Data Analysis Skills or more neutrally Independent Researchers. But he refers to them as bloggers. I feel “bloggers” diminishes the substance of their contributions and suggested that he come up with a less pejorative label. However, this is only my opinion and I would be interested in how others feel
Bernie: On my site I have specifically laid out my interests. Among them is that I felt my role was to plough the field rapidly ahead of others, and if I turned up things of interest, I would be very happy being a ‘footnote” in there published papers. In my opinion, that I’m a “link” and not just a footnote is something of “high praise”. So don’t worry about the “slight” of being called a “blogger”, for to me it is no slight. It is the truth and “The Truth just is. -E.M.Smith” and I am honored to be outside the “peer” system as the emails of “ClimateGate” have shown it to be. There can not be any higher praise than to say “He is not one of them”, at least in “my book”.
Rich Matarese says:
August 3, 2010 at 7:30 pm
Not having been aware of the glossary maintained by Mr. Watts and his associates on this site, I had used a search engine to seek the meaning of “GHCN” on the Web at large.
Naturally, “Wiki-bloody-pedia” came up,….
Has anyone an opinion of when that page should be edited to incorporate the results of Dr. McKitrick’s paper? In order to give the warmists a bit more agita and to help the millions of search engine users gain a more accurate appreciation of the validity of the information provided ….
__________________________________________________________
Unfortunately it will not “stick” thanks to Connolley who is dedicated to maintaining the CAGW propaganda at Wikipoo.
—
At 9:44 PM on 3 August, Gail Combs had replied to my query about editing the “Wiki-bloody-pedia” entry on the Global Historical Climatology Network to reflect the findings reported in the preliminary draft of Dr. McKitrick’s paper presently under discussion, writing:
“Unfortunately it will not “stick” thanks to Connolley, who is dedicated to maintaining the CAGW propaganda at Wikipoo.”
I was aware of the activities of Mr. Connolley and the rest of the Wikipedia Ministry of Truth.
I would advise all and sundry that atheromatous cardiovascular disease is, in considerable part, an inflammatory phenomenon which can be exacerbated by emotional stress.
Not that I would wish Mr. Connolley and his co-religionists to come precipitously to something fatal in the way of a cardiac arrhythmia or myocardial infarction, but every little bit of irritation that might be imposed upon these creatures would, ceteris paribus, help to occupy them with stimuli conducive to an outcome beneficial to the well-being of the human race at large.
Is the Italian slang term “agita” not fully understood?
—
@Mike G: I often use “resounding silence” to mean I’m on to something. It usually means nobody has thought about it. That was my response to your point. “Gee… that’s interesting. Probably something to it. I have no idea what to say as it’s a new idea to me. Hmmm…”
Watch for the “negative space” of things. It’s a great forensic tool…
Dr A Burns says:
I feel the situation is much worse than you suggest. Read page 11 of this document
http://www.srh.noaa.gov/ohx/dad/coop/EQUIPMENT.pdf
Temperatures are now measured to 0.1 deg but recorded to 1.0 deg. What nonsense to suggest fractions of a degree in global temp changes !
[REPLY – “Oversampling” will do it. For example, a million die rolls should yield just about a 3.5 average. (Having said that, I don’t much trust the current adjustments) ~ Evan.]
At the risk of igniting yet another precision firestorm: Oversampling of A Thing works, but sampling of many divergent things is NOT oversampling. Measruring A place at A time 10,000 times is NOT the same as measuring 10,000 time/place sets and averaging them. The first is oversampling, the latter is delusional.
@Steven Mosher:
You sure about that “add them together” angle?
From: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.readme
That sure looks to me like they are stand alone versions. Also the v2.mean.Z size is 12016 KB while the v1.mean.adj.Z size is 8672 KB which implies to me a near parity, not just a few updates to be glued on.
I can download the data and look at it, but I’m pretty sure it’s a stand alone set.
I would urge all readers to send a link to this page to their elected representatives along with a strongly worded “suggestion” that they get off their duffs and kill this AGW foolishness immediately. Come to think of it, send it to your local paper too [with appropriate commentary].
Nick Stokes (August 3, 2010 at 6:42 am)
Sorry, only just spotted your comment Nick.
“Zeke showed that a reconstruction using the adjusted file gave very similar results to the GHCN unadjusted.”
That does not surprise me in the least. I have been following the analyses that you and others have been doing. Re the adjustments – they seem to be so finely balanced that the there is little change overall. At least that is what I have been seeing in GIStemp. It is not just the adjustment, but the spatial and temporal balance of the adjustments. At a local level they can have a big effect but scaled up and averaged over the globe the differences are very small ~0.1 deg of trend. I think it is to do with homogenisation tending to ‘average things’ such that in a location one station may be ‘warmed’ and another ‘cooled’ so that all have a similar trend.
I posted a version of this at tAV earlier – looking at the nightlights radiance adjustment implemented in January. http://diggingintheclay.wordpress.com/2010/07/10/gistemp-plus-ca-change-plus-cest-la-meme-chose/
It caused very little overall difference, and looking at how it affected the data by lattitude there were reasonable differences in the latitude bands, but they cancelled each other out at global level. Dig down to country level, even grid square location and individual station level and there are very major changes. This is something we all need to look into more.
Gail Combs (August 3, 2010 at 7:17 pm)
Thanks for the heads up – link now fixed. And yes the adjustment process is a source of no end of wonder to me (and others).
Peter O’Neill has done some very detailed posts explaining it: http://oneillp.wordpress.com/
In other words:
The surface temp. data sets are far too biased and limited to draw any dire conclusions about long term climate.
Now can we get back to real problems and just ignore the likes of Gore, Hansen, Schmidt and Jones?
David A. Evans says:
August 3, 2010 at 5:26 pm
John Finn says:
August 3, 2010 at 10:38 am
Have you ever actually been to Aldergrove airport?
Yes.
I have & for an airport site it’s not too bad, it’s actually quite rural
But there are still planes and tarmac. Things which weren’t around in 1880. Aldergrove should have been more affected than most other places by the UH influence over the past 50 to 100 years. If you read up on the UH response to urbanisation the effect is logarithmic, i.e. the transition from rural to semi-rural has a far bigger effect than, say, increased urbanisation in a major city.
In other words – the TREND should have been greater. The trend is actually greater at Armagh.
E.M.Smith says:
August 3, 2010 at 11:32 am
The effect varies a bit with wind speed, but one of the problems is that large airports are near large urban centers, so you often can simply have the city UHI blowing over the airport, so it’s both wind speed AND direction (and what’s nearby…) that matters most. But overall, yes, they are hot places. You can do A/B/ compares via Wunderground and see it for yourself some times.
I am not questioning the UH effect. I know UH exists. I’ve experienced it many times. I am, though, questioning it’s effect on the calculated trend(s). A major urban area will generally have higher temperatures than a nearby rural area – especially when it’s very hot or very cold. But does the urban area have a greater warming trend over time? Do airports have a greater warming trend than nearby rural locations? There doesn’t appear to be much evidence that they do.
Raw temperatures are not really relevant. It’s the trend that matters.
—
At 12:52 AM on 4 August, Jeff B. writes:
“Now can we get back to real problems and just ignore the likes of Gore, Hansen, Schmidt and Jones?”
To a considerable extent, “the likes of Gore, Hansen, Schmidt and Jones are the real problems we’re confronting today, by which I mean creatures without conscience or consequence fixated upon exerting control over their fellow human beings out of megalomania and a strange, sick sense of their own “entitlement” to rule those whom they arrogantly consider to be their inferiors.
That, by the bye, emphatically includes us “skeptics.”
Haven’t you noticed their attitude toward us? How dare we question them, dispute them, ridicule them, and treat them with the scorn they have so richly earned?
Heavens, don’t us peasants know our place?
Let us of course add to that listing of warmist charlatans their political allies in both wings of America’s sordid, permanently incumbent Boot-On-Your-Neck Party (especially the National Socialist faction, which we’re no longer calling “Democrat”).
Whenever our President-With-An-Asterisk addresses the American people, do you note that he holds his head as if somebody had smeared his upper lip with a particularly noisome dollop of fresh stool?
Call me a teabagger, but I get the impression that – despite his fervent desire to spread our wealth around – he doesn’t like us very much at all.
—
EM.
It’s a change file. Regardless, nobody uses it.
“sure looks to me like they are stand alone versions. Also the v2.mean.Z size is 12016 KB while the v1.mean.adj.Z size is 8672 KB which implies to me a near parity”
Near parity?
you have a file with 12MB of “raw” data. That data is adjusted. the adjusted lines are written out. 8.6MB. Otherwise you end up with like 12 stations in 2009.
Thats when the lightbulb went on. The majority of adjustments happen early on.
late in the record you have fewer adjustments. fewer lines to replace.
A first I thought it was stand alone, but when you actually plot out area maps of every year you can see that past 2005 there is almost NO DATA. Why? cuase its a change log.
THAT is why Ross chart goes bonkers.
Its a change file.
I believe carrot confirmed this with his contacts inside Noaa, or ron did.
http://rankexploits.com/musings/2010/the-ghcn-adustment-puzzle/#comments
EM Chad appears on that thread which is why I was asking Ross.
Good to see that climate science is finally progressing after years of sabotage and blockade by the hijacked peer review process.
This article should become standard literature for 10% of the climate scientists, while the other 90% will start do something useful, after the global warming swindle is over.
In before the lock as they say, or should that be the crys of the uninformed, ‘But it’s not peer reviewed’.
Saying that if i was a true scientist i would welcome such comment on my work as it can only make it better, whats that flipper it’s not about science but money, power and changing the way every single human being live’s, stop talking nonsense your just a dolphin.
And the ususal pat on the back, bag of peanuts and pint of beer for all the hard and excellent work carried out by Dr McKitrick.
Ross, there are a few areas with typos and expression could be clearer here and there, but who am I to comment?
I seek to raise an additional, possibly major source of error, on the topic of anomaly methods. It’s not novel.
On p.33 of your draft we have “CRU combines obviously duplicated (identical) records, then computes anomalies, then grids the resulting series (Brohan et al. 2006). Anomalies are computed by subtracting the 1961-1990 mean (the so-called “normal”). ”
Elsewhere as in fig 1.1, you point to 1961-1990 as a period when the greatest numbers of observing stations were included in the GHCN data. What, then, is the “normal”? Is it the constant historic number derived from the 1961-1990 stations in fig 1.1, or is it an ever-changing number that changes because stations were later dropped from the 1960-1991 period in great volume? The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
So, how does one treat anomaly calculations before 1961 if the normal is not a constant? OTOH, if the normal is a constant, how does one treat data after 1960? Presumably data have been dropped after 1960 because they have problems, so these problems will affect the normal from 1960-1991 in that it will include defective, rejected data.
Some have commented that the absolute values matter less than trends and that trends are less affected by the anomaly method, a solution with so many problems that it deserves a section of its own in your paper. For a start, should a trend be linear of polynomial and why? What is wrong with using physical real numbers?
There is great value in your paper, because it shows the artificiality of the adjustment methods, spiced up by admissions from climategate (and there are more than you quote). I’m reminded of John Cleese in Fawlty Towers, who is a master at digging deeper and deeper holes by successive slight adjustments to his stories. In the case of these global temperatures, the hole cannot be filled in again because no single person knows who made which adjustment of what size to whatever station data, before or after another adjuster had a go. The reconstruction back to truth is now irreversible.
Unless, of course you have the splendid ability of the Sir Muir Russell group who knocked over the prioblems in a couple of days. What a shallow, pathetic exercise that was.
If you seek another section for your paper, try an examination of the country data adjustments performed before GHCN again adusted their “raw” data.
Still, a double shot cappucino is a good way to start a boring day.
John Finn “Do airports have a greater warming trend than nearby rural locations? There doesn’t appear to be much evidence that they do.”
Do you have any quantitative data?
Why not model an airport in terms of the energy needed to taxi, lift and lower a 100 tonne object to 100m altitude, n times a day, spread the energy over the airport volume and see how many watt per sq m you get at ground level. This can then be converted to a temperature change. I can’t find anyone who has done this simple exercise.
Kevin:
It’s not my data either, and none of us are trying to “guide policy using it”, so this argument is just a distraction.
In the end, I would suggest that the adjustments don’t matter that much, because the two main effects seen in the data is the global mean trend (which is about 10x the corrections to it that people worry about) and secondly the south-to-north variability in the trend.
All data have warts, the $20 question is whether they matter. I would claim for the dominant questions (is it warming? is the northern climes warming faster than the southern ones?), the real effects are big enough that the problems with the data don’t affect those conclusions.
Dr. McKitrick,
Thanks for the clarification. And thank you for the paper. It was easy to read and understand. While I do have a background in statistics (my degree is in Economics), it has been a long time since I have had to use the science as my current profession lies in computer networks. But it was good to exercise some rusty brain cells!
Geoff Sherrington says:
August 4, 2010 at 5:20 am
Do you have any quantitative data?
Why not model an airport in terms of the energy needed to taxi, lift and lower a 100 tonne object to 100m altitude, n times a day, spread the energy over the airport volume and see how many watt per sq m you get at ground level. This can then be converted to a temperature change. I can’t find anyone who has done this simple exercise.
I’m not sure how that would influence the trend. I don’t think we need to go to that much trouble. If the trends at airport stations are broadly similar to trends at nearby rural stations then we can assume UH has little or no influence on the overall warming (or cooling) trend.
Thie issue is not whether airports are warmer than rural locations (the tropics are warmer than the poles) but whether airports have warmed at a faster rate. There appears to be no evidence to suggest there is a significant difference between the two.
Carrick says:August 4, 2010 at 6:15 am
In the end, I would suggest that the adjustments don’t matter that much, because the two main effects seen in the data is the global mean trend (which is about 10x the corrections to it that people worry about) and secondly the south-to-north variability in the trend.
A little bit of an poofy blather to make your point don’t you think. Considering the warming since 1970 is ~.7 F and the adjustments are ~.45 F., (TOBS, Station relocation, and instrumental change, none of which were made by side to side comparison with actual empirical data; but not including the “quality” changes made by GHCN prior to distributing the “raw” data) just doesn’t seem to add up to 10X.
Tim Clark.
TOBS is an empirical model. Validated with real observations. Its applied, as far as we know, in the US. Anyway, TOBS is REQUIRED. this can be proved with observational data. Go look at the CA thread on TOBS. Next, the method was developed by studying hundreds of stations across the US. Stations that had HOURLY data. This allowed researchers to calculate the effect from CHANGING the observer time. That study resulted in a prediction method. Based on the lat/lon time of year, position of the sun, etc, the prediction code would take the recorded min/max at say 7AM and predict the min/max at the prior midnight. The code was then verified against a portioon of the sample that was “held out” during model creation. Very well done. There is a SE ( standard error of prediction) BUT if you did not correct for TOBS you would have a biased record. TOBS removes the bias, but introduces additional uncertainty.
Changes in instrumentation are likewise based on observational studies. Station RELOCATIONS. The station moves that need attention are moves in lat/lon and moves in elevation. Elevation moves are corrected for by looking at lapse rates.
Its a one time adjustment. Same for lat/lon moves.
Geoff.
WRT your desire to model the “energy” required at an airport.
The difficultly is that to do this properly you need to understand the local wind conditions and things like the turbulant mixing that occur when a plane lands.
Also, You need to know the time of day. There are two critical periods.
The Tmin period ( say late night ) and the Tmax period ( say mid day).
If you dont have planes landing or taking off in the Tmin window then the chances of infecting that record are slim. Tmax? I would bet your chances are better looking at that.
Mitigating factors: The wind, clouds, rain. Since airports tend to be located in areas with a long fetch, long open fetch, with consistent wind patterns, you are going to have a lot of heat moved by that nice laminar flow over the undisturbed surface.
Again, People always look at the picture and forget the dynamics of the situation.
One data point. In the CRN study a small warming bias at the airport was found. .25C
Modulated by clouds, winds and rain. Those three factors kill UHI.
So, if you had a 2C warm bias on Tmax.. Then Tave would get a 1C bias. Tave=(tmax+tmin)/2. One BENEFIT of not integrating over time.
Next, if you see a 1C bias on sunny days and half your year is cloudy. that puts the bias down to .5C. Windy days? same thing. rainy days? same thing.
Is there a bias? probably, phsyics seems to indicate as much. How big? on a few days it could be large. Over the course of years and years? We can measure that my looking at lots of airports. Its on the list of things to do.
BUT to do it right, you want that wind data.
Geoff:
Elsewhere as in fig 1.1, you point to 1961-1990 as a period when the greatest numbers of observing stations were included in the GHCN data. What, then, is the “normal”? Is it the constant historic number derived from the 1961-1990 stations in fig 1.1, or is it an ever-changing number that changes because stations were later dropped from the 1960-1991 period in great volume? The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
Some clarity on how anomalies are calculated.
For EVERY station you do the following.
1. See if the station has a minimum number of “years” in the period. I use 15 FULL years. I can change this and make it anything I want. Doesnt change answers much.
I have also used other periods. The 1953-1982 is the “richest” period WRT total station months. ( months of data)
So: now for that station you compute a MEAN Jan, mean feb, mean march etc.
So, now you have a mean FOR THAT STATION.
jan =14C
feb =15.5C
mar=16.2C
And so on.
Now you look at the entire time series.
if Jan 1926 was 13.5C your anomaly is 13.5-14C. or .5C
If the station drops out in 1995.. you get NA
To calculate the global average you:
Take a gridcell: 120-123;45-48: (for example)
Average all the stations within that grid. You got a gridcell average.
do that for all grids with data.
Then: look at all the grids: SUM the area (on the sphere) that those grids cover. If they dont have data that month, then NA.
Then you look at the land area for each grid, on the sphere. Say 100sqkm
You then create a weight: cellarea/total area.
Multiply your grid anomaly by the weight. for every MONTH. Then you sum the weighted grid anomalies.
Thanks Steve (Mosher) for the query about the GHCN adj graph. I don’t have an answer at the moment. I have been tied up today reading a dissertation, and tidying up loose ends for a new paper with Steve and Chad, and tomorrow I am offline most of the day. I’ve asked Chad for help on this.
Personally I find the chimney brush very strange and I wouldn’t be surprised if there’s something we missed in the definition of delta that at least partly explains its bizarre appearance.
Also I hope that one of the points that emerges from my paper is that a lot of bloggers like Chiefio, Chad, Bob Tisdale, Jeff Id, RomanM etc. have done impressive work on explaining these temperature products, and their contributions deserve to be read. Not to say I was convinced by everything I came across, but I did often find myself wondering, why am I learning about this from a blogger, instead of from the IPCC (or NOAA)? They’re the ones that rely on the data to make their points, so they should also show the background information to gives readers a sense of the potential problems with continuity, sampling, measurement quality, etc., instead of always waving them away or making people guess about what’s going on.
When I saw the graph and term “chimney-brush”, I must confess I thought of a more common utensil nowadays – a toilet brush. Or in British parlance, a bog brush.
As E.M. Smith correctly points out, the essence of the bias built into the GHCN data set derives from the splicing together of anomalies from a set of stations that DIFFERS from the beginning to the end of the reconstructed record. If stations little affected by UHI are used in constructing the “norm” in any grid-cell, then virtually any apparent trend can be obtained by varying the ratio of corrupted-to-pristine records near the beginning and end of the reconstruction period. That something akin to this has been done is quite apparent from the steep drop-out of “rural” stations since 1990, especially in Asia, where the reputed “global warming” is most intense. There are also other issues, but this lies at the very core.
Paul Deacon, Christchurch, New Zealand says:
August 4, 2010 at 5:28 pm
When I saw the graph and term “chimney-brush”, I must confess I thought of a more common utensil nowadays – a toilet brush. Or in British parlance, a bog brush.
__________________________________-
Darn it now I have to clean my computer screen again. I think Dr McKitrick was just being polite but your name for it will probably be the one that catches on.
Steven mosher says:
August 4, 2010 at 11:10 am “WRT your desire to model the “energy” required at an airport.”
I agree with what you write, but I’m looking at ballpark figures. I have a suspicion that busy airports do not create enough fuel-derived heat even on a windless day to change the temperature much. If this is the case, it cuts short a lot of speculation about the desirability of airport versus non-airport station siting.
That would cut out quite a lot of verbiage. The present discussion seems to assume significant fuel heat.
The work I have done is limited, but I could pick up no significant difference from comparison sites where there is a genuine rural site close to an airport site. I did not use big hub airports.
WRT time of day, do you think that daily temperature max-mins are those recorded electronically these days – or is there curve fitting and rejection of transients? If the latter, it would be hard to splice mercury records to thermistor/thermocouple records as they were successively replaced. Have you ever sen an instrumental overlap comparisson or any code to cope with the transition?
Steven Mosher – where I speculate “The global absolute mean changes from the march of the thermometers. So does the value of the “normal”.
I am I right or wrong? I know the procedure you patiently describe again, but I am unsure of the fate of the “NA” cases or what effect they have on the global anomaly on a given day like today.
Steve, you are right, it’s a change file. The chimney brush uses a delta series constructed such that v2.mean is computed on its spatial basis, and v2.mean.adj is computed only using the gridcells with adjusted data, rather than filling the remaining cells with unadjusted data. That conflates two sources of difference, the adjustments themselves and the drop in sample size between raw and adj. To isolate the adjustment effect we’re putting together a new version where we only compare based on cells where both raw and adj are present. It looks like a chimney brush after being run over by a truck. The alternative is to assign a zero adjustment to all non-adj gridcells, which we are checking as well. But that assumes that the adjustment is estimated as zero, rather than being set to zero due to a lack of data. It is not reasonable to assume the adjustments would be zero everywhere the data happen to be missing for computing the actual adjustments.
Thanks for the assistance.
This may have already been answered (I know, I know. But it’s past 3 AM in the morning, and I don’t have the time to trawl through all the posts looking to see if this has been covered), but figure 1.6…puzzled me. How, exactly, was ~30% of GHCN data taken from airports in the 1890s? Is it just that stations were situated at sites that were later to become airports (which seems peculiar; why would that be so?) If that’s the case, anyone happen to know why airports were, apparently, often built where old weather-monitoring stations had been constructed?
Another question I had was concerning graph 1.7; were these mean values calculated by giving northern latitudes positive values and southern latitudes negative values, or was it all absolute value (I ask because if it positive for northern and negative for southern, the shift towards the equator could be considered good news, as it would mean that more of the southern hemisphere was being accounted for than before. This supposition doesn’t seem to be born out by the listings of station coverage for northern and southern hemispheres, though. If absolute value…well, less so, since an ideal coverage would (area-wise) yield an average latitude of 30 degrees, assuming I calculated things correctly)? This may have been covered in the appendix, but I’m not particularly computer literate and I don’t really know what the code given there means. Many thanks if anyone is willing to spare the time to answer these questions.
FWIW, I’ve put up a “Volatility” posting that explains the effect I mentioned above:
http://chiefio.wordpress.com/2010/08/04/smiths-volatility-surmise/
And per the issue of airport change / warming trend over time. It think it’s pretty clear that you can look at places like, oh, most any major international airport with dozens (hundreds?) of acres of concrete and tarmac and figure it was a grass field transpiring water to maintain a low leaf temperature 100 years ago. So go from grass field to black solar collector with constant growth over time for most of them. Yeah, I’d call that a ‘warming trend’. FWIW, in my lifetime I’ve watch much of the growth. Populations grow, airports grow. Little grass shack tropical fields are now 10000 foot jet airports with A/C and giant terminals and non-stop traffic of both cars and planes. Far more ‘trend’ than just about any other place and more than the nearby cities (that were often in existence prior to the first airplanes). I’ve got many examples in the links posted above, but one of my favorite is the Seaplane harbor at Milano that became tarmac as intercontinental travel moved from Pan Am Clippers to 707s. Water to concrete and tarmac does kind of count as ‘warming trend’. If you don’t believe me, go stand in the parking lot as long as your bare feet can stand it in August, then run and jump into the lake…
Sam Yates says: August 5, 2010 at 12:45 am
“Another question I had was concerning graph 1.7; were these mean values calculated by giving northern latitudes positive values and southern latitudes negative values, or was it all absolute value”
I think Sam’s suspicion is right. The mean latitude is signed:
lat_mean_unweighted[i] = sum(temp*lat)/sum(temp)
That gives wrong results. It means, for example, that the big dive in mean latitude in 1990 came when a big batch of China/Canada/Turkey stations was eliminated, and the sharp recovery came in 1992 when an even bigger batch of Australian stations was removed.
Is there any reason why this paper or a version of it is not being published in a peer reviewed journal?
Ah, thank you, Nick. So, according to that, a shift towards zero…well, I can hardly call it a good state of affairs, considering that it’s caused by stations dropping out of the record, but it does mean the data sources are getting more evenly distributed over the globe. That might or might not cause an overall warming trend; a lot of US stations were cut out (and the US has been warming faster than the globe on average: http://data.giss.nasa.gov/gistemp/graphs/Fig.D.lrg.gif), so…would that give a cooling bias, overall? Or warming, because the US is cooler than places closer to the equator (then again, Australian stations got cut, as well, and that hardly counts as a cool and clement place).
PolyisTCOandbanned: In response to your question #4, 30 degrees is the average latitude (set the integral of sine theta from 90 degrees to x degrees equal to the integral of sine theta from x degrees to 0 degrees, and solve. If I made a mistake in the setup, please, let me know). According to what Nick Stokes posted, though, that may be irrelevant; if the southern hemisphere latitudes are given negative values, then 0 degrees should be what you’d expect for a perfect global distribution.
Sorry, meant cosine in that last comment.
[snip – policy me@meme.com is not a valid email address ~mod]
It’s a self-evident non-valid one. And funnily so.
BillD: all I’ve tried to do is pull together the strands of work done by others on blogs and in journal articles, and put together a readable story with it. This isn’t what I’d call original research. It’s more of a survey. Based on this info people might want to go on and do some original research, such as trying to quantify the effect of the changing sample distribution/locations on the global average. That kind of work should go to journals.
Ross,
“Steve, you are right, it’s a change file. The chimney brush uses a delta series constructed such that v2.mean is computed on its spatial basis, and v2.mean.adj is computed only using the gridcells with adjusted data, rather than filling the remaining cells with unadjusted data.”
Thanks, Thanks go to Zeke and a guy named zoro80 for figuring it out.
Also, be careful about merging them blindly. Working with R I found some irregularities in the .adj file.
Take a dataline in v2.mean
999123450001 1980 NA NA NA 12 13 14 62 23 23 24 25 67
Thats a line for a GHCNID 99912345000 DUPLICATE 1. year, then months.
A change file, should then have only lines that either MATCH this line or are absent
Like so:
999123450001 1980 NA NA NA 12 13 14 12 23 23 24 25 26
And you can see that I changed some values for that GHCNID/Dup.
BUT, v2.mean.adj has Lines that DONT appear in v2mean?? weird. are these
new records? how do you adjust something that didnt exist?
SECOND. there are duplicate lines in v2.mean.adj. Same ghcnid/dup, same year, but
MULTIPLE occurances.
My guess is the programmer didnt check for this kind of error. But when you work in R and merge based on matching indices these goofs become readily apparent.
have chad double check or write me or zeke.
Geoff:
“WRT time of day, do you think that daily temperature max-mins are those recorded electronically these days – or is there curve fitting and rejection of transients? If the latter, it would be hard to splice mercury records to thermistor/thermocouple records as they were successively replaced. Have you ever sen an instrumental overlap comparisson or any code to cope with the transition?”
Electronic AFAIK. WRT splicing. I’d have to look around but typically there are studies done to estimate the adjustment required at a splice. Rememeber a splice,
even DONE POORLY doesnt hit your trend very much, and depnds upon the year the splice was done.
WRT the energy calculations. Im working on a different study right now, but I’ll return to the airport one. You could just wing it or back of the envelop.
Depending on the altitude a 777 GULPS 100Klbs of fuel per hour
at MAX throttle. ( two engines at 50K)
Go figure 3 minutes for takeoff. The busiest airport has fewer than 3000 movements per day.
The number 30 busiest airport would only have 1000 movements per day.
So I’d start by bounding the problem from above.
5K lbs of fuel burned for every takeoff and landing ( thats a HIGH estimate)
Work from that. None of the top 30 airports are in ghcn. so you can bound the movements to less than 1000 per day.
more later gotta run
A new version of the report is posted at my home page with a few corrections: Fig 1-7 mean latitude is now mean absolute latitude (graph looks similar overall, but without spikes); Fig 1-10 re-done using ghcn.adj as a change file, only using grid cells where both raw and adjusted obs are available. Accompanying text revised as needed; revised codes in appendix. Thanks for the corrections. BTW some of you might be interested in our new ASL paper, also on my homepage.
Right, I just saw news about the new paper of McKitrick, McIntyre and Herman at Pielke Pere’s site. The climate models’ projections don’t look very good.
========================
—
At 6:37 PM on 5 August, Dr. McKitrick had written:
“A new version of the report is posted at my home page with a few corrections: Fig 1-7 mean latitude is now mean absolute latitude (graph looks similar overall, but without spikes); Fig 1-10 re-done using ghcn.adj as a change file, only using grid cells where both raw and adjusted obs are available. Accompanying text revised as needed; revised codes in appendix. Thanks for the corrections. BTW some of you might be interested in our new ASL paper, also on my homepage.”
The exchanges in this thread present the most interesting example of open-forum peer review that I have yet seen. Everything is transparent, and even queries from the “unqualified” are posted and receive response.
What’s that old saw? “The greatest fool may ask more questions than the wisest man could answer.”
Dr. McKitrick’s readiness to read and respond to pertinent interrogatories and comments demonstrates that in a fashion far more flexible and immediate than the formal mechanism of academic peer review, manuscripts can be refined to improve their quality, and as Mr. Watts maintains these exchanges in a Web archive of sorts, all of this is available for later review by anyone interested in the subject under consideration.
This beats the hell out of an editor designating a small cadre of peer reviewers who frequently use their participation in the vetting process to snipe at an academic rival from behind a screen of anonymity.
I would ask that Dr. McKitrick please provide an active link to his revised version of this paper, as a look into his personal Web site does not readily lead to this element.
—
I barely understand half of what you guys are talking about, partly because I have developed a temperaturegraph phobia.
But thanks to all of you for investigating and back-engineering this issue with a calm, reasonable and professional conduct. I believe you are the true scientists. Because you don’t need an approved title to be right.
Ross;
Where has the Sociometric paper now been accepted? At the same journal, or “elsewhere”?
Rich, the current version of the paper is here. I have migrated to http://rossmckitrick.weebly.com/ so the most updated version will be there.
Brian H., “elsewhere.” I’ll post up details when I have final acceptance.