E.M. Smith over at the blog Musings from the Chiefio earlier this month posted an analysis comparing versions 1 and 3 of the GHCN (Global Historical Climate Network) data set. WUWT readers may remember a discussion about GHCN version 3 here. He described why the GHCN data set is important:
There are folks who will assert that there are several sets of data, each independent and each showing the same thing, warming on the order of 1/2 C to 1 C. The Hadley CRUtemp, NASA GIStemp, and NCDC. Yet each of these is, in reality, a ‘variation on a theme’ in the processing done to the single global data set, the GHCN. If that data has an inherent bias in it, by accident or by design, that bias will be reflected in each of the products that do variations on how to adjust that data for various things like population growth ( UHI or Urban Heat Island effect) or for the frequent loss of data in some areas (or loss of whole masses of thermometer records, sometimes the majority all at once).
He goes on to discuss the relative lack of methodological analysis and discussion of the data set and the socio-economic consequences of relying on it. He then poses an interesting question:
What if “the story” of Global Warming were in fact, just that? A story? Based on a set of data that are not “fit for purpose” and simply, despite the best efforts possible, can not be “cleaned up enough” to remove shifts of trend and “warming” from data set changes, of a size sufficient to account for all “Global Warming”; yet known not to be caused by Carbon Dioxide, but rather by the way in which the data are gathered and tabulated?…
…Suppose there were a simple way to view a historical change of the data that is of the same scale as the reputed “Global Warming” but was clearly caused simply by changes of processing of that data.
Suppose this were demonstrable for the GHCN data on which all of NCDC, GISS with GIStemp, and Hadley CRU with HadCRUT depend? Suppose the nature of the change were such that it is highly likely to escape complete removal in the kinds of processing done by those temperature series processing programs?….
He then discusses how to examine the question:
…we will look at how the data change between Version 1 and Version 3 by using the same method on both sets of data. As the Version 1 data end in 1990, the Version 3 data will also be truncated at that point in time. In this way we will be looking at the same period of time, for the same GHCN data set. Just two different versions with somewhat different thermometer records being in and out, of each. Basically, these are supposedly the same places and the same history, so any changes are a result of the thermometer selection done on the set and the differences in how the data were processed or adjusted. The expectation would be that they ought to show fairly similar trends of warming or cooling for any given place. To the extent the two sets diverge, it argues for data processing being the factor we are measuring, not real changes in the global climate..The method used is a variation on a Peer Reviewed method called “First Differences”…
…The code I used to make these audit graphs avoid making splice artifacts in the creation of the “anomaly records” for each thermometer history. Any given thermometer is compared only to itself, so there is little opportunity for a splice artifact in making the anomalies. It then averages those anomalies together for variable sized regions….
What Is Found
What is found is a degree of “shift” of the input data of roughly the same order of scale as the reputed Global Warming.
The inevitable conclusion of this is that we are depending on the various climate codes to be nearly 100% perfect in removing this warming shift, of being insensitive to it, for the assertions about global warming to be real.
Simple changes of composition of the GHCN data set between Version 1 and Version 3 can account for the observed “Global Warming”; and the assertion that those biases in the adjustments are valid, or are adequately removed via the various codes are just that: Assertions….
Smith then walks the reader through a series of comparisons, both global and regional and comes to the conclusion:
Looking at the GHCN data set as it stands today, I’d hold it “not fit for purpose” even just for forecasting crop planting weather. I certainly would not play “Bet The Economy” on it. I also would not bet my reputation and my career on the infallibility of a handful of Global Warming researchers whose income depends on finding global warming; and on a similar handful of computer programmers who’s code has not been benchmarked nor subjected to a validation suite. If we can do it for a new aspirin, can’t we do it for the U.S. Economy writ large?
The article is somewhat technical but well worth the read and can be found here.
h/t to commenters aashfield, Ian W, and rilfeld
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
(Various UHI comments…)
The IPCC data (2007) indicates that UHI has an influence of ~0.006°C/decade. BEST calculates it at -0.019°C ± 0.019/decade. I’m afraid that anyone who’s looked at the data in depth finds that the range of possible UHI effects are an order of magnitude (or more) smaller than the observed trend.
Also note that (as Zeke pointed out in http://wattsupwiththat.com/2012/06/21/chiefio-smith-exqamines-ghcn-and-finds-it/#comment-1015394) raw data gives the same trends, rural stations (by any selection criteria) show the same trends, satellite data shows the same trends in and out of cities, and sea surface temperatures (SST’s) show those trends too (slightly smaller, just as expected per higher thermal mass, but again no cities there).
UHI is just not an issue. That horse is dead, it’s perhaps time to stop beating it. And, back on topic for this thread, adjustments in the GHCN data (again, see http://wattsupwiththat.com/2012/06/21/chiefio-smith-exqamines-ghcn-and-finds-it/#comment-1015394) aren’t a significant issue either.
Adieu.
KR: “t the trend measurements shown by NASA/GISS, HadCRUT, NOAA, etc., are of temperature anomalies, not absolute temperatures.”
True … but:
1) There were no satellites to measure UHI in the past so we don’t know if 8C of UHI was 8C of UHI in 1920 or 1930. It may have bee 1C in 1900 and 3 in 1930 etc.
2) They aren’t the same thermometers and they aren’t in the same location.
3) GIStemp etc are full of stations with no modern data and no early data. How do we know what UHI was. We will never know. The thermometer is gone or ignored.
If we ever want to know what UHI is, then a set of reference thermometers need to be decided on and the metadata kept up to date and satellites need to measure UHI for decades to see the rate of UHI changes. Math or algorithms won’t cut it.
Zeke, the BEST “Analysis Charts” are useless. Can’t they grid the data and then supply the data for each grid so we could actually see if what they are claiming is true.
A 2d graph is useless.
Something like this would be nice.
http://sunshinehours.wordpress.com/2012/03/18/cooling-weather-stations-by-decade-from-1880-to-2000/
David, UK says:
June 22, 2012 at 11:50 am
mfo says:
June 21, 2012 at 7:19 pm
A very interesting and thorough analysis by EM. When considering the accuracy of instruments I like the well known example used for pocket calculators:
0.000,0002 X 0.000,0002 = ?
[Reply: I hate you for making me go get a calculator to do it. ~dbs, mod.]
[PS: kidding about the hating part☺. Interesting!]
[PPS: It also works with .0000002 X .0000002 = ?
Also works for .0002 x .0002 (or any other number to that many decimal places) simply because the answer has more digits than the average calculator can…er…. calculate. .0002 x .002 just fits!
Can someone explain the joke here? When I was a wee boy my father taught me .0 x .0 = 0
Is it, the more you multiply, the more zeroes you add?
KR says:
June 22, 2012 at 2:25 pm
(Various UHI comments…)
The IPCC data (2007) indicates that UHI has an influence of ~0.006°C/decade. BEST calculates it at -0.019°C ± 0.019/decade. I’m afraid that anyone who’s looked at the data in depth finds that the range of possible UHI effects are an order of magnitude (or more) smaller than the observed trend.
____________________________________________________________________________
0.019°C ± 0.019/decade looks trivial. How about expressing it as 0.19C +/- 0.19 per century?
Suddenly we can see that UHI effects are not ” an order of magnitude (or more) smaller than the observed trend”.
In fact you have just provided a reference for what I’d always suspected. UHI effects account for a substantial portion of the observed trend.
Thank you.
clipe,
Maybe this explains it.
And maybe not.
Gee, take a nap and a whole new load of comments show up… many unrelated to v1 vs v3.
I’ll try to address the most relevant ones, but “what some other data do” isn’t all that useful. Most all of it comes from electronic devices at airports, so is inherently biased by the growth of aviation. Note that airplanes and airports go from “nothing” to “almost all the record” in the last 100 years. From grass fields to acres of tarmac and tons of kerosene burnt per day, surrounded by expanding urban jungles. Doesn’t matter who collects the METARS, that same problem persists.
@Keith. Battye:
All UHI ‘gradually accumulates’. Especially at Airports. During the last 40 years, SJC the local San Jose airport, has gone from a small private plane field with a terminal for commercial use where you could leave your car and in less than 100 yards be walking up the roll around stairs to one where there are massive “international terminals” and you get to drive a couple of mile loop to go between terminals. Huge area now paved for “long term parking”.
IMHO, much of what we measure in the instrumental record is the growth of aviation from 1940 to 2012. Changing which instruments are in a set (like v1 vs v3) shifts how much of that kind of effect is measured vs more pristine places.
@Brent Hargreaves:
Near as I can tell, they’ve tossed it in the bin. I saved a copy some long time ago. I think others have too. I’d love to find a public copy still available.
@Barry:
My “start date” is not 1740. The “start date” is the most recent data item. 1990 in this analysis. The slope of the trend line just reaches your desired number of 0.75 C (that matches my rough eyeball estimate) at that date.
@Quinn the Eskimo:
Looks interesting… now if I can just find the time to look into it 😉
@Lazy Teenager:
Not claiming that the ’70s dip was caused by splice artifacts. It was a known cool time. In fact, that the graphs find it (and things like “1800 and froze to death”) act as sanity checks that the code works reasonably well.
My assertion of suspicion that “splice artifacts” ( in quotes as some of the effects are not formally a splice, but an ‘averaging together in a kind of a splice’) are a probable cause of things like the 1987-1990 “shift” seen at the same time that the Duplicate Number changes in v2 indicating a change of instruments or processes. You need to be less lazy and read more completely, please. BTW, I have no need to provide an “explanation” for your misreadings…
@John Doe:
Nice… very nice… TOBS always has bothered me… Over the course of a month, one would expect it to “average out”, since the “climate scientists” seem to think all sorts of other errors can just be average away… ;-0
@TonyTheGeek:
Blush 🙂
But honestly, isn’t it just that we are both Geeks so we understand Geek Speak? ;-0
@KR:
Anyone who thinks UHI doesn’t introduce a bias has never walked barefoot down a paved road in summer… Or had to stand on the tarmac waiting to board a plane…
@Al in Kansas:
BINGO! Now figure in that there is a giant Polar Vortex with megatons of air spinning down the funnel to the cold pole, and similar megatons of hot air rising over the topics, with huge heat transfer via evaporation and condensation and loads of heat dumped via IR at the TOP of the atmosphere, with variable hight, mass, water content, and velocity; and tell me again how surface temperatures explain anything?…
@Bill Illis:
It’s worse than that. Individual Met Offices can change the data prior to sending it to NCDC. So it might or might not be “raw” as it is originated. Many of the METARS are raw, but they also have giant errors in them (like that 144 C value…) so even what IS raw is full of, er, “issues”…
It’s a Hobson’s choice between “crap data” and “fudged data”…
@sunshinehours1 :
Spot on.
@Pascvaks:
You are most welcome.
@David Falkner:
If it doesn’t matter what data you use, you always get the same results, then the data do not say anything, it is all in the processing…. Just use a single thermometer and be done with it…
The “negative space” issue of “they all are the same” is that they all ought not to be the same if they are truly independent different data sets and different processes. The differences might not be dramatic, but if there are no differences, that is a Red Flag of suspicion that things are not as ‘independent’ as they are claimed..
I see that is echoing Luther Wu… Yeah, “my thoughts exactly” 😉
@KR:
What you don’t seem to realize is that GIStemp keeps temperatures AS TEMPERATURES and not anomalies all the way to the last “grid / box” step. Only then does it create grid / box anomalies by comparing a fictional ‘grid box’ value now to an equally fictional one in the past. (They must be predominantly fictional as the total number of grid / boxes is vastly higher than the present number of thermometers in use. 14/16 ths, roughly, of the boxes have no thermometer in them…)
So the argument that “it is all done with anomalies” is just flat out wrong for GIStemp.
Don’t know about the other programs as I’ve not looked inside their code. Have you?
@EthicallyCivil:
Yeah, I love the way ‘urban areas’ never grow in the Warmers World, yet the global population goes from 1 Billion to 9 billion and a massive shift of people move from the rural landscape into the cities over the last 100 years…
@Luther Wu:
I did an analysis of GHCN v2 some years back that shows the percentage of Airports rising over time and by latitude:
http://chiefio.wordpress.com/2009/12/08/ncdc-ghcn-airports-by-year-by-latitude/
While some places are ‘only’ 50% or so Airports, others are a ‘bit more’:
Not like airports in the USA got larger between 1950 and 2010… Or the Jet Age began… or runways grew from 2000 ft to 10,000 ft for jets… or…
@Zeke:
“without trying to correct for these biases.”
In other words, making up a value that we HOPE is better…
“But Hope is not a strategy. -E.M.Smith”.
And that, in a nutshell, is my core complaint about the data and how they are used. There is, as you pointed out, just so much “instrument change” that the basic data are fundamentally useless for climatology (as you illustrated). That by necessity means we are depending on Hope and Trust in the fiddling of the data to “adjust it” and “fix it’.
I prefer to just recognize that the basic data are not “fit for purpose” and that the “fiddling” is error prone and that “hope” is not a decent foundation for Science.
@rilfeld :
You got it! We have two basic POV.
One is that the variations in the data constitution are so large that it just isn’t very reliable nor usable. Zeke ran into that in his example. I demonstrate it in my test suite / comparison.
The other says “But if we adjust, fiddle, and use just the right process, no matter what code is run you get the same results!” (or the analog “No matter what data you feed the code you get the same results!”)
But if the data used don’t change the outcome, what use is the data? If a specific data set fed to different codes give the same results, what trust can be put in the code?
There ought to be variations based on specific data used and based on specific codes run. Those variations ought to be characterized to measure the error bars on each. THEN the best data and / or best codes can be selected / validated / vetted.
It’s like taking 10 cars to the race track with 10 drivers and they all clock the same lap time. Something isn’t right when that happens…
In a valid test, there ought to be outlier codes and there ought to be measurable effects from changes of the data set shown and calibrated by benchmarks. There are no benchmarks, and we are told the individual data used don’t make any difference. That’s just wrong.
@Quinn the Eskimo:
Very well said! That is the basic issue. The “negative space”… what ought to be there but isn’t.
There is an undeniable UHI and Airport Heat Island effect. Anyone with bare feet can find it.
There is an undeniable gross increase in urbanization around the globe on all continents and a massive increase in population.
Yet “no UHI effect is found”? Then the test is broken… which means either the data or the codes or both are broken.
My “BS O’meter” will not allow me to do otherwise than doubt the assertion that UHI doesn’t matter. In fact, to the extent the “climate scientists” claim to believe it doesn’t matter, it just makes them look ever less credible and ever more gullible or both.
@Barry:
In re-reading your comment:
I think you have the dates used by GISS and Hadley off. GIStemp uses an 1880 cutoff on data, Hadley, IIRC, is 1850. BOTH use data from prior to 1900.
For your comparisons to those sets, you need to use those ranges.
@Smokey:
Yeah, that explains everything 😉
Latimer Alder writes @Pouncer
“At the risk of being terribly pedantic, if the average temperature gets to 500K we are all already in a lot of trouble 🙂 You might wish to rephrase it at about 290K (+17C).”
Kelvin. Rankin. Wunderlich. One of those dead white males. I’m old. I get them confused.
Speaking of Wunderlich, his 19th century measures of human body temperature suffered some of the same problems as we now see with global temperatures. The best he could do — and he was a VERY careful guy with a WHOLE LOT of data — was average his result down to 37 degrees. (Uhm, centigrade. avoiding the dead white male name issue, again)
37 degrees German is 98.6 American. But 98.6 is synthetically precise, not actually accurate. That is, plus or minus 0.1 degree Fahrenheit compared to plus or minus 1/2 a degree centigrade is, uhm, almost ten times too precise.
If Chiefio’s work is comparable and the trend, as adjusted, in the GHCN data is ten times more precise than the accuracy of the underlying figures actually supports; well, it won’t be the first time temperature was badly reported to the public.
BY the way, speaking of adjusted data, did y’all see where they’d adjusted the time of Secretariat’s races in the Triple Crown?
Smokey says:
June 22, 2012 at 3:43 pm
clipe,
Maybe this explains it.
And maybe not.
Like dividing .1 by 3 = 0?
@Pouncer:
Adjusting Secretariat? Say it isn’t so!!
Since the non-random error in the GHCN older data is 1 F range for the US data and who knows what up to 1 C for much of the ROW, IMHO the best precision that can be legitimately claimed is about 1 F. 1/2 C. Even that is just a bald guess. The averaging that gives a more precise value can remove the random error, but not any systematic errors, and we don’t know how much systematic error there might be… so must presume it is the range of the width of the recorded precision.
@Barry:
Just for you, custom graphs with trend lines and formulas aligned on the GIStemp and Hadley HadCRUT start of time period:
http://chiefio.wordpress.com/2012/06/23/ghcn-v1-vs-v3-special-alignments/
(Can you tell I grew up in a service business? Family Restaurant and all… Even making custom cut reports for a commenter 😉
Zeke Hausfather – I checked the BEST paper (Muller, Curry et al) on UHE for method (it used MODIS to identify rural stations), and then checked the Australian stations used (I asked BEST for, and received, the set of stations). Of the ~800 Australian stations, over 100 had “airport” or equivalent in their name, and over 100 had “post office” or equivalent in their name. The inevitable conclusion, which I posted in a comment on WUWT at the time, is that their method of assessing UHE was useless.
Some time ago, I did an analysis of Australian temperatures in an attempt to identify the presence, or absence, of UHE. In that analysis, I took all long-term stations, examined their location using Google Earth to see if they looked truly rural, and then compared the temperature trends of the clearly-rural stations with all the others’. The temperature trends of the clearly-rural stations were substantially lower. All the data and all the Google Earth images were posted with my analysis on WUWT, so that others could check, which some did.
It is very clear to me that UHE has not yet been properly identified and quantified.
[I am currently on holiday – about to go and watch Black Caviar – and a long way from my home computer so sorry, but I can’t supply references for the above and please note that it is all from memory.]
Zeke Hausfather
Again, “similar results” is not good enough. All the hysteria over warming is literally based on 1/10ths of a degree. 1/10th of a degree difference in the output in the various ways the data is handled results in panic costing billions because disaster is imminent. And making 1/10 of a degree in warming is oh so easy to produce. Fudging can easily add 1/10th here and there. Extrapolating over 1000 miles can do it too. Whole degrees, let alone 1/10ths, can be fudged in.
Here’s an example that should make anyone wonder if GISTemp, for one, can be trusted with 1/10ths of a degree—and it’s a very easy example to understand:
Does GISTemp change? part 1
Does GISTemp change? part 2
Preaching to the choir, Rev Smith. 🙂 The earth actualy has runaway water vapor feedback heating the planet. It has had it ever since there were oceans to supply an unlimited amount of greenhouse gas. As I have commented previously, I think Miskolczi got it right. The atmosphere adjusts to a fixed optical depth in the far infared that results in maximum energy transfere to space. Any additional heat input results in more clouds(NEGATIVE feedback) and more weather. Not more extreme, just a few more thunderstorms or a small increase in evaporation and precipitation.
@ur momisugly E.M. Smith
But honestly, isn’t it just that we are both Geeks so we understand Geek Speak? ;-0
I think it is a coding thing, (I have done a little C++ for fun) in that strange effects can come from seemingly innocuous code. Oh, and you have NEVER found all the bugs 🙂
TonyTheGeek
E.M.Smith said @ur momisugly June 22, 2012 at 4:01 am
The earliest mention I know of angels dancing on the head of pins is in Chillingworth’s Religion of Protestants: a Safe Way to Salvation (1638). He accuses unnamed scholastics of debating “Whether a Million of Angels may not fit upon a needle’s point?” H.S. Lang, author of Aristotle’s Physics and its Medieval Varieties (1992), wrote (p 284): “The question of how many angels can dance on the point of a needle, or the head of a pin, is often attributed to ‘late medieval writers’ … In point of fact, the question has never been found in this form”.
WUWT commenters seem to love propagating this sort of story: “In the middle ages it was believed the earth was flat”, “Galileo showed medieval physics was wrong by dropping weights from the Leaning Tower of Pisa”, and so on. They are equivalent in the history of ideas to urban myths like the exploding cat/poodle/hamster [delete whichever is inapplicable] in the microwave.
I think, EM, your most important point is that the notion of “average temperature” is just such a myth.
@Mike Jonas:
See the videos here:
http://wattsupwiththat.com/2012/06/22/comparing-ghcn-v1-and-v3/#comment-1016042
for a nice description of how the Russians have found about 1/2 C of “excess warming” in the data used in GHCN for their country due to Urban Heat Effect…. It’s not just Australia…
Oh, and there’s a paper from the Turkey Met folks as well where they used “all the Turkey” data, not just the ones selected for GHCN, and found Turkey was cooling, not warming.
In a comment from here:
http://chiefio.wordpress.com/2010/03/10/lets-talk-turkey/
From that paper Abstract:
A common rule of thumb is that once is an accident. Twice is a pattern. Three times is an investigation and charge…
@Amino Acids in Meteorites:
That’s one of those loverly little things just just needs more awareness. We have a ‘product’ that wanders around by more than the sought after effect, so process is greater than nature; yet some folks want to believe it. Go figure…
@Al in Kansas:
If you look at the increase in precipitation as the sun has gone quiet, I think you can directly see the increased heat flow off planet as rain (the ‘left overs’ of that evaporate and condense cycle). I agree about the feedback and oceans.
Though I suspect that there is a bi-stable oscillator going on. We’re in the less common hot end stage that only comes for a dozen thosand years every 100,000 or so. The other stable state is the cold end, when ice / albedo and desertification dominate and we’re in an Ice Age Glacial mode. That the only “tipping point” we risk is “down to cold” is what folks ought to be looking at.
@tonyfolleyTonyTheGeek:
Well, I certainly have found all the bugs in MY code. (From here on out, any “odd” behaviour will be classed as a ‘feature’ by The Marketing Department 😉 We have an ever growing “feature” list in our products 😉
@The Pompous Git :
Note that I never attributed “Angels and pins” to an old source. I just used the modern form of the metaphor… (Duck, dodge, weave 😉
BTW, you’ve warmed my heart that at least one other soul has “Got It” that GAT is a farce…
E.M.Smith said @ur momisugly June 23, 2012 at 5:06 pm
Not for an instant did it cross my mind that you believed the canard and I took it that you were using the expression metaphorically. No ducking, dodging and weaving necessary 🙂
The phrase: “The Git got it that GAT is a farce” has a certain charm doncha think? I won’t repeat what I said to myself when the penny dropped cos I don’t want to distress the mods who then have to remind me that this is a family blog 😉
@The Pompous Git:
Tee Hee ;=)