Chiefio Smith examines GHCN and finds it “not fit for purpose”

E.M. Smith over at the blog Musings from the Chiefio earlier this month posted an analysis comparing versions 1 and 3 of the GHCN (Global Historical Climate Network) data set.  WUWT readers may remember a discussion about GHCN version 3 here.   He described why the GHCN data set is important:

 There are folks who will assert that there are several sets of data, each independent and each showing the same thing, warming on the order of 1/2 C to 1 C. The Hadley CRUtemp, NASA GIStemp, and NCDC. Yet each of these is, in reality, a ‘variation on a theme’ in the processing done to the single global data set, the GHCN. If that data has an inherent bias in it, by accident or by design, that bias will be reflected in each of the products that do variations on how to adjust that data for various things like population growth ( UHI or Urban Heat Island effect) or for the frequent loss of data in some areas (or loss of whole masses of thermometer records, sometimes the majority all at once).

 He goes on to discuss the relative lack of methodological analysis and discussion of the data set and the socio-economic consequences of relying on it. He then poses an interesting question:

What if “the story” of Global Warming were in fact, just that? A story? Based on a set of data that are not “fit for purpose” and simply, despite the best efforts possible, can not be “cleaned up enough” to remove shifts of trend and “warming” from data set changes, of a size sufficient to account for all “Global Warming”; yet known not to be caused by Carbon Dioxide, but rather by the way in which the data are gathered and tabulated?…

 …Suppose there were a simple way to view a historical change of the data that is of the same scale as the reputed “Global Warming” but was clearly caused simply by changes of processing of that data.

 Suppose this were demonstrable for the GHCN data on which all of NCDC, GISS with GIStemp, and Hadley CRU with HadCRUT depend? Suppose the nature of the change were such that it is highly likely to escape complete removal in the kinds of processing done by those temperature series processing programs?….

 He then discusses how to examine the question:

…we will look at how the data change between Version 1 and Version 3 by using the same method on both sets of data. As the Version 1 data end in 1990, the Version 3 data will also be truncated at that point in time. In this way we will be looking at the same period of time, for the same GHCN data set. Just two different versions with somewhat different thermometer records being in and out, of each. Basically, these are supposedly the same places and the same history, so any changes are a result of the thermometer selection done on the set and the differences in how the data were processed or adjusted. The expectation would be that they ought to show fairly similar trends of warming or cooling for any given place. To the extent the two sets diverge, it argues for data processing being the factor we are measuring, not real changes in the global climate..The method used is a variation on a Peer Reviewed method called “First Differences”…

 …The code I used to make these audit graphs avoid making splice artifacts in the creation of the “anomaly records” for each thermometer history. Any given thermometer is compared only to itself, so there is little opportunity for a splice artifact in making the anomalies. It then averages those anomalies together for variable sized regions….

 What Is Found

What is found is a degree of “shift” of the input data of roughly the same order of scale as the reputed Global Warming.

 The inevitable conclusion of this is that we are depending on the various climate codes to be nearly 100% perfect in removing this warming shift, of being insensitive to it, for the assertions about global warming to be real.

 Simple changes of composition of the GHCN data set between Version 1 and Version 3 can account for the observed “Global Warming”; and the assertion that those biases in the adjustments are valid, or are adequately removed via the various codes are just that: Assertions….

 Smith then walks the reader through a series of comparisons, both global and regional and comes to the conclusion:

 Looking at the GHCN data set as it stands today, I’d hold it “not fit for purpose” even just for forecasting crop planting weather. I certainly would not play “Bet The Economy” on it. I also would not bet my reputation and my career on the infallibility of a handful of Global Warming researchers whose income depends on finding global warming; and on a similar handful of computer programmers who’s code has not been benchmarked nor subjected to a validation suite. If we can do it for a new aspirin, can’t we do it for the U.S. Economy writ large?  

The article is somewhat technical but well worth the read and can be found here.

 h/t to commenters aashfield, Ian W, and  rilfeld

Advertisements

  Subscribe  
newest oldest most voted
Notify of
barry

There is hardly any difference in the global temperature record between the different GHCN versions and the raw data set. Skeptics (like Jeff Conlon and Roman M at the Air Vent) have come up with global temp records that are actually warmer than those produced by the institutes. But they all rest within margin of error.
Adjustments are more readily noticeable at local scale, but the big picture is not qualitatively affected by adjustments. At a few hundredths of a degree per decade, the fixation on slight differences in global trend between one method or another is overly pedantic.

Barry…if this statement is in dispute…“What is found is a degree of “shift” of the input data of roughly the same order of scale as the reputed Global Warming.” Then please clarify why you think there is “hardly any difference”?

barry says: June 21, 2012 at 3:55 pm
Doesn’t look like you’ve even looked at EMS’ work.
I did, and it’s a tour de force. Thank you EMS. But IMHO it needs re-presentation (a) to grasp in one go (b) with say 20-50 bite-size statements that accumulate the wisdom gained. Say, a rewrite like Bishop Hill did for Steve McIntyre.

Rob R

Barry
This is not the point that chiefio (EM Smith) is addressing.
The analyses you mention (Jeff Condon etc) all take the GHCN version 2 or version 3 thermometer compilation as a “given” and produce global trends from a single version.
The Chiefio is looking primarily at the differences between the different GHCN versions. He is not really focussing on the trend that you can get from a single GHCN version.

Kev-in-Uk

A fair analysis of the REAL state of play, IMHO – but really it’s what we have known for ages, i.e that the manipulated data is essentially worthless!
*drums fingers* – waits for someone like Mosh to come along and remind us that it’s all we have, etc, etc….
I don’t actually mind necessary data manipulations IF they are logged and recorded and explained in sufficient detail that they can be reviewed later. To my knowledge such explanation(s) is/are not available to the general public? What is more worrying is that I doubt the reasoning behind the adjustments are still ‘noted’ anywhere – a la ‘the dog ate my homework’, etc – effectively making any and every subsequent use of the ‘data’ pointless!
We have seen various folk analyse and debunk single stations (or a few at a time) – but does anyone think Jones has been through every stations’ data and ‘checked’ through each and every time series, site change, etc? Somehow I think not – it is likely all computer manipulation at the press of a button and we all know what that means…….(where’s Harry? LOL)

Ian W

barry says:
June 21, 2012 at 3:55 pm
There is hardly any difference in the global temperature record between the different GHCN versions and the raw data set. Skeptics (like Jeff Conlon and Roman M at the Air Vent) have come up with global temp records that are actually warmer than those produced by the institutes. But they all rest within margin of error.

Barry I would really suggest you read Chiefio’s post. Then after that come back here and tell us all the errors that you have found in his approach.

Chris

Way back in the olden days during the brief time I was doing science as a grad student, I was taught that measured results can either be accepted, or with good reason rejected. There was no option to “adjust” numbers. Adjusting was fraud.
Now in the temperature records we no longer have data. We have a bunch of numbers, but the numbers are not data. If adjustments are necessary, they should be presented in painstaking detail, if need be as a separate step. Adjustments are never part of the data: data cannot be adjusted and still remain data. Adjustments are part of the model, not part of the data. They have to be documented and justified as part of the model. Anything else is fraud.

u.k.(us)

Chris says:
June 21, 2012 at 4:50 pm
=========
+1
Though using the term “fraud”, leaves no room to maneuver.
I assume many would like to wriggle out of the trap they have entered.

pouncer

barry says:” the big picture is not qualitatively affected by adjustments.”
Yep, Right. The hairs on the horse’s back may be thicker or thinner but the overall height of the horse isn’t affected.
The thing is that the “simple physics” everybody tells me settles the science of the Earth’s black body radiation budget is based on the average temperature of the Earth in degrees KELVIN. An adjustment of one to two degrees to an average temperature of 20 is already small. Such a variation on an average temperature of about 500 is — well, you tell me.
The effect being measured is smaller than the error inherent in the measuring tool. In order to account for that, very strict statistical protocols must be chosen, documented, tested, applied, and observed for all (not cherry picked samples of) the data.
Note that problems, if any, with the historic instrument record propagate into the pre-historic reconstructions. When calibrating ( “screening” if you like, or “cherry picking”) proxies against “the instrumental record” — does the researcher use the world wide average? The closest local station record? A regional, smoothed, aggregated record? As it turns out the regional extracts of the GHCN record are themselves “proxies” (or as Mosher explains, an “index”) for the actual factor of interest in the “big picture” — the T^4 in degree Kelvin in black body models. If you’re matching your speleo, isotopic or tree-ring record to a debatable instrument proxy — the magical teleconnection screen — why wouldn’t you expect debate about temperatures in the past thousand years?
Chiefio says the record is unfit for use. Barry, what use case do you have in mind for which the record is well fit?

KR

This does not address the fact that the various surface temp readings match the satellite readings (see http://tinyurl.com/6nl33kz), with the caveat that satellite readings are known to be more sensitive to ENSO variations, and that land variations are higher than global variations. Those surface temperatures have been confirmed by the entirely separate satellite series.
Sorry, but the opening post is simply absurd.

EMS has put a lot of work into this analysis and I wish to congratulate him. He has posted a great comment at Jonova’s sit here http://joannenova.com.au/2012/06/nature-and-that-problem-of-defining-homo-sapiens-denier-is-it-english-or-newspeak/#comments at comment 49 worth reading. I wish I had time to put up some more technical information at my pathetic attempt of a blog. I would suggest EMS gets together with Steve (McI) and Ross (McK) to publish some of this.
Keep up the good work. Eventually, truth will out but I am concerned that only legal action will stop the flow of misinformation and power seeking by the alarmists and climate (pseudo)scientists.

pat

Seth is all over the MSM with his simple facts!
20 June: Canadian Press: AP: Seth Borenstein: Since Earth summit 20 years ago, world temperatures, carbon pollution rise as disasters mount
http://ca.news.yahoo.com/since-earth-summit-20-years-ago-world-temperatures-155231432.html

Ian W

KR says:
June 21, 2012 at 5:23 pm
This does not address the fact that the various surface temp readings match the satellite readings (see http://tinyurl.com/6nl33kz), with the caveat that satellite readings are known to be more sensitive to ENSO variations, and that land variations are higher than global variations. Those surface temperatures have been confirmed by the entirely separate satellite series.
Sorry, but the opening post is simply absurd.

Note that EMS was just assessing the differences between GHCN V1 and GHCN V3 from 1880 to 1990. So I am somewhat confused by your comment – can you describe what “satellite surface temp readings” were produced from 1880 – 1990? and how they were assimilated into GHCN V1 and GHCN V3? The issue is the accuracy of anomalies and warming rates over a century based on the GHCN records. These show differing ‘adjustments’ to temperature readings back over a century ago between versions. How do satellite temp readings affect this?

u.k.(us)

Chiefio Smith examines GHCN and finds it “not fit for purpose”
===============
When did we switch from charting the weather’s vagaries, to predicting it ?
How will changing previous weather data, enhance our understanding ?
A battle surely lost.

wayne

Thanks E.M., good article.
How does this affect the GHCN or is it a different matter?
http://www.ncdc.noaa.gov/img/climate/research/ushcn/ts.ushcn_anom25_diffs_urb-raw_pg.gif
All I see is the very rise in temperature that has been created by the adjustments themselves! (but a small ~0.3C difference)

Michael R

This does not address the fact that the various surface temp readings match the satellite readings (see http://tinyurl.com/6nl33kz), with the caveat that satellite readings are known to be more sensitive to ENSO variations, and that land variations are higher than global variations. Those surface temperatures have been confirmed by the entirely separate satellite series.
Sorry, but the opening post is simply absurd.

As Ian pointed out above, I am curious to know how surface temperatures and satelite correlation has anything to do with pre-satelite temperature data? The only way that argument is valid is if we could hindcast temperatures using those satelites – which I would love to know how that is done…
Having good correlation between recent ground thermometres and satelites means squat about previous temperature readings. I also find it curious that almost the entire warming that is supposed to have occured in the last 150 years occured BEFORE the satelite era and if that data shows it has been adjusted several times to create higher and higher warming trends, the effect is lots of warming pre satelite area and a sudden flattening of temp rise post satelite era…. but hang on…. that’s exactly how the data looks and your link shows it.
Unfortunately your link does not prove your argument, but it does support E.M. Smith’s. Maybe try a different tact..

Mark T

Grammar nazi comment: who’s is a contraction for who is; the possessive is whose.
Mark

mfo

A very interesting and thorough analysis by EM. When considering the accuracy of instruments I like the well known example used for pocket calculators:
0.000,0002 X 0.000,0002 = ?
[Reply: I hate you for making me go get a calculator to do it. ~dbs, mod.]
[PS: kidding about the hating part☺. Interesting!]
[PPS: It also works with .0000002 X .0000002 = ?

thelastdemocrat

Puh leezze EDIT POSTS!
“Data” is plural!!! “Datum” is singular!!!
“If that data has an inherent bias in it, by accident or by design, that bias will be reflected in each of the products that do variations on how to adjust that data for various things like population growth ( UHI or Urban Heat Island effect) or for the frequent loss of data in some areas (or loss of whole masses of thermometer records, sometimes the majority all at once).”
This datum, these data.
[REPLY: The memo may not have reached you yet, but language evolves. Ursus horribilis, horribilis est. Ursus horribilis est sum. -REP]
[Reply #2: Ever hear of Sisyphus? He would know what it’s like to try and correct spelling & grammar in WUWT posts. ~dbs, mod.]

@Lucy Skywalker:
It’s open and public. The code is published, so folks can decide if there is any hidden ‘bug’ in it as they feel like it. The whole set, including links to the source code, can be easily reached through a short link:
http://chiefio.wordpress.com/v1vsv3/
Anyone who wants to use it as a ‘stepping off point’ for a rewrite as a more approachable non-technical “AGW data have issues” posting is welcome to take a look and leverage off of it.
@Barry:
The individual data items do not need to all be changed for the effect to be a shift of the trend. In particular, “Splice Artifacts”. They are particularly sensitive to ‘end effects’ where the first or last data item (the starting and ending shape of the curve) have excessive effect.
In looking through the GIStemp code (yes, I’ve read it. Ported it to Linux and have if running in my office. Critiqued quite a bit of it.) I found code that claims to fix all sorts of such artifacts, but in looking at what the code does, IMHO, it tries but fails. So if you have a 1 C “splice artifact” effect built into the data, and code like GISTemp is only 50% effective at removal, you are left with a 1/2 C ‘residual’ that isn’t actually “Global Warming” but an artifact of the Splice Artifacts in the data and imperfect removal from codes like GIStemp and HadCRUT / CRUTEMP.
The code I used is particularly designed not to try to remove those splice artifacts. The intent is to “characterized the data”. To find how much “shift” and “artifact” is IN the data, so as to know how much programs like GIStemp must remove, then compare that to what they do. (My attempts to benchmark GIStemp have run into the problem that it is incredibly brittle to station changes, so I still have to figure out a way to feed it test data without having it flat out crash. But in the testing that I have done, it looks like it can remove about 1/2 of the bias in a data set, but perhaps less.)
So depending on how the particular “examination” of the data set is done, you may find that it “doesn’t change much” or that it has about 1 C of “warming” between the data set versions showing up as splice artifacts and subtle changes of key data items, largely at ends of segments.
In particular, the v1 vs v3 data often show surprising increases in the temperature reported in the deep past, while the ‘belly of the curve’ is cooled. If you just look at “average changes” you would find some go up, and some go down, net about nothing. Just that the ones that go up are in the deep past that is thrown away by programs like GIStemp (that tosses anything before 1880) or Hadley (that tosses before 1850). Only if you look at the shape of the changes will you find that the period of time used as the ‘baseline’ in those codes is cooled and the warming data items are thrown away in the deep past; increasing the ‘warming trend’ found.
Furthermore, as pointed out by Rob R, the point of the exercise was to dump the Version One GHCN and the Version Three GHCN data aligned on exactly the same years, through exactly the same code, and see “what changes”. The individual trends and the particular “purity” of the code don’t really matter. What is of interest is how much what is supposedly the SAME data for the SAME period of time is in fact quite different from version to version.
That the changes are about the same as the Global Warming found from V2 and V3 data, and that the data do produce different trends is what is of interest. It shows the particular data used does matter.
@Kev-in-UK:
It is important to realize that the difference highlighted in this comparison is NOT item by item thermometer by thermometer changes of particular data items. ( i.e. June 18 1960 at Alice Springs) but rather the effect of changes of what total data is “in” vs “out” of the combined data set. Yes, some of what it will find will be influenced by “day by day adjustments”, but the bulk of what is observed is due to wholesale changes of what thermometers are in, vs out, over time.
I know, harvesting nits… but it’s what computer guys do 😉
@Ian W:
Good suggestion…
Also note that the v1 data are no longer on line as near as I can tell. I had saved a copy way back when (being prone to archiving data. Old habit.) It may be hard for other folks to find a copy to do a comparison. I’ve sent a copy to at least one other person ‘for safe keeping’ but it is a bit large to post.
Frankly, most folks seem to have forgotten about v1 once v2 was released, and paid little attention to both now that v3 is out.
@Chis:
What I was taught as well. You got an F in my high school Chemistry class if you had an erasure in your lab book. Any change was ONLY allowed via a line-out and note next to it. Then the new data written below.
FWIW, the “adjustments” are now built into GHCN v3. In v2 each station record had a ‘duplicate number’. Those are now “spiced” and QA changes made upstream of the V3 set. This is particularly important as one of the first things I found was that the major shift that is called warming happened with the change of “duplicate number” at about 1987-1990 (depending on station). It is no longer possible to inspect that ‘join’, as it is hidden in the pre-assembly process at NCDC. (But I’ve saved a copy of v2 as well, so I can do it in the future as desired 😉
@Pouncer:
Well said. 😉
@KR:
Reaching for “The Usual Answers” eh?
Did you notice that this is a comparison of v1 and v3 aligned on 1990 when v1 ends? Did you even read the article where that is pointed out? Look at any ONE of the graphs that all start in 1990?
So it has all of about 12 years of overlap with the satellites. Just not relevant. Recent dozen+ years temperatures have shown no warming anyway, so that they match the sats is just fine with me…
@Cementafriend:
Thanks for the support. FWIW, I’ve got a version where I cleaned up a typo or two and got the bolding right posted here:
http://chiefio.wordpress.com/2012/06/21/response-to-paul-bain/
with a couple of ‘lumpy’ sentences cleaned up a bit. It’s a bit of a ‘rant’, but I stand by it.
@UK (US):
It’s not predicting weather that bothers me. Folks like Anthony can do that rather well and it doesn’t need the GHCN. It’s the notion of predicting “climate change” and the notion that we can influence at all the climate that is just broken.
When I learned about “climates”, we were taught that they were determined by: Latitude, distance from water, land form (think mountain ranges between you and water), and altitude. So a desert climate is often found behind a mountain range where rain is squeezed out (often as snow) on the mountains (making an Alpine climate).
In the world of Geology I learned, unless you change one of those factors, you are talking weather, not climate…
CO2 does not change latitude, distance from water, land form, nor altitude. So the Mediterranean still has a Mediterranean Climate and the Sahara Desert still has a Desert Climate and The Rockey Mountains still have an Alpine Climate and the Arctic is still an Arctic Tundra Climate. Then again, I’m old fashioned. I like both my science definitions and my historical data to remain fixed…

Mike Jowsey

KR @ 5:23 says: Those surface temperatures have been confirmed by the entirely separate satellite series.
Were those satellite measuring devices ever calibrated to surface temperature? If so, then the satellite series is not “entirely separate”, but in fact joined at the hip. If not, then how does the proxy of an electronic device in orbit translate to accurate surface land temperature?

barry
Thanks for being the first to comment. It was clear in the past that some global warming believers, or more simply put, those that disagree with this web site, seemed to sit in front of their computers waiting for a new post so they could be the first to comment.

Gail Combs

Chris says:
June 21, 2012 at 4:50 pm
Way back in the olden days during the brief time I was doing science as a grad student, I was taught that measured results can either be accepted, or with good reason rejected…..
_________________________________
BINGO!
That is one of the reasons for doubting the entire con-game in the first place. How the Heck does Jones or Hansen KNOW that all the guys who took the readings in 1910 did it wrong and all the results need to be adjusted DOWN by a couple of hundredths or the guys in 1984 screwed up and all the data needs to be raised by 0.4. A few hundreths in 1910??? The data was never measured that precisely in the first place. http://cdiac.ornl.gov/epubs/ndp/ushcn/ts.ushcn_anom25_diffs_urb-raw_pg.gif
Even more suspicious is the need to CONSTANTLY change the readings. http://jonova.s3.amazonaws.com/graphs/giss/hansen-giss-1940-1980.gif
The data has been so messaged, manipulated and mangled that no honest scientist in his right mind would trust it now especially since Jones says the “dog ate my homework” and New Zealand’s ‘leading’ climate research unit NIWA says the “goat ate my homework”

…In December, NZCSC issued a formal request for the schedule of adjustments under the Official Information Act 1982, specifically seeking copies of “the original worksheets and/or computer records used for the calculations”. On 29 January, NIWA responded that they no longer held any internal records, and merely referred to the scientific literature.
“The only inference that can be drawn from this is that NIWA has casually altered its temperature series from time to time, without ever taking the trouble to maintain a continuous record. The result is that the official temperature record has been adjusted on unknown dates for unknown reasons, so that its probative value is little above that of guesswork. In such a case, the only appropriate action would be reversion to the raw data record, perhaps accompanied by a statement of any known issues,” said Terry Dunleavy, secretary of NZCSC.
“NIWA’s website carries the raw data collected from representative temperature stations, which disclose no measurable change in average temperature over a period of 150 years. But elsewhere on the same website, NIWA displays a graph of the same 150-year period showing a sharp warming trend. The difference between these two official records is a series of undisclosed NIWA-created ‘adjustments’…. http://briefingroom.typepad.com/the_briefing_room/2010/02/breaking-news-niwa-reveals-nz-original-climate-data-missing.html

And if you did not follow ChiefIO’s various links. This one graph of GHCN data set. Version 3 minus Version 1 says it all:comment image
Nothing like lowering the past data by about a half degree and raising the current data by a couple tenths to get that 0.6 degree per century change in temperature….
What is interesting is the latter half of the 1700’s had times that were warmer that to day and overall the temperature was more variable.

KR

E.M.Smith“Reaching for “The Usual Answers” eh?”
You have written a great deal, but included little evidence.
I would point you to http://forums.utsandiego.com/showpost.php?p=4657024&postcount=86 where a reconstruction from _raw_, unadjusted data, from the most rural stations possible (readily available, mind you) has been done. Results? Area weighted temperatures estimated from the most rural 50 GHCN stations closely match the NASA/GISS results. As confirmed by the separately run, separately calibrated satellite data
You have asserted quite a lot – without proving it. You have cast aspersions aplenty – with no evidence. And when uncorrected data is run, the same results as the NASAQ/GISS adjusted data comes out, with only a small reduction in uncertainties and variances.
I don’t often state things so strongly, but your post is b******t. If you don’t agree with the results, show your own reconstruction, show us that the adjustments are incorrect. Failing that, your claims of malfeasance are as worthy as the (zero) meaningful evidence you have presented.
[Reply: Please feel free to submit your own article for posting here. ~dbs, mod.]

Luther Wu

“What is found is a degree of “shift” of the input data of roughly the same order of scale as the reputed Global Warming.”
That’s the “money shot”.

Bennett

@ cementafriend at 5:32 pm
Wow. Thanks so much for the link to E.M. Smith’s comment essay. I’ll be printing and sending that to all of my State reps to DC. It really can’t be laid out any better than that.
http://chiefio.wordpress.com/2012/06/21/response-to-paul-bain/

KR

Clarification – in my previous post (http://wattsupwiththat.com/2012/06/21/chiefio-smith-exqamines-ghcn-and-finds-it/#comment-1015094) the text regarding variances should have been:
“…only a small reduction in uncertainties and variances in the NASA/GISS data due to corrections and much more data.”

Great job of explaining the issues with the data!

@E.M.Smith June 21, 2012 at 7:33 pm
(My attempts to benchmark GIStemp have run into the problem that it is incredibly brittle to station changes, so I still have to figure out a way to feed it test data without having it flat out crash. But in the testing that I have done, it looks like it can remove about 1/2 of the bias in a data set, but perhaps less.)

If I am understanding you correctly the issue is that it is looking for specific station names/ IDs?
If so could you feed it test code by inserting test data into a data file with all the current station names and other information it is depending on. Create a pink noise data set with the individual entries assigned to the station names it is looking for.
Larry

KenB

Thanks Chiefio
I wonder how long this farce of trying for social change by corrupting science will go on. It seems inevitable there will be a massive law suit. With the poor state of climate science, its myths and beliefs that keep changing to protect the guilty/careless,stupid.or just incompetent, how will that fare when put to the test of providing sworn testimony AND data, adjustments to raw data in a civil court of law.
The Harry read me file should have sparked alarm bells among true scientists and immediately lead to a clean up. As it stands now, the harm and potential damages that have occurred since that event, have made it odds on, that the only remedy is, either recant the meme now or prepare to be sued and explain “why the deception continued”? The continuance after exposure of a problem/issue is surely , where the punitive damage and assessment of costs will rest in this donnybrook!

barry

Rob,

The analyses you mention (Jeff Condon etc) all take the GHCN version 2 or version 3 thermometer compilation as a “given” and produce global trends from a single version.
The Chiefio is looking primarily at the differences between the different GHCN versions. He is not really focussing on the trend that you can get from a single GHCN version.

The Air Vent used the raw data (check the code in the post I linked), not any of the adjusted versions. The point is, the results are hardly different from the adjusted results. And this has been discovered over and over again at Lucia’s place (Zeke Hausfather) and by other blog efforts taking raw and adjusted data and comparing. At the global level there are differences in trend between the institutional and blog efforts, but they are minor.
I can’t speak to the acuracy of EM Smiths’ methods, but I note that he points out the African trend has actually decreased in V3 and the same with Europe – this is a shout out to those commentators who think that every adjustment is upwards. Histograms of adjustments show a fairly even split – but you wouldn’t know it because critics tend to focus on the upwards adjustments and usually omit references to downward ones.
I’d be interested to know what EM Smith discerns as the global trend difference between V1 and V3 from 1900. By eyeball it doesn’t appear to be much. Smith appears to have worked out a trend comparison from the 1700s (!) – it would not be a big surprise if the major part of the difference Smith is seeing is due to adjustments made to the early record, which, being sparse, is more prone to bigger adjustments than later in the record when there is much more data averaging out. No one should rely on early data – and indeed trend estimates given by the institutes are usually from about 1900, when the data are firmer. When the data is less sparse, the records match much better. For various reasons, as I mentioned above, adjustments are more obvious at regional and especially local resolution, but gobally these adjustments tend to cancel out, which is why so many trend analyses using raw and adjusted global data from 1900 are so close.

barry

EM Smith,
thanks for the reply. What is the difference between V1 and V3 for the global temperature trend from 1900? And in your initial comparison in the post, what was the start year when you discerned a 0.75C difference in trend?

KR

~dbs, mod – I have pointed to a temperature estimate from rural stations and raw data, showing that the NASA/GISS data is simply a refinement of the results from that subset of _raw_ data.
Given that E.M.Smith has not presented an alternative temperature estimate, let alone one that stands up to scrutiny, I feel that I have fully documented the case against this opening post.

@TheLastDemocrat:
It’s a jargon thing. In Data Processing the use of data is as a ‘mass noun’. The data. These data. That data (item). Datum is never heard (unless from a ‘newbie’ who doesn’t do it long).
On maps, you find a Datum…
So you have a choice: Become a Defender Of Proper Latin! And would that be Law Latin? Vulgar Latin? Latin of the Renaissance? Latin of the Early Roman Empire? Or late? They are all different, you know… there is even a government web site in the UK that details the differences between UK Government Latin from the late Empire usage vs early formal Latin. It is an interesting read, BTW. http://www.nationalarchives.gov.uk/latin/beginners/

The tutorial covers Latin as used in England between 1086 and 1733, when it was the official language used in documents. Please note that this type of Latin can be quite different from classical Latin.

So do be sure to identify exactly which Latin in which we are to be schooled…
Or accept that “things change”.
Ita semper fuit … De rerum transmutatione magis stetisse
As I’m a “computer geek” in Silicon Valley, I’m using “Silicon Valley Standard Geek Speak” in which the plural of data is data and the singular of data is data and the specific indicative is “data item”. YMMV. ( Though I tend to the English usage of ‘should’ and ‘ought’ not the American, as Mum was a Brit. So “I ought to eat less” instead of “I should eat less” and “If I should fall” instead of “If I were to fall” (or worse, the Southernism “If I was to fall”… ) so that can be disconcerting to some of my American neighbors… But I often shift dialect with context, so put me talking to a Texan and it will be “If’n I was ta’ fall”… so don’t expect strict adherence to any one dialect, time period, or ergot.)
@Mark T.:
Unfortunately, I spell more or less phonetically (with a phonemic overlay). I had 6 years of Spanish in grammar school and high school, French to the point of French Literatures IN French at University, and along the way took a class in German and one in Russian (in which I did not do so well – funny characters ;-). Since then I’ve learned a bit, on my own, of Italian (enough to get around Rome), Gaelic (at which I’m still dismal, but trying), some Swedish (enough to read office memos when working at a Swedish company – with enough time and the dictionary) and smatterings of Norwegian. I’ve also looked at, but not learned much of, Greek (where I can puzzle out some words and simple phrases), Icelandic, Portuguese (which I can read as it is close to Spanish but the spoken form is just enough off that I can’t lock on to it) and a few others. (Not to mention a dozen or two computer programming languages).
The upshot of it all is that my use of Standard Grammar can be pretty easily shot as the various systems get a bit blurred… But I try. So I’ll see about changing it to what you like, so it won’t bother you. But I do generally agree with Mark Twain on the whole thing, and if you look back just a couple of hundred years even English had widely variant usage and spelling. It can not be neatly put in a bottle and frozen (though grammarians try…)
http://www.twainquotes.com/Spelling.html
But at least it’s not German…
http://www.cs.utah.edu/~gback/awfgrmlg.html
@Mfo:
In the USA, data were read off the Liquid In Glass thermometers and recorded in Whole Degree F on the reports. These data are compared to current records from ASOS stations at airports that report electronically.
All it takes is a simple tendency for folks to read 67.(whatever) as something to report as 67 for there to be a 1/2 F degree jump in ‘bias’ at the point about 1987 where the change of Duplicate Number happens.
IMHO, the shift from LIG to automated stations accounts for the “jump” at that point and for some of the volatility changes. Proving that belief will take some work.
@Mike Jowsey:
As I understand it, the sats are calibrated to a standard reference inside the enclosure. They read an absolute temperature reference for calibration.
It isn’t particularly relevant to my posting though.
It also doesn’t really help much in comparing sats results to land averages as there is the fundamental fact that temperature is an intrinsic property of A material, and an average of intrinsic properties is meaningless. (That is a point most folks ignore. I occasionally raise it as it is a critical point, but nobody really cares… The best example I’ve found is this: Take two pots of water, one at 0 C the other at 20 C. Mix them. What is the final temperature? How does that relate to the average of the temperatures? The answer is “You can not know” and “not at all”. Missing are the relative masses of water in the two pots and the phase information on the 0 C water. Was it frozen or melted? You simply can not average temperatures and preserve meaning; yet all of “climate science” is based on that process…)
http://chiefio.wordpress.com/2011/07/01/intrinsic-extrinsic-intensive-extensive/
So the sats are calibrated to a standard then look at a field that covers some large area and integrates the incident energy and creates a value we call a temperature. It isn’t, but that’s what we call it. Then the land temperatures are averaged eight ways from Sunday (starting with averaging the min and max each day to get the mean) and we call THAT a temperature, but it isn’t. That the two values end up approximating each other is interesting, but doesn’t mean much. It mostly just shows that both series move more or less in sync with the incident energy level. ( i.e. they very indirectly measure solar output modulated by albedo changes.)
But that’s all a bit geeky and largely ignored other than by a few folks even more uptight about their physics philosophy than a grammar Nazi is about commas quotes, and apostrophes…

Paul Linsay

KR @ 8:03 pm
You missed the point of the article. Differences in instrumental artefacts between v1 and v3 are the source of the rise in global average temperature. It’s the artefacts that produce the rise you plot, which E.M. Smith would get too if he bothered to do the calculation, as does the BEST collaboration. Look around for the 1990 GISS temperature plot versus their 2000 version. You’ll see the shift where the 1930’s go from being the same temperature as 1990 to being cooler. It’s available on this web site somewhere.

Grammar Nazi 2 reporting for duty…..
A poster wrote (in good faith) “Maybe try a different tact..”. Sorry, it’s “tack”. It’s a sailing term. Here beside the St. John River we are familiar with such terminology; the writer might come from a drier region where sailboats are not so common.
IanM

Trevor

@thelastdemocrat:
First of all, you have a lot of fracking nerve criticizing others’ grammar when you open your post with “Puh leezze”, end TWO sentences with multiple exclamation points, and end your post with an incomplete sentence.
Second, “data” is NOT plural. It is a MASS NOUN, and as such, is treated as singular, like “information”. “Data” is, in fact, DEFINED as “information”, and like “information”, though it may be comprised of many individual pieces, when referred to as a whole, it is singular.
So shut the frack up!
I bet you also think it’s wrong to end a sentence with a preposition, don’t you? God, I can’t stand grammar snobs. Especially when they’re WRONG.
[Well, PASS THE POPCORN!! ~dbs, who is sitting back and immensely enjoying all sides of this fracking conversation!☺]

KR

E.M.Smith – Let me put this in clear terms. I would challenge you to show, using uncorrected, well distributed, rural raw data, a temperature estimate that significantly varies from the GHCN estimates. One that shows a consistent bias from other estimates. If you do, I promise to take a look at it, and give an honest response as to my perspective, which will include agreement if it is well supported.
Failing that, I would consider your essay an attempt to raise unwarranted doubts, and treat it just as noise.
Seriously, folks. It’s one thing to argue about attribution (human or natural), or feedback levels (clouds, aerosols, albedo). It’s another thing to attempt to dismiss the masses of data we’ve accumulated over time regarding instrumental temperature records. The temperature is what it is, we’re ~0.7C over pre-industrial levels. We are where we are – claiming that the temperature record(s) are distorted is as unsupportable as the “Slayers” or Gerlich and Tscheuschner claims of 2nd law issues.

barry
I forgot the “sarc” tag on my comment.
What you consider pedantic some other consider checking things for themselves. Being like Al Gore who says everything he says he heard from his science advisers, one was James Hansen, isn’t good enough, and shouldn’t be good enough. Would to God everyone was your kind of pedantic. “Manmade global warming” scares would be completely over!

KR
you say: “closely match the NASA/GISS results”
Closely match is not good enough. Global warming hysteria is based on 1/10ths of a degree. The difference between the earth heading to global warming calamity and nothing out of the ordinary is so little that they “closely match”.

KR
Here’s Richard Lindzen pointing out how global warming alarmists make their case for histrionics on things amounting to ant hills:

Oh well.
Chiefo hasnt kept up with skeptical science. He relies on First differences. Yes, First differences is peer reviewed, but smart skeptics dont trust peer review, we do our own review.
So, lets look at the review that Jeff Id did of first differences
http://noconsensus.wordpress.com/2010/08/21/comparison-and-critique-of-anomaly-combination-methods/
“This post is a comparison of the first difference method with other anomaly methods and an explanation of why first difference methods don’t work well for temperature series as well as comparison to other methods”
basically, Chiefo is using an known inferior method, a proven inferior method. That worse than anything Mann ever did, because it was skeptics that pointed out that First differences was inferior.
The other thing he doesnt realize that an entire global reconstruction can be done without using a single station from GHCN Monthly. Guess what, the answer is the same.

KR
“a temperature estimate that significantly varies from the GHCN”
You are going to have to define what you think significant is. What are you parameters?

Leonard Lane

We hear the statement “chimes against humanity” all the time in a political context, and I am not belitteling real crimes against humanity in any way. They have occured in the past and are occurring now.
But when taxpayers have funded climatic data collection, storage, and publication for decades because the data are essential to agriculture, infrastructure design and protection, etc., what do you call it when crooks and dishonest scientists destroy these vital public records for their own selfish and dishonest reasons?And when the so called adjustments to the historical data are deliberately made to corrupt the data and cannot be reversed to restore the historical data, what do you call it? I think fraud is too weak a term for corrupting these invaluable data.

Mosher,
Bad mouthing the “process” and comparing this to “Mann’s work” is not refutation. In science, we show our work. All I am asking is to show your work…. Here is a testable hypothesis with data and code completely free for you to show the process is bad. Do your own analysis and show that the work is wrong and please do not resort to name-calling and other obfuscation and drawing up of straw men. Use logic my friend. We don’t need to argue on political attack dog mode do we?
In other words, next time try to prove the result wrong directly. We are talking about the differences between GHCN V1 and V3 by the way. The theory is that there should be very little difference between the results and that any varience will have no slope and will not change the actual results in any way. I seem to see that it does in this case with this work. But I will always eat my hat like I say all the time if proven wrong and will appologize.
This is not to say that we have not warmed at all or that this work proved that all warming was the results of instrument problems, you have to read more into the work to come up with what it is saying.
This proves that IT IS POSSIBLE that the entire “postulated” warming caused by humans is all the result of instrumentation and measuring issues along with methodology therein. Heck, its also “just possible” that humans are responsible for the same amount of warming due to “Emitting CO2”.
Doesn’t that make either theory equally likely until one or the other is proven wrong?

jorgekafkazar

The trolls are here in force, a sure sign that they fear the post. They resolutely go round and round the same wrong interpretation of it, hoping to divert readers from the actual message to a straw man of their own devising.

@Wayne:
GIStemp takes in USHCN and GHCN and then does a bewildering set of manipulations on them. The USHCN is just US data, so theoretically only would have effect in the USA. HOWEVER, GIStemp is a ‘Serial Averager” and a “Serial Homogenizer”, so any given thermometer record may have a missing data item “filled in” via extrapolation from another thermometer up to 1200 km away. That record is then spliced onto other records and “UHI correction” applied (that can go in either direction, often the wrong one) based on, yup, records up to 1200 km away that might themselves have data from 1200 km away as in-fill.
Now comes the fun part…. Only AFTER all that averaging of temperatures, it creates “grid box anomalies”. The last version I looked at had 16,000 grid /boxes filled in with ‘temperatures’. The One Small Problem is that the GHCN last I looked had 1280 current thermometers providing data items… and the USHCN only covers 2% of the Earth’s surface. Soo…. by definition, about 14,000 of those “grid /boxes” contain a complete fabrication. A fictional temperature. That value can be “made up” by, yes, you guessed it, looking up to 1200 km away and via the Reference Station Method filling in a value where none exists (in fact, where about 14/16 of them do not exist) which reference may itself be adjusted via stations up to 1200 km away based on infilled data from 1200 km away… so any one data item may have “reach” of up to 3600 km… (Though most of them do not, but you don’t know how many or how far…)
At any rate, as v1 changes to v2 changes to v3, the input to that process changes (the exact thermometers that change which other thermometers and exactly what goes into each grid box). In this way, any “splice artifacts” and any “instrument changes” can directly propagate into the “grid box anomalies” calculated in the last step of GIStemp. The major difference between what I do (using a reverse First Differences) and what it does, is that it uses an average of a middling bit of the data do do the ‘splice’ so will be a little less sensitive to end effects from the final data item. (But it has a bunch of other bad habits that make that a small comfort… for more, see:
http://chiefio.wordpress.com/gistemp/ but it’s a couple of years of work, much of it technical, so can be a bit ‘thick’ at times…)
@Michael R: Tee Hee 😉
@Amino Acids in Meteorites :
You noticed that too, eh? I’ve sometimes privately speculated that some various college professors might be giving students assignments to sit and comment… Who else would have the time and patience to so regularly be “first out the gate”?
Not particularly a pejorative statement, BTW. I’d often give my students an assignment to “Go shop for a computer” for example. I wanted them to report their interactions in the real world with what they had learned of the jargon. So assigning students to “hang out” somewhere and be present is a common assignment.
It would also explain why the “who’s on first” (note: Proper usage of who’s for the grammar Nazis) tends to change just about every semester / school year rotation.
Or it could just be folks who just have to be “in their face” with “the opposition”. Who knows…
@Gail Combs:
As you said: “Bingo!”
Though I do have to stress that part of the greater volatility in the past is an artifact of reducing instrument numbers. Eventually in the 1700s you are down to single digits (and at the very end a single instrument). It is just a mathematical truth that a single instrument will range far more widely than an average of instruments; and that the more instruments and the more widely they are scattered, the less the average will vary. (So a half dozen on the East Coast will move together with a Tropical Heat Wave; but average in the West Coast where we’re in the cooler and that dampens).
So anything before about 1700 is very subject to individual instrument bias. In fact, the perfectly reasonable reason that GIStemp cuts off the data in 1880 and Hadley in 1850 is that before that time there are just not enough instruments to say anything about the Global Average (though regional averages and country averages and individual instrument trends are still valid).
But more interesting to me is the other end of the graph. As we approach the present and numbers of thermometers are more stable OR they drop out of the record, we get even LESS volatility? That is counter to the math… That is the end where something fishy is showing up. Stable or dropping numbers ought to have stable or increasing volatility. Yet it goes the other way. A direct finger print of data artifacts from instrument and processing changes.
OK, with that caveat about old temperatures said: The method I used of comparing each thermometer ONLY to itself, ought to give as good a standardizing of ‘trend over time’ as it is possible to get from an average. Better would be to look at those old records and just plot them as an individual instrument. When you do that, results are in conformance with what I found. In particular, when Linnaeus was doing his work in Sweden (and some folks scoff at the warm climate plants he said he grew) it really WAS warm in Sweden. The record from Upspala from ONE instrument, well tended and vetted, shows that 1720 was rather warm just like now.
(Hope my tendency to state the limits and caveats, then also say “but the result matches” isn’t too disconcerting…. I just think it’s important to say “Watch out for this issue” and yet also say “But this is what is demonstrable”… )
@Mfo and Moderators:
Are you REALLY going to make me find a calculator somewhere and do it? Or can you post the result? Or does it work in Excel too? Sheesh… Put an anomaly in front of an obsessive compulsive and then walk away… with friends like that… 😉
@KR:
No “evidence” eh? The data are published. I’ve published the code that does the anomaly creation. Other than data, process, code, and results, what else would you like?
No, I’m not going to go chasing after Your topic and links. Maybe in a few days. Right now I’m busy… Just realize that in forensics there may be 100 ways the evidence is NOT seen, but what matters is the one that shows the stain… or, in short “Absence of evidence is not evidence of absence.” So about that absence of evidence for issues you posted…
BTW, did you know that Einstein was asserted to be WRONG on General Relativity by one of the early tests of light bending around the sun. They said “we didn’t find anything so he is wrong” in essence. Later that was shown to be in error. They just didn’t have the right tools to see the effect. Asserting from the absence of evidence is a very dicey thing to do.
So I don’t really care if a dozen folks look at things one way or another and see nothing. I care about the one way that shows something. Rather like watching a Magic Show. It is fine that 99%+ of the people will say “I saw him saw the girl in half”; but the interesting view is from the side or stage back where you can see how the trick is done.
BTW, what I have done is to “show my own reconstruction”. I’ve done it including publishing the code, describing the method, and publishing the graphs of what happens when the GHCN data in various “versions” are put through the same process. Take a look at the graphs. Many of them show warming. In particular, North America and Europe show warming trends on the order of 1 C to 3 C (depending on version of data used).
Are you unhappy that I find warming of 1 C to 3 C?
Or is it that the same code finds little to no warming in the whole Southern Hemisphere and finds Africa cooling? That is shows that “instrument selection” in the various versions causes as much shift of results as the supposed “global warming”? That it shows wildly divergent “warming” depending on where in the world you look, and when? All things that are not compatible with the CO2 warming hypothesis…
So it’s easy to toss the “BS Bomb”, but I’m the one who has all cards on the table. In that context, a random “rant” doesn’t look like much.
Look, I use the official GHCN data. I don’t change it. It’s not “my version”. I use the First Differences technique that is an accepted method (with only minor changes – order of time from latest to earliest and not doing a reset on missing data but just waiting for the next valid data item to span dropouts). Not “my” method, really. So using other folks methods and other folks data I look at what the data says. Then publish it all. If that’s “BS” then most of science is “BS” by extension.
@Larry Ledwick (hotrod ):
My first attempts at benchmarking were to remove some station data and see what happened. The code crashed. There’s a place where an errata file is used to update some specific stations and I think that is where it crashes – if one of those stations is missing.
In theory, I think, one could keep all the station entries in place, but re-write them with specific replacement data items and get a benchmark to complete. But that is a heck of a lot of data re-writing.
I was in the middle of thinking through how best to do that when two things happened.
1) I realized that GHCN itself “had issues” and did the GHCN investigation series instead as a more important task.
2) A bit later GIStemp changed to support v3, so I was looking at benchmarking code that was no longer in use. I’d need to do a new port and all, or benchmark obsoleted code as the walnut shell moves to a new pea…
It’s on a ‘someday try that’ list, but frankly, GIStemp was making such out of alignment results from the other codes as to not be very reputable anyway.
I THINK that a benchmark could be run by replacing data in each year with specific values that show a gradual increase, then running it. Then do the same thing with “dropouts” and with the ‘end data’ sculpted so that segments each ‘start low and end high’ but the actual overall trend is the same. Then compare results. But it is a lot of work and there is only one of me. So I put myself on higher value tasks. ( Little things, like making some money to buy food… Despite all the assertions to the contrary, being a Skeptic does not pay well and there isn’t a lot of funding from the NSF… )
@Luther Wu:
It’s what the data show when run through a very direct First Differences process.
@Bennet:
Sometimes I have my moments… Just needs a bit of pins under the skin 😉
@KenB:
My speculation would simply be that they have “Top Cover”. The whitewash in the Climategate “Investigations” pretty much serves as evidence of the same. The boldness with which clear evidence of wrong is simply brushed away speaks of folks who have been assured “We can protect you”… So someone with “Privilege” can assert it as protection. It is what seems to fit, though there is little to prove it. (At least in the present batch of Climategate and FOIA 2011 emails… but who knows what they are holding in the encrypted files…)
One of the interesting bits in forensics is what I call “The Negative Space” issue. What ought to be, but is not. The “Negative Space” of the investigations (the complete lack of what ought to be happening) IMHO is evidence of protection from much higher up the food chain. At least, were I leading an investigation, that’s where I’d look.
@Barry:
Two very important points:
1) I am NOT looking at “adjustments” per se in individual records. I’m looking at the aggregate impact of the totality of the data changes. What changes between v1 and v3. That will include some changes of the data and it will include many changes of the data set composition as to which thermometers are in vs out of the data set. The point is not that “adjustments” are the issue, only that ‘data set composition’ has as much impact as the imputed “global warming” and as much as or more than any “adjustments”.
2) The method I use is specifically designed such that the “trend” is found from the present and does not depend on the oldest data. That is why I start my First Differences in the present (where we supposedly have the best coverage and the best data) and then work backwards in time. That accumulates any “strangeness” in the early records only into the very early parts of the graphs. I go ahead and graph / show it, since in some places (like Europe and North America) there is decent data prior to 1850. In essence, any trend in the graph is most valid in the left most 3/4, but is likely decent in the left most 4/5 and only gets ‘suspect’ in the far right at the oldest thermometers with the fewest instruments. Even then, since each instrument is only compared to itself, and even then only within the same month / year over year, the accumulated anomaly will get wider error bars, but ought to remain representative.
So, for example, the oldest record in Europe ought to come from Uppsala IIRC. That record will have “January 2012” compared to “January 2011”. Then January 2011 compared to 2010. etc. back to the start of time in the 1720 to 17teen area. Similarly for February, March, etc. If a given month has a value missing, it is just skipped. So if January 2011 is missing, 2012 gets compared to 2010 and the difference entered at that point. In this way missing data is just ignored and any trend will continue to be recorded. (The method is very robust to data dropouts). At the end of it all, the monthly anomaly series are averaged together to get annual anomaly series and any collection of instruments can get their anomalies averaged together. At least in theory, the accumulated anomaly ought to be valid even over long ranges of time, and as thermometers drop out of the record, the remaining anomaly in any given year ought to have wider range (as any average of fewer things can have more volatility) but ought to still be valid as to absolute value.
Hopefully that isn’t too complicated an explanation of why the ‘trend’ is valid even as the length of time can be long as as instruments leave the pool. If plotting a ‘trend line’ I’d likely cut off the right most 10% of the graph just to assure that there isn’t too much ‘end effect’ on calculating the fit, but for ‘eyeball’ purposes, the graphs are fine. Just be aware that big swings at the far right are when you are down to single digit or even just one thermometer and they can move more than an aggregate.
Per the trend difference from 1900 in aggregate: Frankly, I don’t find it as interesting as just how the tend varies from region to region and how much some parts of the globe have little to no trend. All of that speaks to data artifacts, not a “well distributed gas” as causal.
But, looking at a large version of the graph, it looks like about 1/4 C to 1/2 C of variation between the two data sets depending on exactly which points in time you choose to inspect.comment image
As the “global warming” we are supposed to be worried about is about 1/2 C, I’d say “that matters”…

KR – you refer to a study using ‘rural’ stations. One of the problems with such studies is they tend to accept the ‘rural’ classification of a station as meaning that it is truly rural – ie, that it will not be affected by urban development. Unfortunately, that is often incorrect (bear in mind that UHE tends to come from urban growth not urban size, ie, small towns and airports can have UHE too). For example, in the BEST study, I did a quick chec on their list of ‘rural’ stations for Australia, and it contained over 100 airports and over 100 post offices.
In the study that you cited, I picked one Australian station at random from the map. It was Onslow, in NW WA, and the station appears to be at the airport. Unfortunately I am on holiday and unable to put in more time, but I think it would be a good idea to check that the stations in your cited study really are rural in the non-UHE sense.

barry

EM,
my eyeball sees little trend in the difference (V3 minus V1) from 1900, but we know how eyeballs are. Rather than guestimate, would it be too much trouble to quantify? The period 1900 to 1990 (incl) is what I’m interested in, because then I can check that against the official trend estimates with some confidence. GISS, NCDC and UK Met Office don’t, AFAIK, offer trend estimates from the middle of the 19th century.
(I understood the rest of your reply, thank you)

@Barry:
Look at the graph in the comment just posted and eyeball it for your self. It looks to me like about 1/2 C of offset with v3 warmer in about 1984-1987.
As for 0.75C, as I’ve never said anything about it, I don’t have any better eyeball of the graph than anyone else. At about 1829 the two are 1 C apart with v3 colder, so about 1.5 C of ‘offset increase” compared with v1 from 1984 or so…, but there is variation from year to year, so how you find a ‘trend’ to get 0.75 C between values will be influenced by exactly what start and end dates you pick.
Then again, that’s sort of the whole point I’ve been making about “end effects” and “splice artifacts”…
@KR:
“Feel” whatever you like. I have no need to dance to whatever tune you care to call. Everything I’ve done is public and posted. You’ve tossed dirt and a link. Others can judge.
I’d only comment that if you think GISS and GIStemp is a ‘refinement’ then you haven’t looked at how the code works. I have. I ported it and have it running (and fixed a bug in how it did F to C conversion that warmed about 1/10 C through the data…)
Painful details here: http://chiefio.wordpress.com/gistemp/
Basically, if you think GIStemp is an improvement of the data, watch out for folks selling bridges in NYC…
Per your “challenge”: So just where to I get the global data to do that calculation? Hmmm?
What I have done is exactly to show trends in the GHCN data, even breaking it out by region (and I can do similar trends down to country and even any other subset of the WMO numbers including individual station data). The whole point of the posting is to show that a reasonable method, based on a peer reviewed and supposedly valid method of First Differences, gets very different trends out of two different version of what is the same time period of data from the same physical locations. The differences being 100% due to changes IN THE DATA SET. You, then, want to get all huffy about my not somehow magically making a different trend out of some other data set. Well, I don’t have that other data set. I have the GHCN (in three versions).
Perhaps you can do me the favor of posting a link to some other global historical temperature data set, not based on the GHCN? I’ll be glad to make trends and graphs from it, I’ll be waiting for your link. Remember: Global coverage. From at least 1750 to date. NOT GHCN based or derived. Publicly available. At least 1200 thermometers world wide in the recent decades.
I do find it funny that you think using the GHCN, publishing the code used, and graphing and showing trends from it is all just “noise”. Guess that makes GISS, Hadley, and NOAA/NCDC just noise makers too… 😉
BTW, I’m not “dismissing the data” we’ve accumulated. I’m illustrating a problem in the way it is assembled and presented for use. The way that it is lacking coverage and has dramatic changes in instruments in it, and how those changes show through accepted methods of making “trends” as what I think are “splice artifacts” and related issues. That the particular assemblages of it have more variation from version to version than the signal being sought.
As to your assertion that the world is 0.7 C warmer: Nope. If you bother to look at the graphs presented, you will find that Europe and North America (and a little bit of South America) have much stronger warming. I would assert it is largely a result of putting most of our present thermometers near the tarmac and concrete at airports. Oceana / Pacific Basin isn’t warming much at all, and Africa doesn’t warm either. I’d also suggest you read the link about averaging intensive properties…
@Mosher:
That First Differences is “inferior” for finding warming doesn’t mean it is inferior for doing comparisons of two versions of the same data. In some ways as a forensic tool it is more valuable.
It is more impacted by ‘end effects’, particularly at the end where you start the comparison. Since part of what I’m trying to find is how much bias is in the data, having a method a bit more sensitive to the common causes of bias (like where each trend starts and ends) is a feature.
In essence, your criticism amounts to: “He didn’t sit in the audience and went behind the stage to look at the magic show”.
I freely grant that if you do everything exactly the same way as the Magician you get the same show. So what?
My purpose is not to create some fictional “Global Average Temperature” (which really is a silly thing to do – again, see that ‘intrinsic properties’ issue) so having a ‘better way’ to calculate something that is fundamentally meaningless is really rather pointless.
My purpose is to ‘compare and contrast’ two versions of GHCN and see how they are different from each other. To find out how much cruft shows up and how much one version changes compared to the next. For that purpose, a very clear and very simple and reliably repeatable system is far more important than one that does some very complicated shifting about and hard to explain manipulations of the data. Furthermore, as the purpose is to find how much of such things as “splice artifacts” are changed, having a tool that shows them is rather important. In that context, the Jeff Id critique of First Differences essentially says that I’ve picked a tool that works well for that purpose. (In that FD is sensitive to the splice point).

The average of all the steps over 10,000 series is about zero, but even over 50 series the average trend is dominated by this random microclimate noise.

Notice those values well. There are about 6000 thermometers at peak in GHCN. As I do FD on each month separately, that’s about 72000 FD segments. 7.2 x the point where Jeff finds the error to be “about zero”. Even at the 50 range, that’s about 5 thermometers. I’ve already pointed out that in the single digit range of instruments the graphs will be a bit off. As that’s pretty much prior to 1700 in places like North America and Europe and prior to 1800 in most of the rest of the world, I’m OK with that.
One of the first rules of forensics is to look at things from a different POV and with different lighting than used in “the show”…
And, pray tell, what publicly available non-GHCN data set might I download and test? Eh?
Link please…
And since GIStemp, Hadley, and NCDC are all just variations on GHCN post processing, they are not reasonable alternatives… Same limits as given to the other person saying to use other data: Public available, a few thousand thermometers, not GHCN based, etc. etc.

barry

EM,

As for 0.75C, as I’ve never said anything about it, I don’t have any better eyeball of the graph than anyone else

Then I may have misunderstood this comment from your post

Overall, about 0.75 C of “Warming Trend” is in the v3 data that was not in the v1 data.

I thought that you had crunched the numbers for that. I’m still interested in the actual, rather than the guessed trend difference from 1900 to 1990 (because it looks to be about zero to me), but if it’s too much trouble then no worries. Thanks for the replies.