Comparing GHCN V1 and V3

@Paul in Sweden
I’m a volunteer, so not a lot of resources consumed by me 😉
But yes, the amount of global waste over the fantasy of a Global Average Temperature is astounding. That it could instead be used to absolutely fix and solve a large number of significant problems instead is a tragedy. That the “answers” that come from AGW Panic are ‘exactly wrong’ is heart wrenching. (For example, if we did Coal To Liquids or Gas To Liquids we could be free of OPEC oil in 5 to 10 years fairly easily and at lower cost than gasoline here last week. Similar fixes to global “fuel poverty” are just as available. Then the “oil wars” could end… as just ONE example of how the misallocation of resources is a travesty.)
FWIW, the very notion of a Global Average Temperature is based in a fundamental error of Philosophy of Science. It is an intrinsic or intensive property. You simply can not average two temperatures from different things and have any meaning in the result. It is an obscure, but vitally important point; that is consistently ignored by the entire Climate Panic Industry…
http://chiefio.wordpress.com/2011/09/27/gives-us-this-day-our-daily-enthalpy/
http://chiefio.wordpress.com/2011/07/01/intrinsic-extrinsic-intensive-extensive/
Take two pans of water, one at 0 C and the other at 20 C, average temperature is 10 C. Mix them. What is the resultant temperature? You do not know. What are the relative masses of water in the two pans? Is that 0 C frozen or melted? Same problem in climate and temperatures. Average a place with 1 C change from below zero to above zero and 2 feet of snow melt with a desert that has a 1 C change the other way, you get “no change”, yet tons of snow melt heat of fusion moved… It is just wrong to use average temperatures for heat flow problems. But “it’s what they do”…
So I’m occasionally a bit “bummed” that so much time is spent examining “average of temperatures” since it really is an “angels and pins” question; even when I do it. But as that is where the “debate” is mired, it’s the only place to participate. Just somewhere in the back of your mind remember that GAT (or any other average of temperatures) is just meaningless.
@david_in_ct:
I could say “They won’t do that as it will find what I found”, but that would be cheeky… 😉
In defense of what they have done: In forensics you always want to look at things from a different point of view. Watching the magician from the audience front row doesn’t illuminate any “issues” as well as looking from back stage (or in the basement under the stage…)
So it IS important to have a divergent look at things.
My only complaint about this critique, really, is that “part two” is typically to look at things in the same way, then “part three” is to “compare and contrast” followed by “why are the things that are different, different? What is going on?”. That is where learning and wisdom come, and that is where the “perp” is typically caught and the Magician has his trick “found out’. All those are missing here.
No curiosity at all about why a ‘grid / box’ method finds similarity while a FD variable assortment method finds interesting patterns of divergence. Just “He must be wrong because he didn’t do it my way”. It isn’t about which way is “right”, it is about how each method illuminates what can be happening in the climate codes. That this ‘grid / box’ method finds similar data, yet the climate codes smear data all over, and my method that blends data also finds difference; that ought to cause much more curiosity about the nature of the data blending and smearing in the climate codes. Instead it just devolves in a more tribal ‘fling poo’ display. “I’m right, he is in error”. Not very useful, IMHO.
So the “look at it different” is a valid step, but only a first step.
It could be that the data are being carefully manicured in just such a way that the distribution of changes about a mean, ignoring date, gives a surprisingly symmetrical curve. ( I’d expect some kind of small bias in ‘accidental’ non-directed adjustment…) It could be that the ‘grid / box’ averages are so close because the data are being manicured up front to assure that result (but knowing that the actual effect will be different when run through an RSM or Homogenizing blending step). Or it could just be a reasonable and randomly collected set of data that has those properties, but tickles an unexpected ‘bug’ like behaviour when spread around. It could be malice, so a more paranoid forensic mind set ought be applied; or it could be simple error and a less paranoid and more subtle analytical examination ought to be applied. Or, in fairness, it might just be that FD has a different result than the grid / box / baseline method (and that, then turns into an argument over which is more ‘accurate’).
We just don’t know. (And “Trust me” from the data set creation folks is NOT acceptable, not now that Hansen has testified that breaking the law is justified if your goal is “good” enough.)
But even IF it is an artifact of FD vs grid/box/baseline: That, then, just raises the question of how do we benchmark the grid/box/baseline codes in use so as to prove THEY are not out of whack with the warming they find? If 1/2C to 1C of “warming” depends on method, how do you prove the “error bars” are less than 1/10 C when you can not run a benchmark through the codes? Or prove the data unbiased to that precision?
At any rate, taking a different POV is valid, just incomplete.
@Geoff Sherrington
That’s the kind of “devil in the details” stuff that gets ignored in the kind of examination done in this critique. It is the kind of thing that the forensics mindset is dedicated to finding. It is why doing a broad comparison of data sets is an important thing to do, and why past versions ought to all be archived and publicly available (but are not).
@Wayne:
Pixel counting? Talk about dedication 😉 Nicely done, though…
Yes, things always cool the past and warm the present (with the only exception being the occasional warming of the very deep past where the data are thrown away in HadCRUT and GIStemp.)
My question can be recast in the context of your finding as “To what extent does the small induced bias found in a non-spreading grid/box examination become a larger bias when spread via homogenizing, UHI “adjusting” and infilling via the RSM?” The critique here can not answer that question. My examination points out that there’s plenty of opportunity for selected data items to have more impact when spread outside the grid / box. I ask the question “Is that enough to account for the imputed ‘warming’?” Others say “It can not be a problem, trust us.” I’d rather not trust. I’d rather verify…
@Phi:
It is exactly that ‘joining of segments’ as a kind of ‘splice artifact’ that IMHO can be a source of issues. In places like Marble Bar Australia, GIStemp finds a ‘warmer than ever’ trend; yet the record temperature there has never been matched. IMHO that is evidence for a “splice artifacty” behaviour in GHCN/GIStemp interactions. So my approach is to look for the potential for that kind of artifact stimulation in the data. (Segments with a low start point and a high end point). FD, being more sensitive to end states, will react more to those “high ends” and show the rest of the past lower in comparison. It will illustrate the “splice artifact” potential in the data.
The critique here takes deliberate action to suppress that effect (uses a baseline from a semi-random spot in the middle) then simply asserts that the various climate codes can’t have a sensitivity to the splice as they do something similar. I find that inadequate (especially in the face of the known data smearing methods and the specific examples of places like Marble Bar where the artifact effect is clearly present.) So theory runs headlong into existence proof…
No, I’ve not been able (yet) to prove exactly how the artifact effect gets through the codes. (GIStemp is brittle to station change so hard to benchmark) but to simply assert it does not matter because a “toy world” version of the code shows only a small part of it is, well, inadequate.
@all:
I see Steven Moser is more interested in tossing insults than thinking. Spending more time with the Warmers, eh?
BTW, I’ve generally been pretty careful to say I use a variation on a peer reviewed method and to NOT claim that that variation was peer reviewed. Please to do not assert that I lie about that. I’ve said FD is peer reviewed, and it is; and then I point out where I do something different from the classical form. That, too, is true.
Per “Didn’t Get The Memo”:
Steve seems to have not read my statements about wanting the end point sensitivity in the look for splice artifact potential. Oh well. He also seems to have ignore my response in the prior thread where I quoted Jeff Id saying 10,000 segment compares had near zero error but down in the 50 range there was an issue; and where I pointed out that using FD on over 6000 thermometers, by month, gives over 72,000 segments and is well over 10,000. Oh Well.
Steve seems fixated on what finds the perfect Global Average Temperature (ignoring that it is impossible and ignoring that it is meaningless even if found). Not “got the memo” on intrinsic properties I guess. Oh Well.
Here’s a hint Steven: Drop the insult tone. Makes you look vindictive. Contributes nothing. I’d also suggest being careful hanging out with Warmers. It is slowly making your presentation negative and bitter in public.
Personally, I don’t think there will ever be a “right way to go”. Just different methods with different uses and different “issues”, so usable for different things. The world is not a nail, even if you love your hammer…
@Andrew:
I spent a while trying to find “raw” data. It doesn’t exist. There are, as Steve calls it, “First report” data. And it has errors in it.
I noted one where I found 144 C for a station in the USA.
Now Steve is happy to say ~’but we can find that 144 C and change it to something valid’. I think “Hmmm… So we catch the 144 C, but do we catch the 40C that ought to be 38 C?” There is a system that looks for some large number of sigmas off and replaces them with an average of “nearby” ASOS stations (think hot airports, then averaged that will by definition suppress things like large down spikes) but this does nothing to catch whole classes of bad data.
That the electronic instruments seem to regularly “fail high” to insane values, THEN are caught and fixed, implies a probably slow fail to higher than would not be caught, THEN a catastrophic fail high. A periodic calibration is supposed to prevent / catch this, but that would just result in a series of segments with ‘start low end high’ effects that, in smearing / splicing, can induce bias into the result.
So while some folks are happy to just say “Trust me, we fix it good.”, I’m less willing to accept that. The data from the electronic thermometers that “sucked their own exhaust” and heated from the humidity measurement device is still in the record, for example.
@Pouncer:
One clarification:
While I do think all the adjustment and such of individual data items is an issue; I think that the larger issue is the potential for changing which records are in, vs out, of the data set, to have impacts. What I show in the v1 vs v3 is that they DO have impacts, and those can be large.
What the critique here does is say in essence, if we avoid the question of station records in a small area having influence over others we don’t find that problem. Yes, don’t look for it, you won’t find it. Then the assertion is left that the various climate codes are also insensitive to the effect (yet we know that they do data spreading).
IMHO that is the larger issue, not things like tossing out a 144 C and replacing it with an interpolation from the neighbors…
Basically, my major point is just that “The way you look at the data has more effect than the presumed Global Warming”. So is GIStemp right, or wrong? You can not know. It is not possible to benchmark their code and do comparative runs on different data. But you can put comparative sets of data through different systems and get an idea of the general range of sensitivity. That shows a range greater than the Global Warming signal. So ‘choice of method’ can have more ‘warming’ than the GW signal. Choice of data set can have more ‘warming’ than the GW signal. I find that “a problem”. Others like to say “Trust us, we have it right”. This critique says “Here is A way that is less sensitive”. Which is sort of a “so what” kind of moment for me…
@Pamela Gray:
IMHO, the “QA method” is even worse than that. It takes a collection of “nearby” ASOS stations at airports and uses them to replace any data found too far from the expected value. IMHO this will incorporate Airport Heat Islands by default AND via averaging them, prevent any low going extreme values from showing up. (An average always suppresses range).
For homogenizing, the techniques vary, but largely use the same kind of “average a bunch” to get a value. That will always tend to put in mid-values and not what would be there in the real world of actual high or low range values. And what if “the bunch” have “issues”? Like Airport Heat Island? Well, you get that too. (And very large percentages of present GHCN data come from Airports, while by definition none came prior to 1914).
The whole issue of how potentially wrong data is detected and handled is, IMHO, not very good. IIRC the threshold for tossing a “bogus” value was a large fixed temperature in one code ( so low excursions will be tossed more often than high excursions… Inspection of the data shows cold spikes are more extreme than high spikes). In others it is several std deviations. That will miss a whole lot of “somewhat off” errors (like those electronic thermometers that “sucked their own exhaust”).
It’s a modestly large source of error that is largely ignored, IMHO.
(Time for another break… back soon.)