Darwin Zero Before and After

Guest Post by Willis Eschenbach

Recapping the story begun at WUWT here and continued at WUWT here, data from the temperature station Darwin Zero in northern Australia was found to be radically adjusted and showing huge warming (red line, adjusted temperature) compared to the unadjusted data (blue line). The unadjusted data showed that Darwin Zero was actually cooling over the period of the record. Here is the adjustment to Darwin Zero:

Figure 1. The GHCN adjustments to the Darwin Zero temperature record.

Many people have written in with questions about my analysis. I thank everyone for their interest. I’m answering them as fast as I can. I cannot answer them all, so I am trying to pick the relevant ones. This post is to answer a few.

• First, there has been some confusion about the data. I am using solely GHCN numbers and methods. They will not match the GISS or the CRU or the HadCRUT numbers.

• Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.

• Next, a number of people have claimed that the reason for the Darwin adjustment was that it is simply the result of the standard homogenization done by GHCN based on comparison with other neighboring station records. This homogenization procedure is described here (PDF).

While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way. Instead they have adjusted them differently (see below). This argues strongly that they were not done by the listed GHCN homogenization process. Any process that changed one of them would change all of them in the same way, as they are nearly identical.

• Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert. You can’t adjust Darwin based on that. The GHCN homogenization method requires five well correlated neighboring “reference stations” to work.

From the reference cited above:

“In creating each year’s first difference reference series, we used the five most highly correlated neighboring stations that had enough data to accurately model the candidate station.”

and  “Also, not all stations could be adjusted. Remote stations for which we could not produce an adequate reference series (the correlation between first-difference station time series and its reference time series must be 0.80 or greater) were not adjusted.”

As I mentioned in my original article, the hard part is not to find five neighboring stations, particularly if you consider a station 1,500 km away as “neighboring”. The hard part is to find similar stations within that distance. We need those stations whose first difference has an 0.80 correlation with the Darwin station first difference.

(A “first difference” is a list of the changes from year to year of the data. For example, if the data is “31, 32, 33, 35, 34”, the first differences are “1, 1, 2, -1”. It is often useful to examine first differences rather than the actual data. See Peterson (PDF) for a discussion of the use of the “first-difference method” in climate science.)

Accordingly, I’ve been looking at the candidate stations. For the 1920 adjustment we need stations starting in 1915 or earlier. Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:

WYNDHAM_(WYNDHAM_PORT) = -0.14

DERBY = -0.10

BURKETOWN = -0.40

CAMOOWEAL = -0.21

NORMANTON = 0.35

DONORS_HILL = 0.35

MT_ISA_AIRPORT = -0.20

ALICE_SPRINGS = 0.06

COEN_(POST_OFFICE) = -0.01

CROYDON = -0.23

CLONCURRY = -0.2

MUSGRAVE_STATION = -0.43

FAIRVIEW = -0.29

As you can see, not one of them is even remotely like Darwin. None of them are adequate for inclusion in a “first-difference reference time series” according to the GHCN. The Economist excoriated me for not including Wyndham in the “neighboring stations” (I had overlooked it in the list). However, the problem is that even if we include Wyndham, Derby, and every other station out to 1,500 km, we still don’t have a single station with a high enough correlation to use the GHCN method for the 1920 adjustment.

Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.

• Next, the GHCN does use a good method to detect inhomogeneities. Here’s their description of their method.

To look for such a change point, a simple linear regression was fitted to the part of the difference series before the year being tested and another after the year being tested. This test is repeated for all years of the time series (with a minimum of 5 yr in each section), and the year with the lowest residual sum of the squares was considered the year with a potential discontinuity.

This is a valid method, so I applied it to the Darwin data itself. Here’s that result:

Figure 2. Possible inhomogeneities in the Darwin Zero record, as indicated by the GHCN algorithm.

As you can see by the upper thin red line, the method indicates a possible discontinuity centered at 1939. However, once that discontinuity is removed, the rest of the record does not indicate any discontinuity (thick red line). By contrast, the GHCN adjusted data (see Fig. 1 above) do not find any discontinuity in 1941. Instead, they claim that there are discontinuities around 1920, 1930, 1950, 1960, and 1980 … doubtful.

• Finally, the main recurring question is, why do I think the adjustments were made manually rather than by the procedure described by the GHCN? There are a number of totally independent lines of evidence that all lead to my conclusion:

1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).

2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).

3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).

4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).

5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. Here are those pictures:

Figure 3. Darwin station data before and after GHCN adjustments. Upper panel shows unadjusted Darwin data, lower panel shows the same data after adjustments.

Before the adjustments we had the station Darwin Zero (blue line line with diamonds), along with four other nearby temperature records from Darwin. They all agreed with each other quite closely. Hardly a whisper of dissent among them, only small differences.

While GHCN were making the adjustment, two stations (Unadj 3 and 4, green and purple) vanished. I don’t know why. GHCN says they don’t use records under 20 years in length, which applies to Darwin 4, but Darwin 3 is twenty years in length. In any case, after removing those two series, the remaining three temperature records were then adjusted into submission.

In the “after” picture, Darwin Zero looks like it was adjusted with Sildenafil. Darwin 2 gets bent down almost to match Darwin Zero. Strangely, Darwin 1 is mostly untouched. It loses the low 1967 temperature, which seems odd, and the central section is moved up a little.

Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value? Seems to me you’d be better off picking any one of the originals, because they all agree with each other.

The reason you adjust is because records don’t agree, not to make them disagree. And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.

So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims. I’m happy to be proven wrong, and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.

Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.


Sponsored IT training links:

Want to pass HP0-J33 at first try? Gets certified 000-377 study material including 199-01 dumps to pass real exam on time.


0 0 votes
Article Rating

Discover more from Watts Up With That?

Subscribe to get the latest posts sent to your email.

303 Comments
Inline Feedbacks
View all comments
temp
December 23, 2009 8:41 am

Willis Eschenbach (22:29:16), you do realize it is Dec. 23rd?
Wait until the 3rd week in Jan., and then send a simple e-mail saying you can’t figure out how the corrected the data with respect to whatever station, and any input/information they can give you beyond the very general details in published reports would be greatly appreciated. Information related specifically to whatever information would be appreciated.

temp
December 23, 2009 8:42 am

“Information related specifically to whatever information would be appreciated.”
should read:
Information related specifically to whatever station would be appreciated.

December 23, 2009 9:41 am

temp (07:28:55)
Thank you for taking the time to look at my NOAA GHCN analysis on digginintheclay. You seem to have not quite got the message I’ve tried to convey.
NOAA has made adjustments to at least 400 WMO stations that are just not physically justified. Have a look at Edson and at Mayo Airport again. Why should such pretty much reasonable, non-problematic raw data be adjusted into oblivion? I’ve analysed several other WMO stations on both lists now and they all get the staircase ‘step function’ treatment. It doesnt matter a jot to me whether or not in the end these all cancel out. According to Giorgio Gilestris anlys they dont. The fact is they are just not physically justifiable and thats what matters to me.
Even if they do by chance cancel out on a global basis, they won’t necessarily cancel out on a regional basis as Willis’s original Darwin thread demonstrates. It ony takes a small number of these in a given region e.g. Northern Australia, New Zealand, Siberia etc for these non-physically justifiable adjustments to make a large difference to claimed warming trends in certain regions of the world.
My next analysis is going to look into this very point. Given Edson, Alberta and Mayo, Yukon it looks like Canada might be a good place to start.
KevinUK

Willis Eschenbach
December 23, 2009 12:01 pm

temp (08:41:11) :

Willis Eschenbach (22:29:16), you do realize it is Dec. 23rd?
Wait until the 3rd week in Jan., and then send a simple e-mail saying you can’t figure out how the corrected the data with respect to whatever station, and any input/information they can give you beyond the very general details in published reports would be greatly appreciated. Information related specifically to whatever information would be appreciated.

What? It’s December 23? My gosh, how could I have missed that? Now the GHCN folks won’t be in the office for a week or so, and my email will get all stale and faded from sitting in their inbox for a week, and they’ll refuse to answer it because it’s so old …
This is great. You object to me not writing GHCN. Now I write, and noooo, that’s not good enough for you. You bitch that I haven’t done it at the right time, or included the magic words, or said “mother may I”, or something.
Look, temp, if you don’t think my timing or my exact phraseology will achieve the desired result, I strongly suggest that you write GHCN yourself.
Here’s what I’d do if I were you, temp. I’d wait until the 3rd week in Jan., and then send a simple e-mail saying you can’t figure out how they corrected the data with respect to whatever station, and any input/information they can give you beyond the very general details in published reports would be greatly appreciated.
I advise that method because I understand that the “be vague” approach works much better than telling them exactly what information you are requesting.
Make sure you tell them, however, that “Information related specifically to whatever information would be appreciated”, because a friend of mine says you’ll get better results that way.
Bear in mind that my email will still be in their system at that time, albeit somewhat stale and faded, and the race is on … we’ll see which one (if either) of us gets any joy from GHCN first.
I love all the good folks who are too tired or bored or busy or uninterested to get up off of their dead keesters and actually do something, but are all too happy to sit on the sidelines and give me the benefit of their vast experience by telling me that I’m doing it oh so wrong …

Dave F
December 23, 2009 12:10 pm

temp (07:07:13) :
On the surface, that sounds like a good idea, but how is the satellite data converted to actual temperature? Isn’t it calibrated against the surface record?

Richard
December 23, 2009 12:10 pm

Willis Eschenbach – a question for you sir. Questions – Have the Met Office commented on your analysis? Have they released their workings and reasons? Have they said anything at all?

Richard
December 23, 2009 12:27 pm

Ignore that – didnt read the posts. Lets see if GHCN reply and if so what they say

Nick Stokes
December 23, 2009 1:14 pm

KevinUK (02:46:24) :
Nick Stokes (20:34:00) :

“KevinUK (13:23:59) :
In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.
And how many instances of turning warming into cooling”
Nick why don’t you visit my thread on ”diggingintheclay‘ and you’ll find out?
That’s exactly what’s wrong with this “analysis”. A process subject to apparently random fluctuation, which can be beaten up into a conspiracy with selection of results.“194 instances of WMO stations where “cooling” has been turned into “warming””! And if you want to find that 216 went the other way, as you’d expect with random movement, has to be discovered from the website.
Willis did the same with his “smoking gun”. “a regular stepped pyramid climbing to heaven” . This echoes far and wide. The observation that it is an extreme case from a large sample, and pyramids are almost as likely to go to hell, comes much later, but which time Willis is saying that all he was complaining about was some apparent difficulty with squaring with the Peterson algorithm in Darwin. 1920.

Willis Eschenbach
December 23, 2009 3:05 pm

Nick Stokes (13:14:00) :


Willis did the same with his “smoking gun”. “a regular stepped pyramid climbing to heaven” . This echoes far and wide. The observation that it is an extreme case from a large sample, and pyramids are almost as likely to go to hell, comes much later, but which time Willis is saying that all he was complaining about was some apparent difficulty with squaring with the Peterson algorithm in Darwin. 1920.

Say what? I reported what I found. What did you expect me to do? I thought then, and I think now, that the Darwin adjustment was not done by the claimed method. I think that for a number of reasons listed above, which I guess you didn’t read, including the one you mentioned – because I can’t find any stations that are suitable for the adjustment by the claimed method. That’s not “some apparent difficulty”, it is a complete inability to replicate their results.
Nor has anyone else been able to find such stations, including yourself. When you do, or when you can explain how the Darwin adjustment was done using their claimed method, you will have something other than your fantasies to be complaining about.
I don’t care if the pyramids go up or down. I do care if they are bogus. I do care if the listed procedures weren’t used, no matter which way the pyramids go.
And “almost as likely” is meaningless when the claimed adjustments add up to a significant part of the claimed global warming. Nor does “almost as likely” mean a damn thing when the results are used to make regional claims.
I got into this from investigating Professor Karlen’s claims about the region used by the IPCC to represent “Northern Australia”. The unadjusted trend for this large region is 0.06C per century, in other words, no trend at all. The adjusted trend for this region is 0.57C per century, which coincidentally agrees with the global trend … convenient, huh?
In other words, the entire trend for this area comes from the GHCN adjustment. Perhaps you find that immaterial, since the adjustments are “almost as likely” to go down as up.
I don’t.
Finally, you excuse it as “an extreme case from a large sample”. However, unless you are claiming that Darwin Zero really, truly did warm at 6C per century, or that it was adjusted to a local trend of 6C per century, I fear you are left with showing just why it is so extreme. The fact that it is so extreme is not an explanation, it is something that needs explanation.

Nick Stokes
December 23, 2009 4:03 pm

Willis Eschenbach (15:05:03) :
Let me remind you again of your “smoking gun”:
They’ve just added a huge artificial totally imaginary trend to the last half of the raw data! Now it looks like the IPCC diagram in Figure 1, all right … but a six degree per century trend? And in the shape of a regular stepped pyramid climbing to heaven? What’s up with that?
Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.

Doesn’t sound like a complaint about how to apply an algorithm. It sounds like you’re saying that someone’s deliberately creating a stairway to heaven. And I’m sure that’s what, say, Megan McArdle was conveying when she ran your plot under the heading “Was data faked?”.
If you apply any break recognition adjustment algorithm to 6000+ instances, you’ll get some false positives and other odd effects. And if you look hard enough, you’ll find some where their effect accumulates. So, I don’t know if Darwin really warmed at 0.6 C/decade over that late 20C. Nor whether Coonabarabran really cooled at such a great rate. I think it is more likely that the algorithm overreacted to something in the data. That doesn’t make it a useless algorithm – just one to be interpreted with care, at some suitable level of aggregation. Which is pretty much what Peterson’s paper says.
[REPLY – Peterson, Parker peeked a poke of packed parking lots. ~ Evan]

Richard
December 23, 2009 4:21 pm

Nick Stokes (13:14:00) :
“KevinUK (13:23:59) : In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.”
And how many instances of turning warming into cooling?
That’s exactly what’s wrong with this “analysis”. A process subject to apparently random fluctuation, which can be beaten up into a conspiracy with selection of results.“194 instances of WMO stations where “cooling” has been turned into “warming””! And if you want to find that 216 went the other way, as you’d expect with random movement, has to be discovered from the website.

Nick Stokes you are not being honest. Maybe this is not deliberate. Maybe you are not being honest with yourself.
Just like justice not only should be done, but it must seem to have been done, so also it is with science. It demands openness and transparency.
That there are 194 instances of WMO stations where “cooling” has been turned into “warming”, is a finding of fact. That this is purely random in nature and thus will cancell out is an assumption.
When a huge amount of stations are dropped over 20 years, coinciding with increase in trends, this is a finding of fact. That this has nothing to do with the increase in trends is conjecture.
That Global Warming has proceeded in step with the march of thermometers southwards, is an observation, (fact). That because the temperatures are anomalies, and thus an anomaly at 20N for example will be identical to an anomaly at 60N is an assumption. At best it is sloppy science.
Whether the assumptions are true or not, we just dont know. Even before, because data, methods and programs were refused to be divulged, there was suspicion that things might not be ok.
After the climate-gate emails, we just cant take the scientists at their word. And that is not the way science is done anyway.
When people like Willis Eschenbach and others take an enormous amount of effort to look into the matter, their efforts should be applauded, not attacked in the way you have. If discrepancies are found, they should either be explained or corrected.
Are you not interested in checking, auditing, quality control and the truth being out?
Because if you are not I will be far less polite with you. I have no patience with crooks. I would like to deal with them in good old kiwi fashion where I can. And you seem to be siding with them for no good reason.

Willis Eschenbach
December 23, 2009 4:27 pm

Nick Stokes (16:03:01) :

Willis Eschenbach (15:05:03) :
Let me remind you again of your “smoking gun”:
They’ve just added a huge artificial totally imaginary trend to the last half of the raw data! Now it looks like the IPCC diagram in Figure 1, all right … but a six degree per century trend? And in the shape of a regular stepped pyramid climbing to heaven? What’s up with that?
Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.
Doesn’t sound like a complaint about how to apply an algorithm. It sounds like you’re saying that someone’s deliberately creating a stairway to heaven. And I’m sure that’s what, say, Megan McArdle was conveying when she ran your plot under the heading “Was data faked?”.
If you apply any break recognition adjustment algorithm to 6000+ instances, you’ll get some false positives and other odd effects. And if you look hard enough, you’ll find some where their effect accumulates. So, I don’t know if Darwin really warmed at 0.6 C/decade over that late 20C. Nor whether Coonabarabran really cooled at such a great rate. I think it is more likely that the algorithm overreacted to something in the data. That doesn’t make it a useless algorithm – just one to be interpreted with care, at some suitable level of aggregation. Which is pretty much what Peterson’s paper says.
[REPLY – Peterson, Parker peeked a poke of packed parking lots. ~ Evan]

Are you just pretending not to get it, or do you really not get it? I don’t think the algorithm was used on Darwin. I might be wrong, but that’s what I think. So no, it was not a complaint about “how to apply an algorithm”, that’s your fantasy. It was a claim that the cited algorithm wasn’t applied.
You say “I don’t know if Darwin really warmed at 0.6C/decade over that late 20th century”. Hey, I don’t know if Darwin really cooled by 3.7C per day, or if there are really fairies in the back garden … but since we have absolutely no evidence for either of those temperature claims, and we do have evidence that neither temperature claim is true, you’re wandering around in “what if” rather than sticking to the science.
You say the algorithm “overreacted” … we’re paying rooms full of scientists to design an algorithm, on the basis of which we are told we must spend billions of dollars to avoid imaginary catastrophe, and your claim is that it “overreacted”? So would you agree that the claim we should spend billions based on this algorithm might possibly be an “overreaction” as well?
However, I note that you have given up your claim that none of this matters because the algorithm is “almost as likely” to overreact on the plus side as on the minus side, which I guess is some progress …
Finally, whether the data was faked by someone manipulating the Darwin data individually or by someone manipulating all of the data en bloc, the Darwin results are indeed fake – Darwin did not do what the GHCN says it did.

Richard
December 23, 2009 4:46 pm

“the UK Met Office (which works closely with the CRU and relies heavily on its product) announced a three-year project to re-examine 160 years of temperature data, signalling its own lack of confidence in its CRU-based temperature record.”

wobble
December 23, 2009 6:30 pm

Nick Stokes (16:03:01) :
“”So, I don’t know if Darwin really warmed at 0.6 C/decade over that late 20C. Nor whether Coonabarabran really cooled at such a great rate. I think it is more likely that the algorithm overreacted to something in the data.””
Nick Stokes, Willis’ efforts have convinced me that the algorithm wasn’t used for Darwin.
If you really believe that the algorithm was used at Coonabarabran, then we have a problem. We have a problem because that would imply that the cooling adjustment at Coonabarabran was appropriate while many of us believe that the warming adjustment at Darwin is not.
If you believe that the algorithm wasN’T used at Coonabarabran, then we still have a problem. We have a problem because that would imply that Darwin isn’t the only case of the algorithm isn’t being applied – this really raises serious questions about possible false claims regarding non-US temperature adjustments.

Nick Stokes
December 23, 2009 7:55 pm

wobble (18:30:48) :
As I said, I believe that all the non-US data was computed with the same algorithm. All Willis has persuaded me of is that there may have been some change to the algorithm since 1997. And yes, if so there is a documentation fault.
Willis Eschenbach (16:27:12) :

You say the algorithm “overreacted” … we’re paying rooms full of scientists to design an algorithm, on the basis of which we are told we must spend billions of dollars to avoid imaginary catastrophe, and your claim is that it “overreacted”? So would you agree that the claim we should spend billions based on this algorithm might possibly be an “overreaction” as well?
However, I note that you have given up your claim that none of this matters because the algorithm is “almost as likely” to overreact on the plus side as on the minus side, which I guess is some progress …

Willis, the Petersen paper included with the data made this perfectly clear:

A great deal of effort went into the homogeneity adjustments. Yet the effects of the homogeneity adjustments on global average temperature trends are minor (Easterling and Peterson 1995b). However, on scales of half a continent or smaller, the homogeneity adjustments can have an impact. On an individual time series, the effects of the adjustments can be enormous. These adjustments are the best we could do given the paucity of historical station history metadataon a global scale. But using an approach based on areference series created from surrounding stations means that the adjusted station’s data is more indicative of regional climate change and less representative of local microclimatic change than an individual station not needing adjustments. Therefore, the best use for homogeneity-adjusted data is regional analyses of long-term climate trends (Easterling et al.1996b).

And no, I do think it’s important that the bias of the effect is small. That’s what P is saying – occasionally enormous effects locally, little effect on global trends (which is what really affects the fate of the billions, not Darwin 1920). He lays out what the difficulties are and when to use it.
Incidentally, I doubt that Peterson had roomfuls of scientists. The whole staff of CRU could fit into a smallish room, and I doubt that GHCN is much better resourced.

Willis Eschenbach
December 23, 2009 9:52 pm

Nick Stokes (19:55:03) :


Willis, the Petersen paper included with the data made this perfectly clear:

A great deal of effort went into the homogeneity adjustments. Yet the effects of the homogeneity adjustments on global average temperature trends are minor (Easterling and Peterson 1995b). However, on scales of half a continent or smaller, the homogeneity adjustments can have an impact. On an individual time series, the effects of the adjustments can be enormous. These adjustments are the best we could do given the paucity of historical station history metadataon a global scale. But using an approach based on areference series created from surrounding stations means that the adjusted station’s data is more indicative of regional climate change and less representative of local microclimatic change than an individual station not needing adjustments. Therefore, the best use for homogeneity-adjusted data is regional analyses of long-term climate trends (Easterling et al.1996b).

And no, I do think it’s important that the bias of the effect is small. That’s what P is saying – occasionally enormous effects locally, little effect on global trends (which is what really affects the fate of the billions, not Darwin 1920). He lays out what the difficulties are and when to use it.

Yes. He says to use it regionally. Which is why I pointed out that it totally changes the regional picture for Northern Australia. Why should an algorithm that claims to merely make the outliers look more like the average radically change the average? To believe that, we’d have to assume that most of the erroneous data erroneously shows cooling. Perhaps you could comment about the change that GHCN makes to the Northern Australia record?

Incidentally, I doubt that Peterson had roomfuls of scientists. The whole staff of CRU could fit into a smallish room, and I doubt that GHCN is much better resourced.

CRU is part of a university. GHCN is a project of the NCDC, which is part of the US Government. NCDC have a staff of 133 including support staff. CRU has a staff of 25 including support staff. Please do your homework before making this kind of claim. In addition to making yourself look foolish, you are making me do your homework.

Nick Stokes
December 24, 2009 12:49 am

Willis,
You showed pictures from the AR4 indicating an upward gradient in Northern Australia, which you apparently blame on homogenization. But you showed no evidence that they used GHCN homogenised data. The graphs indicate that their data source was CRU. Now, as you say, CRU use data equivalent to GHCN raw data. But there’s no evidence that they used GHCN homogenization. In fact, the posted CRU data shows no great rise.

Ryan Stephenson
December 24, 2009 2:12 am

I have used this graph and two others at “Digging in the clay” to look at the behaviour of the algorithm. I can say that the algorithm has several bugs due to sloppy programming, and these bugs are the footprint of a automatic adjustment – the adjustment was not done by hand. The bugs are as follows:-
1] Incorrectly detecting data for adjustment that does not require adjustment, i.e the algorithm detects discontinuities where it is not clear they exist. (e.g. Darwin 1980)
2] Correctly detected adjustments which are then made in the wrong direction, e.g. the algorithm should shift the data down by amount “x” but then shifts it up by the same amount. (e.g. Darwin 1930 – 1940 and Darwin 1920 – 1930)
3] Adjustments of unreliable data or data that is too short in duration to be used reliably – which results in rather bizarre adjustments. (Edson, Alberta)
4] Adjustments of single years which are then not applied to all following years (e.g. Darwin 1890 and Darwin 1905 – suggests the algorithm just didn’t like those two readings for some reason!)
5] Failure to detect some obvious (and known) discontinuities that actually require adjustment (e.g. about 1940 in Darwin)
6] Applying adjustments up to two years too early or up to two years too late based on incorrect detection of the date at which the discontinuity occured. (e.g. Darwin 1963 – two data points just before the adjusted data at that point seem to have been “left behind” )
7] Errors in the magnitude of the correction applied. (e.g. Darwin 1963 -1980 looks like 0.5Celsius shift was needed but 0.7Celsius shift applied)
[8] The algorithm does not shift the adjusted data such that the most recent years are shown as coincident in temperature. (e.g. Darwin 1995 adjustment tells us that the temperature measured by the modern electronic Stevenson screen is 2.5Celsius out and the measurement made in 1880 using a thermometer bolted on the side of the post office was much more accurate)
The algorithm is a complete mess as exposed by these graphs. There is every reason to believe that the bugs in the algorithm have made the same erroneous adjustments with all the data. There is no reason to believe that some of the bugs are not biased towards a particular temperature trend outcome. The adjusted data is not reliable over small regions and therefore could not be used to correlate to tree-ring data or satellite data.

Ryan Stephenson
December 24, 2009 2:24 am

I should perhaps add two further conclusions from the study of the three graphs
A] The adjustments are indeed unjustified and result in data that is actually far less reliable than the underlying data they were supposed to correct.
B] The adjustments are roughly equally likely to cause donward trends in temperature, so there is no real evidence of deliberate malign intent in the adjustments made. It is simply seriously bad programming, although some of the bugs may be biased towards a particular direction of adjustment, perhaps as a result of confirmational bias on the part of the programmer.

Geoff Sherrington
December 24, 2009 3:15 am

Nick Stokes (23:24:59) :
So your answer can be named another “Hide the decline?”
How about a straightforward answer, where there will be no criticism of a “don’t know”. Even if Giss have altered their public presentation of data imcluding GHCN corrections recently, can you be sure that they do not continue to use adjusted data in compiling global patterns?
Second question. Do you have daily or monthly data for the Darwin airport versus Post Office comparison that went from Feb 1941 to January 1942?

wobble
December 24, 2009 8:52 am

Nick Stokes (19:55:03) :
“”All Willis has persuaded me of is that there may have been some change to the algorithm since 1997. And yes, if so there is a documentation fault.””
A documentation fault? Are you suggesting that the Peterson paper isn’t the proper algorithm to use for recreation? Everyone on here was telling Willis to dig into Peterson. Were they wrong?
“”As I said, I believe that all the non-US data was computed with the same algorithm. “”
Nick Stokes, I think Willis has done an adequate job attempting to recreate the Darwin adjustment. If you think it’s possible to recreate the objective use of an algorithm then please show us your work. (And don’t forget to tell us which algorithm you use.)

wobble
December 24, 2009 8:59 am

Ryan Stephenson (02:12:36):
“”I can say that the algorithm has several bugs due to sloppy programming, and these bugs are the footprint of a automatic adjustment – the adjustment was not done by hand.””
If the programming being used isn’t properly employing the algorithm, then the algorithm isn’t being used.
So it would be accurate to state, “The algorithm wasn’t used objectively.”

Nick Stokes
December 24, 2009 1:15 pm

Geoff Sherrington (03:15:31) :
There’s no reason to believe GISS ever used GHCN adjustments, and I think it is very unlikely. They have their own method, which actually makes very little adjustment to Darwin.
I don’t have that Darwin data.

Willis Eschenbach
December 24, 2009 9:06 pm

Well, dang-a-lang. To my immense surprise, I got a before-Christmas present, an email from Tom Peterson of GHCN, one of the authors of the GHCN homogenization method.. I’m usually loath to post emails, but this one was so genteel and was all business so I hope that Dr. Peterson will forgive me for posting it here.
From the sound of it, they were not happy with the algorithm which was used to homogenize the temperature including Darwin, and are replacing it with a new algorithm due out in the spring.
So, I didn’t get a single one of my questions answered, but in a separate email Dr. Peterson said that he would send me the old code when he returns from the holidays. He also sent me a number of papers on the subject, all but one of which I already had, but I greatly appreciate his thoroughness.
I must warmly commend Dr. Peterson for his openness and his willingness to send the (now obsolete) code. He also says that they will be releasing the new code concurrent with the release of the new GHCN series. When the old code arrives I’ll let y’all know, and maybe the questions will be answered at last.
So while we still have not definitively established whether Darwin was adjusted manually, or whether it was just wildly mis-adjusted by a bad algorithm and overlooked by shoddy quality control, it looks possible that we may get an answer when the code arrives.
Given Dr. Peterson’s very quick response, I’d say that Climategate has had a very salubrious effect on the climate science community. I say this because of Peterson’s totally different response given in the CRU emails. In response to Phil Jones saying

I do now wish I’d never sent them the data after their FOIA request!

Peterson replied

Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests.
Russ’ view is that you can never satisfy them so why bother to try?

As a result of having read that response, I didn’t bother to write. Live and learn, it appears that both Dr. Peterson and I have learned something from all of this.
I’m very interested to see what the new algorithm makes of Darwin. Heck, it might even make the obvious 1940 correction that the previous correction didn’t make.
Again, I want to extol the actions of Dr. Peterson. They are in the finest tradition of scientific transparency, a tradition which has been sadly lacking in the climate science community for many years. This is a very welcome development in the field. In particular I call attention to his last paragraph.
Best Christmas wishes to everyone,
w.

Dear Willis Eschenbach,
I received your questions today. They are quite detailed and would take some digging through files from the mid to late 1990s for me to answer all of them. This would take time I don’t have right now (I actually should be on annual leave right now, but had a few things I wanted to get done before I take off for the rest of the year in a few hours). So let me respond in general terms first and provide you with some articles to make sure we’re both starting from the same page.
One of the problems we were trying to address in some of the procedures we developed back in the mid-1990s was how to take advantage of the best climate information we had at each location at each point in time. We had spent a great deal of time and energy digitizing European colonial era data (article sent) which went a great deal towards making global data prior to 1950 more global (see http://www.ncdc.noaa.gov/img/col.gif for a movie loop of the stations we digitized or acquired for GHCN by this project). This means that in some parts of the world, we might have more stations available to build a reference series from prior to the country’s independence than afterwards. To utilize data that did not span the whole period of record, we used what we called the first difference method (article sent). Using this approach we built a reference series (article sent) one year at a time.
There were two concerns about this approach. The first was how to make sure we didn’t incorporate a change in station location (etc.) artifact into the reference series. That aspect was done by using the 5 highest correlated stations for the reference series and removing the value from the highest and lowest of the 5 highest correlated first difference values for that year based on the assumption that the mean of the three center most values provided a robust measure of the climate signal and if a station moved up or down a hill, its value would likely be the highest or lowest due to the impact of the station move that year. (This last part was a later addition and is explained in the homogeneity review paper (paper sent).)
The homogeneity review paper explains the reasons behind adopting this complex reference series creation process. It did indeed maximize the utilization of neighboring station information. The downside was that there was a potential for a random walk to creep into the reference series. For example, if the nearest neighbor, the one with the highest correlation, had a fairly warm year in 1930, its first difference value for 1930 would likely be fairly high. The first difference value for 1931 would therefore likely be low as it probably was colder than that experienced in that very warm year preceding it. So the reference series would go up and then down again. The random walk comes in if the data for 1931 were missing. Then one gets the warming effect but not the cooling of the following year. The likelihood of a warm random walk and a cold random walk are equally possible. Based on the hundreds of reference series plots I looked at during my mid-1990s evaluation of this process, random walks seemed do be either non-existent or very minor. However, they remained a possibility and a concern.
Partly in response to this concern, over the course of many years, a team here at NCDC developed a new approach to make homogeneity adjustments that had several advantages over the old approaches. Rather than building reference series it does a complex series of pairwise comparisons. Rather than using an adjustment technique (paper sent) that saw every change as a step function (which as the homogeneity review paper indicates was pretty standard back in the mid-1990s) the new approach can also look at slight trend differences (e.g., those that might be expected to be caused by the growth of a tree to the west of a station increasingly shading the station site in the late afternoon and thereby cooling maximum temperature data). That work was done by Matt Menne, Claude Williams and Russ Vose with papers published this year in the Journal of Climate (homogeneity adjustments) and the Bulletin of the AMS (USHCN version 2 which uses this technique).
Everyone here at NCDC is very pleased with their work and the rigor they applied to developing and evaluating it. They are currently in the process of applying their adjustment procedure to GHCN. Preliminary evaluation appears very, very promising (though of course some very remote stations like St Helena Island (which has a large discontinuity in the middle of its long record due to moving downhill) will not be able to be adjusted using this approach). GHCN is also undergoing a major update with the addition of newly available data. We currently expect to release the new version of GHCN in February or March along with all the processing software and intermediate files which will dramatically increase the transparency of our process and make the job of people like you who evaluate and try to duplicate surface temperature data processing much easier.
I hope this email and the series of articles I am sending will answer some of your questions at least (e.g., in the homogeneity review paper it clearly states that the first difference correlation threshold of 0.8 is between the candidate station and the final reference series, not the individual stations that make up the reference series). They are likely to also stimulate some additional questions. So if it is all right with you, I won’t follow up on your questions when I return in January but rather will wait until you send in a new set of questions or just send these old ones back to me.
We’re doing a lot of evaluation of our new approach to adjusting global temperature data to remove artificial biases but additional eyes are always welcome. So I would encourage you to consider doing additional GHCN evaluations when we release what we are now calling GHCN version 2.5 in, hopefully, February or March of 2010.
Happy Holidays,
Tom Peterson

REPLY: That’s some Christmas gift, nice to see such an inquiry handled professionally. – Anthony

Nick Stokes
December 24, 2009 10:19 pm

Willis,
A very interesting response. Could you please post a list of the papers that he sent?
Thanks.

Verified by MonsterInsights