Guest Post by Willis Eschenbach
Recapping the story begun at WUWT here and continued at WUWT here, data from the temperature station Darwin Zero in northern Australia was found to be radically adjusted and showing huge warming (red line, adjusted temperature) compared to the unadjusted data (blue line). The unadjusted data showed that Darwin Zero was actually cooling over the period of the record. Here is the adjustment to Darwin Zero:
Figure 1. The GHCN adjustments to the Darwin Zero temperature record.
Many people have written in with questions about my analysis. I thank everyone for their interest. I’m answering them as fast as I can. I cannot answer them all, so I am trying to pick the relevant ones. This post is to answer a few.
• First, there has been some confusion about the data. I am using solely GHCN numbers and methods. They will not match the GISS or the CRU or the HadCRUT numbers.
• Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.
• Next, a number of people have claimed that the reason for the Darwin adjustment was that it is simply the result of the standard homogenization done by GHCN based on comparison with other neighboring station records. This homogenization procedure is described here (PDF).
While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way. Instead they have adjusted them differently (see below). This argues strongly that they were not done by the listed GHCN homogenization process. Any process that changed one of them would change all of them in the same way, as they are nearly identical.
• Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert. You can’t adjust Darwin based on that. The GHCN homogenization method requires five well correlated neighboring “reference stations” to work.
From the reference cited above:
“In creating each year’s first difference reference series, we used the five most highly correlated neighboring stations that had enough data to accurately model the candidate station.”
and “Also, not all stations could be adjusted. Remote stations for which we could not produce an adequate reference series (the correlation between first-difference station time series and its reference time series must be 0.80 or greater) were not adjusted.”
As I mentioned in my original article, the hard part is not to find five neighboring stations, particularly if you consider a station 1,500 km away as “neighboring”. The hard part is to find similar stations within that distance. We need those stations whose first difference has an 0.80 correlation with the Darwin station first difference.
(A “first difference” is a list of the changes from year to year of the data. For example, if the data is “31, 32, 33, 35, 34”, the first differences are “1, 1, 2, -1”. It is often useful to examine first differences rather than the actual data. See Peterson (PDF) for a discussion of the use of the “first-difference method” in climate science.)
Accordingly, I’ve been looking at the candidate stations. For the 1920 adjustment we need stations starting in 1915 or earlier. Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:
WYNDHAM_(WYNDHAM_PORT) = -0.14
DERBY = -0.10
BURKETOWN = -0.40
CAMOOWEAL = -0.21
NORMANTON = 0.35
DONORS_HILL = 0.35
MT_ISA_AIRPORT = -0.20
ALICE_SPRINGS = 0.06
COEN_(POST_OFFICE) = -0.01
CROYDON = -0.23
CLONCURRY = -0.2
MUSGRAVE_STATION = -0.43
FAIRVIEW = -0.29
As you can see, not one of them is even remotely like Darwin. None of them are adequate for inclusion in a “first-difference reference time series” according to the GHCN. The Economist excoriated me for not including Wyndham in the “neighboring stations” (I had overlooked it in the list). However, the problem is that even if we include Wyndham, Derby, and every other station out to 1,500 km, we still don’t have a single station with a high enough correlation to use the GHCN method for the 1920 adjustment.
Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.
• Next, the GHCN does use a good method to detect inhomogeneities. Here’s their description of their method.
To look for such a change point, a simple linear regression was fitted to the part of the difference series before the year being tested and another after the year being tested. This test is repeated for all years of the time series (with a minimum of 5 yr in each section), and the year with the lowest residual sum of the squares was considered the year with a potential discontinuity.
This is a valid method, so I applied it to the Darwin data itself. Here’s that result:
Figure 2. Possible inhomogeneities in the Darwin Zero record, as indicated by the GHCN algorithm.
As you can see by the upper thin red line, the method indicates a possible discontinuity centered at 1939. However, once that discontinuity is removed, the rest of the record does not indicate any discontinuity (thick red line). By contrast, the GHCN adjusted data (see Fig. 1 above) do not find any discontinuity in 1941. Instead, they claim that there are discontinuities around 1920, 1930, 1950, 1960, and 1980 … doubtful.
• Finally, the main recurring question is, why do I think the adjustments were made manually rather than by the procedure described by the GHCN? There are a number of totally independent lines of evidence that all lead to my conclusion:
1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).
2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).
3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).
4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).
5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. Here are those pictures:
Figure 3. Darwin station data before and after GHCN adjustments. Upper panel shows unadjusted Darwin data, lower panel shows the same data after adjustments.
Before the adjustments we had the station Darwin Zero (blue line line with diamonds), along with four other nearby temperature records from Darwin. They all agreed with each other quite closely. Hardly a whisper of dissent among them, only small differences.
While GHCN were making the adjustment, two stations (Unadj 3 and 4, green and purple) vanished. I don’t know why. GHCN says they don’t use records under 20 years in length, which applies to Darwin 4, but Darwin 3 is twenty years in length. In any case, after removing those two series, the remaining three temperature records were then adjusted into submission.
In the “after” picture, Darwin Zero looks like it was adjusted with Sildenafil. Darwin 2 gets bent down almost to match Darwin Zero. Strangely, Darwin 1 is mostly untouched. It loses the low 1967 temperature, which seems odd, and the central section is moved up a little.
Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value? Seems to me you’d be better off picking any one of the originals, because they all agree with each other.
The reason you adjust is because records don’t agree, not to make them disagree. And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.
So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims. I’m happy to be proven wrong, and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.
Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.
Sponsored IT training links:
Want to pass HP0-J33 at first try? Gets certified 000-377 study material including 199-01 dumps to pass real exam on time.
Another possible problem with correlation measurements is that the discontinuities will destroy the correlation structure. You almost have to remove them before measuring the correlation, or measure correlation over periods of time where there aren’t significant discontinuities. This is especially important with relatively few data points……
Marc:
“Why do they use yearly temperatures?
If the temperature stepped at some point, it should be possible to see it in the day by day temperature.”
Daily temperature fluctuations are much larger (one or two orders of magnitude) than, and would mask, the typical effect being adjusted for.
DR:
GHCN isnt doing meteorology, and they arent buildng engines. If you want to disprove GHCN methods, you have to address what they are doing – estimating global surface temperature.
Bill Illis :
“(I can’t be sure but there is long enough overlap between individual station locations in this chart that no adjustments should have been made.)”
Overlap among station locations does not rule out adjustment, much less overlap among duplicate records for a single station …
Bill,
“To get a VALID check on global temperatures will take at least another 150 years and more.”
Then you better get on it. Because if you dont have VALID data, then you are just waving your hands.
“If there is a problem with AGW then you have consigned large populated areas to the scrap heap.”
Nonsense. That aside, your analysis is decidedly one sided. If there is not a problem with AGW, but we spend Hundreds of Trillions of dollars on the pretense that there is, then you have consigned large populated areas to the scrap heap. If you are going to demand a response like that, you need to bring sufficient VALID data to the table. If it takes 150 years, so be it.
************
bill (09:58:40) :
To get a VALID check on global temperatures will take at least another 150 years and more. If there is a problem with AGW then you have consigned large populated areas to the scrap heap.
***************
If the Earth hits a dense swarm of asteroids, the Earth is toast but that isn’t very likely and neither is catastrophic global warming. I’m sure we can come up with a long list of events that might hurt us, but we don’t prepare for every one of them as if they WILL happen.
Why do they have to adjust the temperature readings at all? Temperature at a given site is the temperature at that site. If it goes up or down, so be it. Averaging all sites in a local area to come up with a temperaure report for that area seems reasonable, six or twenty stations in a city or township could be averaged to give a regional temperature. “Adjusting” temp readings seems fraught with problems and opens the path to errors (and to misrepresentation of reality.)
Does anyone really know what the temperature is anywhere on the planet anymore? The graph shown for Darwin is disturbing, raw data shows one thing, adjusted shows something very different, which one is reality and which is an illusion?
Nick,
Any “analysis” such as GG’s which leaves out key variables is worse than meaningless, it’s deceptive. Another good example is the supposed US pay gap between men and women. When factors such as profession, location, education, years of experience, etc. are factored in women in the US make slightly more than men. GG leaves out both location and time and thus should be disregarded immediately as bogus.
On your speculation of Stevenson Screen changes for the later Darwin stepwise adjustments, show us the records. You can’t expect to be taken seriously if you just throw out guesses. Also explain why a new Stevenson screen (not the purported change to a Stevenson screen way back around the turn of the century at Darwin) should result in an upward temperature adjustment or a lesser decrease in adjustment. My understanding is that a new (and thus brighter) screen will, if anything, result in slightly lower temperatures readings than if it was old and discolored. The slats being blocked would also yield higher readings in sunshine. What you see in the Darwin record for the later decade’s temp adjustments is less decrease followed by increase, so what’s your theory there?
Finally, your continued mention that Darwin is but one station is a pathetic and ridiculous diversion tactic. We’re not discussing every single station in the GHCN record here and pointing out the fact that a single one has been adjusted. There are other posts and other sites that cover many other stations. There are also posts that look at the cumulative effect of adjustments in various areas. This discussion is not happening in a vacuum, so pretend like you’re smarter than a fifth grader and drop the pretense that what has been found with Darwin adjustments is everything and therefore can’t really affect the overall data.
Until you can specifically account for the dramatic adjustments at Darwin it remains a smoking gun.
I’ve charted up the difference in the annual temperatures from the GISS unadjusted dataset and the GHCN adjusted data from the ClimateExplorer.
This is presented a little different than Willis did but I think you will get the picture.
http://img192.imageshack.us/img192/521/darwintemps.png
Difference / Changes made by GHCN
http://img14.imageshack.us/img14/8291/darwintempdifference.png
bill (09:58:40) : Was that spaguetti’s temperatures and hockey stick shaped graph made wth Mann’s software?☺
JJ (07:28:55) :
If they were merely duplicates, the GHCN would not adjust them differently. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
Now, do you see how condescending that sounds? If you want people to take you seriously, talk to us like adults, not the way you talk to kids.
If they were merely “duplicate records”, they’d average them to get the final record and be done with it. They don’t. If they were just “duplicate records”, they wouldn’t disagree 90% of the time. They do. If they were only “duplicate records”, they would not have been adjusted separately. They were.
Man, you’ve been hanging around with too many “climate scientists”, you’re all about “could” and “might” and “may” and “could easily” and the like. Yes, the GHCN algorithm could do the things you claim … but it doesn’t. TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!! I am sick of people making claims about what “could easily” and “might” and “may” happen. if you think it might happen, try it and see. I have, and I couldn’t make it happen, “easily” or otherwise.
The GHCN says “neighboring” stations. Perhaps more than 1,500 km away is “neighboring” on your planet. On this planet it is not.
I thought you were following the story. I am clearly talking about the adjustment in 1920. Go back to the records. Which Darwin first difference am I talking about?
No. The assertion is that “neighboring” stations are used. I’ve checked the neighborhood out to 1,500 km. without finding a single suitable station. If you think that there are suitable stations, you dig them out, because I can see that absolutely nothing that I might do will satisfy you. When you five the five stations, let us know. Until then, your fantasies are just that.
Take a deep breath and think about this for a minute, JJ. All the GHCN method can do is to adjust a station to match the trend in neighboring stations. It can’t create a trend out of nothing. For the adjustment to have been used, we would have to find 1) five well correlated neighboring sites that 2) increase at 6C per century.
I can’t find either of those. If you can find those, bring it on. If not, the results indicate that they didn’t use the algorithm, regardless of how frantically you wave your hands and say that they might could easily have used some possible unknown stations 5,000 km away.
Simply put, I have shown it out to a distance of 1,500 km, well beyond any reasonable interpretation of “neighboring”, and well beyond the known intercorrelation distance of temperatures. If you think I’m wrong, COME UP WITH SOME STATIONS THAT FIT THE GHCN CRITERIA!!!
I don’t know if I did, JJ. I just do the best I can, nobody can ever be sure that they are correct. That’s why I put my work out on the web, to let other people find errors in my application of the GHCN algorithm. If you find I’ve done it wrong, let me know. Asking me if I’m sure is just wasting electrons. If you think I’m wrong, do the analysis yourself and show that I’m wrong.
Frankly, without a citation, your “belief” is meaningless.
Make sense with respect to 1) logic 2) mathematics 3) practical considerations 4) the fact that they are nearly identical. If you can’t follow those, I can’t help you.
Say what? My claim is that the listed GHCN methodology was not used. Whether is is “sufficient” for anything at all is your issue, not mine.
Proven? I haven’t “proven” anything in any direction. How could I?
Now you’re just playing at not understanding. The GHCN produce data to be used in global and regional estimates of temperature change. Do you think that they do that without averaging? Sheesh …
Effect on the aggregate? I’m talking about Darwin, not some mythical “aggregate”.
Your speculation and fantasies are worse than useless, as you are basing conclusions on them with absolutely no data.
TRY THE GHCN ALGORITHM!!! Use some dummy data for neighboring stations and see what it does to the different length (but otherwise nearly identical) Darwin datasets. You claim it is possible to get wildly different results simply because the datasets are of different lengths. I have not been able to reproduce that result. No matter what I try, I get very similar results. Get back to us when you have something other than your speculations to report.
First, you need to learn that when you say to someone “you need to learn”, it just pisses them off.
Next, JJ, if you can show how the GHCN methods were used on the 1920 step, bring it on. Until then, your claims about what you understand and what I don’t understand ring hollow.
If you can’t take being proven wrong, don’t go into science. It is an inescapable part of science that some of your claims will be shown to be wrong. I’m happy to be proven wrong because that’s how science progresses. Of course, I’m happier if I can prove someone else wrong …
You should look up the difference between “hope” and “expect” sometimes. I don’t expect the GHCN to do a damn thing.
Look, JJ, if you want someone to go ask GHCN something, be my guest. Report back and tell us how far you have gotten. Me, I’m done with beating my head on closed doors … but if you think it’s a good plan, stop messing around and get on with it. You think I’m wrong, you go ask GHCN to prove it. Let us know how it goes. Me, I’ve tried that route over and over, I find it a useless path. But you might surprise me.
If you truly think it is the right thing to do, why wait? Go ahead and ask the GHCN, and report back with what you find out. Stop trying to be an armchair general, it just makes you look foolish saying “you should do this, you should do that”. Get out and do something other than cavil and carp and whine and find fault.
I’m dead serious here. You are all too willing to point out to me the right path, the decent path, the proper path. You keep telling me about all of the things that might be and could easily be. So show us how it’s done, JJ. Get in touch with GHCN and see how far that gets you. You claim that’s the right path, give us a demonstration.
I await your report from GHCN, although not with bated breath …
Say what? The Economist apologized to me, both in the blog and in private emails, for their unconscionable over-reaching in their first post. Both the guy who wrote the blog and his editor wrote me to say that they were way over the line. Now they are flailing and trying to salvage something of their reputation. They are lucky that I didn’t bring a libel suit against them, I suspect that’s why they rushed to apologize both publicly and privately. If you think that they “spanked” me, you’re not following the story.
In any case, until you actually do some of the things that you so earnestly recommend, like showing that nearly identical records can be warped differently by the GHCN algorithm, or getting in touch with GHCN for an explanation, or finding one single candidate comparison station for the 1920 adjustment to Darwin Zero, please don’t bother me again. I’m not interested in your “may” and “could easily” and “might”, answering your fantasies is taking up way too much time. If it “could easily”, then do it and report the results.
Latest from the UK MET office (to think my taxes directly pay for this lot.)
The HadCRUT database is UNDERSTATING temp increases and the Russians recent complaint confirms the database is correct as its matches their lower temps.
Quote ‘The IEA’s output is consistent with HadCRUT as they both confirm the global warming signal in this region since 1950, which we see in many other variables and has been consistently attributed to human activities’
I want my money back !!!!!
http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091218b.html
JJ will be busy over Christmas… 😉
See all the climategate analysis here: http://69.84.25.250/blogger/post/ClimateGate-Data-Series-Part-5-A-break-down-of-large-data-file-for-manipulating-global-temperature-trends-from-2006-2009.aspx
joe (09:07:01) :
joe, look at the legend in Fig. 1, and at the colors in Fig. 2. Some of the items shown are measured on the left scale, and some on the right.
w.
bill (09:28:20) :
Bill, GISS makes their own very different adjustments to the data. As such, they are not relevant to what GHCN does.
w.
Mesa (11:00:12) :
This is why the correlation is done on the first differences rather than on the underlying data. With the first differences, a step will only affect that year, not the entire record. Because of that, it does not “destroy the correlation structure”.
w.
RE: Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
This isn’t rocket science – adjust the trend of about half of the stations UP since 1930. Adjust the trend of the other half of the stations DOWN prior 1930. Presto, the average adjustment is… Zero!
That’s why Willis’ time-domain analysis is so significant.
On Willis’ last reply to JJ – well done Willis!
I think we will be waiting a long time for either JJ (whoever he might be incidentally, as you post your analysis and comments under your real name) to get a response out of GHCN or he actually does some analysis of his own and posts it for other people to prod into like you have done.
KevinM (08:11:16) :
Prove it! somebody show me one plot from one station with data that is downloadable and verifiable and shows the opposite pattern.
Well, I said above that of stations >80 years adjusted record, 17 were adjusted down by more than Darwin was adjusted up. Here they are:
211357000002 GUR’EV
222234720001 TURUHANSK
222246410000 VILJUJSK
222255510002 MARKOVO
222310880000
222325830000
222325830002
222325830003
403717140040
403717470010
403718360001
414763930000 MONTERREY,N.L
425726710020
501947280000 COONABARABRAN NSW
603113200000 INNSBRUCK/UNIVERSITYAUSTRIA
615076300000 TOULOUSE/BLAG
I didn’t look up all the names, but you can find them, with details, on the v2.temperature,inv file.
I’ve plotted the Australian station, Coonabarabran, here. Adjusted in red.
Richard Sharpe (22:00:48) :
Scott of Melb Australia (20:58:57) said:
(using microfish of available temp data)
Hmmm, is that a new unit of measure for temperature data that I am not aware of or did you mean microfiche?
Thanks for the pickup Richard – no excuses I should have checked my spelling. Hopefully it doesn’t detract from the point I was making.
Scott
Nick Stokes,
Are you a free-lance or paid ‘provocateur’ whose only purpose is pedantic obfuscation? Instead you constantly change the subject returning again and again to pick at non germain nits. Do you get paid to divert rational discussions, or do you do so simply for the jollies? Do you get paid by RC?
Real Climate.org funded by Fenton Communications, a front organization of the Al Gore empire and administered by his ex-campaign treasurer, apparently exists strictly for the purpose of providing an official propaganda house organ. It is run by Gavin Schmidt, even as he double dips on his govermnment job for Astronomer Hansen, as a sort of ‘the Team newspaper’.
It si similar to the publication for those friendly Kremlin chaps and the inestimal publication named ‘Pravda’ or was it ‘Isvestiya’. I could never keep straight which was the mouthpiece for the Party, or the Government.
To all aussies: Feel free to use the expression I invented above:
“Kangaroo Tail” for campaigning against local “warmheads”.
lowercasefred (10:35:59) :
See my response to Kevinm above. I improved the plot of the Coonabarabran result, by adding the adjustment plot in Willis-style (green, no axis marked but same scale as the others) here.
[not sure if this was meant to be fightin words or not, but it didn’t really add to the discussion. ~ ctm]
Folks, keep in mind that Nick Stokes is well aware of the temporal problems with GG’s histogram and was before he made his first comment on this thread. I think that speaks to motive.
David Jay (12:49:46) :
Bruce (08:34:30) :
Gail Combs (07:56:23) :
KeithGuy (07:13:16) :
Jim (06:08:58) :
You’re all asking variants of the same question – doesn’t it matter when the adjustments were made? Yes, it does. What GG and I plotted in those histograms was the change to the trend over the whole time period of the station. One stat that Willis drew attention to was that the change to Darwin made a difference of 1.9 C/century (in my calc, it;s 2.35) The histogram shows the equivalent figure for all the other stations. the measure is the linearised cumulative rate of change over the whole period.
I’ve explained in more detail at my blog here.
Nick Stokes (13:35:35) :
Still not sure what you are trying to prove. If you want to say there is not much of an effect on the record, plot the GHCN unadj against the GHCN adj and see what the difference is.