Darwin Zero Before and After

Guest Post by Willis Eschenbach

Recapping the story begun at WUWT here and continued at WUWT here, data from the temperature station Darwin Zero in northern Australia was found to be radically adjusted and showing huge warming (red line, adjusted temperature) compared to the unadjusted data (blue line). The unadjusted data showed that Darwin Zero was actually cooling over the period of the record. Here is the adjustment to Darwin Zero:

Figure 1. The GHCN adjustments to the Darwin Zero temperature record.

Many people have written in with questions about my analysis. I thank everyone for their interest. I’m answering them as fast as I can. I cannot answer them all, so I am trying to pick the relevant ones. This post is to answer a few.

• First, there has been some confusion about the data. I am using solely GHCN numbers and methods. They will not match the GISS or the CRU or the HadCRUT numbers.

• Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.

• Next, a number of people have claimed that the reason for the Darwin adjustment was that it is simply the result of the standard homogenization done by GHCN based on comparison with other neighboring station records. This homogenization procedure is described here (PDF).

While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way. Instead they have adjusted them differently (see below). This argues strongly that they were not done by the listed GHCN homogenization process. Any process that changed one of them would change all of them in the same way, as they are nearly identical.

• Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert. You can’t adjust Darwin based on that. The GHCN homogenization method requires five well correlated neighboring “reference stations” to work.

From the reference cited above:

“In creating each year’s first difference reference series, we used the five most highly correlated neighboring stations that had enough data to accurately model the candidate station.”

and  “Also, not all stations could be adjusted. Remote stations for which we could not produce an adequate reference series (the correlation between first-difference station time series and its reference time series must be 0.80 or greater) were not adjusted.”

As I mentioned in my original article, the hard part is not to find five neighboring stations, particularly if you consider a station 1,500 km away as “neighboring”. The hard part is to find similar stations within that distance. We need those stations whose first difference has an 0.80 correlation with the Darwin station first difference.

(A “first difference” is a list of the changes from year to year of the data. For example, if the data is “31, 32, 33, 35, 34”, the first differences are “1, 1, 2, -1”. It is often useful to examine first differences rather than the actual data. See Peterson (PDF) for a discussion of the use of the “first-difference method” in climate science.)

Accordingly, I’ve been looking at the candidate stations. For the 1920 adjustment we need stations starting in 1915 or earlier. Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:

WYNDHAM_(WYNDHAM_PORT) = -0.14

DERBY = -0.10

BURKETOWN = -0.40

CAMOOWEAL = -0.21

NORMANTON = 0.35

DONORS_HILL = 0.35

MT_ISA_AIRPORT = -0.20

ALICE_SPRINGS = 0.06

COEN_(POST_OFFICE) = -0.01

CROYDON = -0.23

CLONCURRY = -0.2

MUSGRAVE_STATION = -0.43

FAIRVIEW = -0.29

As you can see, not one of them is even remotely like Darwin. None of them are adequate for inclusion in a “first-difference reference time series” according to the GHCN. The Economist excoriated me for not including Wyndham in the “neighboring stations” (I had overlooked it in the list). However, the problem is that even if we include Wyndham, Derby, and every other station out to 1,500 km, we still don’t have a single station with a high enough correlation to use the GHCN method for the 1920 adjustment.

Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.

• Next, the GHCN does use a good method to detect inhomogeneities. Here’s their description of their method.

To look for such a change point, a simple linear regression was fitted to the part of the difference series before the year being tested and another after the year being tested. This test is repeated for all years of the time series (with a minimum of 5 yr in each section), and the year with the lowest residual sum of the squares was considered the year with a potential discontinuity.

This is a valid method, so I applied it to the Darwin data itself. Here’s that result:

Figure 2. Possible inhomogeneities in the Darwin Zero record, as indicated by the GHCN algorithm.

As you can see by the upper thin red line, the method indicates a possible discontinuity centered at 1939. However, once that discontinuity is removed, the rest of the record does not indicate any discontinuity (thick red line). By contrast, the GHCN adjusted data (see Fig. 1 above) do not find any discontinuity in 1941. Instead, they claim that there are discontinuities around 1920, 1930, 1950, 1960, and 1980 … doubtful.

• Finally, the main recurring question is, why do I think the adjustments were made manually rather than by the procedure described by the GHCN? There are a number of totally independent lines of evidence that all lead to my conclusion:

1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).

2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).

3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).

4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).

5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. Here are those pictures:

Figure 3. Darwin station data before and after GHCN adjustments. Upper panel shows unadjusted Darwin data, lower panel shows the same data after adjustments.

Before the adjustments we had the station Darwin Zero (blue line line with diamonds), along with four other nearby temperature records from Darwin. They all agreed with each other quite closely. Hardly a whisper of dissent among them, only small differences.

While GHCN were making the adjustment, two stations (Unadj 3 and 4, green and purple) vanished. I don’t know why. GHCN says they don’t use records under 20 years in length, which applies to Darwin 4, but Darwin 3 is twenty years in length. In any case, after removing those two series, the remaining three temperature records were then adjusted into submission.

In the “after” picture, Darwin Zero looks like it was adjusted with Sildenafil. Darwin 2 gets bent down almost to match Darwin Zero. Strangely, Darwin 1 is mostly untouched. It loses the low 1967 temperature, which seems odd, and the central section is moved up a little.

Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value? Seems to me you’d be better off picking any one of the originals, because they all agree with each other.

The reason you adjust is because records don’t agree, not to make them disagree. And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.

So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims. I’m happy to be proven wrong, and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.

Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.


Sponsored IT training links:

Want to pass HP0-J33 at first try? Gets certified 000-377 study material including 199-01 dumps to pass real exam on time.


0 0 votes
Article Rating
303 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Glenn
December 20, 2009 7:50 pm

Don’t you mean point six (0.6C) per century?

brnn8r
December 20, 2009 7:51 pm

Very nice analysis. Keep up the good work Mr Eschenbach.

tokyoboy
December 20, 2009 7:54 pm

Willis, could you please lead me to the red “homogenized graph” at the GHCN site? Thanks.

Editor
December 20, 2009 8:03 pm

In general terms, am I correct in understanding that the “time of observation bias” inserted every time Hansen’s GISS program runs artificially lowers temperature data before 1970, and raises (or keeps the same data – but with no Urban Heat Island corrction) for all temperature data after 1970. From what I recall, artificial TOBS changes account for over 0.15 of the total 0.4 degree rise in the “supposed” ground temperature record.
Or over 1/3 of Hansen’s entire “measured” climate change is artifically inserted.
If so, why do they claim a TOBS correction is required at all?
Aren’t the only numbers used by Hansen/NOAA the day’s maximun and minimum values? How would those change based on what time of the day you read the max/min thermometer?
(Never mind whatever his “logic” is in using the same TOBS change for every record in every year. (How many times did these earlier weathermen keep changing the time of day they wrote temperatures down?)

Paul Vaughan
December 20, 2009 8:08 pm

Glenn (19:50:45) “Don’t you mean point six (0.6C) per century?”
See figure 1. If you need a grade 10 math tutorial, speak up.

December 20, 2009 8:10 pm

Good work.
Correction: “highly improbability” should be “highly improbable.”

pat
December 20, 2009 8:10 pm

willis, have u seen this?
19 Dec: TBR: NZ Study may hold key to faulty world temp data
A long-forgotten scientific paper on temperature trends in New Zealand may be the smoking gun on temperature manipulation worldwide.
Since Climategate first broke, we’ve seen scandal over temperature adjustments by NZ’s National Institute of Water and Atmospheric research, NIWA, which in turn prompted a fresh look at raw temperature data from Darwin and elsewhere.
Now, a study published in the NZ Journal of Science back in 1980 reveals weather stations at the heart of NIWA’s claims of massive warming were shown to be unreliable and untrustworthy by a senior Met Office climate scientist 30 years ago, long before global warming became a politically charged issue.
The story is published in tonight’s TGIF Edition, and has international ramifications.
That’s because the study’s author, former Met Office Auckland director Jim Hessell, found a wide range of problems with Stevenson Screen temperature stations and similar types of unit.
Hessell debunked a claim that New Zealand was showing a 1C warming trend between 1940 and 1980, saying the sites showing the biggest increases were either urbanized and suffering from urban heat island effect, or they were faulty Stevenson Screen installations prone to showing hotter than normal temperatures because of their design and location.
One of the conclusions is that urbanized temperature stations are almost a waste of time in climate study:
“For the purpose of assessing climatic change, a ‘rural’ environment needs to be carefully defined. Climatic temperature trends can only be assessed from rural sites which have suffered no major transformations due to changes in shelter or urbanisation, or from sites for which the records have been made homogenous. Urban environments usually suffer continual modifications due to one cause or another.”
“It is concluded that the warming trends in New Zealand previously claimed, are in doubt and that as has been found in Australia (Tucker 1975) no clear evidence for long term secular warming or cooling in Australasia over the last 50 years [1930-1980] exists as yet.”
Hessell divided weather stations in New Zealand into two classes, “A” and “B”, where A class locations had suffered increasing urbanization or significant site changes, and B class locations had not.
“It can be seen immediately that the average increase in mean temperatures at the A class stations is about five times that of the B class”, the study notes.
Among the studies listed as a contaminated A site is Kelburn, which was at the heart of the NIWA scandal a fortnight ago.
A link to the study can be found in TGIF Edition.
http://briefingroom.typepad.com/the_briefing_room/2009/12/nz-study-may-hold-key-to-faulty-world-temp-data.html

Michael Jankowski
December 20, 2009 8:13 pm

Glenn,
No, look at the very top chart…from 1941 to about 1990 (50 yrs), the trend is +3 degrees. So it would be 6 deg/century.

Willis Eschenbach
December 20, 2009 8:15 pm

Glenn (19:50:45) :

Don’t you mean point six (0.6C) per century?

Nope, that’s the amazing part. Six degrees per century.

Willis Eschenbach
December 20, 2009 8:18 pm

tokyoboy (19:54:02) :

Willis, could you please lead me to the red “homogenized graph” at the GHCN site? Thanks.

I made the graph with data from the GHCN site. The data is at
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2
in the file called “v2.mean_adj
w.

December 20, 2009 8:22 pm

Oh, I see.. they adjusted all the temps before 1952? 53? down by some constant. Notice the 1.7C drop about 1940 is intact. Then a smaller adjustment was applied step-wise to each of the subsequent decades. -2C before 1952, -1.2C for 1953 to 1960-ish, etc.

Bill of Brisbane
December 20, 2009 8:26 pm

Thanks Anthony. Darwin has only two seasons each year – “the wet” (now) and “the dry”. Both tend to be pretty hot. It’s been consistently like this just as the unadjusted record accurately indicates.
Is there no end to the fabrications being perpetrated ?

Evan Jones
Editor
December 20, 2009 8:27 pm

How would those change based on what time of the day you read the max/min thermometer?
If you measure too close to the high point, you get cases where the same “high” carries over to both days. And vice versa if you measure at the low point.
If it’s really cold at dawn Monday and that’s when I measure it, I’ll get low a minute after dawn on Monday with the Sunday’s low being a minute before dawn. Two days of lows taken from a single cold interval. If dawn Tuesday is warmer than dawn Monday, it gets “left out”.
To avoid this, you need to measure at a time far removed from either the typical high point (mid-afternoon) or the typical low point (predawn).

Willis Eschenbach
December 20, 2009 8:27 pm

RACookPE1978 (20:03:45) :

In general terms, am I correct in understanding that the “time of observation bias” inserted every time Hansen’s GISS program runs artificially lowers temperature data before 1970, and raises (or keeps the same data – but with no Urban Heat Island corrction) for all temperature data after 1970. From what I recall, artificial TOBS changes account for over 0.15 of the total 0.4 degree rise in the “supposed” ground temperature record.
Or over 1/3 of Hansen’s entire “measured” climate change is artifically inserted.
If so, why do they claim a TOBS correction is required at all?
Aren’t the only numbers used by Hansen/NOAA the day’s maximun and minimum values? How would those change based on what time of the day you read the max/min thermometer?
(Never mind whatever his “logic” is in using the same TOBS change for every record in every year. (How many times did these earlier weathermen keep changing the time of day they wrote temperatures down?)

This is really a GISS question or a USHCN question, as GHCN do not do a TOBS adjustment per se. For the amount of the adjustment, see here.
Whether it is necessary or not depends on the station, the type of thermometer used, and the times of observation. The canonical document on this is Peterson (q.v.).

Willis Eschenbach
December 20, 2009 8:31 pm

evanmjones (20:27:01) :

How would those change based on what time of the day you read the max/min thermometer?
If you measure too close to the high point, you get cases where the same “high” carries over to both days. And vice versa if you measure at the low point.
If it’s really cold at dawn Monday and that’s when I measure it, I’ll get lows a minute after dawn with the previous day’s low being a minute before dawn. Two days of lows taken from a single cold interval.
To avoid this, you need to measure at a time far away from either the typical high point (mid-afternoon) or the typical low point (predawn).

No, the “day” is midnite to midnite, so you will only get one high or low per day. The low is typically shortly after dawn, and the high somewhere in the late afternoon.
[REPLY – The 24-hour interval between times of observation is the “day” so far as the records are concerned, not midnight – midnight. There’s really no other way to do it. The best observation time is probably around 11 A.M. ~ Evan]

Glenn
December 20, 2009 8:44 pm

Willis Eschenbach (20:15:45) :
Glenn (19:50:45) :
Don’t you mean point six (0.6C) per century?
“Nope, that’s the amazing part. Six degrees per century.”
Ah. That would make the region the fastest warming area claim in the world, if memory serves, topping the Arctic and Siberia. About 3 times the global surface estimate in any event.

Peter
December 20, 2009 8:45 pm

Glen the 6C per century slope is correct. No need to have a PhD to see that.

Michael
December 20, 2009 8:50 pm

I already knew everything they explained to me in the special report on Climategate on FOX News except for the way Mann and others referred to McKitrick & McIntyre in the e-mails.
Did those creepy climate scientist walk around the cubicals in their offices and refer to McKitrick & McIntyre as M&M as a joke often?
Apparently so.

tokyoboy
December 20, 2009 8:54 pm

Willis Eschenbach (20:18:15) :
>tokyoboy (19:54:02) :Willis, could you please lead me to the
>red “homogenized graph” at the GHCN site? Thanks.
“I made the graph with data from the GHCN site. The data is at
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2 in the file called “v2.mean_adj”
Thanks….. but my PC can’t open the file, probably due to software mismatch.
Willis, don’t you have the data on such a popular software as PDF?
Thanks again.

Clive
December 20, 2009 8:56 pm

Nice work Willis. Thank you.
A bit OT … carrying the case into the future.
So let us fast forward 30 years or so. If the top chart “adjusted” trend would continue its upward slope, it would be unbearable at Darwin yet folks will be carrying on as usual. “They” will be telling the Arctic ice has disappeared when it is just fine. “They” will be telling us that the Maldives just sank, when the obviously are fine.
Back to Darwin Zero … If that upward trend continues at some point it would start to look actually silly … nonsensical and YET they could not reverse it could they? Because THAT would be a cooling trend. Gotcha!! ☺ Are “they” LOCKED into this big lie? They cannot reverse it now. Ohoh.
What I am getting at is this: at some future point (even if the BIG LIE continues) when will it all finally fall apart? (I am not as optimistic as others that Climategate can carry this…) When will empirical facts outweight the manipulated data? (IPCC told us in 1991 that sea levels would rise by 300 mm by 2030…we are half way there in time and only 20% there in reality. BUT no one remembers their first report do they?)
Sane folks (like us) know we’ve already reached that point (where observation belies fudged data), but when will it become obvious to people in Darwin, New York, London and Inuvik that the official word is BS?
Just musing aloud on a snowy eve. ☺
Thanks again Willis.

December 20, 2009 8:57 pm

I appreciate your analysis.
There seems to be clear evidence of deception about global warming.
Earth’s climate is changing, has changed in the past, and will always change because Earth’s climate is controlled by the stormy Sun – a variable star.
There is also evidence of government involvement in deception about the composition of the Sun, its source of energy and solar neutrino oscillations.
It appears that politicians have trained scientists with grant funds like Pavlov trained dogs with dog biscuits.
Scientific integrity was the first victim.
My greatest fear is that democracy itself cannot survive if scientists become tools for misinformation by politicians.
With kind regards,
Oliver K. Manuel
Former NASA PI for Apollo

Scott of Melb Australia
December 20, 2009 8:58 pm

Hi Willis Love your work by the way.
This temperature profile of Australia from the BOM Link, highlights the problem with a 1500 KM difference to different weather stations. To give you a clue 1500 km drops you from Darwin to Alice springs ie top middle down to near the bottom of the first dotted line middle of Australia. ie different heat zone.
http://www.bom.gov.au/cgi-bin/silo/temp_maps.cgi
I did the temperature modeling work when unleaded petrol was introduced to Australia. The problem was cars and petrol pumps were vapour locking. I developed the model that was adopted by the oil industry to related 90 percentile temperature data (using microfish of available temp data) to vapour pressure.
This was developed because all you had to do was move as little as 150km particularly from down south, to end up with a problem. So temperature zones were constructed and vapour pressure limits were established to stop valour locking as soon as you drove out of a capital city.
Thats how I am aware that small variations in distance will give a big difference in average temp data.
P.S. I left the oil industry back in 1997.

Keith Minto
December 20, 2009 9:00 pm

In the days of analogue temperature measurement, would time of day be used for max/min readings or would a max/min thermometer (illustration only http://www.redhillgeneralstore.com/A31690.htm ) have been used at these stations? The latter would measure max/min at whatever time that this was reached and would need to be reset once a day. Just curious.

GreenIsAHoax
December 20, 2009 9:07 pm

Global Climate Data Tampering Exposed!
Climate Data Tampering In Australia Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/creating_warming_from_cold_austtralian_stats/
Climate Data Tampering In Darwin Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/climategate_how_one_human_caused_darwin_to_warm/
Climate Data Tampering In New Zealand Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/new_zealands_man_made_warming/
Climate Data Tampering In Russia Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/climategate_was_russias_warming_also_man_made/
Climate Data Tampering In Alaska Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/more_man_made_warming_this_time_in_alaska/
Climate Data Tampering In Sweden Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/making_sweden_warmer/
Fraudulent Hockey Sticks And Hidden Data
http://joannenova.com.au/2009/12/fraudulent-hockey-sticks-and-hidden-data/#more-4660
Would You Buy A Used Car From These So-Called “Scientists”?
Our gullible alarmist friends have obviously fallen for this scam hook, line, and sinker!
First we had the IPCC report “The Science of Climate Change 1995”, where lead auther Benjamin D. Santer removed the following conclusions made by genuine scientists, and without the scientists being made aware of this change.
“None of the studies cited above has shown clear evidence that we can attribute the observed climate changes to the specific cause of increases in greenhouse gases.”
“No study to date has positively attributed all or part [of the climate change observed to date] to anthropogenic [man-made] causes.”
“Any claims of positive detection of significant climate change are likely to remain controversial until uncertainties in the total natural variability of the climate system are reduced.”
Then we have some choice quotes from so-called “consensus scientists”.
“The two MMs [Canadian skeptics Steve McIntyre and Ross McKitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.”
Phil Jones email, Feb 2 2005
“I can’t see either of these papers being in the next IPCC report, Kevin and I will keep them out somehow, even if we have to redefine what the peer-review literature is!”
Phil Jones Director, The CRU
[cutting skeptical scientists out of an official UN report]
“The fact is that we can’t account for the lack of warming at the moment, and it is a travesty that we can’t …there should be even more warming… the data are surely wrong”.
Kevin Trenberth, Climatologist, US Centre for Atmospheric Research
“…If anything, I would like to see the climate change happen, so the science could be proved right, regardless of the consequences. This isn’t being political, it is being selfish. “
Phil Jones Director, The CRU
“We have to get rid of the Mediæval Warm Period” Confided to geophysicist David Deming by the IPCC, 1995
[Many believe that man to be Jonathan Overpeck, which Prof. Deming didn’t deny in an email response, who would later also serve as an IPCC lead author.]
“We have 25 years or so invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it?” Phil Jones Director, The CRU
”We have to offer up scary scenarios, make simplified, dramatic statements, and make little mention of any doubts we might have.” Professor Stephen Schneider
“Humans need a common motivation … either a real one or else one invented for the purpose. … In searching for a new enemy to unite us, we came up with the idea that pollution, the threat of global warming, water shortages, famine and the like would fit the bill. All these dangers are caused by human intervention so the real enemy then, is humanity itself.” Club of Rome declaration
“It doesn’t matter what is true, it only matters what people believe is true…. You are what the media define you to be. Greenpeace became a myth and fund generating machine.” Paul Watson, Co-Founder Greenpeace, Forbes, Nov. 1991
Now what conclusion would a rational and sceptical person come to?
Frederick Seitz, president emeritus of Rockefeller University and chairman of the George C. Marshall Institute, summed it up nicely after seeing the changes made to the IPCC report.
“In my more than 60 years as a member of the American scientific community, including service as president of both the National Academy of Sciences and the American Physical Society, I have never witnessed a more disturbing corruption of the peer-review process than the events that led to this IPCC report.”

Editor
December 20, 2009 9:10 pm

“If you measure too close to the high point, you get cases where the same “high” carries over to both days. And vice versa if you measure at the low point.
If it’s really cold at dawn Monday and that’s when I measure it, I’ll get low a minute after dawn on Monday with the Sunday’s low being a minute before dawn. Two days of lows taken from a single cold interval.
To avoid this, you need to measure at a time far removed from either the typical high point (mid-afternoon) or the typical low point (predawn).”
—…—…—
Yes, understood. I’ve heard similar explanation before.
Now, back to the purpose of my question: What (in your answer) or in the physical world of real temperatures and real measurements, actually justifies lowering all of the country’s recorded temperatures prior to 1970?
We get a cold front coming through once every 10 – 16 days (or maybe 25 – 30 days a year), and at that only half the year does that hypothetical cold front change temepratures drastically (in winter really – summer fronts are most often less drastic), and of those few events how many actually affected two days worth of readings, of that small theoretical fraction left, how many actual events really happened?
There is still no reason to use TOBS as a reason to change the earth’s temperature records.

Editor
December 20, 2009 9:12 pm

Willis Eschenbach (20:27:37) :
Thank you.

Doug in Seattle
December 20, 2009 9:12 pm

I just don’t see how any rational person can look at what was done at Darwin and defend the results.
Perhaps its too late for a quick “oops, we’ll fix it”. It does kind of amaze me that they circled the wagons with the Economist article. This will just end up making looking even stupider.

December 20, 2009 9:12 pm

Rural Beeville 1895 – 2005
NOAA adjustments add a +2.8°F / Century Trend.
I will post a link to a graph tommorrow.

Willis Eschenbach
December 20, 2009 9:20 pm

evanmjones (20:27:01) :

No, the “day” is midnite to midnite, so you will only get one high or low per day. The low is typically shortly after dawn, and the high somewhere in the late afternoon.

[REPLY – The 24-hour interval between times of observation is the “day” so far as the records are concerned, not midnight – midnight. There’s really no other way to do it. The best observation time is probably around 11 A.M. ~ Evan]

You are correct. According to Petersen the “first order” stations use a calendar day, midnite to midnite. Understandably, some people don’t want to use that interval. So a number of the stations don’t use midnite to midnite. But as long as you are not taking the observation near the time of min or max, it shouldn’t be a problem. Your suggestion of 11 AM is a good one.
However, all of this is really a subject for another thread.

Willis Eschenbach
December 20, 2009 9:34 pm

tokyoboy (20:54:32) :

“I made the graph with data from the GHCN site. The data is at
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2 in the file called “v2.mean_adj”

Thanks….. but my PC can’t open the file, probably due to software mismatch.
Willis, don’t you have the data on such a popular software as PDF?
Thanks again.

The data is plain text, compressed using “Z”. There are also .zip versions available at
www1.ncdc.noaa.gov/pub/data/ghcn/vw/zipd/
You could try those.

Dan Martin
December 20, 2009 9:42 pm

Can someone explain to me why in the top graph the temp. anomaly scale is different on the right side from the left side?

J.Hansford
December 20, 2009 9:42 pm

Seems to me that there is no way they could accidentally get that result.
The methodology seems to be striving towards a warming bias…. and deliberately so.
….There. I’ve said it.

Willis Eschenbach
December 20, 2009 9:42 pm

My bad, one letter wrong
www1.ncdc.noaa.gov/pub/data/ghcn/v2/zipd/

Mooloo
December 20, 2009 9:44 pm

There must be people who have lived, and more importantly, farmed in Darwin for fifty years. Have these people noticed the climate changing that rapidly?There must be some evidence, such as that they have to farm different things. Anyone from that part of the world here that has noticed things are 3° warmer?
Similarly, I find it hard to look at the figures shown for New Zealand and reconcile that with the actual weather we have. NIWA allege very rapid recent warming, yet there is no discernable difference. I’ve seen no actual evidence presented that our farmers are having to adapt, for example. My parents are keep gardeners, but haven’t noticed that frosts are any lighter and different plants will now grow.
Does this not bother the warming scientists? How do they reconcile their evidence with what they see with their own eyes?

Peter
December 20, 2009 9:46 pm

Clive, I made a similar comment on another thread. yes, eventually the disparity between reality and official will be so obvious one would have to be a lunatic not to notice something smells in climate science. What’s worse is the disparity between the computer model predictions and even the fudged temperature data is already too significant. The computer models must now be dumped. Anyone who still relies on them are fools.

Neil Gibson
December 20, 2009 9:52 pm

Hi Anthony,
As an engineer with a lifetime of experience in electrical measurement I cannot understand that in all the discussion I
have not seen the words Quality Assurance mentioned. All of our measurements and instruments had to have traceability and customers would full access to inspect our labs and our measurements. Every engineering firm dealing with Government of Semi-Government had to have similar QA procedures. Now while I can understand
that it would be difficult to apply QA to scientific theory it appears to me that the measurement of temperature should definitely have QA procedures and documentation. These should be available on request from the authorities concerned as no Government purchasing body in Australia or other Western countries will accept product from firms without QA. It is totally inconcievable to me that Anthony Watts and his crew of amateurs should be required to perform basic site QA .
The fact that not only do we have measurement without any quality assurance, the very processes used are not divulged.
I think all the climate warmists need is a detailed independent Quality Audit and most global warming will disappear!
REPLY: Agreed. I’ve been saying this for years. For example, why doesn’t climate data have an ISO-8000 certificate? Why doesn’t NCDC have an ISO-9000 rating? Private industry does these things, yet goevrnment seems to do everything haphazardly. – Anthony

Paul R
December 20, 2009 9:53 pm

Nothing going on in Gove, It’s at the same latitude as Darwin, roughly. It needs Homer Geneizing.
http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=36&p_display_type=dataFile&p_startYear=&p_stn_num=014508

Keith Minto
December 20, 2009 9:55 pm

May have answered my own question.
This is from the Australian BOM site http://www.bom.gov.au/climate/cdo/about/definitionstemp.shtml They measure at 9am and it seems that the minimum measured is for that day (the reasonable assumption is that the 9am temp is past the morning minimum on a midnight to midnight cycle) and the maximum recorded is that registered on the previous day. So on these thermometers max/mins are recorded whenever they occur during the day and 9am is a convenient way of reading min. on that day and max. the previous day.
Was this always the case ? as it may effect the recorded temp.
Thanks Willis, for your thoughtful analysis.

Richard Sharpe
December 20, 2009 10:00 pm

Scott of Melb Australia (20:58:57) said:

(using microfish of available temp data)

Hmmm, is that a new unit of measure for temperature data that I am not aware of or did you mean microfiche?

Paul Vaughan
December 20, 2009 10:14 pm

Using the link Willis provided, I found this:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
Listed in there are Canadian stations with serious errors. [I took the time to have a few of my concerns acknowledged by government officials (just to see if they would admit to quality issues – & they did without hesitation).]

steven mosher
December 20, 2009 10:31 pm

hey Willis, petersons say the TOBS code can be purchased for a fee

Billyquiz
December 20, 2009 10:31 pm

[snip -waaayyy off topic, sorry]

JP Miller
December 20, 2009 10:39 pm

Willis, I do not understand the two different scales on the left verticla axis and the right vertical axis. They don’t match, should they?

Geoff Sherrington
December 20, 2009 10:40 pm

Willis,
Glad to see you sticking to your guns.
Clarification about 1940. The year 1940 is in my official data as the year the station moved from the Post Office to the airport (014016 to 014015).
Other official BOM data show a comparison which, unless mislabelled by them compares these 2 sites (lats and longs given) from Jan 1967 to Dec 1973. There is inconsequential change in the average temperature, though there is a bigger range in monthly data at one station, particularly in min temp.
Although this study was done after the shift, it seems to be compelling evidence that no significant correction needed to be applied to the long term data because of the station shift.
It is an unresolved matter if there were problems at the older PO station before 1940. I have never seen any explanation. I do not know why people make a correction in the few years before the station shift. Until a reason is given, I cannot see cause for anyone to make a step change before 1940, or more specifically, because of the station change in 1940 (which some people report was 1941, but then…). An algorith that makes a step change for no explained reason is hardly trustworthy.

Nick Stokes
December 20, 2009 10:46 pm

Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
You say that it is improbable that a station would show an apparent gradient change to a late rise of 6C/century. You give no basis for this statement. Giorgio’s kind of analysis does not help with that figure, but it does indicate how often an adjustment can produce a rise comparable to Darwin’s. The result is more meaningful when restricted to longer series, since a small change to a short series can too easily produce a large trend change. So looking at stations with more than 80 years in the adjusted record, this histogram shows the position of Darwin 0. It is an outlier – 31’st in a list of 2074 stations, ordered by change in gradient. But not so improbable as to say that it had to be adjusted manually. And 17 stations were adjusted downward by the same amount or more that Darwin was adjusted upwards.
You are wrong in saying that the GHCN algorithm would have adjusted the duplicates equally. The algorithm identifies changepoints by looking at whether the difference in slope between the sections on each side is over a certain limit. That slope depends on the length of sample.
Do you still maintain your underlined charge that
“Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.??

Michael
December 20, 2009 10:49 pm

“Santa makes his rounds later this week, but I’ve already received one of the best gifts ever: the complete unmasking of one of the most insidious movements of recent history – the radical effort to force reckless and needless constraints onto the human race in an attempt to change the planet’s climate.”
Mark Davis: The Gift of ‘Climategate’ is a Fresh Start
http://www.dallasnews.com/sharedcontent/dws/dn/opinion/viewpoints/stories/DN-markdavis_1220edi.State.Edition1.1d27b2e.html

Glenn
December 20, 2009 10:52 pm

JP Miller (22:39:57) :
“Willis, I do not understand the two different scales on the left verticla axis and the right vertical axis. They don’t match, should they?”
Read the legend for the heavy black line, at the bottom of the graph.

Willis Eschenbach
December 20, 2009 10:53 pm

Dan Martin (21:42:09) :

Can someone explain to me why in the top graph the temp. anomaly scale is different on the right side from the left side?

Merely for clarity, so the black line (adjustments) doesn’t obstruct the view of what’s been done.

Norm in Calgary
December 20, 2009 10:56 pm

Back to Darwin Zero … If that upward trend continues at some point it would start to look actually silly … nonsensical and YET they could not reverse it could they? Because THAT would be a cooling trend. Gotcha!! ☺ Are “they” LOCKED into this big lie? They cannot reverse it now. Ohoh.
I think they want Copenhagen agreed ASAP because they probably see the temperatures declining in the near future and want to be able to claim that CO2 reduction is the reason. Hence the panic to start CO2 reduction ASAP, before the temperatures drop while CO2 continues to rise. I’m not sure how they would manage to record CO2 dropping enough to make a difference, unless they manipulate the data, but they’d never do that; or they claim the positive multiplier effect also works as a negative multiplier effect in reverse so that a teeny drop could save the world.

Michael
December 20, 2009 10:59 pm

“Global warming has of late been a very hot topic in social media, and last week it was hotter than ever. Much of the added fuel came from climate change believers who engaged in the debate that had been dominated by skeptics.”
No Denying the Heat of Global Warming Debate in the Blogosphere
http://pewresearch.org/pubs/1446/global-warming-debate-on-blogs

Willis Eschenbach
December 20, 2009 11:00 pm

Neil Gibson (21:52:04) :

Hi Anthony,
As an engineer with a lifetime of experience in electrical measurement I cannot understand that in all the discussion I
have not seen the words Quality Assurance mentioned. All of our measurements and instruments had to have traceability and customers would full access to inspect our labs and our measurements. Every engineering firm dealing with Government of Semi-Government had to have similar QA procedures. …

Couldn’t agree more. The computer programs (both climate models and the CRU/GISS/GHCN “adjustment” programs) receive no serious investigation at all. There is a whole branch of computer science which develops and uses tools for software Verification and Validation (V&V) and Software Quality Assurance (SQA). These tools are applied to all mission critical software around the world, software for everything from subways to elevators to submarines to airliners to moon shots.
The fact that we are discussing spending many, many billions of dollars based on untested (and in many cases unknown and unexamined) software is a sick joke. We pay more attention to the software running our subways than to global climate models that make unbelievable claims … how bizarre is that?

Michael
December 20, 2009 11:01 pm

Earth’s upper atmosphere cooling dramatically
New observations may help detect changes from carbon dioxide increases
http://www.msnbc.msn.com/id/34479085/ns/technology_and_science-space/

December 20, 2009 11:09 pm

Doesn’t anybody understand? Only the past changes. The current temperature always stays the same.

Dave F
December 20, 2009 11:15 pm

Nick Stokes (22:46:31) :
I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other. The adjustments are supposedly made to correct data because of spurious readings, not preserve a neutral value in the amount of adjustments. It does not make sense to me that there are just as many stations that read too cold as there are that read too hot, in an equal manner. Why would you expect the readings to be off in both directions in an equal amount? What explanation is there for that?

Paul Vaughan
December 20, 2009 11:15 pm

Re: Nick Stokes (22:46:31)
If your intention is to draw attention to problems with the homogenization paradigm, you have succeeded at least twice with your post. Perhaps I have misunderstood and your intention is to give Willis more material…

David44
December 20, 2009 11:22 pm

@ pat (20:10:47) :
I followed your link through to the pdf of Hessell’s paper and note that in addition to his NZ work, he references in the last paragraph a 1975 paper by Tucker finding in Australia “no clear evidence for long term secular warming or cooling in Australasia over the last 50 years.” Tucker, G.B. 1975: Climate: Is Australia’s changing? Search 6 (8); 323-8.
I suspect these two papers are well known to Willis and Anthony (not to mention Salinger), but certainly new to me. Thanks,
David

Nick Stokes
December 20, 2009 11:24 pm

Dave F
No, the distribution is not perfectly symmetric. The mean trend adjustment is 0.0175 C/decade (upwards). For those >80 year histories, 30 were adjust up more than Darwin. 17 had an adjustment which was bigger but downward.

John in Australia
December 20, 2009 11:25 pm

Wyndham? Wyndham is not only a long way from Darwin, but has a much hotter climate. It is in the Kimberley in north-west Australia. This area is sometimes 5-10 degrees C hotter than Darwin. I went to the Kimberley not so far but inland from Wyndham in October (in the Australian Spring) to a place called the Bungle Bungles NP. When I was there the temperature was well over 40C during the day, while in Darwin it was about 32C at the same time. Once you go inland from Darwin the humidity falls and the temperature climbs. As another example in July (winter), I was in Kakadu NP (say) 300km south-east of Darwin in July. The temperature there was 36C, while it was 30C in Darwin. It is entirely fallacious to modify Darwin temperatures using stations around it, especially those even a short distance inland from the city.

Leon Brozyna
December 20, 2009 11:26 pm

Just wondering if there are other low latitude stations (those in the tropics) in which the unadjusted data might show a similar century long decline in temps. Or even a general unchanged trend. It just seems counterintuitive for temps to be showing a decline to the idea that the planet was recovering from the LIA from the 19th through the 20th century.

Neil Gibson
December 20, 2009 11:32 pm

Hi Willis and Anthony,
The Quality Assurance question is surely a weapon to be used in the debate.
I will write to my local ( skeptic ) pollie and put it to him. It might be easier to question the complete lack of Quality Assurance by “scientists “than to argue about accuracy especially about software given the garbage I have seen.
The lack of accuracy would naturally come out later.

Michael R
December 20, 2009 11:33 pm

RE: Nick Stokes (22:46:31) :
You have attempted to divert the argument to make your own point and then try to falsify your own point.
The original post in relation to this temperature set, is relating to the adjustments made at a single station and the fact that they do not conform to either the proposed list of changes that can be made, nor do the adjustments make any sense – hence the question asking how exactly the adjustments were made (as they do not, as i just said conform to the claimed sequence of adjustments).
Instead, you have taken the argument that “well Darwin shows up others show down therefore whats the issue?”. I am unsure of the point you are trying to make? Are you saying that because others adjust downward then the total effect cancels out and therefore the temp record is reliable? In what way, does showing that adjustments were made to other data nulify the fact that there are issues with this station?
The point of this issue, was not to show that the entire database of temperatures have all been adjusted to show warming, rather that this one has, and it does not have adjustments that make sense. In which case, what proportion of the entire dataset contains adjustments that are wrong (either up or down is irrelevent). The fact that the entire response to this station and its issues was to claim that there are an equal number of adjustments to other stations does not excuse the fact that there are issues and is itself, another attempt to mislead and confuse people reading both articles.

Michael
December 20, 2009 11:36 pm

YouNews: Giant snowball rolled into street in Madison, WI

Reply: I loved both of these. ~ ctm

Michael
December 20, 2009 11:40 pm

This would be a great protest strategy.
Snowmen face down the plows on Gilman Street, Madison, WI
The Great Snowman Wall of Gilman.

Carrick
December 20, 2009 11:45 pm

Nick Stokes, things aren’t quite as copacetic as Giorgio Gilestro implies.
If you look at the trend of the adjustment over time,, it is far from random. This trend in the data are the explanation for the obviously leptokurtic distribution obtained by Gilestro.
This sort of residual test is a rather elementary one that any of us who actually do data analysis are taught to perform. I’m not surprised Gilestro failed to do it (if only because I don’t know him from Adam), but I’m a bit surprised the GHCN guys didn’t address it though. The high amount of kurtosis itself is also a bit of a warning sign—usually it is an indicator than non-random processes are involved in the observed residuals. (Think about the extreme example of histogramming a sine wave… you’d get a symmetric distribution, but something that is very non-gaussian.)

Dave F
December 20, 2009 11:47 pm

Nick Stokes (23:24:30) :
The mean trend adjustment is 0.0175 C/decade (upwards).
So there is basically no difference in the trend if you remove all the adjustments, right? It would look the same either way, basically?

Nick Stokes
December 20, 2009 11:50 pm

David44 (23:22:32) :
Oddly enough, in a much earlier life, I was the number-cruncher for a follow-up study to Brian Tucker’s 1975 paper. Despite the title, we also analysed temperature trends. Again the results were negative. With the better knowledge we have nowadays, that still may have been a correct conclusion for that time. However, that kind of analysis could then only be done much less accurately. GISS temp, which came later, was a huge advance.

Willis Eschenbach
December 20, 2009 11:51 pm

Nick Stokes (22:46:31), thanks for your interesting questions.

Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.

Not sure what your point is here. What does the average have to do with what happened to Darwin?

You say that it is improbable that a station would show an apparent gradient change to a late rise of 6C/century. You give no basis for this statement. Giorgio’s kind of analysis does not help with that figure, but it does indicate how often an adjustment can produce a rise comparable to Darwin’s. The result is more meaningful when restricted to longer series, since a small change to a short series can too easily produce a large trend change. So looking at stations with more than 80 years in the adjusted record, this histogram shows the position of Darwin 0. It is an outlier – 31’st in a list of 2074 stations, ordered by change in gradient. But not so improbable as to say that it had to be adjusted manually. And 17 stations were adjusted downward by the same amount or more that Darwin was adjusted upwards.

I say it is improbable simply because the raw data from Darwin and the sites around it clearly indicate that Darwin is in fact not warming at six degrees per century. I know of no other station on the planet which goes along relatively level for half a century, and then suddenly starts rising at six degrees per century and maintains that trend for a half century.
If you know of one, break it out … but if you don’t know of one, then why don’t you think the claimed Darwin rise is very improbable?
And the fact that the GHCN algorithm may produce other huge adjustments is an indictment of the GHCN method, not of my claim.
In addition, in reading Gilestro (your cite above) you have turned off your skepticism, and that is very dangerous in climate science. Gilestro claims that

(0 is the median adjustment and 0.017 C/decade is the average adjustment – the planet warming trend in the last century has been of about 0.2 C/decade). In other words, most adjustment hardly modify the reading, and the warming and cooling adjustments end up compensating each other.

Nonsense. Both you and he should know better. 0.2C/decade is 2 degrees per century, and the planet warming trend in the last century was nowhere near that, it was only on the order of 0.06C/decade.
Nor do “the warming and cooling adjustments end up compensating each other” as he claims. First, you would need to grid and area-adjust the temperatures to determine that. It is very sensitive to where the stations are located.
Next, even the raw average which he uses (0.017C/decade) is more than a quarter of the estimated warming, which is is a significant amount.
Given those clearly untrue claims of his, I fear that I can’t trust Giorgio’s numbers or conclusions, and neither should you.

You are wrong in saying that the GHCN algorithm would have adjusted the duplicates equally. The algorithm identifies changepoints by looking at whether the difference in slope between the sections on each side is over a certain limit. That slope depends on the length of sample.

Well … in a word, no. Read the description again, the algorithm doesn’t do that at all. The algorithm looks at the change in the total of the residual sum of squares of the trends on both sides of each candidate year, which is a very different beast which does not depend on the length of sample. I suspect that’s one of the reasons that they use that particular algorithm. You really should run the algorithm, as I have in Fig. 2 above, before drawing conclusions.

Do you still maintain your underlined charge that
“Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.??

Most assuredly. The main reason for still maintaining my statement is that there are no suitable stations within 1500 km. of Darwin to use for the 1920 step, so they could not have used the algorithm that they claimed to use. That algorithm requires 5 stations whose average correlation with Darwin is 0.80. For the 1920 adjustment I can’t find a single station, much less five stations, within 1500 miles that has a correlation greater than 0.43 with Darwin. So I can only conclude that they didn’t use the method they claimed to use.
I list a number of other reasons for maintaining my statement above.
All the best,
w.

Janis B.
December 20, 2009 11:56 pm

Nick Stokes (22:46:31) :
Can “Giorgio’s kind of analysis” also indicate why should I trust ANY record in GHCN database at all – whether adjusted or “UNADJUSTED” means? 🙂 If the algorithm ostensibly creates data for Darwin Airport (prior to 1941) out of “thin air” in adjusted means, where is the guarantee that it didn’t calculate/select/concatenate something funny while doing so called “unadjusted” means, for other stations, too?
Had i not come across this blog, I’d never have the slightest idea how much the climate scientists are massaging the data. I think it’s a scandal – in my layman opinion, it can hardly be called ‘instrumental record’. It’s rather an estimation based on instrument record, of difficult to judge reliability.

Willis Eschenbach
December 21, 2009 12:00 am

John in Australia (23:25:58) :

Wyndham? Wyndham is not only a long way from Darwin, but has a much hotter climate. It is in the Kimberley in north-west Australia. This area is sometimes 5-10 degrees C hotter than Darwin. I went to the Kimberley not so far but inland from Wyndham in October (in the Australian Spring) to a place called the Bungle Bungles NP. When I was there the temperature was well over 40C during the day, while in Darwin it was about 32C at the same time. Once you go inland from Darwin the humidity falls and the temperature climbs. As another example in July (winter), I was in Kakadu NP (say) 300km south-east of Darwin in July. The temperature there was 36C, while it was 30C in Darwin. It is entirely fallacious to modify Darwin temperatures using stations around it, especially those even a short distance inland from the city

Aussie John, your intuition and experience is amply supported by the lack of correlation between the Wyndham and Darwin records (first differences of temperatures as used by the GHCN), which is -0.14, in other words, they are not significantly correlated at all …
Which is why Wyndham could not have been used in the GHCN algorithm.
However, it is worth noting that the fact that Wyndham is hotter than Darwin is not the issue. It is that they don’t move in parallel, that there is very little correlation between their swings. If Wyndham warmed when Darwin warmed and cooled when Darwin cooled, we could use it to check Darwin for inhomogeneities. But since it doesn’t, we can’t.

John in NZ
December 21, 2009 12:04 am

Thank you Willis. This is very interesting.
A thought though.
When talking about climate and Darwin, has any one mentioned Cyclone Tracey.?
Darwin was pretty much destroyed in 1974. It looks like a bit of a jump in the record about then.
Is there any mention of change of instruments or location about then?
According to
http://www.enjoy-darwin.com/cyclone-tracy.html
“Wind speeds of 217 km/h registered at Darwin Airport before the equipment was itself blown away and there are estimates of maximum speeds of up to 300 km/h.”

mareeS
December 21, 2009 12:08 am

I suppose this has been said before about Darwin temperature records from 1941 (I have been away and not reading this blog), BUT, Darwin was bombed off the map from Feb 19, 1942, until Nov 12, 1943 by Japanese air raids.
The population withdrew south to Adelaide River and points beyond for years before rebuilding began and any reliable weather service was restored.
Darwin was again smashed on Christmas Day 1974 by an intense cyclone that the local military weather station did not detect.
How could Darwin have adequate and reliable weather records with interruptions and failures such as these?

Mark.R
December 21, 2009 12:13 am

I just dont understand why they make adjustments to the data in the first place. And why are the adjustments almost always up. Should they not be adjusted down in the citys?.

Willis Eschenbach
December 21, 2009 12:23 am

Leon Brozyna (23:26:37) :

Just wondering if there are other low latitude stations (those in the tropics) in which the unadjusted data might show a similar century long decline in temps. Or even a general unchanged trend. It just seems counterintuitive for temps to be showing a decline to the idea that the planet was recovering from the LIA from the 19th through the 20th century.

An interesting issue. In general, the tropics has warmed much less than the extratropics. According to the UAH MSU satellite temperatures, there has been no significant warming of the tropics in the thirty years of the satellite record. This inconvenient fact is usually ignored by AGW supporters, who claim that poor tropical countries will bear the brunt of the hypothesized warming.

Michael R
December 21, 2009 12:31 am

“RE Dave F (23:15:19) : I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other.”
It does, to some extent come across odd. The strangness of this standard deviation model is hard to grasp unless explained very simply – something that this standard deviation article fails to (and certainly would not want you to notice). A link to graphical analysis is here http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/ this should make it easier to follow the issue.
I will simplify.
A standard deviation of adjustments will show only one thing – how many adjustments were made and in which direction. The thing is, noone is interested in how MANY adjustments were made, or whether the DIRECTION was equal. What is missing is TIME.
There is no plot or analysis on Giorgio’s page relating to WHEN the adjustments were made and unfortunately that is the entire crux of the argument.
If you want an example, take a single station that was basically neutral trend. If you were to apply several cooling adjustments to prior to 1950 (say 0.3 degrees), then several warming adjustments post 1950 for the same amount, you would get a standard deviation of 0, but the temperature trend would have gone from stable to cooling slightly until zomg increeased CO2 + 0.6 degrees increase BURN!!!.
In addition, if you had 2 temperature series one had a trend of 0.3+ the other was stable up until 1960 then had a slight upward trend of say 0.1. we apply 3 adjustments down to the lower end of the first one and get an amplified warming up to 0.6 and then apply the upward adjustments to the higher end of the second one and again, amplified warming – this time showing hockey stick – but again with a 0 standard deviation.
In fact, mathematically if you wanted to attempt to hide this increase, the best way to do it would be to apply corrections that give a 0 standard deviation – this gives it the illusion that there is no issue – and that is all that paper is – an illusion designed to give the semblance of a rebuttal, framed in such a way that makes it look indisputable (unless you actually investigate the data).
In the end, as i said previously the entire argument is moot from the get go. This station has adjustments that appear to be uncalled for, are not explained through GHCN’s own adjustments “manual” and show that the adjusted data may well have errors. The only real way to fix that is to reassess the entire database – not allude to some standard deviation model that just confuses the issue (even though warmists are very good at doing so).

Brownedoff
December 21, 2009 12:35 am

Commenters,
When going down a thread and encountering the words “Nick Stokes” – right away, go up to Edit, Find on this page, enter Nick and then every hit is highlighted. This means that when you see the highlight approaching you can quickly scroll through, thereby saving precious time.

Michael
December 21, 2009 12:47 am

If the nation was not led down the path of global warming alarmism, but rather given a warning of a fierce winter season based on true science, perhaps the retailers and cleanup crews could have been put in place to deal with the mess in order to help improve customer traffic and boost Christmas sales. Too bad, many retailers will be going out of business due to the nation not being prepaired.
US Blizzard Puts Crimp in Holiday Sales, Travel
http://au.news.yahoo.com/a/-/world/6608283/us-blizzard-puts-crimp-in-holiday-sales-travel/

supercritical
December 21, 2009 1:07 am

Willis, great work!
minor points;
– third para .. too many ‘independent’s?
– re tthe 1500km limit, they wouldn’t have used data from sites further north would they? (i.e. not in Australia)

Michael
December 21, 2009 1:12 am

Beautiful pictures in this article, like postcard pictures.
Think we have it bad? Try Washington for a whole winter’s snow in one day
http://www.dailymail.co.uk/news/worldnews/article-1237351/Think-bad-Try-Washington-winters-snow-day.html

Jeff B.
December 21, 2009 1:16 am

The reason they brought out the heavy fire from The Economist (which by the way, is also very wrong on our ailing economy) is that they are running scared. This gig is up. The public no longer buys the MSM orchestrated AGW scare story.
Panic time. Yell. Scream. Discredit. Ad Hominem.

Rabe
December 21, 2009 1:17 am

I wonder how average temperatures are calculated. It’s not even ingenious let alone scientific. Why don’t we measure those average temperatures directly? Here is how it would work:
Rock is a very good medium to use so drill a horizontal hole into a mountain. Start at one meter or maybe two. Place a temperature sensor at the end and isolate the hole as good as you can. Wait a week. Measure. Repeat for another depth. The age of the averaged temperature should be proportional to the depth. I guess exponential. We would have to calibrate the depth/age relation according to the property of the material like say, Rock Rings.
The nearer you get to the surface the lesser the attenuation will work and you may see the annual, later dayly temperature oscillation. There you may measure in fixed intervals and integrate numerically.
Stevenson screens, pah.

Mooloo
December 21, 2009 1:19 am

“It is entirely fallacious to modify Darwin temperatures using stations around it, especially those even a short distance inland from the city.”
This might be true, but as I understand it, not for the reasons you give.
If two points have a strong correlation, then it doesn’t matter that they are actually 6° different on any one day, or year, or whatever. An analogy: my sister and I never are the same age, but our ages correlate very strongly. I am prepared to bet she will be 45 when I turn 50.)
I would bet that in the short term a station a short distance inland of Darwin correlates very strongly with Darwin. I think a long term bet on that might fail, due to UHI or other effects.

pete m
December 21, 2009 1:25 am

Nick – most general adjustments are justified to account for the UHI effect. I know of no general reason to say why adjustments should be made the other way. Do you?
If most adjustments are required to reduce the impact of the UHI, then the overall trend in the adjustments should be largely negative.
The fact it is positive is completely bizarre and needs a lot of explaining.

MartinGAtkins
December 21, 2009 1:36 am

I found this. I haven’t had time to explore further yet.
A change in the type of thermometer shelter used at many Australian observation sites in the early 20th century resulted in a sudden drop in recorded temperatures which is entirely spurious.
http://www.bom.gov.au/climate/change/datasets/datasets.shtml
There is a PDF on the same page that says.
other known data problems. Such discontinuities can be as large, or larger than, real temperature changes (eg. the approximate 1°C drop in maximum temperatures associated with the switch to Stevenson screen exposure) and consequently confound the true long-term trend. Generally the high-quality records were homogenised from 1910, by which time most stations are believed to have been equipped with the current standard instrument shelter. 224 temperature records were reconstructed to an acceptable standard, 181 of which were identified as being non-urban.

Ken Seton
December 21, 2009 1:45 am

Go Willis.
I am starting to think that one should never adjust station records (i.e. the real observed data) and continue to call them by the same name as the raw data station. For example, if an “adjusted” Darwin0 is required for some purpose – then the resulting series – homogenized, stretched, smeared or tilted should always be called by a name that identifies it as such. For example, DarwinHSST023, where such name can be looked up and shown to represent Darwin Homogenized Stretched Smeared Tilted algorithm 023. Thus, whereas Darwin 0 represents a particular station at a particular location between two dates, and never changes (it is after all – within other constraints – real data), the DarwinHSST023 might (say) represent some sort of area algorithm incorporating data from station thousand(s) of km away. This is possibly acceptable if you state its method and use. What is not acceptable is to pretend (by same-name retention) that some HSST actually IS the correct temperature picture of Darwin – rather than what we know was the actually measured temperature. I don’t know why we let them get away with it.

December 21, 2009 1:51 am

S.1733 Cap-and-Tax Energy Inflation bill text: http://www.opencongress.org/bill/111-s1733/text articles, blog, and can comment on the text of the bill line by line.

Peter
December 21, 2009 2:29 am

AGW proponents have to make a decision now that all the evidence they used to support their case has been discredited and debunked. Either they keep on with the mantra without any evidence in which case they are simply acting like clueless children, or they admit they have very likely got it wrong and they have to go back to square one and try to find evidence to support their case. If you are one of them, which one is it for you?

JustPassing
December 21, 2009 2:36 am

Has anyone wondered if the 1908 Tunguska comet explosion in Siberia could have effected measurements in some way. This article mentions the comet creating Noctilucent clouds forming a day later thousands of miles away.
http://insciences.org/article.php?article_id=5899

john east
December 21, 2009 2:38 am

A few posters have pointed out that the scam will be self defeating over time once it becomes evident that the scare mongering projections have not come to pass. However, I doubt things will turn out that way.
In ten or twenty years, or whenever the evidence of our own eyes refutes the warmists prophecies, the world will be a different place. The UN will have set up its monitoring, tax collection, and enforcement regime, or as I like to think of it “Proto-world government”, the MSM will have moved on, climate scientists will have learned their lessons (no more stray emails/more cunningly fudged data), and very frustrated and angry ruling elites will have neutered the internet.
Our only hope is to kill all of this today before it becomes entrenched.

Rob Vermeulen
December 21, 2009 2:42 am

Following weather undeground, Darwin is about to smash its previous record temperature for December (the previous record dates back to 2001).
Is weather undeground also adjusting its values?

Geoff Sherrington
December 21, 2009 2:58 am

Nick Stokes (22:46:31) : on 20/12
GG’s near-symmetric sort of Poisson distribution:
The weakesses with this study are several fold. One is evident from a station in an older city with UHI. In Melbourne for example, various comparisons with other sites show it likely that UHI ‘maxed out’ at Melbourne Central some time before 1950. That is, from humble beginnings, the temperature from UHI rose steadily to (say) the 1930s and then remained about constant because nothing was explainable or available to push the UHI much higher. So Melbourne would be one of the stations in an 80-year plot that whould be near the median, despite having UHI (from studies already referenced) of up to 10 deg C.
Depending on the geographic location of the thermometer with respect to development around it, there must be a host of other cities in the gg data set that show that same “false” response. That is, one part of the comparison between adjusted and unadjusted is incomplete.
I take you pount that one swallow does not make a Spring – but I am using a town familiar to me to reduce the likelihood of spurious effects unknown to me. I would strongly suspect that Adelaide and Sydney have the same problem. Even a 100-year analysis might fail to pick up the main part of UHI.
One swallow might make a tablet go down better.

niphredilflower
December 21, 2009 3:02 am

Your accurate analysis has been invalidated by governement computer models and a global consensus, therefore your work remains accurate but unacceptable.
Is it possible to incorrect something to make it ‘correct’? In anycase make sure you continue not doing this, but I apologise that this is simply not acceptable!
Good work Willis (the unacceptable)

JB Williamson
December 21, 2009 3:18 am

Dr David Bellamy, well known BBC presenter (who no longer seems to appear on the BBC) has written to the Daily Telegraph thus…
SIR – The only good news to come out of Copenhagen is that, in the words of Greenpeace: “There are no targets for carbon cuts and no agreement on a legally binding treaty.”
Hooray! Along with tens of thousands of global-warming sceptics, the world can now breathe a sigh of relief and return to the sanity of real science, which counsels that carbon dioxide is not a poison, let alone likely to cause a heat-driven Armageddon.
We can now burn non-sulphurous coal again to ameliorate the effects of the colder climate that has already been with us for the past decade and is likely to stay for the next 30 years.
Dr David Bellamy
Bedburn, Co Durham
http://www.telegraph.co.uk/comment/letters/6851649/After-a-decade-of-global-cooling-we-should-sigh-with-relief-at-Copenhagens-failure.html

rbateman
December 21, 2009 3:42 am

Michael (00:47:30) :
An excellent case example of the end result of AGW driving public policy the wrong way on the freeway. The MSM is glued to the Agenda of pumping as much Global Warming through Going Green advertising to an obsession with “Scientists find more evidence of Global Warming”. What the MSM should be reporting is the increasing cold & snows of Southern and Northern Hemisphere winters.
i.e. – the MSM should be focused on what is going on rather than what is being wrongly predicted year after year after year.

Nick Stokes
December 21, 2009 3:49 am

Willis Eschenbach (23:51:16) :
The reason why I believe that GG’s histogram is highly relevant goes back to your reaffirmed statement

they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.

You can’t change the global record by changing one station. If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception.
Anyway, I’ve collected the argument in a post on my blog site. I’ve also calculated the statistics of trends since 1940, to show that Darwin’s post-adjustment trend is in the upper tail, but not extreme. And it would not stand out among the unadjusted station data either.

Nick Stokes
December 21, 2009 3:58 am

pete m (01:25:23) :
GHCN homogenization does not adjust for UHI. It corrects for discrete events that it detects by time series analysis. Things like station moves, equipment changes, MMTS introduction. There’s no particular reason to expect those to go up or down on balance, but no reason not to either.
Someone mentioned Bethell’s paper upthread, in which he said back in the ’70’s that some noted NZ stations had decaying Stevenson screens. Well, those would have been replaced in the years since. Each replacement would have caused an adjustment. It would say that there was spurious warming in the past (screen decay), and push the readings down. etc

Aligner
December 21, 2009 4:06 am

O/T: Mayon looking ominous, raised to alert level 4 yesterday.
Volcano Live
Guardian
Times
ABC News

Nick Stokes
December 21, 2009 4:09 am

Carrick (23:45:16) :
Check the comments at Romanm’s blog for discussion of his diagram. The pattern that he shows is what you expect, because if plots the cumulative effect of adjustments as you go back in time. And the slope is about equal to the 0.0175 C/decade of GG. As I said there, they represent two different variable plots of the same data. If you take trends over time, and look at the distribution over stations, you get GG’s. If you average over stations and plot over time, you get Roman’s. The trend of Roman’s is equal to (approx) the average of GG’s.

Nick Stokes
December 21, 2009 4:16 am

Michael R (23:33:12) :
I am unsure of the point you are trying to make? Are you saying that because others adjust downward then the total effect cancels out and therefore the temp record is reliable? In what way, does showing that adjustments were made to other data nulify the fact that there are issues with this station?

No. I think that the adjustments are made for a good reason. I’m answering the suspicion that they are done to deliberately enhance warming. The fact that the nett trend is small means that they don’t have that effect (or only to a small extent). Crudely, if someone was trying to rig the effect, they aren’t doing very well.

Max
December 21, 2009 4:17 am

Did the Met office ever release the global datasets of raw temperature?

RoHa
December 21, 2009 4:31 am

It’s actually quite simple if you stop and think about it.
Darwin is in Australia. Things are reversed here. Christmas is in midsummer. It gets cooler as you go South. The water goes down the toilet in the opposite direction.
So when the raw data show temperature going down, the graph should be inverted to match Northern Hemisphere norms.
Isn’t that standard practice for climate scientists?

Ox-FordPrefect
December 21, 2009 4:46 am

Is there a record of the number of corrections for inhomogeneity and what direction these are in? An assessment of the formal guidelines used should be revealing – are there more reasons to adjust data upwards than down?

Rob
December 21, 2009 4:46 am

The GHCN, GISS, HadCrut and all there are tries to find the trend line in any global temperature mean. Why would that require any adjustments at all apart from those trying to adjust for biases induced by human activity. That would be solely UHI-effects or land use changes I would assert.
For example, a specific station time of observation would have a trend line which would look the same or at least as much erroneous regardless of the time of day it was observed? The trend for a 3 o clock observation would contain as much global warming as any trend for a 5 o clock series of observation, wouldn’t it?
I try to read all the rationale for adjustments but cannot make any sense out of it. The gridding for example, what is that for? The neighboring station mumbo-jumbo, how is that justified? It seems that when the task is to try to find a trend line in a global mean temperature, all the data should be included since any error in the data would go in both direction. The Darwin station for example, wouldn’t there be a station somewhere in the world that has a funny up spike in 1939 the compensates the Darwin down spike?
As a long time engineer, I try to find KISS solutions an all similar problems, the “state of the art ” science in this field has complicated matters beyond recognition.
I would love to be educated on this.

Ed Moran
December 21, 2009 4:51 am

Great work!!
Has there been any reaction to your open letter to the Ecomomist?

Rob
December 21, 2009 4:53 am

Amendment:
I read the abstract of the Peterson paper, it seems to me that the method described therein is to be able to find the actual temperatures, i.e. the absolute values that are comparable with other stations, when such comparison is needed/wanted/called for, it wouldn’t apply to finding high pass filtered temperature trends, why would it?
Maybe I am jumping to conclusions, I always do that…

Phil A
December 21, 2009 4:53 am

“You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. ”
Cogent? GG appears to believe that if a merchant pushes down on one arm of the scales as heavily as he pulls up on the other, that that’s a fair transaction.
Someone else posted the graph above from RomanM’s analysis, here:
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/
How you can look at that timewise adjustment ‘trend’ and pretend this is innocent is beyond me.

Antony
December 21, 2009 5:08 am

@Ken Seton
Station records that are unclear labeled resemble GM food in the US and Canada:
from the label you do NOT know what you are dealing with.

Gail Combs
December 21, 2009 5:40 am

Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up….

YOU miss the point entirely. There is absolutely no reason to adjust these records period. There is no reason to adjust ANY of the records unless a very good VALIDATED reason exists. Saying some pie in the sky method was used to do the adjustment does not justify tampering with the data.
I ran a quality lab. If I look at graphs of chemical analysis and find one does not “match” the typical data does that mean I have reason to “adjust ” the data or do I go look for the actual cause? If I do not find a problem with calibration, sampling method or whatever am I justified in arbitrarily “correcting” the data. Do I place the corrected numbers on the Certificate of Analysis or do I take it the data is telling me something – like the product should be rejected.

3x2
December 21, 2009 5:46 am

I’m still left with a couple of questions.
For those who say that individual stations don’t matter, the global stats will sort things out
If, when scaled globally, local “adjustments” have no real impact … why bother “adjusting” in the first place? Adjustments at the individual station level obviously do impact both regional and global results. Especially if the end result is in the 1/10th degree range.
Specifically for Darwin adjustments.
If you take the difference between the “winter” and “summer” means over the entire record (25/28 from v2.mean) as being about 3°C. Is a 2.25°C “adjustment”, in any direction, at any point, even physically possible?
To put the level of Darwin adjustment in context. If you applied it to Helsinki the “Darwin adjustment” level would be around 17°C. ({v2.mean} ((summer 17) – (winter -6))*0.75) over almost 200 years.
Both adjustments seem pretty much impossible.
(real Helsinki adjustment looks to be (peak/trough) +1.25 to -0.7 (in nice clear edged steps) over 200 years – pretty much neutral overall)

Stacey
December 21, 2009 6:01 am

Dear Mr Eschenbach
Another erudite post please keep up the good work. Is there any chance of you revisiting your work on the CET record?
Sorry my hobby horse as always been that a global temperature as such is a nonsense given the spread of data. Howver temperature records of individual countries could show if an area is warming or cooling.
Your forensic work has the beauty of showing that hockey stick graphs are being mad up out of measured temperatures, just as in New Zealand.
In the past four weeks, we have an area of Australia where a hockey stick can be demonstrated to have been manufacture, Russia where they say that much data has been ignored and badly manipultated, New Zealand.
You couldn’t make this up, they did.

Jim
December 21, 2009 6:08 am

*************
Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
***********************
Nick: An adjustment histogram just hides the actual adjustments – it is meaningless. Why? Because WHEN the adjustment is made is key. If most of the downward adjustments are made before half the time of the total length of the record and most of the upwards adjustments after half the time, then the trend up will be enhanced. I’m really tired of this statistical shell game and this isn’t the first time I’ve seen these histograms trotted out as proof of fairness. They are bogus and designed to deceive.

Curiousgeorge
December 21, 2009 6:30 am

Your’s and other independent analysis will soon be irrelevant if Gordon Brown and other world leaders have their way.
Excerpt:
A new global body dedicated to environmental stewardship is needed to prevent a repeat of the deadlock which undermined the Copenhagen climate change summit, Gordon Brown will say tomorrow.
The UN’s consensual method of negotiation, which requires all 192 countries to reach agreement, needs to be reformed to ensure that the will of the majority prevails, he feels.
The Prime Minister will say: “Never again should we face the deadlock that threatened to pull down those talks. Never again should we let a global deal to move towards a greener future be held to ransom by only a handful of countries. One of the frustrations for me was the lack of a global body with the sole responsibility for environmental stewardship.
“I believe that in 2010 we will need to look at reforming our international institutions to meet the common challenges we face as a global community.” The summit failed to produce a political agreement among all the countries. Delegates instead passed a motion on Saturday “taking note” of an accord drawn up the night before by five countries: the US, China, India, Brazil and South Africa. ”
End quote
Gordon Brown (UK ) is following the “Belgrade Process” script. Right on schedule. http://www.timesonline.co.uk/tol/news/uk/article6963482.ece#… .

JonesII
December 21, 2009 6:41 am

So Aussies did their own..I would not say “Hockey Stick” but “Kangaroo Tail”

Mike G
December 21, 2009 6:45 am

Glenn (19:50:45) :
Ah. That would make the region the fastest warming area claim in the world, if memory serves, topping the Arctic and Siberia. About 3 times the global surface estimate in any event.
Last I heard, all the warmining the warmists were claiming was 0.6. So, this would be 10 X that.

Gail Combs
December 21, 2009 7:08 am

Norm in Calgary (22:56:52) :
…. I’m not sure how they would manage to record CO2 dropping enough to make a difference, unless they manipulate the data, but they’d never do that; or they claim the positive multiplier effect also works as a negative multiplier effect in reverse so that a teeny drop could save the world.
That is REAL easy. Mauna Loa has active vents putting out CO2. The official CO2 numbers we get are “calculated” to remove the amount of CO2 contributed by the volcano.

KeithGuy
December 21, 2009 7:13 am

Nick Stokes (22:46:31) :
“I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other.
When you say ‘balance out’, I presume that you mean over the entire instrumental temperature history. ”
Couldn’t you get a balancing out even if, let’s say, the pre 1960s (I choice that year arbitrarily) temperatures were adjusted significantly lower and the post 1960s were adjusted significantly higher?
Has anyone tested this possibility?

MartinGAtkins
December 21, 2009 7:22 am

Bold are my highlight, italics my comments.
1120593115.txt
From: Phil Jones xxxxxxxx
To: John Christy xxxxxxxx
Subject: This and that
Date: Tue Jul 5 15:51:55 2005
Also this load of rubbish!
This is from an Australian at BMRC (not Neville Nicholls). It began from the attached article. What an idiot. The scientific community would come down on me in no uncertain terms if I said the world had cooled from 1998. OK it has but it is only 7 years of data and it isn’t statistically significant.
The Australian also alerted me to this blogging! I think this is the term! Luckily
I don’t live in Australia.
[1]http://mustelid.blogspot.com/2005/06/first-look-at-scs-msu-vn52.html
Unlike the UK, the public in Australia is very very naive about climate change, mostly because of our governments Kyoto stance, and because there is a proliferation of people with no climate knowledge at all that are prepared to do the gov bidding.
Australian farmers and Aboriginal people know all about floods and droughts. This urban [snip] wants us to believe it’s caused by cow farts.
Hence the general populace is at best confused, and at worst, antagonistic about climate change – for instance, at a recent rural meeting on drought, attended by politicians and around 2000 farmers, a Qld collegue – Dr Roger Stone – spoke about drought from a climatologist point of view, and suggested that climate change may be playing a role in Australias continuing drought+water problem. He was booed and heckled (and unfortunately some politicians applauded when this happened) – that’s what we’re dealing with due to columists such as the one I sent to you.
Need I say more? The useless [snip] thinks he can teach Australians about the climate of a continent he has never set foot in.

Richard M
December 21, 2009 7:23 am

It appears Nick Stokes only answers the easy questions. I think that pretty much demonstrates his INTENT. His only reason for being here is to insert some measure of doubt. Sorry, Nick, not working.
Yes, Darwin is a single station … but … we have all of New Zealand in doubt, we have all of Scandinavia in doubt, we have most of Russia in doubt and it seems just about every time anyone looks at a new location it has been adjusted upward by a questionable amount.
I don’t think there’s much doubt left anymore.

JJ
December 21, 2009 7:28 am

Good work, Willis.
Some issues that require reconsideration:
“Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.”
GHCN treats these as ‘duplicate’ station records, not as separate stations. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
“While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way.”
That is not true. The GHCN algorithm could easily operate differently on different records, even if they are very close (or perhaps even if they are identical) in their values.
The records are of different lengths. This could easily change which ‘neighboring’ stations correlate with the records and thus which stations are used for the identification of discontinuities and application of adjustments. Even when the same stations are used for the adjustment, record length may also affect how discontinuities are identified and how adjustments are applied. You may not yet conlcude that the standard GHCN adjustment proceedure was not correctly applied.
“Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert.”
You repeat your mistake of the first analysis, by applying your own criteria for suitability, rather than GHCNs. GHCN does not appear to have any specific distance limit. GHCN appears to only be concerned if the stations are in the same ‘region’ (climatalogically) . You cannot say that GHCN did not apply its standard method. You need to ask.
“Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:”
Which Darwin first difference? The correlation between any given station and any of the five Darwin duplicate records may be very different. Which one of the five do you report here? What do the other four look like?
“Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.”
So?
The assertion is that stations within the same ‘region’ are sufficient for use as reference stations, when making homogenizations of datasets desgned to be used at ‘regional’ or larger scales. You have not addressed that, not that it matters one whit to your claim that the adjustment wasnt applied correctly.
“1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).”
True. This is a potential indictment of the adjustment method, not proof that the adjustment wasnt properly applied. Note that to serve as an attack on the adjustment method, you would need to demonstrate that the adjustment method was properly applied here, and that the adjustment had a significant effect on the temperature measurement *at regional and larger scales*.
Recall from our previous discussions that the GHCN method specifically allows for oddball results that do not track local temps. They assert that these weird adjustments are rare (i.e. ‘highly improbable’ results are consistent with the methodology), and have little if any effect on the aggregate results.
“2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).”
Simply put, you have not shown that. You conclusion is not warranted.
“3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).”
Are you sure you applied that completely and correctly?
“4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).”
I believe that you may be incorrect about that.
“5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. ”
Sense with respect to what? If your claim is that the GHCN methodology was not applied, then you must prove that they do not make sense with respect to the GHCN methodology. You have not done that.
If your claim is that the GHCN methodology is not sufficient for estimating long term trends at large scales (regional and larger), then you need to prove that the adjustments do not make sense with respect to estimating long term trends at large scales (regional and larger). You have not done that either.
At this point, all you have proven is that they do not make sense to you. If they do make sense with respect to the methodology and its stated goals, then all you have proven is your own ignorance.
“Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value?”
When did they average them to get an adjusted value?
“The reason you adjust is because records don’t agree, not to make them disagree.”
That may be a successful line of attack – you have to demonstrate the effect on the aggregate result first.
“And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.”
The datasets are not nearly identical. They differ greatly in length.
“So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims.”
Once again, you are over reaching. Your conclusions are not properly supported. As shown above, they are founded in part on your (potentially valid) criticism of the GHCN method, which does not demonstrate that the method was not applied correctly. You need to learn to separate those two issues.
As shown above, your conclusions regarding manual adjustment are also founded in part on your own ignorance regardin some aspect of the method. You really need to ask GHCN to document the specific adjustment for these station records, and replicate them yourself, before making such claims.
“I’m happy to be proven wrong, …”
I doubt that. Being proven wrong would be very embarassing to you. And the rest of us.
“… and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.”
Uh, yeah. Thats what GHCN does. Troll all of the blogs on the planet, looking to see if anyone is talking about them. And then rush out to defend themself against yahoos that call them criminals.
If you have a question (and you do) of GHCN – ASK.
“Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.”
Over reaching and illegitimate accusation of wrongdoing. Invites another spanking, such as you received from the Economist. Stop setting yourself up… and the rest of us by extension.

Gail Combs
December 21, 2009 7:33 am

Rabe (01:17:17) : said
I wonder how average temperatures are calculated. It’s not even ingenious let alone scientific. Why don’t we measure those average temperatures directly? Here is how it would work:….
Yes I have always wondered why they do not use the temperature in caves. I remember from my college caving days Indiana caves are 54.5 F year round, while Texas is around 70 and England was somewhere in the 40’s.
I wonder if the caving societies around the world would have records. I helped collect cave fauna for a caving buddies PhD Thesis and I am pretty sure the temperature was one of the data points recorded. This was in 1969 to 1972.

3x2
December 21, 2009 7:37 am

The more threads I read re: anything factual, the more I see how some individuals tour the “sites” and simply throw tear gas canisters into the discussion.
No. I think that the adjustments are made for a good reason. I’m answering the suspicion that they are done to deliberately enhance warming. The fact that the nett trend is small means that they don’t have that effect (or only to a small extent). Crudely, if someone was trying to rig the effect, they aren’t doing very well.
The net trend in DARWIN (the station under the microscope here) is anything but “small” we are talking here about an adjustment of almost the difference between “summer” and “winter” not “oh we painted the SS the wrong colour” or “oh we moved site from PO to airport”.
I don’t believe for one minute that there is one conspiracy looking to “rig the game” but it often looks like a hive mind trying desperately to “run the game into extra time”
Don’t get me wrong here – where somebody (a critic?) has pointed to a paper that sheds light on the subject in hand (DARWIN), I am more than willing to read. My problem is that I see why Willis started all this thread and all most of the critics have done is to point to the chirping cricket under the elephant so far.

Henry chance
December 21, 2009 7:40 am

Mann Jones scam has only been exposed widely for 30 days. It will spread and crumble further.
It will take a huge surge or crusade for them to even partially recover.
The quality control in the gathering recording business for the Meth, CRU and NASA is just not there.

imapopulist
December 21, 2009 7:42 am

If I were an Aussie, I would be very perturbed about now.
A homogeneity adjustment of over 2 degrees centigrade in a stair step fashion of about one half degree every decade? Right!
How can there be any reasonable explanation other than it has been intentionally manipulated?
Now do other stations indicate similar adjustments have been made?
How has this station adjustment affected temperature statistics for the continent?

Gail Combs
December 21, 2009 7:46 am

Rob Vermeulen (02:42:36) :
Is weather undeground also adjusting its values?
Rob, This summer, I found that the high and low temp for my area, as reported by Weather Underground, changed by 2-3 F (higher) when reported as the high and low for the area the next day. There is no way I could have missed the low since I get up before dawn, check the weather and then go feed my animals.

JJ
December 21, 2009 7:49 am

Nick Stokes:
“If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception.”
Not necessarily true.
These data have not only magnitude, but a spatial and temporal component. That the adjustments have a nearly equal distribution in magnitude does not demonstrate that they have an equal temporal distribtion, or an equal spatial distribution.
It is entirely possible, in fact likely, that data represented in GG’s graph could have the ‘ups and downs’ at opposite ends of the time period, and/or that the ‘ups’ may be applied to larger areas in the area-weighted, aggregate temp estimate. The analogy above of a push up on one end of the balance beam and an equal push down on the other end is an apt illustration.
Another unwarrented conclusion…

wobble
December 21, 2009 7:53 am

Nick Stokes,
GG doesn’t prove anything.
If Darwin’s adjustments don’t adhere to Peterson then GG would need to show that the offsetting cooling adjustments ALSO don’t adhere to Peterson. If the offsetting cooling adjustments DO adhere to Peterson, then it’s obvious we have a problem.
In fact, we have a problem even if the cooling adjustments don’t adhere to Peterson – wouldn’t you agree?

Gail Combs
December 21, 2009 7:56 am

Ox-FordPrefect (04:46:08) :
Is there a record of the number of corrections for inhomogeneity and what direction these are in? An assessment of the formal guidelines used should be revealing – are there more reasons to adjust data upwards than down?
As others above stated, you can have a zero net adjustment and cause a large change in the graphs. Just adjust a reading before 1950 DOWN by the same amount you adjust a reading after 1950 UP. The net adjustment is zero but you lowered the first half of the graph and raised the second giving a flat line graph of raw data a nice upward trend so you can yell GLOBAL WARMING.

beng
December 21, 2009 8:09 am

*******
21 12 2009
Geoff Sherrington (02:58:12) :
The weakesses with this study are several fold. One is evident from a station in an older city with UHI. In Melbourne for example, various comparisons with other sites show it likely that UHI ‘maxed out’ at Melbourne Central some time before 1950. That is, from humble beginnings, the temperature from UHI rose steadily to (say) the 1930s and then remained about constant because nothing was explainable or available to push the UHI much higher. So Melbourne would be one of the stations in an 80-year plot that whould be near the median, despite having UHI (from studies already referenced) of up to 10 deg C.
*******
Geoff, I don’t think UHI effects will “max out” in this way in most situations. Even if an urban area becomes static, if the surrounding suburban areas continue to build up, this would still increase (admittedly by a smaller amount) the UHI effect in the center area by increasing the temps of incoming air toward it. One exception might be under conditions of no wind or no surrounding growth — then the effect might max out.
So, tho I agree the UHI in the urban center would increase at a lesser rate (how much, I don’t know) once it’s fairly static, I don’t think it would max out if surrounding areas continue to build up.

KevinM
December 21, 2009 8:11 am

Nick Stokes:
Prove it! somebody show me one plot from one station with data that is downloadable and verifiable and shows the opposite pattern.
Get the data Nick, if it exists.

amicus curiae
December 21, 2009 8:16 am

well its 14C outside a few days off Xmas in aus..hmm
it could be expected to be 25. instead I am heating wheat bags for arthritic aches from the cold..
and if Mayon goes boom..I guess I wont be bothering to plant melons for a few summers

Entrecuesto
December 21, 2009 8:29 am

Nick Stokes
I find your histograms very interesting. Especially the unadjusted and
the adjusted ones at the bottom of the page. Could you please also indicate
in your blog the mean, mode and median of both histograms?. Or even better,
could you subtract both histograms so we can see what we are left with?

Bruce
December 21, 2009 8:34 am

Nick Stokes, isn’t the reality that the cooling adjustments were done to old records and warming adjustments done to newer records creating the illusion of warming when there was none?

KeithGuy
December 21, 2009 8:36 am

When considering the possibility for bias in the GHCN temperature reconstruction, which would not affect the overall temperature balance
JJ (07:49:00) described it clearly earlier:
“These data have not only magnitude, but a spatial and temporal component. That the adjustments have a nearly equal distribution in magnitude does not demonstrate that they have an equal temporal distribution, or an equal spatial distribution.”
I understand the importance of considering the temporal component, which is dealt with well in this article:
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/
Am I right to assume the possibility of the following bias:
Is there another important possibility of bias? Since one remote station has more weight attached to it than a number of stations grouped together. A large adjustment upwards on the remote station can be ‘averaged out’ by several small movements down on the grouped station but the effect it has when creating the temperature profile for a geographical area is significantly larger (I’m assuming that GHCN use a method similar to the grid system used by GISS to provide a global temperature metric.)
Could this be the motivation for adjusting Darwin’s temperature history with a view to the creation of a warming profile?

3x2
December 21, 2009 8:47 am

(somebody mentioned DB earlier in the thread)
I don’t have Dr Bellamy’s e-mail address so I will post this message here.
Dr David Bellamy
Bedburn, Co Durham
Dr. Bellamy,
Stick to your guns.
I (only slightly younger) have very fond memories of your work. You have always been an excellent bridge between “hard science” and the “public”. Opening up worlds that, without you pointing them out, most of the “public” would just never see or care about. Now they do, thanks to you.
It is a real travesty that the current set of “bong heads” over at the BBC do not recognise that you have done more for the real “environment”, at least in the UK, than most any individual has. Don’t worry though, you have many allies.
Can’t help you much with the science but if you need me to go ’round the BBC and kick (Dr. M boots wise) some sense into one of their current “environment workshops” and flush their Jamaican weed down the nearest multi-cultural, multi-sexual WC- just give me a call.
Like I say – Stick to your guns.
(In the mean time why not post your views over here at WUWT?)

Dave F
December 21, 2009 8:49 am

Nick Stokes (04:09:42) :
And the slope is about equal to the 0.0175 C/decade of GG.
Your point should be relatively simple to prove visually. Plot the adjusted data next to the unadjusted data and the trend should be .0175 C/dec different.

Larry K
December 21, 2009 8:55 am

— Nick Stokes wrote : If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception. —
This is simply not the case as (Michael R 00:31:00) post showed. Just because the overall averaged temperature adjustment = 0.0C, it doesn’t mean the temperature data wasn’t skewed to make it look like there is a warming trend. When the adjustments were made makes all the difference in the world.
For example, if one averaged a 20-year period like 1921 – 1940 downward -1.0C, and then averaged the 20-year period after that, 1941 – 1960 upward +1.0C. While the overall average adjustment = 0.0C, one has skewed the temperature data and introduced a 2.0C warming trend into the data for the overall 40 year period. If the 40 year temperature data for 1921- 1960 previously showed a flat trend, it now shows a 2.0C warming trend where there was none before.
(moderator please delete my previous post, it got messed up due to formatting)

Gail Combs
December 21, 2009 9:05 am

Let me see if I have this correct. To determine the global temperature, climate scientists have cut the earth up into grids. They then use mysterious mathematical manipulations to prepare “raw data” so a temperature can be assigned to each grid and use that temperature to compute the global temp.
GHCN makes adjustments to the raw Darwin data according to these mysterious mathematical manipulation techniques presumably to prepare it for use in the global grids and then reports this as the temperature for the Darwin location instead of using the raw data.
Nick Stokes is defending the adjustments made to the Darwin data by saying they were made according to the mysterious mathematical manipulation techniques of GHCN and therefore were justified while Willis Eschenbach has analyzed the data set for Darwin and see no justification for any adjustments to be made at all. Nick Stokes then states the mysterious mathematical manipulation techniques called GHCN homogenization “corrects for discrete events that it detects by time series analysis. Things like station moves, equipment changes, MMTS introduction.” (So I guess I was wrong it is not adjusted so it helps represent a grid area)
But if that is the case and if homogenization is supposed to detect and correct for “discrete events” then why does fig 3 raw data look completely reasonable and figure 3 adjusted data look like a lot of noise has been added to the data and not filtered out? If there were “discrete events” requiring “adjustment I would expect the graphs to be reversed.

joe
December 21, 2009 9:07 am

There is an error in the picture:
The left and the right labels are not equal.

Billy
December 21, 2009 9:14 am

Can’t this issue be easily resolved by just looking at some tree ring, mussel shell and sediment proxies in order to determine what the true temperature was at Darwin Zero over the last 50 years? I mean according to the chart at the top of this page,
http://www.usatoday.com/weather/climate/2006-06-01-wine-warming_x.htm
proxies can measure temps from 2000 years ago to the nearest 1/10000th of a degree whereas modern day thermometers can only measure today’s temps to the nearest 1/1000th of a degree. And you can believe it because it’s “based on scientific analysis”.
Heck, I don’t even know why we bother with thermometers and billion dollar satellites at all.

Jim
December 21, 2009 9:15 am

I’ve seen this same “histogram of adjustments” approach on other blogs. Do these people have a play book or do they just copy each other? It’s time to call a spade a spade. This one is pure BS meant to divert the discussion from more fruitful paths. I think the best idea is to just ignore them.

Marc
December 21, 2009 9:16 am

When I look at the homogenization technic, some questions come to mind.
Why do they use yearly temperatures?
If the temperature stepped at some point, it should be possible to see it in the day by day temperature.
Have they proven that isolated hot spot are equally likely than isolated cold spot on the planet?
If isolated cold spot are more likely on the planet, the technic could generate a false warming signal.

DR
December 21, 2009 9:21 am

“The sum of anecdotes is not data.” -Roger Brinner
One could only imagine if we who work in metrology (not to be confused with the other -ology) applied the techniques for measurement of temperature to that of measuring engine components, how various engine parts would fit together.
Sorry Nick Stokes and other GHCN apologists, but from a metrology perspective, derived temperature values can’t even be considered data. It is a complete joke and goes against every principle in basic metrology concepts.

bill
December 21, 2009 9:28 am
who cares
December 21, 2009 9:44 am

Haven’t checked all the comments, but maybe this is useful.
I have found, but misplaced, a study in google scholar where there was an analysys of the impact of changing from averaging max and min temperature (probably from mercury and alcohol thermomethers) to using regular timed temperatures. Nevertheless that change was in 1995, so it cannot be used to justify previous adjustments.
On the other hand on the thread about the economist I left a link to a study on the adjustments to radiosondes and how the first attempts on purely manual adjustments “failled” I think you will find many interetsing tricks of the trade from the usual suspects.
best regards

Bill Illis
December 21, 2009 9:44 am

Other sources using this data confirm Willis charts.
The raw data from GISS.
http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=501941200000&data_set=0&num_neighbors=1
(I can’t be sure but there is long enough overlap between individual station locations in this chart that no adjustments should have been made.)
And then the ClimateExplorer lets one look at and download the monthly adjusted GHCN data for all/most of the stations in the network.
Here are the individual stations (4771 with a handy Get Data link where you can also download the monthly GHCN data) .
http://climexp.knmi.nl/allstations.cgi?someone@somewhere+temperature+12
Here is the adjusted monthly data for Darwin (+2.0C increase in the trend compared to GISS’ unhomogenized data).
http://climexp.knmi.nl/gettemp.cgi?someone@somewhere+94120+DARWIN_AIRPOR+

bill
December 21, 2009 9:58 am

DR (09:21:43) :
…from a metrology perspective, derived temperature values can’t even be considered data. It is a complete joke and goes against every principle in basic metrology concepts.
Many say you must adjust for UHI
It is obvious you must correct for instrument change
It is obvious you must correct for location change
To me there is an obvious error in max/min/average due to time of observation. Using a continuous reading electronic thermometer max and min can be “easily” determined. A once/twice daily reading is wrong perhaps 3 consecutive days
So should adjustments be made?
The record we have is all there is – warts and all.
There is no way to improve this. For future a few millions spent on new equipment re-sited to ideal locations (some of these MUST be in cities and near roads as these are a valid part of the environment).
To get a VALID check on global temperatures will take at least another 150 years and more. If there is a problem with AGW then you have consigned large populated areas to the scrap heap.
What is the way you suggest that we take?
Just for info (I’m sure you will shout INVALID) here are a few UK places and de Bilt. No adjustments made to individual UK stations This shows the typical warming response.
http://img410.imageshack.us/img410/8996/ukspaghetti.jpg

Bill Illis
December 21, 2009 10:01 am

You can also get the annual GHCN adjusted data for Darwin
At the link (Click on “Make Time Series” at the bottom of the page and the annual data will be shown).
http://climexp.knmi.nl/gettemp.cgi?someone@somewhere+94120+DARWIN_AIRPOR+

pat
December 21, 2009 10:20 am

It is extraordinary bad science to do any adjustment to thermometer readings. If there is a noted environmental or instumentational alteration to the readings, such as the construction of nearby blacktop or the reorientation of the thermometer, the same should be noted with an asterisk and a footer.

lowercasefred
December 21, 2009 10:35 am

To Nick Stokes:
I would also appreciate your pointing out stations with downward adjustments equal to or greater than that of Darwin. I’ll bet Anthony would arrange for the graphics to be shown if you cannot provide a link.

geo
December 21, 2009 10:49 am

As somebody formally asked whoever is in charge of GHCN?
I still think I want the first question answered “how did this actually happen?” before I move on to any discussion of “was the result an intentional inflation?”.
I still have this feeling -possibly entirely incorrect- that there exists a significant chance this kind of thing (we’re talking about how many thousands of stations in GHCN?) is much more likely to be done by herds of grad students working from scripts they barely understand and with no authority, opportunity, nor confidence in their own judgements to challenge results that their older, more experienced, more confident colleagues would immediately discard as an unacceptable result.

Mesa
December 21, 2009 10:53 am

The number of negative correlations on an annual basis is pretty surprising…
Here is a significance test for correlations.
http://faculty.vassar.edu/lowry/rsig.html
For 100 data points correlations of absolute value < .2 are not significantly different from zero.
It's possible that Australia has a particularly low spatial correlation of temperature changes compared to other regions?

Mesa
December 21, 2009 11:00 am

Another possible problem with correlation measurements is that the discontinuities will destroy the correlation structure. You almost have to remove them before measuring the correlation, or measure correlation over periods of time where there aren’t significant discontinuities. This is especially important with relatively few data points……

JJ
December 21, 2009 11:02 am

Marc:
“Why do they use yearly temperatures?
If the temperature stepped at some point, it should be possible to see it in the day by day temperature.”
Daily temperature fluctuations are much larger (one or two orders of magnitude) than, and would mask, the typical effect being adjusted for.
DR:
GHCN isnt doing meteorology, and they arent buildng engines. If you want to disprove GHCN methods, you have to address what they are doing – estimating global surface temperature.
Bill Illis :
“(I can’t be sure but there is long enough overlap between individual station locations in this chart that no adjustments should have been made.)”
Overlap among station locations does not rule out adjustment, much less overlap among duplicate records for a single station …
Bill,
“To get a VALID check on global temperatures will take at least another 150 years and more.”
Then you better get on it. Because if you dont have VALID data, then you are just waving your hands.
“If there is a problem with AGW then you have consigned large populated areas to the scrap heap.”
Nonsense. That aside, your analysis is decidedly one sided. If there is not a problem with AGW, but we spend Hundreds of Trillions of dollars on the pretense that there is, then you have consigned large populated areas to the scrap heap. If you are going to demand a response like that, you need to bring sufficient VALID data to the table. If it takes 150 years, so be it.

Jim
December 21, 2009 11:03 am

************
bill (09:58:40) :
To get a VALID check on global temperatures will take at least another 150 years and more. If there is a problem with AGW then you have consigned large populated areas to the scrap heap.
***************
If the Earth hits a dense swarm of asteroids, the Earth is toast but that isn’t very likely and neither is catastrophic global warming. I’m sure we can come up with a long list of events that might hurt us, but we don’t prepare for every one of them as if they WILL happen.

John Nicklin
December 21, 2009 11:10 am

Why do they have to adjust the temperature readings at all? Temperature at a given site is the temperature at that site. If it goes up or down, so be it. Averaging all sites in a local area to come up with a temperaure report for that area seems reasonable, six or twenty stations in a city or township could be averaged to give a regional temperature. “Adjusting” temp readings seems fraught with problems and opens the path to errors (and to misrepresentation of reality.)
Does anyone really know what the temperature is anywhere on the planet anymore? The graph shown for Darwin is disturbing, raw data shows one thing, adjusted shows something very different, which one is reality and which is an illusion?

Chris
December 21, 2009 11:34 am

Nick,
Any “analysis” such as GG’s which leaves out key variables is worse than meaningless, it’s deceptive. Another good example is the supposed US pay gap between men and women. When factors such as profession, location, education, years of experience, etc. are factored in women in the US make slightly more than men. GG leaves out both location and time and thus should be disregarded immediately as bogus.
On your speculation of Stevenson Screen changes for the later Darwin stepwise adjustments, show us the records. You can’t expect to be taken seriously if you just throw out guesses. Also explain why a new Stevenson screen (not the purported change to a Stevenson screen way back around the turn of the century at Darwin) should result in an upward temperature adjustment or a lesser decrease in adjustment. My understanding is that a new (and thus brighter) screen will, if anything, result in slightly lower temperatures readings than if it was old and discolored. The slats being blocked would also yield higher readings in sunshine. What you see in the Darwin record for the later decade’s temp adjustments is less decrease followed by increase, so what’s your theory there?
Finally, your continued mention that Darwin is but one station is a pathetic and ridiculous diversion tactic. We’re not discussing every single station in the GHCN record here and pointing out the fact that a single one has been adjusted. There are other posts and other sites that cover many other stations. There are also posts that look at the cumulative effect of adjustments in various areas. This discussion is not happening in a vacuum, so pretend like you’re smarter than a fifth grader and drop the pretense that what has been found with Darwin adjustments is everything and therefore can’t really affect the overall data.
Until you can specifically account for the dramatic adjustments at Darwin it remains a smoking gun.

Bill Illis
December 21, 2009 11:41 am

I’ve charted up the difference in the annual temperatures from the GISS unadjusted dataset and the GHCN adjusted data from the ClimateExplorer.
This is presented a little different than Willis did but I think you will get the picture.
http://img192.imageshack.us/img192/521/darwintemps.png
Difference / Changes made by GHCN
http://img14.imageshack.us/img14/8291/darwintempdifference.png

JonesII
December 21, 2009 11:44 am

bill (09:58:40) : Was that spaguetti’s temperatures and hockey stick shaped graph made wth Mann’s software?☺

Willis Eschenbach
December 21, 2009 12:16 pm

JJ (07:28:55) :

Good work, Willis.
Some issues that require reconsideration:

“Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.”

GHCN treats these as ‘duplicate’ station records, not as separate stations. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.

If they were merely duplicates, the GHCN would not adjust them differently. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
Now, do you see how condescending that sounds? If you want people to take you seriously, talk to us like adults, not the way you talk to kids.
If they were merely “duplicate records”, they’d average them to get the final record and be done with it. They don’t. If they were just “duplicate records”, they wouldn’t disagree 90% of the time. They do. If they were only “duplicate records”, they would not have been adjusted separately. They were.

“While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way.”

That is not true. The GHCN algorithm could easily operate differently on different records, even if they are very close (or perhaps even if they are identical) in their values.
The records are of different lengths. This could easily change which ‘neighboring’ stations correlate with the records and thus which stations are used for the identification of discontinuities and application of adjustments. Even when the same stations are used for the adjustment, record length may also affect how discontinuities are identified and how adjustments are applied. You may not yet conlcude that the standard GHCN adjustment proceedure was not correctly applied.

Man, you’ve been hanging around with too many “climate scientists”, you’re all about “could” and “might” and “may” and “could easily” and the like. Yes, the GHCN algorithm could do the things you claim … but it doesn’t. TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!! I am sick of people making claims about what “could easily” and “might” and “may” happen. if you think it might happen, try it and see. I have, and I couldn’t make it happen, “easily” or otherwise.

“Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert.”

You repeat your mistake of the first analysis, by applying your own criteria for suitability, rather than GHCNs. GHCN does not appear to have any specific distance limit. GHCN appears to only be concerned if the stations are in the same ‘region’ (climatalogically) . You cannot say that GHCN did not apply its standard method. You need to ask.

The GHCN says “neighboring” stations. Perhaps more than 1,500 km away is “neighboring” on your planet. On this planet it is not.

“Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:”
Which Darwin first difference? The correlation between any given station and any of the five Darwin duplicate records may be very different. Which one of the five do you report here? What do the other four look like?

I thought you were following the story. I am clearly talking about the adjustment in 1920. Go back to the records. Which Darwin first difference am I talking about?

“Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.”

So?
The assertion is that stations within the same ‘region’ are sufficient for use as reference stations, when making homogenizations of datasets desgned to be used at ‘regional’ or larger scales. You have not addressed that, not that it matters one whit to your claim that the adjustment wasnt applied correctly.

No. The assertion is that “neighboring” stations are used. I’ve checked the neighborhood out to 1,500 km. without finding a single suitable station. If you think that there are suitable stations, you dig them out, because I can see that absolutely nothing that I might do will satisfy you. When you five the five stations, let us know. Until then, your fantasies are just that.

“1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).”

True. This is a potential indictment of the adjustment method, not proof that the adjustment wasnt properly applied. Note that to serve as an attack on the adjustment method, you would need to demonstrate that the adjustment method was properly applied here, and that the adjustment had a significant effect on the temperature measurement *at regional and larger scales*.
Recall from our previous discussions that the GHCN method specifically allows for oddball results that do not track local temps. They assert that these weird adjustments are rare (i.e. ‘highly improbable’ results are consistent with the methodology), and have little if any effect on the aggregate results.

Take a deep breath and think about this for a minute, JJ. All the GHCN method can do is to adjust a station to match the trend in neighboring stations. It can’t create a trend out of nothing. For the adjustment to have been used, we would have to find 1) five well correlated neighboring sites that 2) increase at 6C per century.
I can’t find either of those. If you can find those, bring it on. If not, the results indicate that they didn’t use the algorithm, regardless of how frantically you wave your hands and say that they might could easily have used some possible unknown stations 5,000 km away.

“2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).”

Simply put, you have not shown that. You conclusion is not warranted.

Simply put, I have shown it out to a distance of 1,500 km, well beyond any reasonable interpretation of “neighboring”, and well beyond the known intercorrelation distance of temperatures. If you think I’m wrong, COME UP WITH SOME STATIONS THAT FIT THE GHCN CRITERIA!!!

“3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).”
Are you sure you applied that completely and correctly?

I don’t know if I did, JJ. I just do the best I can, nobody can ever be sure that they are correct. That’s why I put my work out on the web, to let other people find errors in my application of the GHCN algorithm. If you find I’ve done it wrong, let me know. Asking me if I’m sure is just wasting electrons. If you think I’m wrong, do the analysis yourself and show that I’m wrong.

“4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).”

I believe that you may be incorrect about that.

Frankly, without a citation, your “belief” is meaningless.

“5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. ”

Sense with respect to what? If your claim is that the GHCN methodology was not applied, then you must prove that they do not make sense with respect to the GHCN methodology. You have not done that.

Make sense with respect to 1) logic 2) mathematics 3) practical considerations 4) the fact that they are nearly identical. If you can’t follow those, I can’t help you.

If your claim is that the GHCN methodology is not sufficient for estimating long term trends at large scales (regional and larger), then you need to prove that the adjustments do not make sense with respect to estimating long term trends at large scales (regional and larger). You have not done that either.

Say what? My claim is that the listed GHCN methodology was not used. Whether is is “sufficient” for anything at all is your issue, not mine.

At this point, all you have proven is that they do not make sense to you. If they do make sense with respect to the methodology and its stated goals, then all you have proven is your own ignorance.

Proven? I haven’t “proven” anything in any direction. How could I?

“Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value?”
When did they average them to get an adjusted value?

Now you’re just playing at not understanding. The GHCN produce data to be used in global and regional estimates of temperature change. Do you think that they do that without averaging? Sheesh …

“The reason you adjust is because records don’t agree, not to make them disagree.”

That may be a successful line of attack – you have to demonstrate the effect on the aggregate result first.

Effect on the aggregate? I’m talking about Darwin, not some mythical “aggregate”.

“And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.”

The datasets are not nearly identical. They differ greatly in length.

Your speculation and fantasies are worse than useless, as you are basing conclusions on them with absolutely no data.
TRY THE GHCN ALGORITHM!!! Use some dummy data for neighboring stations and see what it does to the different length (but otherwise nearly identical) Darwin datasets. You claim it is possible to get wildly different results simply because the datasets are of different lengths. I have not been able to reproduce that result. No matter what I try, I get very similar results. Get back to us when you have something other than your speculations to report.

“So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims.”

Once again, you are over reaching. Your conclusions are not properly supported. As shown above, they are founded in part on your (potentially valid) criticism of the GHCN method, which does not demonstrate that the method was not applied correctly. You need to learn to separate those two issues.
As shown above, your conclusions regarding manual adjustment are also founded in part on your own ignorance regardin some aspect of the method. You really need to ask GHCN to document the specific adjustment for these station records, and replicate them yourself, before making such claims.

First, you need to learn that when you say to someone “you need to learn”, it just pisses them off.
Next, JJ, if you can show how the GHCN methods were used on the 1920 step, bring it on. Until then, your claims about what you understand and what I don’t understand ring hollow.

“I’m happy to be proven wrong, …”

I doubt that. Being proven wrong would be very embarassing to you. And the rest of us.

If you can’t take being proven wrong, don’t go into science. It is an inescapable part of science that some of your claims will be shown to be wrong. I’m happy to be proven wrong because that’s how science progresses. Of course, I’m happier if I can prove someone else wrong …

“… and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.”

Uh, yeah. Thats what GHCN does. Troll all of the blogs on the planet, looking to see if anyone is talking about them. And then rush out to defend themself against yahoos that call them criminals.
If you have a question (and you do) of GHCN – ASK.

You should look up the difference between “hope” and “expect” sometimes. I don’t expect the GHCN to do a damn thing.
Look, JJ, if you want someone to go ask GHCN something, be my guest. Report back and tell us how far you have gotten. Me, I’m done with beating my head on closed doors … but if you think it’s a good plan, stop messing around and get on with it. You think I’m wrong, you go ask GHCN to prove it. Let us know how it goes. Me, I’ve tried that route over and over, I find it a useless path. But you might surprise me.
If you truly think it is the right thing to do, why wait? Go ahead and ask the GHCN, and report back with what you find out. Stop trying to be an armchair general, it just makes you look foolish saying “you should do this, you should do that”. Get out and do something other than cavil and carp and whine and find fault.
I’m dead serious here. You are all too willing to point out to me the right path, the decent path, the proper path. You keep telling me about all of the things that might be and could easily be. So show us how it’s done, JJ. Get in touch with GHCN and see how far that gets you. You claim that’s the right path, give us a demonstration.
I await your report from GHCN, although not with bated breath …

“Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.”

Over reaching and illegitimate accusation of wrongdoing. Invites another spanking, such as you received from the Economist. Stop setting yourself up… and the rest of us by extension.

Say what? The Economist apologized to me, both in the blog and in private emails, for their unconscionable over-reaching in their first post. Both the guy who wrote the blog and his editor wrote me to say that they were way over the line. Now they are flailing and trying to salvage something of their reputation. They are lucky that I didn’t bring a libel suit against them, I suspect that’s why they rushed to apologize both publicly and privately. If you think that they “spanked” me, you’re not following the story.
In any case, until you actually do some of the things that you so earnestly recommend, like showing that nearly identical records can be warped differently by the GHCN algorithm, or getting in touch with GHCN for an explanation, or finding one single candidate comparison station for the 1920 adjustment to Darwin Zero, please don’t bother me again. I’m not interested in your “may” and “could easily” and “might”, answering your fantasies is taking up way too much time. If it “could easily”, then do it and report the results.

JohnH
December 21, 2009 12:32 pm

Latest from the UK MET office (to think my taxes directly pay for this lot.)
The HadCRUT database is UNDERSTATING temp increases and the Russians recent complaint confirms the database is correct as its matches their lower temps.
Quote ‘The IEA’s output is consistent with HadCRUT as they both confirm the global warming signal in this region since 1950, which we see in many other variables and has been consistently attributed to human activities’
I want my money back !!!!!
http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091218b.html

Antonio San
December 21, 2009 12:33 pm

JJ will be busy over Christmas… 😉

Willis Eschenbach
December 21, 2009 12:40 pm

joe (09:07:01) :

There is an error in the picture:
The left and the right labels are not equal.

joe, look at the legend in Fig. 1, and at the colors in Fig. 2. Some of the items shown are measured on the left scale, and some on the right.
w.

Willis Eschenbach
December 21, 2009 12:42 pm

bill (09:28:20) :

KevinM (08:11:16) :
wil you accept Giss?
down:
http://img379.imageshack.us/img379/5074/newarknycprawgiss.jpg
http://img197.imageshack.us/img197/8283/honolulu.jpg
some up some down some twiddly:
http://img11.imageshack.us/img11/7440/gissrawtemps.jpg

Bill, GISS makes their own very different adjustments to the data. As such, they are not relevant to what GHCN does.
w.

Willis Eschenbach
December 21, 2009 12:48 pm

Mesa (11:00:12) :

Another possible problem with correlation measurements is that the discontinuities will destroy the correlation structure. You almost have to remove them before measuring the correlation, or measure correlation over periods of time where there aren’t significant discontinuities. This is especially important with relatively few data points……

This is why the correlation is done on the first differences rather than on the underlying data. With the first differences, a step will only affect that year, not the entire record. Because of that, it does not “destroy the correlation structure”.
w.

December 21, 2009 12:49 pm

RE: Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.

This isn’t rocket science – adjust the trend of about half of the stations UP since 1930. Adjust the trend of the other half of the stations DOWN prior 1930. Presto, the average adjustment is… Zero!
That’s why Willis’ time-domain analysis is so significant.

Neil Crafter
December 21, 2009 1:00 pm

On Willis’ last reply to JJ – well done Willis!
I think we will be waiting a long time for either JJ (whoever he might be incidentally, as you post your analysis and comments under your real name) to get a response out of GHCN or he actually does some analysis of his own and posts it for other people to prod into like you have done.

Nick Stokes
December 21, 2009 1:11 pm

KevinM (08:11:16) :
Prove it! somebody show me one plot from one station with data that is downloadable and verifiable and shows the opposite pattern.

Well, I said above that of stations >80 years adjusted record, 17 were adjusted down by more than Darwin was adjusted up. Here they are:
211357000002 GUR’EV
222234720001 TURUHANSK
222246410000 VILJUJSK
222255510002 MARKOVO
222310880000
222325830000
222325830002
222325830003
403717140040
403717470010
403718360001
414763930000 MONTERREY,N.L
425726710020
501947280000 COONABARABRAN NSW
603113200000 INNSBRUCK/UNIVERSITYAUSTRIA
615076300000 TOULOUSE/BLAG
I didn’t look up all the names, but you can find them, with details, on the v2.temperature,inv file.
I’ve plotted the Australian station, Coonabarabran, here. Adjusted in red.

Scott of Melb Australia
December 21, 2009 1:17 pm

Richard Sharpe (22:00:48) :
Scott of Melb Australia (20:58:57) said:
(using microfish of available temp data)
Hmmm, is that a new unit of measure for temperature data that I am not aware of or did you mean microfiche?
Thanks for the pickup Richard – no excuses I should have checked my spelling. Hopefully it doesn’t detract from the point I was making.
Scott

Stas Peterson
December 21, 2009 1:31 pm

Nick Stokes,
Are you a free-lance or paid ‘provocateur’ whose only purpose is pedantic obfuscation? Instead you constantly change the subject returning again and again to pick at non germain nits. Do you get paid to divert rational discussions, or do you do so simply for the jollies? Do you get paid by RC?
Real Climate.org funded by Fenton Communications, a front organization of the Al Gore empire and administered by his ex-campaign treasurer, apparently exists strictly for the purpose of providing an official propaganda house organ. It is run by Gavin Schmidt, even as he double dips on his govermnment job for Astronomer Hansen, as a sort of ‘the Team newspaper’.
It si similar to the publication for those friendly Kremlin chaps and the inestimal publication named ‘Pravda’ or was it ‘Isvestiya’. I could never keep straight which was the mouthpiece for the Party, or the Government.

JonesII
December 21, 2009 1:31 pm

To all aussies: Feel free to use the expression I invented above:
“Kangaroo Tail” for campaigning against local “warmheads”.

Nick Stokes
December 21, 2009 1:35 pm

lowercasefred (10:35:59) :
See my response to Kevinm above. I improved the plot of the Coonabarabran result, by adding the adjustment plot in Willis-style (green, no axis marked but same scale as the others) here.

DaveC
December 21, 2009 1:43 pm

[not sure if this was meant to be fightin words or not, but it didn’t really add to the discussion. ~ ctm]

Richard M
December 21, 2009 1:43 pm

Folks, keep in mind that Nick Stokes is well aware of the temporal problems with GG’s histogram and was before he made his first comment on this thread. I think that speaks to motive.

Nick Stokes
December 21, 2009 1:47 pm

David Jay (12:49:46) :
Bruce (08:34:30) :
Gail Combs (07:56:23) :
KeithGuy (07:13:16) :
Jim (06:08:58) :

You’re all asking variants of the same question – doesn’t it matter when the adjustments were made? Yes, it does. What GG and I plotted in those histograms was the change to the trend over the whole time period of the station. One stat that Willis drew attention to was that the change to Darwin made a difference of 1.9 C/century (in my calc, it;s 2.35) The histogram shows the equivalent figure for all the other stations. the measure is the linearised cumulative rate of change over the whole period.
I’ve explained in more detail at my blog here.

Dave F
December 21, 2009 1:51 pm

Nick Stokes (13:35:35) :
Still not sure what you are trying to prove. If you want to say there is not much of an effect on the record, plot the GHCN unadj against the GHCN adj and see what the difference is.

Gary Hladik
December 21, 2009 1:57 pm

Willis Eschenbach (12:16:55) : “I’m happy to be proven wrong because that’s how science progresses. Of course, I’m happier if I can prove someone else wrong …”
Wow, I gotta get meself inta this here science-type career thingy…no downside! 🙂
Congrats on another great article, Willis.

Willis Eschenbach
December 21, 2009 1:58 pm

JohnH (12:32:02) :

Latest from the UK MET office (to think my taxes directly pay for this lot.)
The HadCRUT database is UNDERSTATING temp increases and the Russians recent complaint confirms the database is correct as its matches their lower temps.
Quote ‘The IEA’s output is consistent with HadCRUT as they both confirm the global warming signal in this region since 1950, which we see in many other variables and has been consistently attributed to human activities’
I want my money back !!!!!
http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091218

You deserve your money back. I loved their explanation:

The lower figure is the ECMWF analysis which uses all available observations, including satellite and weather balloon records, synthesised in a physically- and meteorologically-consistent way, and the upper figure represents the same period from our HadCRUT record. The ECMWF analysis shows that in data-sparse regions such as Russia, Africa and Canada, warming over land is more extreme than in regions sampled by HadCRUT. If we take this into account, the last decade shows a global-mean trend of 0.1 °C to 0.2 °C per decade. We therefore infer with high confidence that the HadCRUT record is at the lower end of likely warming.

When someone says that the observations have been “synthesized in a physically- and meteorologically-consistent way”, wise men run.
To review the bidding, HadCRUT says there is no significant warming in the last decade. GISS says there’s no significant warming in the last decade. Both the RSS and the MSU satellites say no significant warming in the last decade.
However, the magic “synthesis” of the above data by ECMWF shows “the last decade shows a global-mean trend of 0.1 °C to 0.2 °C per decade”.
Next, the met office is not showing the HadCRUT data, which includes the ocean. They are only using land stations … what’s up with that?
Their conclusion is that the EMCWF “synthesis” shows that the Met Office is right.
My conclusion is that all that has been shown is that the EMCWF “synthesis” is bovine excrement … however, having said that, I suspect someone will show up soon to say the EMCWF “could easily” be right.
Finally, being the suspicious sod that I am, I went to the EMCWF site and found … model results. Which makes sense, since they are the European Centre for Medium-Range Weather Forecasts, which they do with models.
It is also not clear what data the EMCWF are using. The met says that EMCWF are using “all available surface temperature measurements, together with data from sources such as satellites, radiosondes, ships and buoys.” However, their analysis does not show anything but grid boxes containing land stations … if they are using all available datasets, why is there nothing shown over the ocean?
In their catalog (175kb PDF) of products, they have:

Observation data
Data available from both the raw observation archive and the ECMWF operational feedback archive.
Marine observations
The subtypes available are surface BUOY, surface BATHY, surface TESAC and SYNOP ship. Oceanographic (sub-surface) data are available from DRIBU/BUOY, BATHY and TESAC.
Aircraft observations
The subtypes available are AMDAR, AIREP and ACAR.
Upper air soundings
The subtypes available are PILOT (Land), PILOT (Ship), TEMP (Land), TEMP (Ship), TEMP (mobile), TEMP (Drop), ROCOB (Land) and ROCOB (Ship).
Satellite data
Data from observational satellites that are agreed with satellite operators are available. The subtypes available are SATEM and SATOBS.

I don’t see any land station observational data there, but it is possible that they only give that to their friends … it certainly is not available on their data server, which only gives us plebians access to model results.
Finally, what “surface temperature measurements” are they using? GHCN raw? GHCN adjusted? Their own “adjustment” of the GHCN raw data? Anyone’s guess.

Janis B.
December 21, 2009 2:15 pm

Nick Stokes (13:47:31) :
“You’re all asking variants of the same question – doesn’t it matter when the adjustments were made? Yes, it does. What GG and I plotted in those histograms was the change to the trend over the whole time period of the station.”
Interesting… Take a look at: Musgrave (50194187000, and if I understood correctly, supposed to be -0,39C a decade, according to GG result.xls), adjusted graph here: http://www.appinsys.com/GlobalWarming/climgraph.aspx?pltparms=GHCNT100XJanDecI188020080600101AS50194187000x , years with less than 6 months of data excluded.
How does that “change to the trend” applies “over the whole time period of the station”?
“the measure is the linearised cumulative rate of change over the whole period.”
Don’t the different “whole periods” for different stations matter?

JJ
December 21, 2009 2:17 pm

Willis,
“If they were merely duplicates, the GHCN would not adjust them differently.”
Willis, they are duplicates. GHCN identifies them as such. Read the GHCN station metadata documentation, rather than making up stories based on suppositions.
You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
“Now, do you see how condescending that sounds?”
That is not condescending. It is simply the fact of the matter. You need to know what a duplicate record is, and how it is ultimately used, before you can make claims about those duplicate records, and how they are used. You dont know those things. Find out.
“If you want people to take you seriously, talk to us like adults, not the way you talk to kids.”
I am talking to you as an adult. You are responding like a child. Stop being defensive when someone is trying to help you.
“If they were merely “duplicate records”, they’d average them to get the final record and be done with it. They don’t. If they were just “duplicate records”, they wouldn’t disagree 90% of the time. They do. If they were only “duplicate records”, they would not have been adjusted separately. They were.”
You’re making stuff up again. You dont need to do that. Read the GHCN documentation. If you find something that isnt well documented, ask. Dont try to reason what you dont know, from the limited amount that you do, when you can simply look it up, or ask.
“Man, you’ve been hanging around with too many “climate scientists”, you’re all about “could” and “might” and “may” and “could easily” and the like. Yes, the GHCN algorithm could do the things you claim … ”
Yes, the GHCN adjustment method can do those things I claim. And because it can, your claims that assume it cannot are false.
“TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!”
Excuse me? Please take your own advice. TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!”
You have yet to replicate a single GHCN adjustment. Until you have done that, you cannot say what they did. This is my fundamental point to you. Take a page from Steve Mc. He takes great pains to replicate results before he starts making claims about what someone else did.
“I am sick of people making claims about what “could easily” and “might” and “may” happen.”
Then you need to stop making claims supported by further claims about what “cannot” happen. You claim that the GHCN adjustment cannot have been applied, because the GHCN method cannot adjust duplicate records differently. As you admit, it can. Your claims are false.
“The GHCN says “neighboring” stations. Perhaps more than 1,500 km away is “neighboring” on your planet. On this planet it is not.”
Again, making stuff up. GHCN apparently does not have a distance limit. You cannot just make one up, and then claim they didnt follow their own rules because they didnt follow yours.
“I thought you were following the story. I am clearly talking about the adjustment in 1920. Go back to the records. Which Darwin first difference am I talking about?”
Answer the questions. Which of the five duplicate records was your correlation calculated for? And where is the similar calc for the other four? The records with shorter time frames may be matched to reference stations with shorter timeframes. What does that do to the availability of ‘neighboring’ stations? What does that do for correlations? Hint: It is much easier to find high correlations between shorter segments …
“No. The assertion is that “neighboring” stations are used. I’ve checked the neighborhood out to 1,500 km. without finding a single suitable station.”
So? Limiting your serach to 1500km is not the GHCN method. That is your method. If you want to claim that GHCN did not follow GHCN method, you have to check against the GHCN method. GHCN can apprently use any station within GHCN’s definition of ‘region’. What is that, for Australia?
“Take a deep breath and think about this for a minute, JJ. All the GHCN method can do is to adjust a station to match the trend in neighboring stations. It can’t create a trend out of nothing. For the adjustment to have been used, we would have to find 1) five well correlated neighboring sites that 2) increase at 6C per century. ”
No, you would have to find five well correlated neighboring sites that when homogenized into a reference series result in a potential adjustment of 6C per century.
Now see, you may be on to something there. I am not sure that it is true that the GHCN adjustment method limits the adjustment to the trend of the reference series. Or for that matter, if the trend in the reference series is limited to the trend in any one of the reference station’s homogenized data.
Hint: You arent sure either.
If you can:
a) prove (mathematically) that the GHCN adjustment method has those limits, and
b) prove that there are no stations that meet GHCN’s requirements for a reference station with that trend, THEN you will have proven that the GHCN methodology wasnt followed. And that would be something, wouldnt it?
And here’s the fun part: If you instead end up proving that the GHCN methodology CAN produce a 6C adjustment where no 6C trend exists in any of the reference stations … you’d have something there, too, now wouldnt you?
If you actually DO THE WORK to rigorously prove your point, you stand to come away with something valuable either way it turns out …
Sounds like a productive line of attack. Give it a go.
More later.

Mesa
December 21, 2009 2:19 pm

WE:
I work with first differenced data all the time in finance (returns actually) – just pointing out that a jump discontinuity in the undifferenced series will lead to one outsized difference and affect correlation measurements for small data sets….
I’d be curious to know if you think the extremely low correlations for the nearest neighboring stations are typical for the rest of the globe – ie any idea if its an Australia specific effect or are the distances just too great? It’s surprising given how they have been justifying using fewer and fewer thermometers globally…
Tks.

wobble
December 21, 2009 2:19 pm

Nick Stokes (13:11:12) :
“”Well, I said above that of stations >80 years adjusted record, 17 were adjusted down by more than Darwin was adjusted up. “”
Were any of these adjustments made outside of the Peterson algorithm?
If Peterson was followed for all of these adjustments, then Darwin is unfairly offsetting them.

DaveC
December 21, 2009 2:24 pm

[not sure if this was meant to be fightin words or not, but it didn’t really add to the discussion. ~ ctm]

Since it’s really hard to get into a fight with a computer screen, my comment was meant to be nothing more than a poignant observation directed at someone displaying boorish behavior.
Reply: Food fights are discouraged. Looks like I made the right choice. ~ ctm

Nick Stokes
December 21, 2009 2:47 pm

Janis B. (14:15:46) :
Don’t the different “whole periods” for different stations matter?

Yes, they do. You’ve plotted a smooth for that blue Musgrave data. If you had worked out the adjustment difference for that period and fitted a line, and got the slope, that is the number that goes into the histogram. There are two time issues:
1. The short period means that a relatively small adjustment overall can make a big slope
2. Short periods could be recent or long ago.
That’s why I’ve tended to focus on periods >80 years. It’s rare to find such a record that doesn’t cover most of the 20th C. And you only get a big gradient if the adjustment is also large.

Nick Stokes
December 21, 2009 2:51 pm

wobble (14:19:56) :
Were any of these adjustments made outside of the Peterson algorithm?

I don’t know. I’m analysing the data as found on the posted file. But I doubt that they were made manually. You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.

Dave Springer
December 21, 2009 2:54 pm

Nick Stokes @ (22:46:31)
Giorgio Gilestro’s “most cogent criticism” is bogus in a couple of ways.
First of all if there is any bias in upward/downward adjustments the bias should be downward if the claimed corrections for urban heat islands were really made. There are going to be very few cases of air conditioner vents or blacktop or buildings removed from near a temp station but very many cases of these things being added near one. Gilestro indeed finds a small bias but its magnitude is the opposite of what one would expect to see. Correction for urban heat islands not only appears to be absent but there appears to be some small correction applied for what I can only describe as urban cold islands.
Secondly he makes no analysis of the temporal distribution of the adjustments. This makes a HUGE difference in what the trend looks like. If we adjust temperatures upward in the sceond half of the 20th century and adjust them downward by a commensurate amount in the first half of the 20th century then by Gilestro’s analysis all is well because they cancel each other out. In fact they don’t cancel out in the trend line. It’ll look colder than it really was in the first half of the century and hotter than it actually was in the second half. This should be obvious to anyone who can apply a triple digit IQ to it for a few minutes at most.

Nick Stokes
December 21, 2009 2:56 pm

Dave F (13:51:22) :
Still not sure what you are trying to prove. If you want to say there is not much of an effect on the record, plot the GHCN unadj against the GHCN adj and see what the difference is.

That’s basically what Romanm’s plot does. It shows the difference, which reaches a maximum size of about 0.2C back in about 1900. Noticeable, but not huge.

JJ
December 21, 2009 3:06 pm

Nick Stokes:
“You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.”
Of course you can. In fact that, is exactly what USHCN does.
And the USHCN adjustments, whatever the distribution of their +/- magnitudes is, adds a strong, nearly hockey stick shaped warming trend into the final US temp estimate.
Where is the similar plot of GHCN gridded temps, comparing the net effect of raw vs adjusted data on the global estimate?

Willis Eschenbach
December 21, 2009 3:07 pm

JJ (14:17:03) :

Willis, …

I told you I’m not interested in your cavilling until you actually do something. You say that I haven’t been able to replicate the GHCN method. You are right, I haven’t been able to replicate the GHCN results. Why? Because despite an extensive search, I can’t find the stations to do so, nor have you. So sue me.
But on the other hand, you haven’t done a damn thing.
Come back when you have news from your enquiring emails to GHCN, or when you have identified possible stations for use in adjusting Darwin, or when you have shown that the GHCN method can take nearly identical records and twist them in different directions.
Because until then, you are just waving your hands and whining, and frankly, Scarlett, I don’t give a damn.

Dave Springer
December 21, 2009 3:08 pm

Nick Stokes @ 14:51:17
There are only two reasonable explanations for the adjustments seen at Darwin Zero. One is that a deliberate manual adjustment was applied to make the temperature trend appear to be an increasing one. The second is there’s a bug in the automated system that finds and applies the adjustments. In either case any honest researcher involved in the adjustment process should react to this by saying “This appears to be inconsistent with our published method of adjusting raw data. Let me figure out what happened and I’ll get back to you as soon as possible with an answer.” But instead we get nothing but rationalizations and denials. The so-called denialists are the warm mongers. The skeptics now appear to be the only ones NOT in denial.

Nick Stokes
December 21, 2009 3:14 pm

Dave Springer (14:54:13) :
First of all if there is any bias in upward/downward adjustments the bias should be downward if the claimed corrections for urban heat islands were really made.
Secondly he makes no analysis of the temporal distribution of the adjustments. This makes a HUGE difference in what the trend looks like. If we adjust temperatures upward in the sceond half of the 20th century and adjust them downward by a commensurate amount in the first half of the 20th century then by Gilestro’s analysis all is well because they cancel each other out. In fact they don’t cancel out in the trend line.

First, the GHCN corrections don’t claim to correct for UHI, and they don’t. They try to detect and correct discrete events.
Your second is the fallacy that I’ve been trying to explain over and over, “In fact they don’t cancel out in the trend line.” What GG is plotting is the distribution of trend line changes. The rate of change over time, for the whole period of the station, in C/decade. In your example, a big upward trend would go into GG’s histogram. No cancellation.
What do you think it is a distribution of?

John McDonald
December 21, 2009 3:17 pm

I have a odd thought about how it might have happened.
It looks sort of like a positive feedback loop.
Maybe something like this … they run the data look for a discontinuity. Clearly one is found around 1939. However, one can assume the old data should be lower or the new data higher … which raises the questions as to which one to change.
So it appears a 2nd order filtering algorithm runs in reverse starting in 1939 and ending 1920 with damping factor. It appears they got their – and + sign mixed up so instead of lowering 1930 to 1939 they raised it and instead of raising 1920 to 1930 ish they lowered it.
Next it appears they ran another algorithm going forward in time from 1939, this looks like a classical positive feedback — again probably a – and + signed mixed up somewhere so instead of damping you get ramping. (If someone can find another station with a similar discontinuity and see if the same thing occurs that would be great)
IF my speculation is correct — then this is not human caused, it is poor software coding, a lack of a software check for positive feedback, and an algorithm that goes unstable when such a profound step change is encountered. Engineers of all types accidentally make oscillating and positively feedback algorithms. Next time you hear a squeal at a concert from the speaker – think Darwin 1.
But who knows …

Dave F
December 21, 2009 3:20 pm

Nick Stokes (14:56:44) :
That’s basically what Romanm’s plot does. It shows the difference, which reaches a maximum size of about 0.2C back in about 1900. Noticeable, but not huge.
Still, what does the graph of raw data look like next to the graph of adjusted data? Is there a difference of 0.0175C, 0.2C, or some other difference? Isn’t this the best way to determine what effect the adjustments have, not making statistical sausage out of the data?

John McDonald
December 21, 2009 3:21 pm

For you filter guys out there … sort of like running a notch filter that goes unstable when you hit with a big enough step change and starts a positive feedback instead of filtering because the Q is set to high.

Dave F
December 21, 2009 3:24 pm

Nick Stokes (14:56:44) :
Also for consideration as to why the distribution was the wrong approach to take, does not GHCN use grids, and wouldn’t the weight of the station also need to be taken into account for its use in the grid when an adjustment is applied?

Janis B.
December 21, 2009 3:29 pm

Nick Stokes:
“Don’t the different “whole periods” for different stations matter?
Yes, they do.”
But that does not get reflected in histogram – at least not in GG’s, or does it?
“Length” of Musgrave record is more than 80 years – 25 years of data between 1907-1992, it’s just that there is a huge gap of those -999.9 in the middle.

Nick Stokes
December 21, 2009 3:33 pm

JJ (15:06:05) :
“You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.”
Of course you can. In fact that, is exactly what USHCN does.

Yes, but USHCN uses metadata files. GHCN explicitly doesn’t, because it says that the way they are kept is just too varied across countries. If GHCN wanted to compile a history of manual changes, they would have to in effect create their own metddata files.

Nick Stokes
December 21, 2009 3:36 pm

Janis B. (15:29:27) :
By length of record, I mean length after adjustment. This acts as a filter for interrupted sets, and would eliminate Musgrave.

wobble
December 21, 2009 3:37 pm

Nick Stokes (14:51:17) :
“”I doubt that they were made manually. You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.””
So don’t you think it’s a problem that the cooling adjustments you highlighted were made algorithmically according to Peterson, yet Darwin (an offsetting adjustment) was not?
Doesn’t this lead you to believe that GG’s plot should have revealed a net cooling adjustment if the temperature record was being objectively corrected?

Nick Stokes
December 21, 2009 3:41 pm

Dave F (15:24:42) :
Also for consideration as to why the distribution was the wrong approach to take, does not GHCN use grids, and wouldn’t the weight of the station also need to be taken into account for its use in the grid when an adjustment is applied?

It has to be better to look at the distribution of all stations than to pick out individual (extreme) stations. The grid argument applies even more to just focussing on Darwin. Incorporating grid weights would be a good thing in calculating the derived mean. It doesn’t change the distribution.

Nick Stokes
December 21, 2009 3:43 pm

wobble (15:37:07) :
I don’t believe that any of these adjustments were done manually.

wobble
December 21, 2009 3:51 pm

Nick Stokes (15:43:36) :
“”I don’t believe that any of these adjustments were done manually.””
Does that include Darwin?

wobble
December 21, 2009 4:00 pm

Nick Stokes (15:43:36) :
“”I don’t believe that any of these adjustments were done manually.””
Actually, cancel my last question. It doesn’t matter.
Willis is attempting to show a warming adjustment (Darwin) which may not have been made objectively by the Peterson algorithm.
GG attempted to demonstrate that it didn’t matter since all adjustment trends offset each other. And you claim on his blog that this is true post 1940 and post 1970 (but you didn’t show your work).
So neither GG nor you really doesn’t address the issue. What happened with the Darwin adjustments DOES matter even IF GG is right.

Nick Stokes
December 21, 2009 4:07 pm

wobble (15:51:21) :
Yes. I believe they were all done by the same computer algorithm, which may or may not have been exactly Peterson’s.

Dave F
December 21, 2009 4:18 pm

Nick Stokes (15:41:22) :
Still, what does the graph of raw data look like next to the graph of adjusted data? Is there a difference of 0.0175C, 0.2C, or some other difference? Isn’t this the best way to determine what effect the adjustments have, not making statistical sausage out of the data?

wobble
December 21, 2009 4:28 pm

Nick Stokes (16:07:54) :
“”Yes. I believe they were all done by the same computer algorithm,””
Well Willis disagrees with you.
Now we’re back to 1st base, right?

JJ
December 21, 2009 4:45 pm

Willis,
Help is being offered to you. Do not take offense when the holes in your argument are pointed out, and do not lash out. Fix the holes. It results in a stronger argument.
GHCN station nomenclature documents that the first three digits ID the country. The next five digits are the nearest WMO station number. The next three digits are a modifier, and are zero if that is a WMO station. The final digit (the 0, 1, 2, 3 and 4 that differentiate the five Darwin records) is called the ‘duplicate number’. You need to know how multiple duplicates are treated in the GHCN, as opposed to multiple stations.
GHCN does not appear to have a numeric limit on the distance for ‘neighboring’. Perhaps you think they should. You can make your case for that, and it might prove fruitful. But you cannot impose a limit that is not present in their methods, and use that to claim that they didnt follow their methods. Those are two different issues.
You claim that the GHCN methodology cannot adjust different record duplicates differently. You rely on that as an assumption to make further claims. However, you provide no support whatsoever for that claim.
On the other hand, it is patently obvious how that might occur: Two different record duplicates, having different lengths, can easily have different correlation with a potential reference site. This is true even if those records overlap for some portion of their length, and have strong agreement in the overlap. Note that the reason behind the ‘hiding of the decline’ was to conceal just such a record length induced change in correlation …
Two record duplicates correlating differently to potential reference sites would result in them being adjusted with different reference series, which would obviously permit differing adjustments. Instead of assuming that this did not happen, find out. It would be valuable for you to know if that occurred …
Similarly, instead of simply assuming that the GHCN methodology limits its adjustments to the trend of the reference series, find out. It would be valuable for you to know if that is true …
On the same note, instead of simply assuming that the GHCN methodology limits the trend of the reference series to the trend of one of its inputs, find out. It would be valuable for you to know if that is true …
You are making many unwarranted assumptions in order to make your points. This not only results in a false or flimsy arguments for those points, it prevents you from making other points that may be more supportive of your ultimate goal.
You are not discussing Darwin in a vacuum. The only reason that anyone would give a gluteus rattius about what you are doing here is that it implies there is something wrong with the aggregate.
Currently, there is no justification for that implication. The GHCN methodology document explicitly acknowledges that there may be large adjustments to individual stations like Darwin. The GHCN methodology document explicitly acknowledges that those large adjustments to individual stations like Darwin may not track the local temperatures well. The GHCN methodology document also asserts that these are not an issue with large scale, long term temperature estimates.
So far, you have confirmed what the GHCN methodology states. If your point were to confirm GHCN, you would be well on your way. But that isnt your point …
Finally, in regards to your ‘you go out and do something before you complain about my work’ meme: That is precisely the refuge that the Hokey Team has taken with Steve Mc. “We’re not going to pay any attention to the deficiencies that you are bringing to light in our work. Who are you to show us holes in our methods? Go publish yourself.” Then they shut themselves away and demand that only people who agree with them 100% should dare speak to them.
This is not scientific behaviour. Nor a winning strategy.
Accept help.

JJ
December 21, 2009 5:00 pm

Nick Stokes:
“Yes, but USHCN uses metadata files.”
Explaining how they do it is not proof that they cant 🙂
“GHCN explicitly doesn’t, because it says that the way they are kept is just too varied across countries. If GHCN wanted to compile a history of manual changes, they would have to in effect create their own metddata files.”
Which they could do. But they dont, opting instead to cut effort and use statsitical methods that can apparently produce a 6C trend where none exists in the data. Yet they stick with the metadata based adjustments in the US. If the statistical methods are sufficient to polish crappy third world data into a gem of a temp estimate, why not apply them everywhere?
Does the answer lie near the fact that the USHCN adjustments, whatever the distribution of their +/- magnitudes is, add a strong, nearly hockey stick shaped warming trend into the final US temp estimate?
BTW, where is the similar plot of GHCN gridded temps, comparing the net effect of raw vs adjusted data on the global estimate? Put the ‘nearly symmetric distribution’ and other handwaving aside. What is the net effect of GHCN temp adjustments on GHCN temp estimates?

Geoff Sherrington
December 21, 2009 5:08 pm

To Nick Stokes
Coonabarabran NSW is given as an example of an NOAA NEGATIVE adjustment, to offest the positive 6 deg per century rate over 50 years at Darwin.
Here is a summary of Coonabarabran from recent Australian Burau of Meteorology files:
st,064008,64 ,COONABARABRAN (NAMOI STREET) ,01/1879, ,-31.2712, 149.2714,GPS ,NSW, 505.0, 510.0,94728,1957,2007, 99, 99, *, 0, *, 0,#
In short, although people setlled Coonabarabran about 1870s, the weather station was judged unreliable for use by the BOM except for the period 1957 to now. (or 2007 as it shows above).
On the other hand, the BOM accept the Darwin record back to 1869, with gaps for events like bombing in WWII of a couple of months.
Also, see an explanation for Coonabarabran:
“A Notable Frost Hollow at Coonabarabran,
New South Wales
Blair Trewin
National Climate Centre, Australian Bureau of Meteorology, Melbourne, Victoria
E-mail: b.trewin@bom.gov.au
Parallel observations, taken for 28 months between July 2001 and October 2003 at two sites at
Coonabarabran, New South Wales, show that topography has a dramatic influence on minimum
temperatures at the two locations. Over the period of the study mean minimum temperatures at a valley
site were 4.9°C lower than those at a plateau site 6.6 km away and 133 metres higher in elevation, with
differences of up to 14.3°C occurring on individual nights.
The observed minimum temperature differences were greater in winter (mean difference 6.0°C) than in
summer (3.0°C). Strong relationships were found between the magnitude of the minimum temperature
difference at individual nights and wind speed and cloud amount at 0300 and 0600 (local time), with
the minimum temperature difference largely disappearing on nights when the wind speed exceeded 8
m s-1 at the plateau site.
There was also a marked tendency, particularly in winter, for the largest temperature differences to occur
on the coldest nights, with the 10th percentile minimum temperatures at the two sites differing by 7.6°C
during the period of overlap. This corresponds to a dramatic difference in the frost risk between the two
sites, with minima falling below 0°C on 196 occasions during the 28 months of the study at the valley
site, but not at all at the plateau site. For a 2°C threshold the figures are 282 and 14 days respectively.
Mean maximum temperatures at the plateau site were 1.0°C lower than those at the valley site, approximately
consistent with the environmental lapse rate. There is little seasonal or day-to-day variation, with the
standard deviation of the daily differences being only 0.6°C, compared with 3.7°C for minima.
The large differences in minimum temperature identified in this study reinforce the fact that minimum
temperatures are highly dependent on local topography. This has substantial implications for the mapping
of frost risk, and other related climatic variables, at resolutions finer than the spacing of the station network
in areas with significant local relief.”
One suspects that there was an event like a site change in the 1950s. There might not have been. The absence of such information is used with dubious intent by pseuso-scientists to cast doubt. Such information alsoexplains the benefit of understatding individual stations well before jamming them into supercomputers with dumb inputs.
NOAA is therefore wrong again to use rejected data and GG was scientifically slack for not checking his sources.
Yours was not a quality post, Nick. The first example I tried gets 0/10.

Nick Stokes
December 21, 2009 5:10 pm

\ JJ (17:00:37) :\What is the net effect of GHCN temp adjustments on GHCN temp estimates?
See above: Nick Stokes (14:56:44) :

Nick Stokes
December 21, 2009 5:14 pm

wobble (16:28:53) :
Well Willis disagrees with you. Now we’re back to 1st base, right?

Yes. But look at the Coonabarabran plot. It’s even jumpier than the Darwin plot, but heads downward. If there’s a case for saying Darwin is manual, even more so for Coona. But what malign hand would be pushing it down?

Paul Vaughan
December 21, 2009 5:20 pm

Nick Stokes, you seem to be arguing that it is ok if the Darwin record gets messed up, so long as the amount that it gets messed up is offset by adjustments (perhaps valid ones) for other stations.
You’ve certainly succeeded in making me very suspicious of the assumptions upon which this changepoint method is based. When I have time/funding, I’m now curious to take a painstaking look at the fundamental assumptions underpinning the adjustment paradigm. We need accurate records at the local level to figure out the complexities of natural climate variations. (See the works of Currie.)

Dave Springer
December 21, 2009 5:32 pm

Bill Illis @ 11:41:23
Nice graphs. Shows it all. Interestingly some small fudge factor ~+-.25C is applied every year superimposed over a large fixed fudge factor spanning 10 to 40 year intervals where each interval steps up a half degree.
I was a prolific programmer for 30 years. If I wrote something that produced crap out like that adjusted temperature record and let it get released for use I would have been mortified, horrified, wouldn’t sleep until I found the flaw, fixed it ASAP, offer my most sincere and abject apologies to each and every harmed user, and beg for forgiveness.
An eyeball glance at the raw data reveals just one anomaly near 1940 where the temperature drops half a degree and remains down half a degree for the next 60 years. Software should have easily caught and corrected that and then there’s nothing else to catch. 120 years of very consistent data with little if any overall trend either up or down. I understand in 1940 the station moved which explains the one-time drop. The adjustment, however it was done, introduced serious flaws into what was an almost pristine record.

Nick Stokes
December 21, 2009 5:36 pm

Geoff Sherrington (17:08:48) :
There’s nothing to indicate that your frost hollow story involves the Coona weather station. The town is by the Castlereagh river, but in a fairly flat region.
I suspect the reason for the big change is a possible merger with data from the Siding Springs Observatory. But that’s just a guess.

JJ
December 21, 2009 5:37 pm

Nick Stokes:
“That’s basically what Romanm’s plot does.”
No it doesnt. That attempts to deal with the temporal issue, but in no way addresses the spatial. Already at 0.2C with just the temporal – a third of the whole value of the alleged ‘global warming’. What happens when those sites get area weighted differentially? When they’re used to fill in missing data cells? Another third?
How much are the UHI and othe runaccounted for local anthropogenic effects? Another third?
Until you can produce a plot of final GHCN global temps that compares adjusted vs raw, you’re just handwaving.

wobble
December 21, 2009 5:49 pm

Nick Stokes (17:14:45):
“”look at the Coonabarabran plot. It’s even jumpier than the Darwin plot, but heads downward. If there’s a case for saying Darwin is manual, even more so for Coona.””
Willis did quite a bit of work in an attempt to convince people that the Darwin adjustment is manual. He showed correlations to neighboring stations and described the distances involved with those further away.
You merely claim that Coonabarabran is “jumpier?”
Gee, who puts forth a better argument?

Willis Eschenbach
December 21, 2009 5:58 pm

JJ (16:45:42) :

Willis,
Help is being offered to you. Do not take offense when the holes in your argument are pointed out, and do not lash out. Fix the holes. It results in a stronger argument. …

JJ, as I said before, what you are offering is not help. It is handwaving and objections. I will gladly accept your help, but to date, you haven’t offered any. I’m not “lashing out”, I’m trying to stem your flood of well-meaning but meaningless platitudes. Let me give you some examples to clarify what I’m trying to say.
Despite trying, I am unable to create a reference series for Darwin because I can’t find relevant well-correlated neighboring stations. I know that. You know that. Help would be a list of appropriate stations, which you have not provided. Instead you keep saying that “neighboring” means over 1,500 km., well beyond the known limit of correlation between temperature stations. That’s no help at all, as any correlation beyond that distance would be by chance.
Despite trying, I have not been able to use the GHCN algorithm to twist nearly identical records in different directions. Help would be a demonstration from you that it can be done. Your statement that it can be done is meaningless. That’s just handwaving, and it is both useless and irritating. If you think it can be done, show us that it can be done.
I say that the GHCN algorithm cannot adjust the trend beyond that of its inputs. Why do I say that? Because I’ve read the description and I’ve tried it. Also, because the point of the algorithm is to make the target station agree with the inputs, not exceed them. You claim the algorithm can adjust it beyond the trend of the inputs. Help would be a demonstration that what you claim is true. Mindlessly repeating “yes it can, yes it can” is not help.
Finally, you keep bringing up Steve McIntyre. You say

Take a page from Steve Mc. He takes great pains to replicate results before he starts making claims about what someone else did.

As someone who has posted extensively on Climate Audit and who corresponds regularly with Steve, I can tell you that’s a fragrant vase of excrement. What Steve does is try to replicate results, as I have done. Often, he is in the situation that I am in … he can’t replicate the results. What he does then is post the fact that he can’t replicate the results. As I have done. Help would be you posting some way to actually replicate the results. Saying I should be like Steve is no help at all.
And no, I’m not using the Hokey Team excuse of “you go and publish before we can talk about this”. I’m asking for help, I don’t give a rat’s fundamental orifice if you publish. However, help does not mean saying “I really really really really think you are wrong about X”. That does nothing at all. If you think I am wrong, don’t talk about how right you know you are and provide endless justifications and reasons. Instead, demonstrate how right you are, show us how it can be done, bring in the citations that support your position, post a method, list the stations, anything but your unending litany of objections.
w.
PS – In researching this article I found this interesting report:

I’d also like to report that over a year ago, I wrote to GHCN asking for a copy of their adjustment code:

I’m interested in experimenting with your Station History Adjustment algorithm and would like to ensure that I can replicate an actual case before thinking about the interesting statistical issues. Methodological descriptions in academic articles are usually very time-consuming to try to replicate, if indeed they can be replicated at all. Usually it’s a lot faster to look at source code in order to clarify the many little decisions that need to be made in this sort of enterprise. In econometrics, it’s standard practice to archive code at the time of publication of an article – a practice that I’ve (by and large unsuccessfully) tried to encourage in climate science, but which may interest you. Would it be possible to send me the code for the existing and the forthcoming Station History adjustments. I’m interested in both USHCN and GHCN if possible.

To which I received the following reply from a GHCN employee:

You make an interesting point about archiving code, and you might be encouraged to hear that Configuration Management is an increasingly high priority here. Regarding your request — I’m not in a position to distribute any of the code because I have not personally written any homogeneity adjustment software. I also don’t know if there are any “rules” about distributing code, simply because it’s never come up with me before.

I never did receive any code from them.

So I’m not the only poor fool who can’t replicate the GHCN algorithm, despite your repeated claim that it can be done. However, I wish you the best of luck in your inquiries with GHCN, and I await your report on their response.

Willis Eschenbach
December 21, 2009 6:10 pm

Nick Stokes (17:14:45):

… look at the Coonabarabran plot. It’s even jumpier than the Darwin plot, but heads downward. If there’s a case for saying Darwin is manual, even more so for Coona.

Coonabarabran has no less than thirty stations within 1,200 km. which are candidates for the earliest adjustment. Darwin has two …

Ryan Stephenson
December 21, 2009 6:28 pm

@JJ: You repeat your mistake of the first analysis, by applying your own criteria for suitability, rather than GHCNs. GHCN does not appear to have any specific distance limit. GHCN appears to only be concerned if the stations are in the same ‘region’ (climatalogically) . You cannot say that GHCN did not apply its standard method.
Actually JJ, if you took the smallest trouble of looking at a map of Oz, you would see it is rather difficult to move more than 1500km from Darwin and still be in the same climatic region – its a big country, but not that big.
You strike me as a proponent of AGW posing as a sceptic that is using the rhetorical technique of challenging the minutae of every assertion as a form of obfuscation. It is similar to another form of obfuscation used on blogs that runs like this “before you challenge my simple assertion A then you must read obscure and difficult to obtain text B in great detail”. In your case you are pushing Willis to do more work to “prove” a point he has, in fact, already made very effectively. The condescending tone is no doubt intended to give the impression of great learning whilst not actually demonstrating anything other than fallacious argument. The fact that you clearly have no idea just how big Australia is exposes you as an ignoramus.
I accuse you of being a troll, and until you post something of real value to this blog then I suggest everyone treats you as a troll.

Dave Springer
December 21, 2009 6:40 pm

Nick,
The raw data can be adjusted so that it shows the famous hockey stick on a rolling average temperature plot and that wouldn’t change the trend line histogram.
Clearly the Darwin Zero data was badly misadjusted and in this case results in a drastic positive slope change on the trend line. The best case you can make from the trend line histogram is that the adjustments are equally flawed in both positive and negative slope changes. That’s not much of a defense for a broken procedure – saying it’s equally flawed in both directions so the flaws cancel out. It hardly inspires confidence that it isn’t broken in other ways. In any case the trend line histogram won’t reveal a temporal distribution problem where the adjustments are excessively negative in earlier records and equally excessive positive adjustments in the later records.

Ryan Stephenson
December 21, 2009 6:46 pm

The entire analysis used at Darwin is wrong-headed anyway. Given we know what todays temperatures are (we can just go out and measure them) we only need to know the rate of change of past temperatures to get an idea of future trends. Therefore we don’t need to adjust to find the absolute temperatures at any site. If a site has been moved, had its measurement kit changed or whatever, then the discontinuities can be detected and the rate of change between those discontinuities can still be calculated without resorting to back-filling data with complicated computer algorithms that clearly can’t be relied upon not to do “wierd stuff”.
The UHI is a different matter, but in each case it should be possible to measure the effect on a specific site by taking actual observation at the site and near to the site but outside the urbanised area. The only places this would not be possible would be in very large cities.
The reason we are having to fiddle with all this data is because there is no good data. All the Stephenson screens were designed to monitor weather, not climate. They were put in places that were easily accessible to the chap that had to read the thermometers that then became urbanised and affected by heavy traffic over the last hundred years. They then had all the instrumentation updated to electronic thermometers with remote reading. Unfortunately this has given climatologists the excuse to fiddle with the data in the most bizarre ways possible to support their ridiculous claims, then dress it up as science to difficult to understand for the layman. It’s not difficult to understand – any high school kid could do a better job.

Dave Springer
December 21, 2009 6:56 pm

Given that much of the record at Darwin is the only record for a large geographical land area doesn’t that give it greater weight in computing global average temperature? It surely should. Was Darwin cherry picked for manual adjustment due to its larger weighting? Again this wouldn’t show up in a trend line historgram of all stations in the worldwide record. The trend line histogram (I presume) treats all stations equally where in the global average calculation each station must be weighted for how much of the earth’s surface it represents.

Dave F
December 21, 2009 7:38 pm

Anyway, Nick Stokes, I would still expect there to be a larger bias in the trends if the adjustments were not being overly managed. The fact that they are so close to 0 is not comforting. Like I said before, if the adjustments are made because of faulty siting, equipment failure, and so forth, I think it is very unlikely that it would work out to be a practically nil contribution to the trend. That it does needs an explanation because I seriously doubt the failures in need of adjustment work out to .0175C of difference. It honestly begs the question, why adjust the data in the first place? Shouldn’t unadjusted and adjusted look basically the same? Do you know if they do, Mr. Stokes?

Grizzled Wrenchbender
December 21, 2009 7:54 pm

Dave Springer: BINGO! Just as Antarctica came to be represented by one station on the Palmer peninsula, Darwin too has high weighting in the gridding step. This is a promising alley for more sleuthing by concerned people like Willis and Ed Smith:
Which temperature stations have large impacts on the global temperature anomaly? Of those stations, how many have “adjustments” that tend to exaggerate GW? I suspect that the answers to these questions may be enlightening. Alas, like Fermat, I’m too busy to do the research myself, but I hope Willis and Ed can make some headway.

DR
December 21, 2009 8:10 pm

Recall the CRUTAR index from Africa
http://climateaudit.org/2009/07/19/christy-et-al-2009-surface-temperature-variations-in-east-africa/
http://wattsupwiththat.com/2009/07/18/out-of-africa-a-new-paper-by-christy-on-surface-temperature-issues/
Why Nick Stokes et al continue regurgitating the same old arguments, it’s the same old problems that plague the surface station network.

Geoff Sherrington
December 21, 2009 8:16 pm

Nick Stokes,
Your response to my observation that the BOM declines to use pre-1950s data for Coonanbarabran is?
If you go to St Kilda market on a Sunday you can pick up many sets of thimbles that are useful for hiding peas. Classically, the magician used 3. You might need a few dozen.
It’s not science to answer a comment with many points with one speculative answer to one selected point.
Yours was a very poor post.

Geoff Sherrington
December 21, 2009 8:23 pm

Nick Stokes
RE my post Geoff Sherrington (22:40:07) : on 20/12
This refers to a 7-year BOM study which BOM documentation shows that the old Darwin site and the Airport location have identical mean temperatures. I have said that there could be reporting errors, though I suspect none will make a difference, for the BOM continues to make the comparison available.
If you have access to more information than I do (I have very little – I have to search for it), then perhaps you can explain why it is a good idea to put a step change into the data at about the 1940 changeover.

Willis Eschenbach
December 21, 2009 10:57 pm

Ooooh, new information. I love new information.
I just paid the outrageous sum of $29.95 for Peterson’s paper describing the GHCN homogenization method. This is CREATION OF HOMOGENEOUS COMPOSITE CLIMATOLOGICAL REFERENCE SERIES, THOMAS C. PETERSON AND DAVID R. EASTERLING, INTERNATIONAL JOURNAL OF CLIMATOLOGY, VOL. 14. 671-679 (1994).
Regarding reference series and distance from the station in question, the paper says:

Distance test
When a reference time series is used to test and adjust for inhomogeneities, the assumption must be made that any regional climate trends and fluctuations are present in the data for each station in the reference series. Reference stations, therefore, need to be chosen within an area in which a climate trend would be reasonably homogeneous. Without a priori knowledge of scale factors in climate change, this can pose major problems. A simple distance test may not be appropriate because gradients in climate may vary considerably with direction. For example, stations some distance apart along a coast may experience similar climate variations while variations at nearby inland stations are quite unrelated.
The first step we can take to avoid such problems is to use only stations with positive correlations. Even if current climate conditions are strongly negatively correlated, it should not be assumed that future climatic trends would affect these stations oppositely. Positive correlations, however, do not necessarily imply that the two stations’ climates are influenced by similar factors; the correlations could be due to remote teleconnections.
Therefore, we also need some distance function included in our selection process to ensure that the stations are within a region that should experience similar climate fluctuations. For our initial work with the GHCNs 6000 station global temperature network, our only test based solely on distance was limiting the list of potential reference stations to the 100 stations nearest to each candidate station.

Now, for Darwin, the hundred nearest stations includes stations out to 1,656 km from Darwin. Since I had already looked at the stations out to 1,500 km from Darwin, this did not add many possibilities. Here are the first difference calculations for the final few of the possible candidate stations per Peterson for the 1920 adjustment to Darwin:
PALMERVILLE = -0.39
BOULIA = 0.31
NULLAGINE (NULLAGINE POST OFFICE) = -0.01
COOKTOWN AMO = -0.22
PORT DOUGLAS POST OFFICE = -0.07
LOW ISLES = -0.35
HERBERTON POST OFFICE = -0.31
As you can see, none of these stations are suitable. In addition to having low correlations, the correlation in all of these is negative, and Peterson says stations with negative correlation can’t be used.
So that complete the demonstration. We cannot construct a reference series for the Darwin 1920 adjustment using the GHCN method and criteria. There’s only three candidate stations with positive correlations for that time, and they have a correlation of 0.06, 0.35, and 0.35 … far too low for a reference series which, according to Peterson, must have a correlation of 0.80 with Darwin.
JJ, you happy now?

Alexander
December 21, 2009 11:10 pm

JJ:
GHCN does not appear to have a numeric limit on the distance for ‘neighboring’
JJ:
GHCN can apprently use any station within GHCN’s definition of ‘region’
I think, I got the point: “Neighboring” and “region” for GHCN does mean: somewhere in the same climatic region.
Ok, let it be so. But this brings us immediatedly to another severe problem. As I understand it the GHCN method is the following: 6 stations in this “region2 are behaving the same way, i.e. being correlated >0,8. The data of 5 are used to “correct” the data of No. 6. In the case of thermometer faults, station movements, all is fine, and this method can give hints to explore the chronic of No 6 for such effects. For Darwin is this for the year 1939 or 1940.
But if the deviation at No. 6 is caused by different reaction on climatic changes at its specific location, the latter will be wiped out! Then we end up which a situation we call in German: Die Katze beisst sich in den Schwanz (The cat bites in its own tail; I dont know an appropriate english expression). We start with the definition of “climatic region”, but if a station shows deviant behaviour because of different local climate, it will be corrected and corrected and corrected again, until it follows the proposition of being in the same “region”.
Best regards from Germany and Austria (not Australia ;-))

Nick Stokes
December 21, 2009 11:24 pm

Geoff Sherrington (20:16:46) :
Nick Stokes,
Your response to my observation that the BOM declines to use pre-1950s data for Coonanbarabran is?

None. I don’t know who’s right. All I’m pointing out is that the mechanics that produce the Darwin rise can also produce the Coona fall. And these are outliers. They are in the outer 1% or so.
Dave F (19:38:26) :
why adjust the data in the first place?

Good question. GHCN themselves say that for a lot of purposes, you might prefer the unadjusted data. They say the adjustments help with regional attribution.
But put it the other way around. What would you folk be saying if they made changes, updated equipment etc, and didn’t adjust.

December 21, 2009 11:26 pm

3×2 (05:46:54) :
I’m still left with a couple of questions.
For those who say that individual stations don’t matter, the global stats will sort things out
If, when scaled globally, local “adjustments” have no real impact … why bother “adjusting” in the first place? Adjustments at the individual station level obviously do impact both regional and global results. Especially if the end result is in the 1/10th degree range.
Specifically for Darwin adjustments.
If you take the difference between the “winter” and “summer” means over the entire record (25/28 from v2.mean) as being about 3°C. Is a 2.25°C “adjustment”, in any direction, at any point, even physically possible?

I would expect so, in ’41 the station was moved from the old PO (destroyed by bombing shortly after) to the airport. In the dry season the prevailing wind at Darwin is southeasterly, the old PO was situated on the end of a peninsula so the southeasterlies blow over water onto the land. This is shown by the data, the record low at the airport is 10.4, the record at the PO was 13.4. The move was made because the PO site was no longer satisfactory due to overhanging trees, subsequent to ’41 the site was relocated several times on the airport.

Willis Eschenbach
December 22, 2009 12:05 am

Nick Stokes (23:24:59) :


Dave F (19:38:26) :

why adjust the data in the first place?

Good question. GHCN themselves say that for a lot of purposes, you might prefer the unadjusted data. They say the adjustments help with regional attribution.
But put it the other way around. What would you folk be saying if they made changes, updated equipment etc, and didn’t adjust.

Nick, I don’t know about Dave, but I have no problem with adjusting for known discontinuities. Such things as equipment changes and station moves definitely introduce false signals into the data.
However, the devil is in the details. My own feeling is that we should take the first difference of the data, and eliminate the one observation from the year of the discontinuity. That way, the trend of the data (which is all that we are interested in) is maintained, and we can average the first difference of the observations directly with other stations to get an average trend over a period of time.
The averaging, of course, introduces other difficulties … but that’s a subject for another thread. My preferred method also doesn’t deal with slow changes, like buildings and trees and paving around the station … but then the GHCN method doesn’t deal with those either.
w.

Ryan Stephenson
December 22, 2009 2:45 am

GHCN have to adjust the data because they have almost no sites that have not changed over the last 100 years. They have sites that have had new screens, they have sites that went from mercury to electronic thermometers, they have sites that were in small villages 100 years ago that find themselves in large cities today, they have sites that had horse and cart going by in 1900 and then three lanes of articulated trucks in 1980, they have sites that were near coastal lighthouses that moved to being near inland airports. All sorts of changes that affected 99% of all the sites. What is worse, from a GHCN point of view, is that the few sites that didn’t change much over 100 years show no warming at all! They should be held up as “gold standards” but GHCN prefers to sublimate them under a mountain of dodgy data from less reliable sites.
Now, if GHCN could reliably detect when the site changes occured, they could simply look for the trend in temperature between those changes when it can be assumed that no changes to the measurement of any kind occurred and therefore the relative temperature change over time could be considered accurate. The fact that they choose not to adopt this simple approach speaks volumes in itself – copying data from one site 1500km away to another site has no scientific validity at all, especially when it intoduces huge adjustments in the trend to the final output. Remember, it is the trend that actually matters, not the absolute temperature. We know today’s absolute temperature and that’s all we need to know if we know the trend, so an adjustment to an absolute temperature that makes a dramatic change in the observed trend is a very bad adjustment approach indeed.
UHI must also be considered, but this can be detected at each site by direct measurement within the UHI affected area and just outside. If the UHI is measured this way over one year then it canb be used to introduce a downard uncertainty in any measured warming trend.

Ryan Stephenson
December 22, 2009 3:31 am

Here is how this particular aspect of the fraud works. Lets say we have two sites that in 1900 were edge of town. By 1960 these towns have expanded and there is heavy traffic causing the urban heat island effect. We then decide that a site is so bad that we move one site out of this urban heat island effect, and the other remains where it is, thus affected by the urban heat island effect.
The algorithm used here is looking for discontinuities, so it will detect the discontinuity in the site that is moved. It will detect no discontinuity in the site that was not moved, but which is affected by urban heat island effect. This is because the urban heat island effect will grow gradually over time and will be indistinguishable from other climate effects.
The algorithm will detect that the site that has been moved has dubious data. It will then seek to correct the data with nearby sites that have not shown a discontinuity. If this is the site with the urban heat island effect, it will copy the data from that site to the moved site and smooth it into one continuous trend.
The impact of this is that rural sites will appear to show a warming trend (which is asctually due to the urban heat island effect of neighbouring sites). Since the site is now rural it will appear as if it can’t be affected by urban heat island effect, but in fact it will have been affected by the copying of data from precisely those sites that are affected by urban heat island effect.
This algorithm is biased in favour of trashing rural site data unaffected by urban heat island effect in favour of data that is affected by urban heat island effect. It has no credibility whatever.

Shane P
December 22, 2009 3:41 am

Nick Stokes (17:36:22) :
“There’s nothing to indicate that your frost hollow story involves the Coona weather station. The town is by the Castlereagh river, but in a fairly flat region.”
Coona is a cold hole in winter. It’s surrounded by a ring of low hills apart from the path of the Castlereagh. I don’t doubt at all that it’s in a frost hollow, although the magnitude of the cooling surprised me. It also has the Warrumbungles immediately to the west.
You might be confusing it with Coonamble which is also on the Castlereagh and *is* dead flat. It’s only about 100km NW of Coona, so it’s easy to confuse them if you aren’t familiar with the area.

Dave Springer
December 22, 2009 4:00 am

Nick,
Your point that Darwin is among a small percentage of outliers may be pointless in light of the fact that not all stations are equal in the impact they have on the global average. As it stands now Darwin is a high impact station where the adjusted data contains a very large and very false warming trend inserted.
If you could provide an example of a similarly high impact station where the adjusted data has a false cooling trend of similar magnitude (-0.6C/decade) then your point about the flaws in the adjusted data cancelling each other out** might be valid. Otherwise that dog doesn’t hunt.
**They still don’t cancel out completely. It’s already been accepted at face value that the adjustments result in an average bias of approximately 0.2C/century of additional warming. That is a significant number when we’re trying to dig out what, if any, human-caused warming has happened.
So far you’ve persuaded me that at least 0.2C of the last century’s warming isn’t man-made but rather is Mann-made, if you know what I mean, and I think you do.

Gail Combs
December 22, 2009 5:44 am

Ryan Stephenson (02:45:27) :
I certainly agree with your approach.
The idea of using stations up to 1500 km or more away “to adjust” a site seems very “dodgy” at best. I do not know Australia, but the whole idea that weather in one location is similar to that in another does not pass the smell test. Within 100 miles of where I live we have oceans, lakes, the piedmont and sand hills. Five miles away ALWAYS gets more rain that I or the airport weather station down the street does. The idea might work OK on a flat plain but not on mountains and ocean front areas. And yes I do understand a significance test for correlations. As a test to determine if the station may have a problem that needs investigation – fine. As the basis for “adjustments” NO.
The whole thing reeks of lazy data handling techniques.

Dave F
December 22, 2009 6:26 am

Nick Stokes (23:24:59) :
I am aware of that, my point was not that the data shouldn’t be adjusted, just that you are saying that it does not contribute much. If that were true, then wouldn’t there be a minute difference in the way unadjusted and adjusted data look plotted next to each other for the entire GHCN dataset? Is there a .0175C difference in the trend when you do that, which is what you and GG are saying in your histogram?

wobble
December 22, 2009 6:53 am

Willis Eschenbach (22:57:54) :
“”our only test based solely on distance was limiting the list of potential reference stations to the 100 stations nearest to each candidate station.””
Great work, Willis.
1. I think you’ve shown that the Peterson algorithm couldn’t have been followed. (Can you just tell us where you read about the 0.8 correlation requirement? Isn’t it possible that they accepted a correlation of 0.4 as long as it was the “highest correlated neighboring station?”)
2. I think you’ve also shown at least one flaw in the Peterson algorithm even if it is utilized objectively. Testing 100 stations for adequate correlation doesn’t seem reasonable. As you hinted at before, even if 1,000 adjustments are using correlations with a p-value = 0.05, then that implies 50 of those 1,000 correlations were a coincidence.
Overall, I think JJ did you a favor. He was definitely condescending, but he did push you to strengthen your argument quite a bit. However, I agree that it would be nice for him to help out with some of these efforts. He’s obviously smart enough. Unless he’s working on something of his own right now. JJ, do you care to share?

wobble
December 22, 2009 7:01 am

Nick Stokes, I appreciate some of the analysis you’ve done, but it seems that your entire argument boils down to the fact that Coonanbarabran is jumpier than Darwin.
All your histograms of trends are quite useless without being able to prove that Coonanbarabran was also manually adjusted. Sorry, but this can’t be done using a visual jumpiness test. Maybe we should start by looking at the nearest 100 stations and finding the most highly correlated.

Ryan Stephenson
December 22, 2009 8:27 am

@Willis: Have you tried taking the approach you/we suggest? Take the data between the discontinuities detected by the algorithm, then check the gradient of that data? Maybe it is difficult because there are only 10 or so datapoints and it is dependent on where you start and stop exactly, but it seems that the gradients between the discontinuities are about 0Celsius/Century compared to the gradient imposed by the corrections. Excel can plot a line of regression between the points, e.g. for the last 12 years we seem to have stable data and Excel can plot a line of regression between the last 12 points of the raw data and the gradient of this line can be derived. Sorry to dump the extra work on you but if you have the data tabulated already it should take you long.
That would really blow this whole thing out of the water. After all, what the algorithm itself is telling us is the site changed about three times after 1940 but was stable as a measuring site between those changes. So that is where the quality data is – the raw data between the site changes. Measure the gradient between the changes and its job done. It sure isn’t 6Celsius/century. So the algorithm will be shown to fail the sanity check badly.

temp
December 22, 2009 8:34 am

Without having looked at this particular station specifically, a couple of points, just because A and B look similar doesn’t mean they will pass/fail the same statistical test.
Given an artificial cut off (e.g. p-value 0.01), it is possible that one station will “fail” the test (e.g p-value = 0.0098) and needs correction, and one that “looks” it like won’t (e.g. p-value = 0.011 (which means no correction).
The samething happens with the correlations. Even assuming that two stations that look a like both “fail”, there is no guarantee that they won’t be corrected differently. A and B are highly correlated. A is correlated to C with a value of 0.81. C is used to correct A. B has a correlation to C of a value of 0.79. C isn’t used to correct B. As C is likely highly correlated to whatever else is used to correct B, it is likely the correction is similar, but over a number of years with enough corrections, it is possible to start with two things that look very similar, but don’t end up being very similar.
The last issue is your given correlations. I’m not sure why 1915 is relevant, or giving the correlation over the whole rest of the life of the stations (e.g 1915-until the data ends) (which is what I presume you did as you didn’t give any years).
My reading of the work was that they were doing a “local” (in time) tests to do a “local” correction based on pretty “local” correlations (I don’t remeber the exact times frames, but 5 years sticks in my mind). In other words, if I’m going to correct something in 1945 (just an example), then I don’t need the correlation back to 1915. I do need a station that has a rather long data set (25 years sticks in my mind), but that I don’t think the that 25 years is even post-correction data (e.g. I could use a station that ran from 1940 to 1965 to correct an issue with another station with an issue starting in 1945).
There are issues with exactly reproducing what they did. For example, it wasn’t clear to me if they were doing correlations to find “good” stations based on the corrected data for the station or the uncorrected data for the station.
And you are right, they’ve AT BEST poorly defined “neighborhood” and what stations they weren’t able to find a “neighborhood” for and didn’t correct and which ones they did are unclear.
I generally don’t think they expected anybody to try and EXACTLY reproduce what they did when they published the method (the publication before this really exploded into the issue it has). They assumed that people would either simply take their corrected data and use it, OR take the raw data and come up with their own method.
I will point out that isn’t exactly rare. In biology, for example, papers rarely truely give all the details. They rarely state, for example, how solutions are stored, and what quality of water was used to make them.
I understand what you are trying to do, but it would be much more convincing if you developed a method based on the raw data that didn’t show warming. There were always be points of contention with any method (e.g. why use the time frame the did to do the tests, why use the p-value cut off they used to say a particluar set of data “failed” and need a correction, why use a correlation of 0.8, etc.).
If for a some set of stations, their method “fails” that isnt’ shocking and doesn’t change the total result.
A reasonable method that took the same data and showed that golbally warming isn’t happening would be a big deal.

December 22, 2009 9:48 am

Using sites 1500KM away?
This shows the difference in sites across the UK,
Ten years with the hottest months from the Met Office Station Data.
Order of maximum temperature
Oxford back to 1853
2006 1983 1995 1911 1921 1976 1868 1947 2003 1874
Stornaway back to 1873
1899 1947 1997 1880 1955 2006 1933 1955 1976 1901
Durham back to 1880
2006 1995 1887 1947 1983 1975 1989 1901 1940 1990
Sheffield back to 1883
2006 1975 1983 1995 1976 1947 1921 1934 1995 1887
Eskdalemuir back to 1911
2006 1947 1955 1983 1989 1995 1975 1976 1934 1940

wobble
December 22, 2009 10:04 am

temp (08:34:27) :
“”If for a some set of stations, their method “fails” that isnt’ shocking and doesn’t change the total result.””
I don’t think Willis was trying to show that their method failed. I think he was trying to show that they failed to use the method.

temp
December 22, 2009 11:54 am

Yeah, but he hasn’t done that. One of his simple claims is that since one was corrected and the other wasn’t, and they look the same that is evidence that they didn’t use their method, but that isn’t true.
You actually have to run the method and see which if either fail the test that is described. If they all fail the test (again, fail meaning they need correction), then you have issues, and if none of them fail the test, then you have issues.
Once you’ve determine if (and where) they fail the test, then you can start to worry about identifying the relevent stations to do the correction.

temp
December 22, 2009 11:57 am

To simply state that they didn’t because it doesn’t “look” they did, and without actually trying to carry out the steps they do describe doesn’t mean you’ve reached a valid conclusion.

wobble
December 22, 2009 12:16 pm

temp (11:57:47) :
“”To simply state that they didn’t because it doesn’t “look” they did, and without actually trying to carry out the steps they do describe doesn’t mean you’ve reached a valid conclusion.””
Check is post at (22:57:54).
I think it completes the effort of trying to carry out the steps they do describe.
He tested the 100 closest stations (which is prescribed by the Peterson method) and found that none had a correlation adequate enough to make adjustments.
What more do you want him to do to prove that the method wasn’t objectively used?

wobble
December 22, 2009 12:17 pm

Sorry, I meant, check HIS post at (22:57:54).

temp
December 22, 2009 12:33 pm

As I said originally, I am assuming he is giving correlations over the entire time series because he isn’t giving starting and ending years for producing the correlation, which isn’t what they did, I think.
If that is wrong, then fine, but it isn’t clear from what he has written.

wobble
December 22, 2009 12:57 pm

temp (12:33:00) :
“”which isn’t what they did, I think.””
What do you think they did? Do you think they truncated time series in order to improve the correlation? If so, then many people agree with you, and you might be right.
So what exactly would you recommend? When calculating correlations of 100 neighboring stations to be used to test for a discontinuity in a specific year you will be required to perform thousands of correlation calculations by truncating each time series by one data point at a time.
Then, you’ll need to repeat that process in order to test for a discontinuity for each point of the station’s time series. 70 years worth of time series data discontinuity tests represents hundreds of thousands of calculations in order to test a single station record.
If you have a better solution for replication, then please share it.

December 22, 2009 1:23 pm

Willis,
With vjones’s help and with the aid of EMSmith’s excellent documentation, I’ve been carrying out my own analysis of the NOAA GHCN data. My first step was to reproduce you excellent analysis for Darwin (unlike the Team who think that ‘there’s nothing to see here, move on’). I’ve therefore been applying the scientiic method and have attempted to falsify your analysis. I’m sorry (actually I’m glad) to say that I failed! I’ve reproduced your charts and results almost 100% and have documented my efforts on vjones blog ‘diggingintheclay‘. You can read the thread in which I reproduce your analysis by clicking on the link below.
Reproducing Willis Eschenbach’s WUWT Darwin analysis
As I’m sure you already know and appreciate science progresses by ‘standing on the shoulders of giants’ so I’ve taken the liberty of further extending you excellent analysis for Darwin to all the WMO stations in the NOAA GHCN dataset.
Specifically I’ve attempte dto answer the question posed by others on your original Darwin thread as to whether or not Darwin is a special case or not?
Well judge for yourself by clicking on the link below which documents my extension of your analysis to include the whole NOAA GHCN dataset.
Physically unjustifiable NOAA GHCN adjustments
The following is an excerpt from the thread
“In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data. As can be seen from the following “Cooling turned into warming” table (Table 1) below, which lists the Top 30 WMO station on the “cooling to warming” list, Darwin is ranked in only 26th place! The list is sorted by the absolute difference in the magnitude of the raw to adjusted slopes i.e. the list is ranked so that the worst case of “cooling” converted to significant “warming” comes first, followed by the next worse etc.
It’s clear from looking at the list that Darwin is certainly not “just a special case” and that in fact that there are many other cases of WMO stations where (as with Darwin) NOAA have performed physically unjustifiable adjustments to the raw data. As can been seen from Table 1 many of these adjustments result in trend slopes which are greater than the IPCC’s claimed 0.6 deg. C/century warming during the 20th century said by the IPCC to be caused by man’s emissions of CO2 through the burning of fossil fuels.

KevinUK

Willis Eschenbach
December 22, 2009 2:32 pm

Gail Combs (05:44:57), you raise an interesting and often misunderstood point:

Ryan Stephenson (02:45:27) :
I certainly agree with your approach.
The idea of using stations up to 1500 km or more away “to adjust” a site seems very “dodgy” at best. I do not know Australia, but the whole idea that weather in one location is similar to that in another does not pass the smell test. Within 100 miles of where I live we have oceans, lakes, the piedmont and sand hills. Five miles away ALWAYS gets more rain that I or the airport weather station down the street does. The idea might work OK on a flat plain but not on mountains and ocean front areas. And yes I do understand a significance test for correlations. As a test to determine if the station may have a problem that needs investigation – fine. As the basis for “adjustments” NO.
The whole thing reeks of lazy data handling techniques.

What is used to adjust the data is not the absolute value of the data. It is the change in the data. As you point out, someplace five miles away may get very different temperatures from where you are.
However, what is related is the change in temperature. If you are having a hot month, it is very likely that someplace five miles away is having a hot month as well. It is this relationship that is used for the adjustment.
However, that doesn’t mean that the adjustment is valid. As someone pointed out above, what it can do is re-institute a UHI signal on a station which has been moved.
Someone else pointed out that Darwin is an outlier. While this is true, such outliers should be a huge warning flag to whoever is using the algorithm. People often say “the exception proves the rule” without realizing that “prove” in this expression has the meaning used in the phrase “proving ground”, as in “testing ground”. The real meaning of the phrase is “the exception tests the rule”. In the current case, the rule seems to be failing the test …

Willis Eschenbach
December 22, 2009 2:52 pm

wobble (06:53:46) :

Willis Eschenbach (22:57:54) :

“”our only test based solely on distance was limiting the list of potential reference stations to the 100 stations nearest to each candidate station.””

Great work, Willis.
1. I think you’ve shown that the Peterson algorithm couldn’t have been followed. (Can you just tell us where you read about the 0.8 correlation requirement? Isn’t it possible that they accepted a correlation of 0.4 as long as it was the “highest correlated neighboring station?”)

Not according to the GHCN document, An Overview of the Global Historical Climatology Network Temperature Database, which says (emphasis mine):

Our approach to adjusting historical data is to make
them homogeneous with present-day observations, so
that new data points can easily be added to homogeneity-
adjusted time series. Since the primary purpose
of homogeneity-adjusted data is long-term climate
analysis, we only adjusted time series that had at least
20 yr of data. Also, not all stations could be adjusted.
Remote stations for which we could not produce an
adequate reference series (the correlation between
first-difference station time series and its reference
time series must be 0.80 or greater)
were not adjusted.

2. I think you’ve also shown at least one flaw in the Peterson algorithm even if it is utilized objectively. Testing 100 stations for adequate correlation doesn’t seem reasonable. As you hinted at before, even if 1,000 adjustments are using correlations with a p-value = 0.05, then that implies 50 of those 1,000 correlations were a coincidence.

There are many flaws in the Peterson algorithm, both theoretical and practical.

Overall, I think JJ did you a favor. He was definitely condescending, but he did push you to strengthen your argument quite a bit. However, I agree that it would be nice for him to help out with some of these efforts. He’s obviously smart enough. Unless he’s working on something of his own right now. JJ, do you care to share?

I’m kinda weird, in that I do all the scientific work that I do simply because I’m passionate about climate science. I have persevered and gone forward with my analysis not because of JJ, but in spite of him. These kinds of studies take time, as unlike Peterson and Hanson and their ilk, I have a regular job that does not involve climate science.
I don’t think JJ is a bad guy, I just get upset by constant cavilling with no corresponding productive input. Lead, follow, or get out of the way.

Willis Eschenbach
December 22, 2009 3:10 pm

Ryan Stephenson (08:27:36) :

@Willis: Have you tried taking the approach you/we suggest? Take the data between the discontinuities detected by the algorithm, then check the gradient of that data? Maybe it is difficult because there are only 10 or so datapoints and it is dependent on where you start and stop exactly, but it seems that the gradients between the discontinuities are about 0Celsius/Century compared to the gradient imposed by the corrections. Excel can plot a line of regression between the points, e.g. for the last 12 years we seem to have stable data and Excel can plot a line of regression between the last 12 points of the raw data and the gradient of this line can be derived. Sorry to dump the extra work on you but if you have the data tabulated already it should take you long.
That would really blow this whole thing out of the water. After all, what the algorithm itself is telling us is the site changed about three times after 1940 but was stable as a measuring site between those changes. So that is where the quality data is – the raw data between the site changes. Measure the gradient between the changes and its job done. It sure isn’t 6Celsius/century. So the algorithm will be shown to fail the sanity check badly.

The problem is that the method you suggest assumes that there really were some kind of problems needing adjustment in those years. I find nothing, either in the record itself or in the station metadata, indicating any problem in those years. The Aussies make adjustments based (very loosely, it appears) on the metadata … but they don’t make a single adjustment in any of the years adjusted by GHCN.

Willis Eschenbach
December 22, 2009 3:36 pm

temp (12:33:00) :

As I said originally, I am assuming he is giving correlations over the entire time series because he isn’t giving starting and ending years for producing the correlation, which isn’t what they did, I think.
If that is wrong, then fine, but it isn’t clear from what he has written.

My apologies for the lack of clarity, temp, you raise an important point. For the 1920 step, I used the correlation for the period 1900-1940, or as much of that period as the station covered. There is nothing I could find in their documentation which describes what period they actually used.
Nor does their documentation specify a couple of other important things. One is whether the correlation is statistically significant. If we flip two coins, and then flip two other coins, they may come up all heads (correlation = 1.0 between the two sets of two). However, because of the length of the dataset (N=2), the correlation is not statistically significant. A ten year temperature dataset is generally not long enough to establish significance.
The other is the effect of autocorrelation on the significance. (Autocorrelation measures how much this year’s data resembles last year’s data.) Temperature records tend to have fairly high autocorrelation, a warm year is likely to be followed by a warm year and vice versa. As a result, we are more likely to get a spurious correlation between temperature series.
I chose a forty-year period because of those two factors, which make a shorter time period more likely to produce spurious correlations. I did not use the entire period of the overlap between two stations because the correlation on that is likely to be lower than on the 40 year period, and I wanted to give the algorithm every chance. However, I still found nothing resembling Darwin’s record in the “neighboring” stations.
Which brings me back to my point that nature is not homogeneous. It is freckled and fractured, a “ghost of rags and patches”. Trying to force it into a homogeneous strait-jacket can be done … but what we end up with, almost by definition, will be “un-natural”. Nature specializes in jumps and changes, in instant inversions and retroversions, and not in bland homogeneity. As the poet said:

Pied Beauty
GLORY be to God for dappled things—
For skies of couple-colour as a brinded cow;
For rose-moles all in stipple upon trout that swim;
Fresh-firecoal chestnut-falls; finches’ wings;
Landscape plotted and pieced—fold, fallow, and plough;
And áll trádes, their gear and tackle and trim.
All things counter, original, spare, strange;
Whatever is fickle, freckled (who knows how?)
With swift, slow; sweet, sour; adazzle, dim;
He fathers-forth whose beauty is past change:
Praise him.
Gerard Manley Hopkins (1844–89). Poems. 1918

Homogenize that and see how it all looks … not pretty.

wobble
December 22, 2009 3:37 pm

Willis Eschenbach (14:52:53) :
“”according to the GHCN document,…the correlation between first-difference station time series and its reference time series must be 0.80 or greater””
Thanks! Again, great job.

Ryan Stephenson
December 22, 2009 4:38 pm

@Willis: I think maybe you misunderstand me. I am not suggesting that the algorithm that detects the discontinuities was right or wrong. That doesn’t matter to me. What matters to me is if the adjustment that was made, automatically or manually, and whether that adjustment was reasonable. Up till now you have been asking “the Team” to justify their adjustments, and waiting…. an waiting….. But I have suggested an approach that will indicate if their adjustments were wrong or not without their help.
So lets assume the algorithm that detects the discontinuities is doing so “willy-nilly”. So what? The Teams algorithm tells us that between 1961 and 1980 there was no need for any adjustment – because the black line is perfectly flat here. This means that the algorithm is telling us that the site was perfectly stable during that time and that the temperature measurments were valid. So let us look at the raw data between those dates. We can pick out the specific years that are between the adjustments that were made – about 17 years altogether – then we can plot a line of regression between them. The gradient of that line of regression will give us the rate of change of temperature. I haven’t done the plot, but just looking at it suggests about 2Celsius per century during that period*. But the gradient of the adjusted data shows a gradient of 6Celsius per century. These two facts are in direct contradiction. The site is detected to have been apparently stable for 17 years, by the Teams own algorithm, so the measured rate of change during those 17 years should be valid, by the Teams own criteria. But their own adjustment method comes up with a rate of change three times higher than this. The adjustment method fails this simple sanity check. There is far more adjustment than is necessary.
*naturally the measured rate of change is not attributable to any specific source. Could be CO2, could be UHI, could be something else.

Ryan Stephenson
December 22, 2009 5:10 pm

@Willis: OK, I tried to do it myself, but GISS/GHCN are both unable to deliver me the raw data 😉
I read the results from the graph you have shown of the raw data during about 1962 to 1979 and stuffed them into Excel, which was then used to add a linear line of regression. The gradient of that line came out at 3.5Celsius per century. Very high, certainly, but definitely not 6Celsius per century. The algorithm used to make the adjustment fails the sanity check – the adjustments used at Darwin are not reasonable.
Before anybody says “wow, 3.5Celsius per century- we’re all going to fry” remember this is data over only a 17 year period, it is preceeded by a long cooling period of about 1Celsius per century lasting some 50 years or more, and we aren’t able to indentify the source of the warming trend – it may not be CO2. I think if you did the same analysis of the full dataset using this approach then you would probably see that temperatures today are about the same as they were in 1880. Shame we can’t get hold of the raw data as it seems to have been, um, taken off line.