Guest Post by Willis Eschenbach
Recapping the story begun at WUWT here and continued at WUWT here, data from the temperature station Darwin Zero in northern Australia was found to be radically adjusted and showing huge warming (red line, adjusted temperature) compared to the unadjusted data (blue line). The unadjusted data showed that Darwin Zero was actually cooling over the period of the record. Here is the adjustment to Darwin Zero:
Figure 1. The GHCN adjustments to the Darwin Zero temperature record.
Many people have written in with questions about my analysis. I thank everyone for their interest. I’m answering them as fast as I can. I cannot answer them all, so I am trying to pick the relevant ones. This post is to answer a few.
• First, there has been some confusion about the data. I am using solely GHCN numbers and methods. They will not match the GISS or the CRU or the HadCRUT numbers.
• Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.
• Next, a number of people have claimed that the reason for the Darwin adjustment was that it is simply the result of the standard homogenization done by GHCN based on comparison with other neighboring station records. This homogenization procedure is described here (PDF).
While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way. Instead they have adjusted them differently (see below). This argues strongly that they were not done by the listed GHCN homogenization process. Any process that changed one of them would change all of them in the same way, as they are nearly identical.
• Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert. You can’t adjust Darwin based on that. The GHCN homogenization method requires five well correlated neighboring “reference stations” to work.
From the reference cited above:
“In creating each year’s first difference reference series, we used the five most highly correlated neighboring stations that had enough data to accurately model the candidate station.”
and “Also, not all stations could be adjusted. Remote stations for which we could not produce an adequate reference series (the correlation between first-difference station time series and its reference time series must be 0.80 or greater) were not adjusted.”
As I mentioned in my original article, the hard part is not to find five neighboring stations, particularly if you consider a station 1,500 km away as “neighboring”. The hard part is to find similar stations within that distance. We need those stations whose first difference has an 0.80 correlation with the Darwin station first difference.
(A “first difference” is a list of the changes from year to year of the data. For example, if the data is “31, 32, 33, 35, 34”, the first differences are “1, 1, 2, -1”. It is often useful to examine first differences rather than the actual data. See Peterson (PDF) for a discussion of the use of the “first-difference method” in climate science.)
Accordingly, I’ve been looking at the candidate stations. For the 1920 adjustment we need stations starting in 1915 or earlier. Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:
WYNDHAM_(WYNDHAM_PORT) = -0.14
DERBY = -0.10
BURKETOWN = -0.40
CAMOOWEAL = -0.21
NORMANTON = 0.35
DONORS_HILL = 0.35
MT_ISA_AIRPORT = -0.20
ALICE_SPRINGS = 0.06
COEN_(POST_OFFICE) = -0.01
CROYDON = -0.23
CLONCURRY = -0.2
MUSGRAVE_STATION = -0.43
FAIRVIEW = -0.29
As you can see, not one of them is even remotely like Darwin. None of them are adequate for inclusion in a “first-difference reference time series” according to the GHCN. The Economist excoriated me for not including Wyndham in the “neighboring stations” (I had overlooked it in the list). However, the problem is that even if we include Wyndham, Derby, and every other station out to 1,500 km, we still don’t have a single station with a high enough correlation to use the GHCN method for the 1920 adjustment.
Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.
• Next, the GHCN does use a good method to detect inhomogeneities. Here’s their description of their method.
To look for such a change point, a simple linear regression was fitted to the part of the difference series before the year being tested and another after the year being tested. This test is repeated for all years of the time series (with a minimum of 5 yr in each section), and the year with the lowest residual sum of the squares was considered the year with a potential discontinuity.
This is a valid method, so I applied it to the Darwin data itself. Here’s that result:
Figure 2. Possible inhomogeneities in the Darwin Zero record, as indicated by the GHCN algorithm.
As you can see by the upper thin red line, the method indicates a possible discontinuity centered at 1939. However, once that discontinuity is removed, the rest of the record does not indicate any discontinuity (thick red line). By contrast, the GHCN adjusted data (see Fig. 1 above) do not find any discontinuity in 1941. Instead, they claim that there are discontinuities around 1920, 1930, 1950, 1960, and 1980 … doubtful.
• Finally, the main recurring question is, why do I think the adjustments were made manually rather than by the procedure described by the GHCN? There are a number of totally independent lines of evidence that all lead to my conclusion:
1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).
2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).
3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).
4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).
5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. Here are those pictures:
Figure 3. Darwin station data before and after GHCN adjustments. Upper panel shows unadjusted Darwin data, lower panel shows the same data after adjustments.
Before the adjustments we had the station Darwin Zero (blue line line with diamonds), along with four other nearby temperature records from Darwin. They all agreed with each other quite closely. Hardly a whisper of dissent among them, only small differences.
While GHCN were making the adjustment, two stations (Unadj 3 and 4, green and purple) vanished. I don’t know why. GHCN says they don’t use records under 20 years in length, which applies to Darwin 4, but Darwin 3 is twenty years in length. In any case, after removing those two series, the remaining three temperature records were then adjusted into submission.
In the “after” picture, Darwin Zero looks like it was adjusted with Sildenafil. Darwin 2 gets bent down almost to match Darwin Zero. Strangely, Darwin 1 is mostly untouched. It loses the low 1967 temperature, which seems odd, and the central section is moved up a little.
Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value? Seems to me you’d be better off picking any one of the originals, because they all agree with each other.
The reason you adjust is because records don’t agree, not to make them disagree. And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.
So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims. I’m happy to be proven wrong, and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.
Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.
Sponsored IT training links:
Want to pass HP0-J33 at first try? Gets certified 000-377 study material including 199-01 dumps to pass real exam on time.
Don’t you mean point six (0.6C) per century?
Very nice analysis. Keep up the good work Mr Eschenbach.
Willis, could you please lead me to the red “homogenized graph” at the GHCN site? Thanks.
In general terms, am I correct in understanding that the “time of observation bias” inserted every time Hansen’s GISS program runs artificially lowers temperature data before 1970, and raises (or keeps the same data – but with no Urban Heat Island corrction) for all temperature data after 1970. From what I recall, artificial TOBS changes account for over 0.15 of the total 0.4 degree rise in the “supposed” ground temperature record.
Or over 1/3 of Hansen’s entire “measured” climate change is artifically inserted.
If so, why do they claim a TOBS correction is required at all?
Aren’t the only numbers used by Hansen/NOAA the day’s maximun and minimum values? How would those change based on what time of the day you read the max/min thermometer?
(Never mind whatever his “logic” is in using the same TOBS change for every record in every year. (How many times did these earlier weathermen keep changing the time of day they wrote temperatures down?)
Glenn (19:50:45) “Don’t you mean point six (0.6C) per century?”
See figure 1. If you need a grade 10 math tutorial, speak up.
Good work.
Correction: “highly improbability” should be “highly improbable.”
willis, have u seen this?
19 Dec: TBR: NZ Study may hold key to faulty world temp data
A long-forgotten scientific paper on temperature trends in New Zealand may be the smoking gun on temperature manipulation worldwide.
Since Climategate first broke, we’ve seen scandal over temperature adjustments by NZ’s National Institute of Water and Atmospheric research, NIWA, which in turn prompted a fresh look at raw temperature data from Darwin and elsewhere.
Now, a study published in the NZ Journal of Science back in 1980 reveals weather stations at the heart of NIWA’s claims of massive warming were shown to be unreliable and untrustworthy by a senior Met Office climate scientist 30 years ago, long before global warming became a politically charged issue.
The story is published in tonight’s TGIF Edition, and has international ramifications.
That’s because the study’s author, former Met Office Auckland director Jim Hessell, found a wide range of problems with Stevenson Screen temperature stations and similar types of unit.
Hessell debunked a claim that New Zealand was showing a 1C warming trend between 1940 and 1980, saying the sites showing the biggest increases were either urbanized and suffering from urban heat island effect, or they were faulty Stevenson Screen installations prone to showing hotter than normal temperatures because of their design and location.
One of the conclusions is that urbanized temperature stations are almost a waste of time in climate study:
“For the purpose of assessing climatic change, a ‘rural’ environment needs to be carefully defined. Climatic temperature trends can only be assessed from rural sites which have suffered no major transformations due to changes in shelter or urbanisation, or from sites for which the records have been made homogenous. Urban environments usually suffer continual modifications due to one cause or another.”
“It is concluded that the warming trends in New Zealand previously claimed, are in doubt and that as has been found in Australia (Tucker 1975) no clear evidence for long term secular warming or cooling in Australasia over the last 50 years [1930-1980] exists as yet.”
Hessell divided weather stations in New Zealand into two classes, “A” and “B”, where A class locations had suffered increasing urbanization or significant site changes, and B class locations had not.
“It can be seen immediately that the average increase in mean temperatures at the A class stations is about five times that of the B class”, the study notes.
Among the studies listed as a contaminated A site is Kelburn, which was at the heart of the NIWA scandal a fortnight ago.
A link to the study can be found in TGIF Edition.
http://briefingroom.typepad.com/the_briefing_room/2009/12/nz-study-may-hold-key-to-faulty-world-temp-data.html
Glenn,
No, look at the very top chart…from 1941 to about 1990 (50 yrs), the trend is +3 degrees. So it would be 6 deg/century.
Glenn (19:50:45) :
Nope, that’s the amazing part. Six degrees per century.
tokyoboy (19:54:02) :
I made the graph with data from the GHCN site. The data is at
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2
in the file called “v2.mean_adj
w.
Oh, I see.. they adjusted all the temps before 1952? 53? down by some constant. Notice the 1.7C drop about 1940 is intact. Then a smaller adjustment was applied step-wise to each of the subsequent decades. -2C before 1952, -1.2C for 1953 to 1960-ish, etc.
Thanks Anthony. Darwin has only two seasons each year – “the wet” (now) and “the dry”. Both tend to be pretty hot. It’s been consistently like this just as the unadjusted record accurately indicates.
Is there no end to the fabrications being perpetrated ?
How would those change based on what time of the day you read the max/min thermometer?
If you measure too close to the high point, you get cases where the same “high” carries over to both days. And vice versa if you measure at the low point.
If it’s really cold at dawn Monday and that’s when I measure it, I’ll get low a minute after dawn on Monday with the Sunday’s low being a minute before dawn. Two days of lows taken from a single cold interval. If dawn Tuesday is warmer than dawn Monday, it gets “left out”.
To avoid this, you need to measure at a time far removed from either the typical high point (mid-afternoon) or the typical low point (predawn).
RACookPE1978 (20:03:45) :
This is really a GISS question or a USHCN question, as GHCN do not do a TOBS adjustment per se. For the amount of the adjustment, see here.
Whether it is necessary or not depends on the station, the type of thermometer used, and the times of observation. The canonical document on this is Peterson (q.v.).
evanmjones (20:27:01) :
No, the “day” is midnite to midnite, so you will only get one high or low per day. The low is typically shortly after dawn, and the high somewhere in the late afternoon.
[REPLY – The 24-hour interval between times of observation is the “day” so far as the records are concerned, not midnight – midnight. There’s really no other way to do it. The best observation time is probably around 11 A.M. ~ Evan]
Willis Eschenbach (20:15:45) :
Glenn (19:50:45) :
Don’t you mean point six (0.6C) per century?
“Nope, that’s the amazing part. Six degrees per century.”
Ah. That would make the region the fastest warming area claim in the world, if memory serves, topping the Arctic and Siberia. About 3 times the global surface estimate in any event.
Glen the 6C per century slope is correct. No need to have a PhD to see that.
I already knew everything they explained to me in the special report on Climategate on FOX News except for the way Mann and others referred to McKitrick & McIntyre in the e-mails.
Did those creepy climate scientist walk around the cubicals in their offices and refer to McKitrick & McIntyre as M&M as a joke often?
Apparently so.
Willis Eschenbach (20:18:15) :
>tokyoboy (19:54:02) :Willis, could you please lead me to the
>red “homogenized graph” at the GHCN site? Thanks.
“I made the graph with data from the GHCN site. The data is at
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2 in the file called “v2.mean_adj”
Thanks….. but my PC can’t open the file, probably due to software mismatch.
Willis, don’t you have the data on such a popular software as PDF?
Thanks again.
Nice work Willis. Thank you.
A bit OT … carrying the case into the future.
So let us fast forward 30 years or so. If the top chart “adjusted” trend would continue its upward slope, it would be unbearable at Darwin yet folks will be carrying on as usual. “They” will be telling the Arctic ice has disappeared when it is just fine. “They” will be telling us that the Maldives just sank, when the obviously are fine.
Back to Darwin Zero … If that upward trend continues at some point it would start to look actually silly … nonsensical and YET they could not reverse it could they? Because THAT would be a cooling trend. Gotcha!! ☺ Are “they” LOCKED into this big lie? They cannot reverse it now. Ohoh.
What I am getting at is this: at some future point (even if the BIG LIE continues) when will it all finally fall apart? (I am not as optimistic as others that Climategate can carry this…) When will empirical facts outweight the manipulated data? (IPCC told us in 1991 that sea levels would rise by 300 mm by 2030…we are half way there in time and only 20% there in reality. BUT no one remembers their first report do they?)
Sane folks (like us) know we’ve already reached that point (where observation belies fudged data), but when will it become obvious to people in Darwin, New York, London and Inuvik that the official word is BS?
Just musing aloud on a snowy eve. ☺
Thanks again Willis.
I appreciate your analysis.
There seems to be clear evidence of deception about global warming.
Earth’s climate is changing, has changed in the past, and will always change because Earth’s climate is controlled by the stormy Sun – a variable star.
There is also evidence of government involvement in deception about the composition of the Sun, its source of energy and solar neutrino oscillations.
It appears that politicians have trained scientists with grant funds like Pavlov trained dogs with dog biscuits.
Scientific integrity was the first victim.
My greatest fear is that democracy itself cannot survive if scientists become tools for misinformation by politicians.
With kind regards,
Oliver K. Manuel
Former NASA PI for Apollo
Hi Willis Love your work by the way.
This temperature profile of Australia from the BOM Link, highlights the problem with a 1500 KM difference to different weather stations. To give you a clue 1500 km drops you from Darwin to Alice springs ie top middle down to near the bottom of the first dotted line middle of Australia. ie different heat zone.
http://www.bom.gov.au/cgi-bin/silo/temp_maps.cgi
I did the temperature modeling work when unleaded petrol was introduced to Australia. The problem was cars and petrol pumps were vapour locking. I developed the model that was adopted by the oil industry to related 90 percentile temperature data (using microfish of available temp data) to vapour pressure.
This was developed because all you had to do was move as little as 150km particularly from down south, to end up with a problem. So temperature zones were constructed and vapour pressure limits were established to stop valour locking as soon as you drove out of a capital city.
Thats how I am aware that small variations in distance will give a big difference in average temp data.
P.S. I left the oil industry back in 1997.
In the days of analogue temperature measurement, would time of day be used for max/min readings or would a max/min thermometer (illustration only http://www.redhillgeneralstore.com/A31690.htm ) have been used at these stations? The latter would measure max/min at whatever time that this was reached and would need to be reset once a day. Just curious.
Global Climate Data Tampering Exposed!
Climate Data Tampering In Australia Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/creating_warming_from_cold_austtralian_stats/
Climate Data Tampering In Darwin Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/climategate_how_one_human_caused_darwin_to_warm/
Climate Data Tampering In New Zealand Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/new_zealands_man_made_warming/
Climate Data Tampering In Russia Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/climategate_was_russias_warming_also_man_made/
Climate Data Tampering In Alaska Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/more_man_made_warming_this_time_in_alaska/
Climate Data Tampering In Sweden Detected
http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/making_sweden_warmer/
Fraudulent Hockey Sticks And Hidden Data
http://joannenova.com.au/2009/12/fraudulent-hockey-sticks-and-hidden-data/#more-4660
Would You Buy A Used Car From These So-Called “Scientists”?
Our gullible alarmist friends have obviously fallen for this scam hook, line, and sinker!
First we had the IPCC report “The Science of Climate Change 1995”, where lead auther Benjamin D. Santer removed the following conclusions made by genuine scientists, and without the scientists being made aware of this change.
“None of the studies cited above has shown clear evidence that we can attribute the observed climate changes to the specific cause of increases in greenhouse gases.”
“No study to date has positively attributed all or part [of the climate change observed to date] to anthropogenic [man-made] causes.”
“Any claims of positive detection of significant climate change are likely to remain controversial until uncertainties in the total natural variability of the climate system are reduced.”
Then we have some choice quotes from so-called “consensus scientists”.
“The two MMs [Canadian skeptics Steve McIntyre and Ross McKitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.”
Phil Jones email, Feb 2 2005
“I can’t see either of these papers being in the next IPCC report, Kevin and I will keep them out somehow, even if we have to redefine what the peer-review literature is!”
Phil Jones Director, The CRU
[cutting skeptical scientists out of an official UN report]
“The fact is that we can’t account for the lack of warming at the moment, and it is a travesty that we can’t …there should be even more warming… the data are surely wrong”.
Kevin Trenberth, Climatologist, US Centre for Atmospheric Research
“…If anything, I would like to see the climate change happen, so the science could be proved right, regardless of the consequences. This isn’t being political, it is being selfish. “
Phil Jones Director, The CRU
“We have to get rid of the Mediæval Warm Period” Confided to geophysicist David Deming by the IPCC, 1995
[Many believe that man to be Jonathan Overpeck, which Prof. Deming didn’t deny in an email response, who would later also serve as an IPCC lead author.]
“We have 25 years or so invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it?” Phil Jones Director, The CRU
”We have to offer up scary scenarios, make simplified, dramatic statements, and make little mention of any doubts we might have.” Professor Stephen Schneider
“Humans need a common motivation … either a real one or else one invented for the purpose. … In searching for a new enemy to unite us, we came up with the idea that pollution, the threat of global warming, water shortages, famine and the like would fit the bill. All these dangers are caused by human intervention so the real enemy then, is humanity itself.” Club of Rome declaration
“It doesn’t matter what is true, it only matters what people believe is true…. You are what the media define you to be. Greenpeace became a myth and fund generating machine.” Paul Watson, Co-Founder Greenpeace, Forbes, Nov. 1991
Now what conclusion would a rational and sceptical person come to?
Frederick Seitz, president emeritus of Rockefeller University and chairman of the George C. Marshall Institute, summed it up nicely after seeing the changes made to the IPCC report.
“In my more than 60 years as a member of the American scientific community, including service as president of both the National Academy of Sciences and the American Physical Society, I have never witnessed a more disturbing corruption of the peer-review process than the events that led to this IPCC report.”
“If you measure too close to the high point, you get cases where the same “high” carries over to both days. And vice versa if you measure at the low point.
If it’s really cold at dawn Monday and that’s when I measure it, I’ll get low a minute after dawn on Monday with the Sunday’s low being a minute before dawn. Two days of lows taken from a single cold interval.
To avoid this, you need to measure at a time far removed from either the typical high point (mid-afternoon) or the typical low point (predawn).”
—…—…—
Yes, understood. I’ve heard similar explanation before.
Now, back to the purpose of my question: What (in your answer) or in the physical world of real temperatures and real measurements, actually justifies lowering all of the country’s recorded temperatures prior to 1970?
We get a cold front coming through once every 10 – 16 days (or maybe 25 – 30 days a year), and at that only half the year does that hypothetical cold front change temepratures drastically (in winter really – summer fronts are most often less drastic), and of those few events how many actually affected two days worth of readings, of that small theoretical fraction left, how many actual events really happened?
There is still no reason to use TOBS as a reason to change the earth’s temperature records.
Willis Eschenbach (20:27:37) :
Thank you.
I just don’t see how any rational person can look at what was done at Darwin and defend the results.
Perhaps its too late for a quick “oops, we’ll fix it”. It does kind of amaze me that they circled the wagons with the Economist article. This will just end up making looking even stupider.
Rural Beeville 1895 – 2005
NOAA adjustments add a +2.8°F / Century Trend.
I will post a link to a graph tommorrow.
evanmjones (20:27:01) :
You are correct. According to Petersen the “first order” stations use a calendar day, midnite to midnite. Understandably, some people don’t want to use that interval. So a number of the stations don’t use midnite to midnite. But as long as you are not taking the observation near the time of min or max, it shouldn’t be a problem. Your suggestion of 11 AM is a good one.
However, all of this is really a subject for another thread.
tokyoboy (20:54:32) :
The data is plain text, compressed using “Z”. There are also .zip versions available at
www1.ncdc.noaa.gov/pub/data/ghcn/vw/zipd/
You could try those.
Can someone explain to me why in the top graph the temp. anomaly scale is different on the right side from the left side?
Seems to me that there is no way they could accidentally get that result.
The methodology seems to be striving towards a warming bias…. and deliberately so.
….There. I’ve said it.
My bad, one letter wrong
www1.ncdc.noaa.gov/pub/data/ghcn/v2/zipd/
There must be people who have lived, and more importantly, farmed in Darwin for fifty years. Have these people noticed the climate changing that rapidly?There must be some evidence, such as that they have to farm different things. Anyone from that part of the world here that has noticed things are 3° warmer?
Similarly, I find it hard to look at the figures shown for New Zealand and reconcile that with the actual weather we have. NIWA allege very rapid recent warming, yet there is no discernable difference. I’ve seen no actual evidence presented that our farmers are having to adapt, for example. My parents are keep gardeners, but haven’t noticed that frosts are any lighter and different plants will now grow.
Does this not bother the warming scientists? How do they reconcile their evidence with what they see with their own eyes?
Clive, I made a similar comment on another thread. yes, eventually the disparity between reality and official will be so obvious one would have to be a lunatic not to notice something smells in climate science. What’s worse is the disparity between the computer model predictions and even the fudged temperature data is already too significant. The computer models must now be dumped. Anyone who still relies on them are fools.
Hi Anthony,
As an engineer with a lifetime of experience in electrical measurement I cannot understand that in all the discussion I
have not seen the words Quality Assurance mentioned. All of our measurements and instruments had to have traceability and customers would full access to inspect our labs and our measurements. Every engineering firm dealing with Government of Semi-Government had to have similar QA procedures. Now while I can understand
that it would be difficult to apply QA to scientific theory it appears to me that the measurement of temperature should definitely have QA procedures and documentation. These should be available on request from the authorities concerned as no Government purchasing body in Australia or other Western countries will accept product from firms without QA. It is totally inconcievable to me that Anthony Watts and his crew of amateurs should be required to perform basic site QA .
The fact that not only do we have measurement without any quality assurance, the very processes used are not divulged.
I think all the climate warmists need is a detailed independent Quality Audit and most global warming will disappear!
REPLY: Agreed. I’ve been saying this for years. For example, why doesn’t climate data have an ISO-8000 certificate? Why doesn’t NCDC have an ISO-9000 rating? Private industry does these things, yet goevrnment seems to do everything haphazardly. – Anthony
Nothing going on in Gove, It’s at the same latitude as Darwin, roughly. It needs Homer Geneizing.
http://www.bom.gov.au/jsp/ncc/cdio/weatherData/av?p_nccObsCode=36&p_display_type=dataFile&p_startYear=&p_stn_num=014508
May have answered my own question.
This is from the Australian BOM site http://www.bom.gov.au/climate/cdo/about/definitionstemp.shtml They measure at 9am and it seems that the minimum measured is for that day (the reasonable assumption is that the 9am temp is past the morning minimum on a midnight to midnight cycle) and the maximum recorded is that registered on the previous day. So on these thermometers max/mins are recorded whenever they occur during the day and 9am is a convenient way of reading min. on that day and max. the previous day.
Was this always the case ? as it may effect the recorded temp.
Thanks Willis, for your thoughtful analysis.
Scott of Melb Australia (20:58:57) said:
Hmmm, is that a new unit of measure for temperature data that I am not aware of or did you mean microfiche?
Using the link Willis provided, I found this:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
Listed in there are Canadian stations with serious errors. [I took the time to have a few of my concerns acknowledged by government officials (just to see if they would admit to quality issues – & they did without hesitation).]
hey Willis, petersons say the TOBS code can be purchased for a fee
[snip -waaayyy off topic, sorry]
Willis, I do not understand the two different scales on the left verticla axis and the right vertical axis. They don’t match, should they?
Willis,
Glad to see you sticking to your guns.
Clarification about 1940. The year 1940 is in my official data as the year the station moved from the Post Office to the airport (014016 to 014015).
Other official BOM data show a comparison which, unless mislabelled by them compares these 2 sites (lats and longs given) from Jan 1967 to Dec 1973. There is inconsequential change in the average temperature, though there is a bigger range in monthly data at one station, particularly in min temp.
Although this study was done after the shift, it seems to be compelling evidence that no significant correction needed to be applied to the long term data because of the station shift.
It is an unresolved matter if there were problems at the older PO station before 1940. I have never seen any explanation. I do not know why people make a correction in the few years before the station shift. Until a reason is given, I cannot see cause for anyone to make a step change before 1940, or more specifically, because of the station change in 1940 (which some people report was 1941, but then…). An algorith that makes a step change for no explained reason is hardly trustworthy.
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
You say that it is improbable that a station would show an apparent gradient change to a late rise of 6C/century. You give no basis for this statement. Giorgio’s kind of analysis does not help with that figure, but it does indicate how often an adjustment can produce a rise comparable to Darwin’s. The result is more meaningful when restricted to longer series, since a small change to a short series can too easily produce a large trend change. So looking at stations with more than 80 years in the adjusted record, this histogram shows the position of Darwin 0. It is an outlier – 31’st in a list of 2074 stations, ordered by change in gradient. But not so improbable as to say that it had to be adjusted manually. And 17 stations were adjusted downward by the same amount or more that Darwin was adjusted upwards.
You are wrong in saying that the GHCN algorithm would have adjusted the duplicates equally. The algorithm identifies changepoints by looking at whether the difference in slope between the sections on each side is over a certain limit. That slope depends on the length of sample.
Do you still maintain your underlined charge that
“Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.??
“Santa makes his rounds later this week, but I’ve already received one of the best gifts ever: the complete unmasking of one of the most insidious movements of recent history – the radical effort to force reckless and needless constraints onto the human race in an attempt to change the planet’s climate.”
Mark Davis: The Gift of ‘Climategate’ is a Fresh Start
http://www.dallasnews.com/sharedcontent/dws/dn/opinion/viewpoints/stories/DN-markdavis_1220edi.State.Edition1.1d27b2e.html
JP Miller (22:39:57) :
“Willis, I do not understand the two different scales on the left verticla axis and the right vertical axis. They don’t match, should they?”
Read the legend for the heavy black line, at the bottom of the graph.
Dan Martin (21:42:09) :
Merely for clarity, so the black line (adjustments) doesn’t obstruct the view of what’s been done.
Back to Darwin Zero … If that upward trend continues at some point it would start to look actually silly … nonsensical and YET they could not reverse it could they? Because THAT would be a cooling trend. Gotcha!! ☺ Are “they” LOCKED into this big lie? They cannot reverse it now. Ohoh.
I think they want Copenhagen agreed ASAP because they probably see the temperatures declining in the near future and want to be able to claim that CO2 reduction is the reason. Hence the panic to start CO2 reduction ASAP, before the temperatures drop while CO2 continues to rise. I’m not sure how they would manage to record CO2 dropping enough to make a difference, unless they manipulate the data, but they’d never do that; or they claim the positive multiplier effect also works as a negative multiplier effect in reverse so that a teeny drop could save the world.
“Global warming has of late been a very hot topic in social media, and last week it was hotter than ever. Much of the added fuel came from climate change believers who engaged in the debate that had been dominated by skeptics.”
No Denying the Heat of Global Warming Debate in the Blogosphere
http://pewresearch.org/pubs/1446/global-warming-debate-on-blogs
Neil Gibson (21:52:04) :
Couldn’t agree more. The computer programs (both climate models and the CRU/GISS/GHCN “adjustment” programs) receive no serious investigation at all. There is a whole branch of computer science which develops and uses tools for software Verification and Validation (V&V) and Software Quality Assurance (SQA). These tools are applied to all mission critical software around the world, software for everything from subways to elevators to submarines to airliners to moon shots.
The fact that we are discussing spending many, many billions of dollars based on untested (and in many cases unknown and unexamined) software is a sick joke. We pay more attention to the software running our subways than to global climate models that make unbelievable claims … how bizarre is that?
Earth’s upper atmosphere cooling dramatically
New observations may help detect changes from carbon dioxide increases
http://www.msnbc.msn.com/id/34479085/ns/technology_and_science-space/
Doesn’t anybody understand? Only the past changes. The current temperature always stays the same.
Nick Stokes (22:46:31) :
I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other. The adjustments are supposedly made to correct data because of spurious readings, not preserve a neutral value in the amount of adjustments. It does not make sense to me that there are just as many stations that read too cold as there are that read too hot, in an equal manner. Why would you expect the readings to be off in both directions in an equal amount? What explanation is there for that?
Re: Nick Stokes (22:46:31)
If your intention is to draw attention to problems with the homogenization paradigm, you have succeeded at least twice with your post. Perhaps I have misunderstood and your intention is to give Willis more material…
@ pat (20:10:47) :
I followed your link through to the pdf of Hessell’s paper and note that in addition to his NZ work, he references in the last paragraph a 1975 paper by Tucker finding in Australia “no clear evidence for long term secular warming or cooling in Australasia over the last 50 years.” Tucker, G.B. 1975: Climate: Is Australia’s changing? Search 6 (8); 323-8.
I suspect these two papers are well known to Willis and Anthony (not to mention Salinger), but certainly new to me. Thanks,
David
Dave F
No, the distribution is not perfectly symmetric. The mean trend adjustment is 0.0175 C/decade (upwards). For those >80 year histories, 30 were adjust up more than Darwin. 17 had an adjustment which was bigger but downward.
Wyndham? Wyndham is not only a long way from Darwin, but has a much hotter climate. It is in the Kimberley in north-west Australia. This area is sometimes 5-10 degrees C hotter than Darwin. I went to the Kimberley not so far but inland from Wyndham in October (in the Australian Spring) to a place called the Bungle Bungles NP. When I was there the temperature was well over 40C during the day, while in Darwin it was about 32C at the same time. Once you go inland from Darwin the humidity falls and the temperature climbs. As another example in July (winter), I was in Kakadu NP (say) 300km south-east of Darwin in July. The temperature there was 36C, while it was 30C in Darwin. It is entirely fallacious to modify Darwin temperatures using stations around it, especially those even a short distance inland from the city.
Just wondering if there are other low latitude stations (those in the tropics) in which the unadjusted data might show a similar century long decline in temps. Or even a general unchanged trend. It just seems counterintuitive for temps to be showing a decline to the idea that the planet was recovering from the LIA from the 19th through the 20th century.
Hi Willis and Anthony,
The Quality Assurance question is surely a weapon to be used in the debate.
I will write to my local ( skeptic ) pollie and put it to him. It might be easier to question the complete lack of Quality Assurance by “scientists “than to argue about accuracy especially about software given the garbage I have seen.
The lack of accuracy would naturally come out later.
RE: Nick Stokes (22:46:31) :
You have attempted to divert the argument to make your own point and then try to falsify your own point.
The original post in relation to this temperature set, is relating to the adjustments made at a single station and the fact that they do not conform to either the proposed list of changes that can be made, nor do the adjustments make any sense – hence the question asking how exactly the adjustments were made (as they do not, as i just said conform to the claimed sequence of adjustments).
Instead, you have taken the argument that “well Darwin shows up others show down therefore whats the issue?”. I am unsure of the point you are trying to make? Are you saying that because others adjust downward then the total effect cancels out and therefore the temp record is reliable? In what way, does showing that adjustments were made to other data nulify the fact that there are issues with this station?
The point of this issue, was not to show that the entire database of temperatures have all been adjusted to show warming, rather that this one has, and it does not have adjustments that make sense. In which case, what proportion of the entire dataset contains adjustments that are wrong (either up or down is irrelevent). The fact that the entire response to this station and its issues was to claim that there are an equal number of adjustments to other stations does not excuse the fact that there are issues and is itself, another attempt to mislead and confuse people reading both articles.
YouNews: Giant snowball rolled into street in Madison, WI
Reply: I loved both of these. ~ ctm
This would be a great protest strategy.
Snowmen face down the plows on Gilman Street, Madison, WI
The Great Snowman Wall of Gilman.
Nick Stokes, things aren’t quite as copacetic as Giorgio Gilestro implies.
If you look at the trend of the adjustment over time,, it is far from random. This trend in the data are the explanation for the obviously leptokurtic distribution obtained by Gilestro.
This sort of residual test is a rather elementary one that any of us who actually do data analysis are taught to perform. I’m not surprised Gilestro failed to do it (if only because I don’t know him from Adam), but I’m a bit surprised the GHCN guys didn’t address it though. The high amount of kurtosis itself is also a bit of a warning sign—usually it is an indicator than non-random processes are involved in the observed residuals. (Think about the extreme example of histogramming a sine wave… you’d get a symmetric distribution, but something that is very non-gaussian.)
Nick Stokes (23:24:30) :
The mean trend adjustment is 0.0175 C/decade (upwards).
So there is basically no difference in the trend if you remove all the adjustments, right? It would look the same either way, basically?
David44 (23:22:32) :
Oddly enough, in a much earlier life, I was the number-cruncher for a follow-up study to Brian Tucker’s 1975 paper. Despite the title, we also analysed temperature trends. Again the results were negative. With the better knowledge we have nowadays, that still may have been a correct conclusion for that time. However, that kind of analysis could then only be done much less accurately. GISS temp, which came later, was a huge advance.
Nick Stokes (22:46:31), thanks for your interesting questions.
Not sure what your point is here. What does the average have to do with what happened to Darwin?
I say it is improbable simply because the raw data from Darwin and the sites around it clearly indicate that Darwin is in fact not warming at six degrees per century. I know of no other station on the planet which goes along relatively level for half a century, and then suddenly starts rising at six degrees per century and maintains that trend for a half century.
If you know of one, break it out … but if you don’t know of one, then why don’t you think the claimed Darwin rise is very improbable?
And the fact that the GHCN algorithm may produce other huge adjustments is an indictment of the GHCN method, not of my claim.
In addition, in reading Gilestro (your cite above) you have turned off your skepticism, and that is very dangerous in climate science. Gilestro claims that
Nonsense. Both you and he should know better. 0.2C/decade is 2 degrees per century, and the planet warming trend in the last century was nowhere near that, it was only on the order of 0.06C/decade.
Nor do “the warming and cooling adjustments end up compensating each other” as he claims. First, you would need to grid and area-adjust the temperatures to determine that. It is very sensitive to where the stations are located.
Next, even the raw average which he uses (0.017C/decade) is more than a quarter of the estimated warming, which is is a significant amount.
Given those clearly untrue claims of his, I fear that I can’t trust Giorgio’s numbers or conclusions, and neither should you.
Well … in a word, no. Read the description again, the algorithm doesn’t do that at all. The algorithm looks at the change in the total of the residual sum of squares of the trends on both sides of each candidate year, which is a very different beast which does not depend on the length of sample. I suspect that’s one of the reasons that they use that particular algorithm. You really should run the algorithm, as I have in Fig. 2 above, before drawing conclusions.
Most assuredly. The main reason for still maintaining my statement is that there are no suitable stations within 1500 km. of Darwin to use for the 1920 step, so they could not have used the algorithm that they claimed to use. That algorithm requires 5 stations whose average correlation with Darwin is 0.80. For the 1920 adjustment I can’t find a single station, much less five stations, within 1500 miles that has a correlation greater than 0.43 with Darwin. So I can only conclude that they didn’t use the method they claimed to use.
I list a number of other reasons for maintaining my statement above.
All the best,
w.
Nick Stokes (22:46:31) :
Can “Giorgio’s kind of analysis” also indicate why should I trust ANY record in GHCN database at all – whether adjusted or “UNADJUSTED” means? 🙂 If the algorithm ostensibly creates data for Darwin Airport (prior to 1941) out of “thin air” in adjusted means, where is the guarantee that it didn’t calculate/select/concatenate something funny while doing so called “unadjusted” means, for other stations, too?
Had i not come across this blog, I’d never have the slightest idea how much the climate scientists are massaging the data. I think it’s a scandal – in my layman opinion, it can hardly be called ‘instrumental record’. It’s rather an estimation based on instrument record, of difficult to judge reliability.
John in Australia (23:25:58) :
Aussie John, your intuition and experience is amply supported by the lack of correlation between the Wyndham and Darwin records (first differences of temperatures as used by the GHCN), which is -0.14, in other words, they are not significantly correlated at all …
Which is why Wyndham could not have been used in the GHCN algorithm.
However, it is worth noting that the fact that Wyndham is hotter than Darwin is not the issue. It is that they don’t move in parallel, that there is very little correlation between their swings. If Wyndham warmed when Darwin warmed and cooled when Darwin cooled, we could use it to check Darwin for inhomogeneities. But since it doesn’t, we can’t.
Thank you Willis. This is very interesting.
A thought though.
When talking about climate and Darwin, has any one mentioned Cyclone Tracey.?
Darwin was pretty much destroyed in 1974. It looks like a bit of a jump in the record about then.
Is there any mention of change of instruments or location about then?
According to
http://www.enjoy-darwin.com/cyclone-tracy.html
“Wind speeds of 217 km/h registered at Darwin Airport before the equipment was itself blown away and there are estimates of maximum speeds of up to 300 km/h.”
I suppose this has been said before about Darwin temperature records from 1941 (I have been away and not reading this blog), BUT, Darwin was bombed off the map from Feb 19, 1942, until Nov 12, 1943 by Japanese air raids.
The population withdrew south to Adelaide River and points beyond for years before rebuilding began and any reliable weather service was restored.
Darwin was again smashed on Christmas Day 1974 by an intense cyclone that the local military weather station did not detect.
How could Darwin have adequate and reliable weather records with interruptions and failures such as these?
I just dont understand why they make adjustments to the data in the first place. And why are the adjustments almost always up. Should they not be adjusted down in the citys?.
Leon Brozyna (23:26:37) :
An interesting issue. In general, the tropics has warmed much less than the extratropics. According to the UAH MSU satellite temperatures, there has been no significant warming of the tropics in the thirty years of the satellite record. This inconvenient fact is usually ignored by AGW supporters, who claim that poor tropical countries will bear the brunt of the hypothesized warming.
“RE Dave F (23:15:19) : I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other.”
It does, to some extent come across odd. The strangness of this standard deviation model is hard to grasp unless explained very simply – something that this standard deviation article fails to (and certainly would not want you to notice). A link to graphical analysis is here http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/ this should make it easier to follow the issue.
I will simplify.
A standard deviation of adjustments will show only one thing – how many adjustments were made and in which direction. The thing is, noone is interested in how MANY adjustments were made, or whether the DIRECTION was equal. What is missing is TIME.
There is no plot or analysis on Giorgio’s page relating to WHEN the adjustments were made and unfortunately that is the entire crux of the argument.
If you want an example, take a single station that was basically neutral trend. If you were to apply several cooling adjustments to prior to 1950 (say 0.3 degrees), then several warming adjustments post 1950 for the same amount, you would get a standard deviation of 0, but the temperature trend would have gone from stable to cooling slightly until zomg increeased CO2 + 0.6 degrees increase BURN!!!.
In addition, if you had 2 temperature series one had a trend of 0.3+ the other was stable up until 1960 then had a slight upward trend of say 0.1. we apply 3 adjustments down to the lower end of the first one and get an amplified warming up to 0.6 and then apply the upward adjustments to the higher end of the second one and again, amplified warming – this time showing hockey stick – but again with a 0 standard deviation.
In fact, mathematically if you wanted to attempt to hide this increase, the best way to do it would be to apply corrections that give a 0 standard deviation – this gives it the illusion that there is no issue – and that is all that paper is – an illusion designed to give the semblance of a rebuttal, framed in such a way that makes it look indisputable (unless you actually investigate the data).
In the end, as i said previously the entire argument is moot from the get go. This station has adjustments that appear to be uncalled for, are not explained through GHCN’s own adjustments “manual” and show that the adjusted data may well have errors. The only real way to fix that is to reassess the entire database – not allude to some standard deviation model that just confuses the issue (even though warmists are very good at doing so).
Commenters,
When going down a thread and encountering the words “Nick Stokes” – right away, go up to Edit, Find on this page, enter Nick and then every hit is highlighted. This means that when you see the highlight approaching you can quickly scroll through, thereby saving precious time.
If the nation was not led down the path of global warming alarmism, but rather given a warning of a fierce winter season based on true science, perhaps the retailers and cleanup crews could have been put in place to deal with the mess in order to help improve customer traffic and boost Christmas sales. Too bad, many retailers will be going out of business due to the nation not being prepaired.
US Blizzard Puts Crimp in Holiday Sales, Travel
http://au.news.yahoo.com/a/-/world/6608283/us-blizzard-puts-crimp-in-holiday-sales-travel/
Willis, great work!
minor points;
– third para .. too many ‘independent’s?
– re tthe 1500km limit, they wouldn’t have used data from sites further north would they? (i.e. not in Australia)
Beautiful pictures in this article, like postcard pictures.
Think we have it bad? Try Washington for a whole winter’s snow in one day
http://www.dailymail.co.uk/news/worldnews/article-1237351/Think-bad-Try-Washington-winters-snow-day.html
The reason they brought out the heavy fire from The Economist (which by the way, is also very wrong on our ailing economy) is that they are running scared. This gig is up. The public no longer buys the MSM orchestrated AGW scare story.
Panic time. Yell. Scream. Discredit. Ad Hominem.
I wonder how average temperatures are calculated. It’s not even ingenious let alone scientific. Why don’t we measure those average temperatures directly? Here is how it would work:
Rock is a very good medium to use so drill a horizontal hole into a mountain. Start at one meter or maybe two. Place a temperature sensor at the end and isolate the hole as good as you can. Wait a week. Measure. Repeat for another depth. The age of the averaged temperature should be proportional to the depth. I guess exponential. We would have to calibrate the depth/age relation according to the property of the material like say, Rock Rings.
The nearer you get to the surface the lesser the attenuation will work and you may see the annual, later dayly temperature oscillation. There you may measure in fixed intervals and integrate numerically.
Stevenson screens, pah.
“It is entirely fallacious to modify Darwin temperatures using stations around it, especially those even a short distance inland from the city.”
This might be true, but as I understand it, not for the reasons you give.
If two points have a strong correlation, then it doesn’t matter that they are actually 6° different on any one day, or year, or whatever. An analogy: my sister and I never are the same age, but our ages correlate very strongly. I am prepared to bet she will be 45 when I turn 50.)
I would bet that in the short term a station a short distance inland of Darwin correlates very strongly with Darwin. I think a long term bet on that might fail, due to UHI or other effects.
Nick – most general adjustments are justified to account for the UHI effect. I know of no general reason to say why adjustments should be made the other way. Do you?
If most adjustments are required to reduce the impact of the UHI, then the overall trend in the adjustments should be largely negative.
The fact it is positive is completely bizarre and needs a lot of explaining.
I found this. I haven’t had time to explore further yet.
A change in the type of thermometer shelter used at many Australian observation sites in the early 20th century resulted in a sudden drop in recorded temperatures which is entirely spurious.
http://www.bom.gov.au/climate/change/datasets/datasets.shtml
There is a PDF on the same page that says.
other known data problems. Such discontinuities can be as large, or larger than, real temperature changes (eg. the approximate 1°C drop in maximum temperatures associated with the switch to Stevenson screen exposure) and consequently confound the true long-term trend. Generally the high-quality records were homogenised from 1910, by which time most stations are believed to have been equipped with the current standard instrument shelter. 224 temperature records were reconstructed to an acceptable standard, 181 of which were identified as being non-urban.
Go Willis.
I am starting to think that one should never adjust station records (i.e. the real observed data) and continue to call them by the same name as the raw data station. For example, if an “adjusted” Darwin0 is required for some purpose – then the resulting series – homogenized, stretched, smeared or tilted should always be called by a name that identifies it as such. For example, DarwinHSST023, where such name can be looked up and shown to represent Darwin Homogenized Stretched Smeared Tilted algorithm 023. Thus, whereas Darwin 0 represents a particular station at a particular location between two dates, and never changes (it is after all – within other constraints – real data), the DarwinHSST023 might (say) represent some sort of area algorithm incorporating data from station thousand(s) of km away. This is possibly acceptable if you state its method and use. What is not acceptable is to pretend (by same-name retention) that some HSST actually IS the correct temperature picture of Darwin – rather than what we know was the actually measured temperature. I don’t know why we let them get away with it.
S.1733 Cap-and-Tax Energy Inflation bill text: http://www.opencongress.org/bill/111-s1733/text articles, blog, and can comment on the text of the bill line by line.
AGW proponents have to make a decision now that all the evidence they used to support their case has been discredited and debunked. Either they keep on with the mantra without any evidence in which case they are simply acting like clueless children, or they admit they have very likely got it wrong and they have to go back to square one and try to find evidence to support their case. If you are one of them, which one is it for you?
Has anyone wondered if the 1908 Tunguska comet explosion in Siberia could have effected measurements in some way. This article mentions the comet creating Noctilucent clouds forming a day later thousands of miles away.
http://insciences.org/article.php?article_id=5899
A few posters have pointed out that the scam will be self defeating over time once it becomes evident that the scare mongering projections have not come to pass. However, I doubt things will turn out that way.
In ten or twenty years, or whenever the evidence of our own eyes refutes the warmists prophecies, the world will be a different place. The UN will have set up its monitoring, tax collection, and enforcement regime, or as I like to think of it “Proto-world government”, the MSM will have moved on, climate scientists will have learned their lessons (no more stray emails/more cunningly fudged data), and very frustrated and angry ruling elites will have neutered the internet.
Our only hope is to kill all of this today before it becomes entrenched.
Following weather undeground, Darwin is about to smash its previous record temperature for December (the previous record dates back to 2001).
Is weather undeground also adjusting its values?
Nick Stokes (22:46:31) : on 20/12
GG’s near-symmetric sort of Poisson distribution:
The weakesses with this study are several fold. One is evident from a station in an older city with UHI. In Melbourne for example, various comparisons with other sites show it likely that UHI ‘maxed out’ at Melbourne Central some time before 1950. That is, from humble beginnings, the temperature from UHI rose steadily to (say) the 1930s and then remained about constant because nothing was explainable or available to push the UHI much higher. So Melbourne would be one of the stations in an 80-year plot that whould be near the median, despite having UHI (from studies already referenced) of up to 10 deg C.
Depending on the geographic location of the thermometer with respect to development around it, there must be a host of other cities in the gg data set that show that same “false” response. That is, one part of the comparison between adjusted and unadjusted is incomplete.
I take you pount that one swallow does not make a Spring – but I am using a town familiar to me to reduce the likelihood of spurious effects unknown to me. I would strongly suspect that Adelaide and Sydney have the same problem. Even a 100-year analysis might fail to pick up the main part of UHI.
One swallow might make a tablet go down better.
Your accurate analysis has been invalidated by governement computer models and a global consensus, therefore your work remains accurate but unacceptable.
Is it possible to incorrect something to make it ‘correct’? In anycase make sure you continue not doing this, but I apologise that this is simply not acceptable!
Good work Willis (the unacceptable)
Dr David Bellamy, well known BBC presenter (who no longer seems to appear on the BBC) has written to the Daily Telegraph thus…
SIR – The only good news to come out of Copenhagen is that, in the words of Greenpeace: “There are no targets for carbon cuts and no agreement on a legally binding treaty.”
Hooray! Along with tens of thousands of global-warming sceptics, the world can now breathe a sigh of relief and return to the sanity of real science, which counsels that carbon dioxide is not a poison, let alone likely to cause a heat-driven Armageddon.
We can now burn non-sulphurous coal again to ameliorate the effects of the colder climate that has already been with us for the past decade and is likely to stay for the next 30 years.
Dr David Bellamy
Bedburn, Co Durham
http://www.telegraph.co.uk/comment/letters/6851649/After-a-decade-of-global-cooling-we-should-sigh-with-relief-at-Copenhagens-failure.html
Michael (00:47:30) :
An excellent case example of the end result of AGW driving public policy the wrong way on the freeway. The MSM is glued to the Agenda of pumping as much Global Warming through Going Green advertising to an obsession with “Scientists find more evidence of Global Warming”. What the MSM should be reporting is the increasing cold & snows of Southern and Northern Hemisphere winters.
i.e. – the MSM should be focused on what is going on rather than what is being wrongly predicted year after year after year.
Willis Eschenbach (23:51:16) :
The reason why I believe that GG’s histogram is highly relevant goes back to your reaffirmed statement
You can’t change the global record by changing one station. If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception.
Anyway, I’ve collected the argument in a post on my blog site. I’ve also calculated the statistics of trends since 1940, to show that Darwin’s post-adjustment trend is in the upper tail, but not extreme. And it would not stand out among the unadjusted station data either.
pete m (01:25:23) :
GHCN homogenization does not adjust for UHI. It corrects for discrete events that it detects by time series analysis. Things like station moves, equipment changes, MMTS introduction. There’s no particular reason to expect those to go up or down on balance, but no reason not to either.
Someone mentioned Bethell’s paper upthread, in which he said back in the ’70’s that some noted NZ stations had decaying Stevenson screens. Well, those would have been replaced in the years since. Each replacement would have caused an adjustment. It would say that there was spurious warming in the past (screen decay), and push the readings down. etc
O/T: Mayon looking ominous, raised to alert level 4 yesterday.
Volcano Live
Guardian
Times
ABC News
Carrick (23:45:16) :
Check the comments at Romanm’s blog for discussion of his diagram. The pattern that he shows is what you expect, because if plots the cumulative effect of adjustments as you go back in time. And the slope is about equal to the 0.0175 C/decade of GG. As I said there, they represent two different variable plots of the same data. If you take trends over time, and look at the distribution over stations, you get GG’s. If you average over stations and plot over time, you get Roman’s. The trend of Roman’s is equal to (approx) the average of GG’s.
Michael R (23:33:12) :
I am unsure of the point you are trying to make? Are you saying that because others adjust downward then the total effect cancels out and therefore the temp record is reliable? In what way, does showing that adjustments were made to other data nulify the fact that there are issues with this station?
No. I think that the adjustments are made for a good reason. I’m answering the suspicion that they are done to deliberately enhance warming. The fact that the nett trend is small means that they don’t have that effect (or only to a small extent). Crudely, if someone was trying to rig the effect, they aren’t doing very well.
Did the Met office ever release the global datasets of raw temperature?
It’s actually quite simple if you stop and think about it.
Darwin is in Australia. Things are reversed here. Christmas is in midsummer. It gets cooler as you go South. The water goes down the toilet in the opposite direction.
So when the raw data show temperature going down, the graph should be inverted to match Northern Hemisphere norms.
Isn’t that standard practice for climate scientists?
Is there a record of the number of corrections for inhomogeneity and what direction these are in? An assessment of the formal guidelines used should be revealing – are there more reasons to adjust data upwards than down?
The GHCN, GISS, HadCrut and all there are tries to find the trend line in any global temperature mean. Why would that require any adjustments at all apart from those trying to adjust for biases induced by human activity. That would be solely UHI-effects or land use changes I would assert.
For example, a specific station time of observation would have a trend line which would look the same or at least as much erroneous regardless of the time of day it was observed? The trend for a 3 o clock observation would contain as much global warming as any trend for a 5 o clock series of observation, wouldn’t it?
I try to read all the rationale for adjustments but cannot make any sense out of it. The gridding for example, what is that for? The neighboring station mumbo-jumbo, how is that justified? It seems that when the task is to try to find a trend line in a global mean temperature, all the data should be included since any error in the data would go in both direction. The Darwin station for example, wouldn’t there be a station somewhere in the world that has a funny up spike in 1939 the compensates the Darwin down spike?
As a long time engineer, I try to find KISS solutions an all similar problems, the “state of the art ” science in this field has complicated matters beyond recognition.
I would love to be educated on this.
Great work!!
Has there been any reaction to your open letter to the Ecomomist?
Amendment:
I read the abstract of the Peterson paper, it seems to me that the method described therein is to be able to find the actual temperatures, i.e. the absolute values that are comparable with other stations, when such comparison is needed/wanted/called for, it wouldn’t apply to finding high pass filtered temperature trends, why would it?
Maybe I am jumping to conclusions, I always do that…
“You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. ”
Cogent? GG appears to believe that if a merchant pushes down on one arm of the scales as heavily as he pulls up on the other, that that’s a fair transaction.
Someone else posted the graph above from RomanM’s analysis, here:
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/
How you can look at that timewise adjustment ‘trend’ and pretend this is innocent is beyond me.
@Ken Seton
Station records that are unclear labeled resemble GM food in the US and Canada:
from the label you do NOT know what you are dealing with.
Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up….
YOU miss the point entirely. There is absolutely no reason to adjust these records period. There is no reason to adjust ANY of the records unless a very good VALIDATED reason exists. Saying some pie in the sky method was used to do the adjustment does not justify tampering with the data.
I ran a quality lab. If I look at graphs of chemical analysis and find one does not “match” the typical data does that mean I have reason to “adjust ” the data or do I go look for the actual cause? If I do not find a problem with calibration, sampling method or whatever am I justified in arbitrarily “correcting” the data. Do I place the corrected numbers on the Certificate of Analysis or do I take it the data is telling me something – like the product should be rejected.
I’m still left with a couple of questions.
For those who say that individual stations don’t matter, the global stats will sort things out
If, when scaled globally, local “adjustments” have no real impact … why bother “adjusting” in the first place? Adjustments at the individual station level obviously do impact both regional and global results. Especially if the end result is in the 1/10th degree range.
Specifically for Darwin adjustments.
If you take the difference between the “winter” and “summer” means over the entire record (25/28 from v2.mean) as being about 3°C. Is a 2.25°C “adjustment”, in any direction, at any point, even physically possible?
To put the level of Darwin adjustment in context. If you applied it to Helsinki the “Darwin adjustment” level would be around 17°C. ({v2.mean} ((summer 17) – (winter -6))*0.75) over almost 200 years.
Both adjustments seem pretty much impossible.
(real Helsinki adjustment looks to be (peak/trough) +1.25 to -0.7 (in nice clear edged steps) over 200 years – pretty much neutral overall)
Dear Mr Eschenbach
Another erudite post please keep up the good work. Is there any chance of you revisiting your work on the CET record?
Sorry my hobby horse as always been that a global temperature as such is a nonsense given the spread of data. Howver temperature records of individual countries could show if an area is warming or cooling.
Your forensic work has the beauty of showing that hockey stick graphs are being mad up out of measured temperatures, just as in New Zealand.
In the past four weeks, we have an area of Australia where a hockey stick can be demonstrated to have been manufacture, Russia where they say that much data has been ignored and badly manipultated, New Zealand.
You couldn’t make this up, they did.
*************
Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
***********************
Nick: An adjustment histogram just hides the actual adjustments – it is meaningless. Why? Because WHEN the adjustment is made is key. If most of the downward adjustments are made before half the time of the total length of the record and most of the upwards adjustments after half the time, then the trend up will be enhanced. I’m really tired of this statistical shell game and this isn’t the first time I’ve seen these histograms trotted out as proof of fairness. They are bogus and designed to deceive.
Your’s and other independent analysis will soon be irrelevant if Gordon Brown and other world leaders have their way.
Excerpt:
A new global body dedicated to environmental stewardship is needed to prevent a repeat of the deadlock which undermined the Copenhagen climate change summit, Gordon Brown will say tomorrow.
The UN’s consensual method of negotiation, which requires all 192 countries to reach agreement, needs to be reformed to ensure that the will of the majority prevails, he feels.
The Prime Minister will say: “Never again should we face the deadlock that threatened to pull down those talks. Never again should we let a global deal to move towards a greener future be held to ransom by only a handful of countries. One of the frustrations for me was the lack of a global body with the sole responsibility for environmental stewardship.
“I believe that in 2010 we will need to look at reforming our international institutions to meet the common challenges we face as a global community.” The summit failed to produce a political agreement among all the countries. Delegates instead passed a motion on Saturday “taking note” of an accord drawn up the night before by five countries: the US, China, India, Brazil and South Africa. ”
End quote
Gordon Brown (UK ) is following the “Belgrade Process” script. Right on schedule. http://www.timesonline.co.uk/tol/news/uk/article6963482.ece#… .
So Aussies did their own..I would not say “Hockey Stick” but “Kangaroo Tail”
Glenn (19:50:45) :
Ah. That would make the region the fastest warming area claim in the world, if memory serves, topping the Arctic and Siberia. About 3 times the global surface estimate in any event.
Last I heard, all the warmining the warmists were claiming was 0.6. So, this would be 10 X that.
Norm in Calgary (22:56:52) :
…. I’m not sure how they would manage to record CO2 dropping enough to make a difference, unless they manipulate the data, but they’d never do that; or they claim the positive multiplier effect also works as a negative multiplier effect in reverse so that a teeny drop could save the world.
That is REAL easy. Mauna Loa has active vents putting out CO2. The official CO2 numbers we get are “calculated” to remove the amount of CO2 contributed by the volcano.
Nick Stokes (22:46:31) :
“I am sorry, but I still find it a bit odd that adjustments to temperature balance out. I would expect there to be a bias in one direction or the other.
When you say ‘balance out’, I presume that you mean over the entire instrumental temperature history. ”
Couldn’t you get a balancing out even if, let’s say, the pre 1960s (I choice that year arbitrarily) temperatures were adjusted significantly lower and the post 1960s were adjusted significantly higher?
Has anyone tested this possibility?
Bold are my highlight, italics my comments.
1120593115.txt
From: Phil Jones xxxxxxxx
To: John Christy xxxxxxxx
Subject: This and that
Date: Tue Jul 5 15:51:55 2005
Also this load of rubbish!
This is from an Australian at BMRC (not Neville Nicholls). It began from the attached article. What an idiot. The scientific community would come down on me in no uncertain terms if I said the world had cooled from 1998. OK it has but it is only 7 years of data and it isn’t statistically significant.
The Australian also alerted me to this blogging! I think this is the term! Luckily
I don’t live in Australia.
[1]http://mustelid.blogspot.com/2005/06/first-look-at-scs-msu-vn52.html
Unlike the UK, the public in Australia is very very naive about climate change, mostly because of our governments Kyoto stance, and because there is a proliferation of people with no climate knowledge at all that are prepared to do the gov bidding.
Australian farmers and Aboriginal people know all about floods and droughts. This urban [snip] wants us to believe it’s caused by cow farts.
Hence the general populace is at best confused, and at worst, antagonistic about climate change – for instance, at a recent rural meeting on drought, attended by politicians and around 2000 farmers, a Qld collegue – Dr Roger Stone – spoke about drought from a climatologist point of view, and suggested that climate change may be playing a role in Australias continuing drought+water problem. He was booed and heckled (and unfortunately some politicians applauded when this happened) – that’s what we’re dealing with due to columists such as the one I sent to you.
Need I say more? The useless [snip] thinks he can teach Australians about the climate of a continent he has never set foot in.
It appears Nick Stokes only answers the easy questions. I think that pretty much demonstrates his INTENT. His only reason for being here is to insert some measure of doubt. Sorry, Nick, not working.
Yes, Darwin is a single station … but … we have all of New Zealand in doubt, we have all of Scandinavia in doubt, we have most of Russia in doubt and it seems just about every time anyone looks at a new location it has been adjusted upward by a questionable amount.
I don’t think there’s much doubt left anymore.
Good work, Willis.
Some issues that require reconsideration:
“Next, some people have said that these are not separate temperature stations. However, GHCN adjusts them and uses them as separate temperature stations, so you’ll have to take that question up with GHCN.”
GHCN treats these as ‘duplicate’ station records, not as separate stations. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
“While it sounds plausible that Darwin was adjusted as the GHCN claims, if that were the case the GHCN algorithm would have adjusted all five of the Darwin records in the same way.”
That is not true. The GHCN algorithm could easily operate differently on different records, even if they are very close (or perhaps even if they are identical) in their values.
The records are of different lengths. This could easily change which ‘neighboring’ stations correlate with the records and thus which stations are used for the identification of discontinuities and application of adjustments. Even when the same stations are used for the adjustment, record length may also affect how discontinuities are identified and how adjustments are applied. You may not yet conlcude that the standard GHCN adjustment proceedure was not correctly applied.
“Next, there are no “neighboring records” for a number of the Darwin adjustments simply because in the early part of the century there were no suitable neighboring stations. It’s not enough to have a random reference station somewhere a thousand km away from Darwin in the middle of the desert.”
You repeat your mistake of the first analysis, by applying your own criteria for suitability, rather than GHCNs. GHCN does not appear to have any specific distance limit. GHCN appears to only be concerned if the stations are in the same ‘region’ (climatalogically) . You cannot say that GHCN did not apply its standard method. You need to ask.
“Here are all of the candidate stations within 1,500 km of Darwin that start in 1915 or before, along with the correlation of their first difference with the Darwin first difference:”
Which Darwin first difference? The correlation between any given station and any of the five Darwin duplicate records may be very different. Which one of the five do you report here? What do the other four look like?
“Now I suppose you could argue that you can adjust 1920 Darwin records based on stations 2,000 km away, but even 1,500 km seems too far away to do a reliable job. So while it is theoretically possible that the GHCN described method was used on Darwin, you’ll be a long, long ways from Darwin before you find your five candidates.”
So?
The assertion is that stations within the same ‘region’ are sufficient for use as reference stations, when making homogenizations of datasets desgned to be used at ‘regional’ or larger scales. You have not addressed that, not that it matters one whit to your claim that the adjustment wasnt applied correctly.
“1. It is highly improbability that a station would suddenly start warming at 6 C per century for fifty years, no matter what legitimate adjustment method were used (see Fig. 1).”
True. This is a potential indictment of the adjustment method, not proof that the adjustment wasnt properly applied. Note that to serve as an attack on the adjustment method, you would need to demonstrate that the adjustment method was properly applied here, and that the adjustment had a significant effect on the temperature measurement *at regional and larger scales*.
Recall from our previous discussions that the GHCN method specifically allows for oddball results that do not track local temps. They assert that these weird adjustments are rare (i.e. ‘highly improbable’ results are consistent with the methodology), and have little if any effect on the aggregate results.
“2. There are no neighboring stations that are sufficiently similar to the Darwin station to be used in the listed GHCN homogenization procedure (see above).”
Simply put, you have not shown that. You conclusion is not warranted.
“3. The Darwin Zero raw data does not contain visible inhomogeneities (as determined by the GHCN’s own algorithm) other than the 1936-1941 drop (see Fig. 2).”
Are you sure you applied that completely and correctly?
“4. There are a number of adjustments to individual years. The listed GHCN method does not make individual year adjustments (see Fig. 1).”
I believe that you may be incorrect about that.
“5. The “Before” and “After” pictures of the adjustment don’t make any sense at all. ”
Sense with respect to what? If your claim is that the GHCN methodology was not applied, then you must prove that they do not make sense with respect to the GHCN methodology. You have not done that.
If your claim is that the GHCN methodology is not sufficient for estimating long term trends at large scales (regional and larger), then you need to prove that the adjustments do not make sense with respect to estimating long term trends at large scales (regional and larger). You have not done that either.
At this point, all you have proven is that they do not make sense to you. If they do make sense with respect to the methodology and its stated goals, then all you have proven is your own ignorance.
“Call me crazy, but from where I stand, that looks like an un-adjustment of the data. They take five very similar datasets, throw two away, wrench the remainder apart, and then average them to get back to the “adjusted” value?”
When did they average them to get an adjusted value?
“The reason you adjust is because records don’t agree, not to make them disagree.”
That may be a successful line of attack – you have to demonstrate the effect on the aggregate result first.
“And in particular, if you apply an adjustment algorithm to nearly identical datasets, the results should be nearly identical as well.”
The datasets are not nearly identical. They differ greatly in length.
“So that’s why I don’t believe the Darwin records were adjusted in the way that GHCN claims.”
Once again, you are over reaching. Your conclusions are not properly supported. As shown above, they are founded in part on your (potentially valid) criticism of the GHCN method, which does not demonstrate that the method was not applied correctly. You need to learn to separate those two issues.
As shown above, your conclusions regarding manual adjustment are also founded in part on your own ignorance regardin some aspect of the method. You really need to ask GHCN to document the specific adjustment for these station records, and replicate them yourself, before making such claims.
“I’m happy to be proven wrong, …”
I doubt that. Being proven wrong would be very embarassing to you. And the rest of us.
“… and I hope that someone from the GHCN shows up to post whatever method that they actually used, the method that could produce such an unusual result.”
Uh, yeah. Thats what GHCN does. Troll all of the blogs on the planet, looking to see if anyone is talking about them. And then rush out to defend themself against yahoos that call them criminals.
If you have a question (and you do) of GHCN – ASK.
“Until someone can point out that mystery method, however, I maintain that the Darwin Zero record was adjusted manually, and that it is not a coincidence that it shows (highly improbable) warming.”
Over reaching and illegitimate accusation of wrongdoing. Invites another spanking, such as you received from the Economist. Stop setting yourself up… and the rest of us by extension.
Rabe (01:17:17) : said
I wonder how average temperatures are calculated. It’s not even ingenious let alone scientific. Why don’t we measure those average temperatures directly? Here is how it would work:….
Yes I have always wondered why they do not use the temperature in caves. I remember from my college caving days Indiana caves are 54.5 F year round, while Texas is around 70 and England was somewhere in the 40’s.
I wonder if the caving societies around the world would have records. I helped collect cave fauna for a caving buddies PhD Thesis and I am pretty sure the temperature was one of the data points recorded. This was in 1969 to 1972.
The more threads I read re: anything factual, the more I see how some individuals tour the “sites” and simply throw tear gas canisters into the discussion.
No. I think that the adjustments are made for a good reason. I’m answering the suspicion that they are done to deliberately enhance warming. The fact that the nett trend is small means that they don’t have that effect (or only to a small extent). Crudely, if someone was trying to rig the effect, they aren’t doing very well.
The net trend in DARWIN (the station under the microscope here) is anything but “small” we are talking here about an adjustment of almost the difference between “summer” and “winter” not “oh we painted the SS the wrong colour” or “oh we moved site from PO to airport”.
I don’t believe for one minute that there is one conspiracy looking to “rig the game” but it often looks like a hive mind trying desperately to “run the game into extra time”
Don’t get me wrong here – where somebody (a critic?) has pointed to a paper that sheds light on the subject in hand (DARWIN), I am more than willing to read. My problem is that I see why Willis started all this thread and all most of the critics have done is to point to the chirping cricket under the elephant so far.
Mann Jones scam has only been exposed widely for 30 days. It will spread and crumble further.
It will take a huge surge or crusade for them to even partially recover.
The quality control in the gathering recording business for the Meth, CRU and NASA is just not there.
If I were an Aussie, I would be very perturbed about now.
A homogeneity adjustment of over 2 degrees centigrade in a stair step fashion of about one half degree every decade? Right!
How can there be any reasonable explanation other than it has been intentionally manipulated?
Now do other stations indicate similar adjustments have been made?
How has this station adjustment affected temperature statistics for the continent?
Rob Vermeulen (02:42:36) :
Is weather undeground also adjusting its values?
Rob, This summer, I found that the high and low temp for my area, as reported by Weather Underground, changed by 2-3 F (higher) when reported as the high and low for the area the next day. There is no way I could have missed the low since I get up before dawn, check the weather and then go feed my animals.
Nick Stokes:
“If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception.”
Not necessarily true.
These data have not only magnitude, but a spatial and temporal component. That the adjustments have a nearly equal distribution in magnitude does not demonstrate that they have an equal temporal distribtion, or an equal spatial distribution.
It is entirely possible, in fact likely, that data represented in GG’s graph could have the ‘ups and downs’ at opposite ends of the time period, and/or that the ‘ups’ may be applied to larger areas in the area-weighted, aggregate temp estimate. The analogy above of a push up on one end of the balance beam and an equal push down on the other end is an apt illustration.
Another unwarrented conclusion…
Nick Stokes,
GG doesn’t prove anything.
If Darwin’s adjustments don’t adhere to Peterson then GG would need to show that the offsetting cooling adjustments ALSO don’t adhere to Peterson. If the offsetting cooling adjustments DO adhere to Peterson, then it’s obvious we have a problem.
In fact, we have a problem even if the cooling adjustments don’t adhere to Peterson – wouldn’t you agree?
Ox-FordPrefect (04:46:08) :
Is there a record of the number of corrections for inhomogeneity and what direction these are in? An assessment of the formal guidelines used should be revealing – are there more reasons to adjust data upwards than down?
As others above stated, you can have a zero net adjustment and cause a large change in the graphs. Just adjust a reading before 1950 DOWN by the same amount you adjust a reading after 1950 UP. The net adjustment is zero but you lowered the first half of the graph and raised the second giving a flat line graph of raw data a nice upward trend so you can yell GLOBAL WARMING.
*******
21 12 2009
Geoff Sherrington (02:58:12) :
The weakesses with this study are several fold. One is evident from a station in an older city with UHI. In Melbourne for example, various comparisons with other sites show it likely that UHI ‘maxed out’ at Melbourne Central some time before 1950. That is, from humble beginnings, the temperature from UHI rose steadily to (say) the 1930s and then remained about constant because nothing was explainable or available to push the UHI much higher. So Melbourne would be one of the stations in an 80-year plot that whould be near the median, despite having UHI (from studies already referenced) of up to 10 deg C.
*******
Geoff, I don’t think UHI effects will “max out” in this way in most situations. Even if an urban area becomes static, if the surrounding suburban areas continue to build up, this would still increase (admittedly by a smaller amount) the UHI effect in the center area by increasing the temps of incoming air toward it. One exception might be under conditions of no wind or no surrounding growth — then the effect might max out.
So, tho I agree the UHI in the urban center would increase at a lesser rate (how much, I don’t know) once it’s fairly static, I don’t think it would max out if surrounding areas continue to build up.
Nick Stokes:
Prove it! somebody show me one plot from one station with data that is downloadable and verifiable and shows the opposite pattern.
Get the data Nick, if it exists.
well its 14C outside a few days off Xmas in aus..hmm
it could be expected to be 25. instead I am heating wheat bags for arthritic aches from the cold..
and if Mayon goes boom..I guess I wont be bothering to plant melons for a few summers
Nick Stokes
I find your histograms very interesting. Especially the unadjusted and
the adjusted ones at the bottom of the page. Could you please also indicate
in your blog the mean, mode and median of both histograms?. Or even better,
could you subtract both histograms so we can see what we are left with?
Nick Stokes, isn’t the reality that the cooling adjustments were done to old records and warming adjustments done to newer records creating the illusion of warming when there was none?
When considering the possibility for bias in the GHCN temperature reconstruction, which would not affect the overall temperature balance
JJ (07:49:00) described it clearly earlier:
“These data have not only magnitude, but a spatial and temporal component. That the adjustments have a nearly equal distribution in magnitude does not demonstrate that they have an equal temporal distribution, or an equal spatial distribution.”
I understand the importance of considering the temporal component, which is dealt with well in this article:
http://statpad.wordpress.com/2009/12/12/ghcn-and-adjustment-trends/
Am I right to assume the possibility of the following bias:
Is there another important possibility of bias? Since one remote station has more weight attached to it than a number of stations grouped together. A large adjustment upwards on the remote station can be ‘averaged out’ by several small movements down on the grouped station but the effect it has when creating the temperature profile for a geographical area is significantly larger (I’m assuming that GHCN use a method similar to the grid system used by GISS to provide a global temperature metric.)
Could this be the motivation for adjusting Darwin’s temperature history with a view to the creation of a warming profile?
(somebody mentioned DB earlier in the thread)
I don’t have Dr Bellamy’s e-mail address so I will post this message here.
Dr David Bellamy
Bedburn, Co Durham
Dr. Bellamy,
Stick to your guns.
I (only slightly younger) have very fond memories of your work. You have always been an excellent bridge between “hard science” and the “public”. Opening up worlds that, without you pointing them out, most of the “public” would just never see or care about. Now they do, thanks to you.
It is a real travesty that the current set of “bong heads” over at the BBC do not recognise that you have done more for the real “environment”, at least in the UK, than most any individual has. Don’t worry though, you have many allies.
Can’t help you much with the science but if you need me to go ’round the BBC and kick (Dr. M boots wise) some sense into one of their current “environment workshops” and flush their Jamaican weed down the nearest multi-cultural, multi-sexual WC- just give me a call.
Like I say – Stick to your guns.
(In the mean time why not post your views over here at WUWT?)
Nick Stokes (04:09:42) :
And the slope is about equal to the 0.0175 C/decade of GG.
Your point should be relatively simple to prove visually. Plot the adjusted data next to the unadjusted data and the trend should be .0175 C/dec different.
— Nick Stokes wrote : If the adjustment moves trends up and down almost equally, they won’t reinforce anyone’s preconception. —
This is simply not the case as (Michael R 00:31:00) post showed. Just because the overall averaged temperature adjustment = 0.0C, it doesn’t mean the temperature data wasn’t skewed to make it look like there is a warming trend. When the adjustments were made makes all the difference in the world.
For example, if one averaged a 20-year period like 1921 – 1940 downward -1.0C, and then averaged the 20-year period after that, 1941 – 1960 upward +1.0C. While the overall average adjustment = 0.0C, one has skewed the temperature data and introduced a 2.0C warming trend into the data for the overall 40 year period. If the 40 year temperature data for 1921- 1960 previously showed a flat trend, it now shows a 2.0C warming trend where there was none before.
(moderator please delete my previous post, it got messed up due to formatting)
Let me see if I have this correct. To determine the global temperature, climate scientists have cut the earth up into grids. They then use mysterious mathematical manipulations to prepare “raw data” so a temperature can be assigned to each grid and use that temperature to compute the global temp.
GHCN makes adjustments to the raw Darwin data according to these mysterious mathematical manipulation techniques presumably to prepare it for use in the global grids and then reports this as the temperature for the Darwin location instead of using the raw data.
Nick Stokes is defending the adjustments made to the Darwin data by saying they were made according to the mysterious mathematical manipulation techniques of GHCN and therefore were justified while Willis Eschenbach has analyzed the data set for Darwin and see no justification for any adjustments to be made at all. Nick Stokes then states the mysterious mathematical manipulation techniques called GHCN homogenization “corrects for discrete events that it detects by time series analysis. Things like station moves, equipment changes, MMTS introduction.” (So I guess I was wrong it is not adjusted so it helps represent a grid area)
But if that is the case and if homogenization is supposed to detect and correct for “discrete events” then why does fig 3 raw data look completely reasonable and figure 3 adjusted data look like a lot of noise has been added to the data and not filtered out? If there were “discrete events” requiring “adjustment I would expect the graphs to be reversed.
There is an error in the picture:
The left and the right labels are not equal.
Can’t this issue be easily resolved by just looking at some tree ring, mussel shell and sediment proxies in order to determine what the true temperature was at Darwin Zero over the last 50 years? I mean according to the chart at the top of this page,
http://www.usatoday.com/weather/climate/2006-06-01-wine-warming_x.htm
proxies can measure temps from 2000 years ago to the nearest 1/10000th of a degree whereas modern day thermometers can only measure today’s temps to the nearest 1/1000th of a degree. And you can believe it because it’s “based on scientific analysis”.
Heck, I don’t even know why we bother with thermometers and billion dollar satellites at all.
I’ve seen this same “histogram of adjustments” approach on other blogs. Do these people have a play book or do they just copy each other? It’s time to call a spade a spade. This one is pure BS meant to divert the discussion from more fruitful paths. I think the best idea is to just ignore them.
When I look at the homogenization technic, some questions come to mind.
Why do they use yearly temperatures?
If the temperature stepped at some point, it should be possible to see it in the day by day temperature.
Have they proven that isolated hot spot are equally likely than isolated cold spot on the planet?
If isolated cold spot are more likely on the planet, the technic could generate a false warming signal.
“The sum of anecdotes is not data.” -Roger Brinner
One could only imagine if we who work in metrology (not to be confused with the other -ology) applied the techniques for measurement of temperature to that of measuring engine components, how various engine parts would fit together.
Sorry Nick Stokes and other GHCN apologists, but from a metrology perspective, derived temperature values can’t even be considered data. It is a complete joke and goes against every principle in basic metrology concepts.
KevinM (08:11:16) :
wil you accept Giss?
down:
http://img379.imageshack.us/img379/5074/newarknycprawgiss.jpg
http://img197.imageshack.us/img197/8283/honolulu.jpg
some up some down some twiddly:
http://img11.imageshack.us/img11/7440/gissrawtemps.jpg
Haven’t checked all the comments, but maybe this is useful.
I have found, but misplaced, a study in google scholar where there was an analysys of the impact of changing from averaging max and min temperature (probably from mercury and alcohol thermomethers) to using regular timed temperatures. Nevertheless that change was in 1995, so it cannot be used to justify previous adjustments.
On the other hand on the thread about the economist I left a link to a study on the adjustments to radiosondes and how the first attempts on purely manual adjustments “failled” I think you will find many interetsing tricks of the trade from the usual suspects.
best regards
Other sources using this data confirm Willis charts.
The raw data from GISS.
http://data.giss.nasa.gov/cgi-bin/gistemp/gistemp_station.py?id=501941200000&data_set=0&num_neighbors=1
(I can’t be sure but there is long enough overlap between individual station locations in this chart that no adjustments should have been made.)
And then the ClimateExplorer lets one look at and download the monthly adjusted GHCN data for all/most of the stations in the network.
Here are the individual stations (4771 with a handy Get Data link where you can also download the monthly GHCN data) .
http://climexp.knmi.nl/allstations.cgi?someone@somewhere+temperature+12
Here is the adjusted monthly data for Darwin (+2.0C increase in the trend compared to GISS’ unhomogenized data).
http://climexp.knmi.nl/gettemp.cgi?someone@somewhere+94120+DARWIN_AIRPOR+
DR (09:21:43) :
…from a metrology perspective, derived temperature values can’t even be considered data. It is a complete joke and goes against every principle in basic metrology concepts.
Many say you must adjust for UHI
It is obvious you must correct for instrument change
It is obvious you must correct for location change
To me there is an obvious error in max/min/average due to time of observation. Using a continuous reading electronic thermometer max and min can be “easily” determined. A once/twice daily reading is wrong perhaps 3 consecutive days
So should adjustments be made?
The record we have is all there is – warts and all.
There is no way to improve this. For future a few millions spent on new equipment re-sited to ideal locations (some of these MUST be in cities and near roads as these are a valid part of the environment).
To get a VALID check on global temperatures will take at least another 150 years and more. If there is a problem with AGW then you have consigned large populated areas to the scrap heap.
What is the way you suggest that we take?
Just for info (I’m sure you will shout INVALID) here are a few UK places and de Bilt. No adjustments made to individual UK stations This shows the typical warming response.
http://img410.imageshack.us/img410/8996/ukspaghetti.jpg
You can also get the annual GHCN adjusted data for Darwin
At the link (Click on “Make Time Series” at the bottom of the page and the annual data will be shown).
http://climexp.knmi.nl/gettemp.cgi?someone@somewhere+94120+DARWIN_AIRPOR+
It is extraordinary bad science to do any adjustment to thermometer readings. If there is a noted environmental or instumentational alteration to the readings, such as the construction of nearby blacktop or the reorientation of the thermometer, the same should be noted with an asterisk and a footer.
To Nick Stokes:
I would also appreciate your pointing out stations with downward adjustments equal to or greater than that of Darwin. I’ll bet Anthony would arrange for the graphics to be shown if you cannot provide a link.
As somebody formally asked whoever is in charge of GHCN?
I still think I want the first question answered “how did this actually happen?” before I move on to any discussion of “was the result an intentional inflation?”.
I still have this feeling -possibly entirely incorrect- that there exists a significant chance this kind of thing (we’re talking about how many thousands of stations in GHCN?) is much more likely to be done by herds of grad students working from scripts they barely understand and with no authority, opportunity, nor confidence in their own judgements to challenge results that their older, more experienced, more confident colleagues would immediately discard as an unacceptable result.
The number of negative correlations on an annual basis is pretty surprising…
Here is a significance test for correlations.
http://faculty.vassar.edu/lowry/rsig.html
For 100 data points correlations of absolute value < .2 are not significantly different from zero.
It's possible that Australia has a particularly low spatial correlation of temperature changes compared to other regions?
Another possible problem with correlation measurements is that the discontinuities will destroy the correlation structure. You almost have to remove them before measuring the correlation, or measure correlation over periods of time where there aren’t significant discontinuities. This is especially important with relatively few data points……
Marc:
“Why do they use yearly temperatures?
If the temperature stepped at some point, it should be possible to see it in the day by day temperature.”
Daily temperature fluctuations are much larger (one or two orders of magnitude) than, and would mask, the typical effect being adjusted for.
DR:
GHCN isnt doing meteorology, and they arent buildng engines. If you want to disprove GHCN methods, you have to address what they are doing – estimating global surface temperature.
Bill Illis :
“(I can’t be sure but there is long enough overlap between individual station locations in this chart that no adjustments should have been made.)”
Overlap among station locations does not rule out adjustment, much less overlap among duplicate records for a single station …
Bill,
“To get a VALID check on global temperatures will take at least another 150 years and more.”
Then you better get on it. Because if you dont have VALID data, then you are just waving your hands.
“If there is a problem with AGW then you have consigned large populated areas to the scrap heap.”
Nonsense. That aside, your analysis is decidedly one sided. If there is not a problem with AGW, but we spend Hundreds of Trillions of dollars on the pretense that there is, then you have consigned large populated areas to the scrap heap. If you are going to demand a response like that, you need to bring sufficient VALID data to the table. If it takes 150 years, so be it.
************
bill (09:58:40) :
To get a VALID check on global temperatures will take at least another 150 years and more. If there is a problem with AGW then you have consigned large populated areas to the scrap heap.
***************
If the Earth hits a dense swarm of asteroids, the Earth is toast but that isn’t very likely and neither is catastrophic global warming. I’m sure we can come up with a long list of events that might hurt us, but we don’t prepare for every one of them as if they WILL happen.
Why do they have to adjust the temperature readings at all? Temperature at a given site is the temperature at that site. If it goes up or down, so be it. Averaging all sites in a local area to come up with a temperaure report for that area seems reasonable, six or twenty stations in a city or township could be averaged to give a regional temperature. “Adjusting” temp readings seems fraught with problems and opens the path to errors (and to misrepresentation of reality.)
Does anyone really know what the temperature is anywhere on the planet anymore? The graph shown for Darwin is disturbing, raw data shows one thing, adjusted shows something very different, which one is reality and which is an illusion?
Nick,
Any “analysis” such as GG’s which leaves out key variables is worse than meaningless, it’s deceptive. Another good example is the supposed US pay gap between men and women. When factors such as profession, location, education, years of experience, etc. are factored in women in the US make slightly more than men. GG leaves out both location and time and thus should be disregarded immediately as bogus.
On your speculation of Stevenson Screen changes for the later Darwin stepwise adjustments, show us the records. You can’t expect to be taken seriously if you just throw out guesses. Also explain why a new Stevenson screen (not the purported change to a Stevenson screen way back around the turn of the century at Darwin) should result in an upward temperature adjustment or a lesser decrease in adjustment. My understanding is that a new (and thus brighter) screen will, if anything, result in slightly lower temperatures readings than if it was old and discolored. The slats being blocked would also yield higher readings in sunshine. What you see in the Darwin record for the later decade’s temp adjustments is less decrease followed by increase, so what’s your theory there?
Finally, your continued mention that Darwin is but one station is a pathetic and ridiculous diversion tactic. We’re not discussing every single station in the GHCN record here and pointing out the fact that a single one has been adjusted. There are other posts and other sites that cover many other stations. There are also posts that look at the cumulative effect of adjustments in various areas. This discussion is not happening in a vacuum, so pretend like you’re smarter than a fifth grader and drop the pretense that what has been found with Darwin adjustments is everything and therefore can’t really affect the overall data.
Until you can specifically account for the dramatic adjustments at Darwin it remains a smoking gun.
I’ve charted up the difference in the annual temperatures from the GISS unadjusted dataset and the GHCN adjusted data from the ClimateExplorer.
This is presented a little different than Willis did but I think you will get the picture.
http://img192.imageshack.us/img192/521/darwintemps.png
Difference / Changes made by GHCN
http://img14.imageshack.us/img14/8291/darwintempdifference.png
bill (09:58:40) : Was that spaguetti’s temperatures and hockey stick shaped graph made wth Mann’s software?☺
JJ (07:28:55) :
If they were merely duplicates, the GHCN would not adjust them differently. You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
Now, do you see how condescending that sounds? If you want people to take you seriously, talk to us like adults, not the way you talk to kids.
If they were merely “duplicate records”, they’d average them to get the final record and be done with it. They don’t. If they were just “duplicate records”, they wouldn’t disagree 90% of the time. They do. If they were only “duplicate records”, they would not have been adjusted separately. They were.
Man, you’ve been hanging around with too many “climate scientists”, you’re all about “could” and “might” and “may” and “could easily” and the like. Yes, the GHCN algorithm could do the things you claim … but it doesn’t. TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!! I am sick of people making claims about what “could easily” and “might” and “may” happen. if you think it might happen, try it and see. I have, and I couldn’t make it happen, “easily” or otherwise.
The GHCN says “neighboring” stations. Perhaps more than 1,500 km away is “neighboring” on your planet. On this planet it is not.
I thought you were following the story. I am clearly talking about the adjustment in 1920. Go back to the records. Which Darwin first difference am I talking about?
No. The assertion is that “neighboring” stations are used. I’ve checked the neighborhood out to 1,500 km. without finding a single suitable station. If you think that there are suitable stations, you dig them out, because I can see that absolutely nothing that I might do will satisfy you. When you five the five stations, let us know. Until then, your fantasies are just that.
Take a deep breath and think about this for a minute, JJ. All the GHCN method can do is to adjust a station to match the trend in neighboring stations. It can’t create a trend out of nothing. For the adjustment to have been used, we would have to find 1) five well correlated neighboring sites that 2) increase at 6C per century.
I can’t find either of those. If you can find those, bring it on. If not, the results indicate that they didn’t use the algorithm, regardless of how frantically you wave your hands and say that they might could easily have used some possible unknown stations 5,000 km away.
Simply put, I have shown it out to a distance of 1,500 km, well beyond any reasonable interpretation of “neighboring”, and well beyond the known intercorrelation distance of temperatures. If you think I’m wrong, COME UP WITH SOME STATIONS THAT FIT THE GHCN CRITERIA!!!
I don’t know if I did, JJ. I just do the best I can, nobody can ever be sure that they are correct. That’s why I put my work out on the web, to let other people find errors in my application of the GHCN algorithm. If you find I’ve done it wrong, let me know. Asking me if I’m sure is just wasting electrons. If you think I’m wrong, do the analysis yourself and show that I’m wrong.
Frankly, without a citation, your “belief” is meaningless.
Make sense with respect to 1) logic 2) mathematics 3) practical considerations 4) the fact that they are nearly identical. If you can’t follow those, I can’t help you.
Say what? My claim is that the listed GHCN methodology was not used. Whether is is “sufficient” for anything at all is your issue, not mine.
Proven? I haven’t “proven” anything in any direction. How could I?
Now you’re just playing at not understanding. The GHCN produce data to be used in global and regional estimates of temperature change. Do you think that they do that without averaging? Sheesh …
Effect on the aggregate? I’m talking about Darwin, not some mythical “aggregate”.
Your speculation and fantasies are worse than useless, as you are basing conclusions on them with absolutely no data.
TRY THE GHCN ALGORITHM!!! Use some dummy data for neighboring stations and see what it does to the different length (but otherwise nearly identical) Darwin datasets. You claim it is possible to get wildly different results simply because the datasets are of different lengths. I have not been able to reproduce that result. No matter what I try, I get very similar results. Get back to us when you have something other than your speculations to report.
First, you need to learn that when you say to someone “you need to learn”, it just pisses them off.
Next, JJ, if you can show how the GHCN methods were used on the 1920 step, bring it on. Until then, your claims about what you understand and what I don’t understand ring hollow.
If you can’t take being proven wrong, don’t go into science. It is an inescapable part of science that some of your claims will be shown to be wrong. I’m happy to be proven wrong because that’s how science progresses. Of course, I’m happier if I can prove someone else wrong …
You should look up the difference between “hope” and “expect” sometimes. I don’t expect the GHCN to do a damn thing.
Look, JJ, if you want someone to go ask GHCN something, be my guest. Report back and tell us how far you have gotten. Me, I’m done with beating my head on closed doors … but if you think it’s a good plan, stop messing around and get on with it. You think I’m wrong, you go ask GHCN to prove it. Let us know how it goes. Me, I’ve tried that route over and over, I find it a useless path. But you might surprise me.
If you truly think it is the right thing to do, why wait? Go ahead and ask the GHCN, and report back with what you find out. Stop trying to be an armchair general, it just makes you look foolish saying “you should do this, you should do that”. Get out and do something other than cavil and carp and whine and find fault.
I’m dead serious here. You are all too willing to point out to me the right path, the decent path, the proper path. You keep telling me about all of the things that might be and could easily be. So show us how it’s done, JJ. Get in touch with GHCN and see how far that gets you. You claim that’s the right path, give us a demonstration.
I await your report from GHCN, although not with bated breath …
Say what? The Economist apologized to me, both in the blog and in private emails, for their unconscionable over-reaching in their first post. Both the guy who wrote the blog and his editor wrote me to say that they were way over the line. Now they are flailing and trying to salvage something of their reputation. They are lucky that I didn’t bring a libel suit against them, I suspect that’s why they rushed to apologize both publicly and privately. If you think that they “spanked” me, you’re not following the story.
In any case, until you actually do some of the things that you so earnestly recommend, like showing that nearly identical records can be warped differently by the GHCN algorithm, or getting in touch with GHCN for an explanation, or finding one single candidate comparison station for the 1920 adjustment to Darwin Zero, please don’t bother me again. I’m not interested in your “may” and “could easily” and “might”, answering your fantasies is taking up way too much time. If it “could easily”, then do it and report the results.
Latest from the UK MET office (to think my taxes directly pay for this lot.)
The HadCRUT database is UNDERSTATING temp increases and the Russians recent complaint confirms the database is correct as its matches their lower temps.
Quote ‘The IEA’s output is consistent with HadCRUT as they both confirm the global warming signal in this region since 1950, which we see in many other variables and has been consistently attributed to human activities’
I want my money back !!!!!
http://www.metoffice.gov.uk/corporate/pressoffice/2009/pr20091218b.html
JJ will be busy over Christmas… 😉
See all the climategate analysis here: http://69.84.25.250/blogger/post/ClimateGate-Data-Series-Part-5-A-break-down-of-large-data-file-for-manipulating-global-temperature-trends-from-2006-2009.aspx
joe (09:07:01) :
joe, look at the legend in Fig. 1, and at the colors in Fig. 2. Some of the items shown are measured on the left scale, and some on the right.
w.
bill (09:28:20) :
Bill, GISS makes their own very different adjustments to the data. As such, they are not relevant to what GHCN does.
w.
Mesa (11:00:12) :
This is why the correlation is done on the first differences rather than on the underlying data. With the first differences, a step will only affect that year, not the entire record. Because of that, it does not “destroy the correlation structure”.
w.
RE: Nick Stokes (22:46:31) :
Willis,
You still haven’t recognised the most cogent criticism of Giorgio Gilestro, who showed that if you looked at the whole distribution of GHCN adjustments, and not just one station, the distribution is fairly symmetric, with stations almost as likely to have the trend adjusted down than up. The average upward adjustment to trend was 0.0175 C/decade; much less than the Darwin figure.
This isn’t rocket science – adjust the trend of about half of the stations UP since 1930. Adjust the trend of the other half of the stations DOWN prior 1930. Presto, the average adjustment is… Zero!
That’s why Willis’ time-domain analysis is so significant.
On Willis’ last reply to JJ – well done Willis!
I think we will be waiting a long time for either JJ (whoever he might be incidentally, as you post your analysis and comments under your real name) to get a response out of GHCN or he actually does some analysis of his own and posts it for other people to prod into like you have done.
KevinM (08:11:16) :
Prove it! somebody show me one plot from one station with data that is downloadable and verifiable and shows the opposite pattern.
Well, I said above that of stations >80 years adjusted record, 17 were adjusted down by more than Darwin was adjusted up. Here they are:
211357000002 GUR’EV
222234720001 TURUHANSK
222246410000 VILJUJSK
222255510002 MARKOVO
222310880000
222325830000
222325830002
222325830003
403717140040
403717470010
403718360001
414763930000 MONTERREY,N.L
425726710020
501947280000 COONABARABRAN NSW
603113200000 INNSBRUCK/UNIVERSITYAUSTRIA
615076300000 TOULOUSE/BLAG
I didn’t look up all the names, but you can find them, with details, on the v2.temperature,inv file.
I’ve plotted the Australian station, Coonabarabran, here. Adjusted in red.
Richard Sharpe (22:00:48) :
Scott of Melb Australia (20:58:57) said:
(using microfish of available temp data)
Hmmm, is that a new unit of measure for temperature data that I am not aware of or did you mean microfiche?
Thanks for the pickup Richard – no excuses I should have checked my spelling. Hopefully it doesn’t detract from the point I was making.
Scott
Nick Stokes,
Are you a free-lance or paid ‘provocateur’ whose only purpose is pedantic obfuscation? Instead you constantly change the subject returning again and again to pick at non germain nits. Do you get paid to divert rational discussions, or do you do so simply for the jollies? Do you get paid by RC?
Real Climate.org funded by Fenton Communications, a front organization of the Al Gore empire and administered by his ex-campaign treasurer, apparently exists strictly for the purpose of providing an official propaganda house organ. It is run by Gavin Schmidt, even as he double dips on his govermnment job for Astronomer Hansen, as a sort of ‘the Team newspaper’.
It si similar to the publication for those friendly Kremlin chaps and the inestimal publication named ‘Pravda’ or was it ‘Isvestiya’. I could never keep straight which was the mouthpiece for the Party, or the Government.
To all aussies: Feel free to use the expression I invented above:
“Kangaroo Tail” for campaigning against local “warmheads”.
lowercasefred (10:35:59) :
See my response to Kevinm above. I improved the plot of the Coonabarabran result, by adding the adjustment plot in Willis-style (green, no axis marked but same scale as the others) here.
[not sure if this was meant to be fightin words or not, but it didn’t really add to the discussion. ~ ctm]
Folks, keep in mind that Nick Stokes is well aware of the temporal problems with GG’s histogram and was before he made his first comment on this thread. I think that speaks to motive.
David Jay (12:49:46) :
Bruce (08:34:30) :
Gail Combs (07:56:23) :
KeithGuy (07:13:16) :
Jim (06:08:58) :
You’re all asking variants of the same question – doesn’t it matter when the adjustments were made? Yes, it does. What GG and I plotted in those histograms was the change to the trend over the whole time period of the station. One stat that Willis drew attention to was that the change to Darwin made a difference of 1.9 C/century (in my calc, it;s 2.35) The histogram shows the equivalent figure for all the other stations. the measure is the linearised cumulative rate of change over the whole period.
I’ve explained in more detail at my blog here.
Nick Stokes (13:35:35) :
Still not sure what you are trying to prove. If you want to say there is not much of an effect on the record, plot the GHCN unadj against the GHCN adj and see what the difference is.
Willis Eschenbach (12:16:55) : “I’m happy to be proven wrong because that’s how science progresses. Of course, I’m happier if I can prove someone else wrong …”
Wow, I gotta get meself inta this here science-type career thingy…no downside! 🙂
Congrats on another great article, Willis.
JohnH (12:32:02) :
You deserve your money back. I loved their explanation:
When someone says that the observations have been “synthesized in a physically- and meteorologically-consistent way”, wise men run.
To review the bidding, HadCRUT says there is no significant warming in the last decade. GISS says there’s no significant warming in the last decade. Both the RSS and the MSU satellites say no significant warming in the last decade.
However, the magic “synthesis” of the above data by ECMWF shows “the last decade shows a global-mean trend of 0.1 °C to 0.2 °C per decade”.
Next, the met office is not showing the HadCRUT data, which includes the ocean. They are only using land stations … what’s up with that?
Their conclusion is that the EMCWF “synthesis” shows that the Met Office is right.
My conclusion is that all that has been shown is that the EMCWF “synthesis” is bovine excrement … however, having said that, I suspect someone will show up soon to say the EMCWF “could easily” be right.
Finally, being the suspicious sod that I am, I went to the EMCWF site and found … model results. Which makes sense, since they are the European Centre for Medium-Range Weather Forecasts, which they do with models.
It is also not clear what data the EMCWF are using. The met says that EMCWF are using “all available surface temperature measurements, together with data from sources such as satellites, radiosondes, ships and buoys.” However, their analysis does not show anything but grid boxes containing land stations … if they are using all available datasets, why is there nothing shown over the ocean?
In their catalog (175kb PDF) of products, they have:
I don’t see any land station observational data there, but it is possible that they only give that to their friends … it certainly is not available on their data server, which only gives us plebians access to model results.
Finally, what “surface temperature measurements” are they using? GHCN raw? GHCN adjusted? Their own “adjustment” of the GHCN raw data? Anyone’s guess.
Nick Stokes (13:47:31) :
“You’re all asking variants of the same question – doesn’t it matter when the adjustments were made? Yes, it does. What GG and I plotted in those histograms was the change to the trend over the whole time period of the station.”
Interesting… Take a look at: Musgrave (50194187000, and if I understood correctly, supposed to be -0,39C a decade, according to GG result.xls), adjusted graph here: http://www.appinsys.com/GlobalWarming/climgraph.aspx?pltparms=GHCNT100XJanDecI188020080600101AS50194187000x , years with less than 6 months of data excluded.
How does that “change to the trend” applies “over the whole time period of the station”?
“the measure is the linearised cumulative rate of change over the whole period.”
Don’t the different “whole periods” for different stations matter?
Willis,
“If they were merely duplicates, the GHCN would not adjust them differently.”
Willis, they are duplicates. GHCN identifies them as such. Read the GHCN station metadata documentation, rather than making up stories based on suppositions.
You need to understand what that means, and how it affects how these data are ultimately used in the global anomaly calc.
“Now, do you see how condescending that sounds?”
That is not condescending. It is simply the fact of the matter. You need to know what a duplicate record is, and how it is ultimately used, before you can make claims about those duplicate records, and how they are used. You dont know those things. Find out.
“If you want people to take you seriously, talk to us like adults, not the way you talk to kids.”
I am talking to you as an adult. You are responding like a child. Stop being defensive when someone is trying to help you.
“If they were merely “duplicate records”, they’d average them to get the final record and be done with it. They don’t. If they were just “duplicate records”, they wouldn’t disagree 90% of the time. They do. If they were only “duplicate records”, they would not have been adjusted separately. They were.”
You’re making stuff up again. You dont need to do that. Read the GHCN documentation. If you find something that isnt well documented, ask. Dont try to reason what you dont know, from the limited amount that you do, when you can simply look it up, or ask.
“Man, you’ve been hanging around with too many “climate scientists”, you’re all about “could” and “might” and “may” and “could easily” and the like. Yes, the GHCN algorithm could do the things you claim … ”
Yes, the GHCN adjustment method can do those things I claim. And because it can, your claims that assume it cannot are false.
“TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!”
Excuse me? Please take your own advice. TRY IT BEFORE YOU MAKE CLAIMS ABOUT IT!!”
You have yet to replicate a single GHCN adjustment. Until you have done that, you cannot say what they did. This is my fundamental point to you. Take a page from Steve Mc. He takes great pains to replicate results before he starts making claims about what someone else did.
“I am sick of people making claims about what “could easily” and “might” and “may” happen.”
Then you need to stop making claims supported by further claims about what “cannot” happen. You claim that the GHCN adjustment cannot have been applied, because the GHCN method cannot adjust duplicate records differently. As you admit, it can. Your claims are false.
“The GHCN says “neighboring” stations. Perhaps more than 1,500 km away is “neighboring” on your planet. On this planet it is not.”
Again, making stuff up. GHCN apparently does not have a distance limit. You cannot just make one up, and then claim they didnt follow their own rules because they didnt follow yours.
“I thought you were following the story. I am clearly talking about the adjustment in 1920. Go back to the records. Which Darwin first difference am I talking about?”
Answer the questions. Which of the five duplicate records was your correlation calculated for? And where is the similar calc for the other four? The records with shorter time frames may be matched to reference stations with shorter timeframes. What does that do to the availability of ‘neighboring’ stations? What does that do for correlations? Hint: It is much easier to find high correlations between shorter segments …
“No. The assertion is that “neighboring” stations are used. I’ve checked the neighborhood out to 1,500 km. without finding a single suitable station.”
So? Limiting your serach to 1500km is not the GHCN method. That is your method. If you want to claim that GHCN did not follow GHCN method, you have to check against the GHCN method. GHCN can apprently use any station within GHCN’s definition of ‘region’. What is that, for Australia?
“Take a deep breath and think about this for a minute, JJ. All the GHCN method can do is to adjust a station to match the trend in neighboring stations. It can’t create a trend out of nothing. For the adjustment to have been used, we would have to find 1) five well correlated neighboring sites that 2) increase at 6C per century. ”
No, you would have to find five well correlated neighboring sites that when homogenized into a reference series result in a potential adjustment of 6C per century.
Now see, you may be on to something there. I am not sure that it is true that the GHCN adjustment method limits the adjustment to the trend of the reference series. Or for that matter, if the trend in the reference series is limited to the trend in any one of the reference station’s homogenized data.
Hint: You arent sure either.
If you can:
a) prove (mathematically) that the GHCN adjustment method has those limits, and
b) prove that there are no stations that meet GHCN’s requirements for a reference station with that trend, THEN you will have proven that the GHCN methodology wasnt followed. And that would be something, wouldnt it?
And here’s the fun part: If you instead end up proving that the GHCN methodology CAN produce a 6C adjustment where no 6C trend exists in any of the reference stations … you’d have something there, too, now wouldnt you?
If you actually DO THE WORK to rigorously prove your point, you stand to come away with something valuable either way it turns out …
Sounds like a productive line of attack. Give it a go.
More later.
WE:
I work with first differenced data all the time in finance (returns actually) – just pointing out that a jump discontinuity in the undifferenced series will lead to one outsized difference and affect correlation measurements for small data sets….
I’d be curious to know if you think the extremely low correlations for the nearest neighboring stations are typical for the rest of the globe – ie any idea if its an Australia specific effect or are the distances just too great? It’s surprising given how they have been justifying using fewer and fewer thermometers globally…
Tks.
Nick Stokes (13:11:12) :
“”Well, I said above that of stations >80 years adjusted record, 17 were adjusted down by more than Darwin was adjusted up. “”
Were any of these adjustments made outside of the Peterson algorithm?
If Peterson was followed for all of these adjustments, then Darwin is unfairly offsetting them.
Since it’s really hard to get into a fight with a computer screen, my comment was meant to be nothing more than a poignant observation directed at someone displaying boorish behavior.
Reply: Food fights are discouraged. Looks like I made the right choice. ~ ctm
Janis B. (14:15:46) :
Don’t the different “whole periods” for different stations matter?
Yes, they do. You’ve plotted a smooth for that blue Musgrave data. If you had worked out the adjustment difference for that period and fitted a line, and got the slope, that is the number that goes into the histogram. There are two time issues:
1. The short period means that a relatively small adjustment overall can make a big slope
2. Short periods could be recent or long ago.
That’s why I’ve tended to focus on periods >80 years. It’s rare to find such a record that doesn’t cover most of the 20th C. And you only get a big gradient if the adjustment is also large.
wobble (14:19:56) :
Were any of these adjustments made outside of the Peterson algorithm?
I don’t know. I’m analysing the data as found on the posted file. But I doubt that they were made manually. You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.
Nick Stokes @ (22:46:31)
Giorgio Gilestro’s “most cogent criticism” is bogus in a couple of ways.
First of all if there is any bias in upward/downward adjustments the bias should be downward if the claimed corrections for urban heat islands were really made. There are going to be very few cases of air conditioner vents or blacktop or buildings removed from near a temp station but very many cases of these things being added near one. Gilestro indeed finds a small bias but its magnitude is the opposite of what one would expect to see. Correction for urban heat islands not only appears to be absent but there appears to be some small correction applied for what I can only describe as urban cold islands.
Secondly he makes no analysis of the temporal distribution of the adjustments. This makes a HUGE difference in what the trend looks like. If we adjust temperatures upward in the sceond half of the 20th century and adjust them downward by a commensurate amount in the first half of the 20th century then by Gilestro’s analysis all is well because they cancel each other out. In fact they don’t cancel out in the trend line. It’ll look colder than it really was in the first half of the century and hotter than it actually was in the second half. This should be obvious to anyone who can apply a triple digit IQ to it for a few minutes at most.
Dave F (13:51:22) :
Still not sure what you are trying to prove. If you want to say there is not much of an effect on the record, plot the GHCN unadj against the GHCN adj and see what the difference is.
That’s basically what Romanm’s plot does. It shows the difference, which reaches a maximum size of about 0.2C back in about 1900. Noticeable, but not huge.
Nick Stokes:
“You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.”
Of course you can. In fact that, is exactly what USHCN does.
And the USHCN adjustments, whatever the distribution of their +/- magnitudes is, adds a strong, nearly hockey stick shaped warming trend into the final US temp estimate.
Where is the similar plot of GHCN gridded temps, comparing the net effect of raw vs adjusted data on the global estimate?
JJ (14:17:03) :
I told you I’m not interested in your cavilling until you actually do something. You say that I haven’t been able to replicate the GHCN method. You are right, I haven’t been able to replicate the GHCN results. Why? Because despite an extensive search, I can’t find the stations to do so, nor have you. So sue me.
But on the other hand, you haven’t done a damn thing.
Come back when you have news from your enquiring emails to GHCN, or when you have identified possible stations for use in adjusting Darwin, or when you have shown that the GHCN method can take nearly identical records and twist them in different directions.
Because until then, you are just waving your hands and whining, and frankly, Scarlett, I don’t give a damn.
Nick Stokes @ 14:51:17
There are only two reasonable explanations for the adjustments seen at Darwin Zero. One is that a deliberate manual adjustment was applied to make the temperature trend appear to be an increasing one. The second is there’s a bug in the automated system that finds and applies the adjustments. In either case any honest researcher involved in the adjustment process should react to this by saying “This appears to be inconsistent with our published method of adjusting raw data. Let me figure out what happened and I’ll get back to you as soon as possible with an answer.” But instead we get nothing but rationalizations and denials. The so-called denialists are the warm mongers. The skeptics now appear to be the only ones NOT in denial.
Dave Springer (14:54:13) :
First of all if there is any bias in upward/downward adjustments the bias should be downward if the claimed corrections for urban heat islands were really made.
Secondly he makes no analysis of the temporal distribution of the adjustments. This makes a HUGE difference in what the trend looks like. If we adjust temperatures upward in the sceond half of the 20th century and adjust them downward by a commensurate amount in the first half of the 20th century then by Gilestro’s analysis all is well because they cancel each other out. In fact they don’t cancel out in the trend line.
First, the GHCN corrections don’t claim to correct for UHI, and they don’t. They try to detect and correct discrete events.
Your second is the fallacy that I’ve been trying to explain over and over, “In fact they don’t cancel out in the trend line.” What GG is plotting is the distribution of trend line changes. The rate of change over time, for the whole period of the station, in C/decade. In your example, a big upward trend would go into GG’s histogram. No cancellation.
What do you think it is a distribution of?
I have a odd thought about how it might have happened.
It looks sort of like a positive feedback loop.
Maybe something like this … they run the data look for a discontinuity. Clearly one is found around 1939. However, one can assume the old data should be lower or the new data higher … which raises the questions as to which one to change.
So it appears a 2nd order filtering algorithm runs in reverse starting in 1939 and ending 1920 with damping factor. It appears they got their – and + sign mixed up so instead of lowering 1930 to 1939 they raised it and instead of raising 1920 to 1930 ish they lowered it.
Next it appears they ran another algorithm going forward in time from 1939, this looks like a classical positive feedback — again probably a – and + signed mixed up somewhere so instead of damping you get ramping. (If someone can find another station with a similar discontinuity and see if the same thing occurs that would be great)
IF my speculation is correct — then this is not human caused, it is poor software coding, a lack of a software check for positive feedback, and an algorithm that goes unstable when such a profound step change is encountered. Engineers of all types accidentally make oscillating and positively feedback algorithms. Next time you hear a squeal at a concert from the speaker – think Darwin 1.
But who knows …
Nick Stokes (14:56:44) :
That’s basically what Romanm’s plot does. It shows the difference, which reaches a maximum size of about 0.2C back in about 1900. Noticeable, but not huge.
Still, what does the graph of raw data look like next to the graph of adjusted data? Is there a difference of 0.0175C, 0.2C, or some other difference? Isn’t this the best way to determine what effect the adjustments have, not making statistical sausage out of the data?
For you filter guys out there … sort of like running a notch filter that goes unstable when you hit with a big enough step change and starts a positive feedback instead of filtering because the Q is set to high.
Nick Stokes (14:56:44) :
Also for consideration as to why the distribution was the wrong approach to take, does not GHCN use grids, and wouldn’t the weight of the station also need to be taken into account for its use in the grid when an adjustment is applied?
Nick Stokes:
“Don’t the different “whole periods” for different stations matter?
Yes, they do.”
But that does not get reflected in histogram – at least not in GG’s, or does it?
“Length” of Musgrave record is more than 80 years – 25 years of data between 1907-1992, it’s just that there is a huge gap of those -999.9 in the middle.
JJ (15:06:05) :
“You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.”
Of course you can. In fact that, is exactly what USHCN does.
Yes, but USHCN uses metadata files. GHCN explicitly doesn’t, because it says that the way they are kept is just too varied across countries. If GHCN wanted to compile a history of manual changes, they would have to in effect create their own metddata files.
Janis B. (15:29:27) :
By length of record, I mean length after adjustment. This acts as a filter for interrupted sets, and would eliminate Musgrave.
Nick Stokes (14:51:17) :
“”I doubt that they were made manually. You can’t maintain an ongoing record with many thousands of stations if you have to keep track of manual changes.””
So don’t you think it’s a problem that the cooling adjustments you highlighted were made algorithmically according to Peterson, yet Darwin (an offsetting adjustment) was not?
Doesn’t this lead you to believe that GG’s plot should have revealed a net cooling adjustment if the temperature record was being objectively corrected?
Dave F (15:24:42) :
Also for consideration as to why the distribution was the wrong approach to take, does not GHCN use grids, and wouldn’t the weight of the station also need to be taken into account for its use in the grid when an adjustment is applied?
It has to be better to look at the distribution of all stations than to pick out individual (extreme) stations. The grid argument applies even more to just focussing on Darwin. Incorporating grid weights would be a good thing in calculating the derived mean. It doesn’t change the distribution.
wobble (15:37:07) :
I don’t believe that any of these adjustments were done manually.
Nick Stokes (15:43:36) :
“”I don’t believe that any of these adjustments were done manually.””
Does that include Darwin?
Nick Stokes (15:43:36) :
“”I don’t believe that any of these adjustments were done manually.””
Actually, cancel my last question. It doesn’t matter.
Willis is attempting to show a warming adjustment (Darwin) which may not have been made objectively by the Peterson algorithm.
GG attempted to demonstrate that it didn’t matter since all adjustment trends offset each other. And you claim on his blog that this is true post 1940 and post 1970 (but you didn’t show your work).
So neither GG nor you really doesn’t address the issue. What happened with the Darwin adjustments DOES matter even IF GG is right.
wobble (15:51:21) :
Yes. I believe they were all done by the same computer algorithm, which may or may not have been exactly Peterson’s.
Nick Stokes (15:41:22) :
Still, what does the graph of raw data look like next to the graph of adjusted data? Is there a difference of 0.0175C, 0.2C, or some other difference? Isn’t this the best way to determine what effect the adjustments have, not making statistical sausage out of the data?
Nick Stokes (16:07:54) :
“”Yes. I believe they were all done by the same computer algorithm,””
Well Willis disagrees with you.
Now we’re back to 1st base, right?
Willis,
Help is being offered to you. Do not take offense when the holes in your argument are pointed out, and do not lash out. Fix the holes. It results in a stronger argument.
GHCN station nomenclature documents that the first three digits ID the country. The next five digits are the nearest WMO station number. The next three digits are a modifier, and are zero if that is a WMO station. The final digit (the 0, 1, 2, 3 and 4 that differentiate the five Darwin records) is called the ‘duplicate number’. You need to know how multiple duplicates are treated in the GHCN, as opposed to multiple stations.
GHCN does not appear to have a numeric limit on the distance for ‘neighboring’. Perhaps you think they should. You can make your case for that, and it might prove fruitful. But you cannot impose a limit that is not present in their methods, and use that to claim that they didnt follow their methods. Those are two different issues.
You claim that the GHCN methodology cannot adjust different record duplicates differently. You rely on that as an assumption to make further claims. However, you provide no support whatsoever for that claim.
On the other hand, it is patently obvious how that might occur: Two different record duplicates, having different lengths, can easily have different correlation with a potential reference site. This is true even if those records overlap for some portion of their length, and have strong agreement in the overlap. Note that the reason behind the ‘hiding of the decline’ was to conceal just such a record length induced change in correlation …
Two record duplicates correlating differently to potential reference sites would result in them being adjusted with different reference series, which would obviously permit differing adjustments. Instead of assuming that this did not happen, find out. It would be valuable for you to know if that occurred …
Similarly, instead of simply assuming that the GHCN methodology limits its adjustments to the trend of the reference series, find out. It would be valuable for you to know if that is true …
On the same note, instead of simply assuming that the GHCN methodology limits the trend of the reference series to the trend of one of its inputs, find out. It would be valuable for you to know if that is true …
You are making many unwarranted assumptions in order to make your points. This not only results in a false or flimsy arguments for those points, it prevents you from making other points that may be more supportive of your ultimate goal.
You are not discussing Darwin in a vacuum. The only reason that anyone would give a gluteus rattius about what you are doing here is that it implies there is something wrong with the aggregate.
Currently, there is no justification for that implication. The GHCN methodology document explicitly acknowledges that there may be large adjustments to individual stations like Darwin. The GHCN methodology document explicitly acknowledges that those large adjustments to individual stations like Darwin may not track the local temperatures well. The GHCN methodology document also asserts that these are not an issue with large scale, long term temperature estimates.
So far, you have confirmed what the GHCN methodology states. If your point were to confirm GHCN, you would be well on your way. But that isnt your point …
Finally, in regards to your ‘you go out and do something before you complain about my work’ meme: That is precisely the refuge that the Hokey Team has taken with Steve Mc. “We’re not going to pay any attention to the deficiencies that you are bringing to light in our work. Who are you to show us holes in our methods? Go publish yourself.” Then they shut themselves away and demand that only people who agree with them 100% should dare speak to them.
This is not scientific behaviour. Nor a winning strategy.
Accept help.
Nick Stokes:
“Yes, but USHCN uses metadata files.”
Explaining how they do it is not proof that they cant 🙂
“GHCN explicitly doesn’t, because it says that the way they are kept is just too varied across countries. If GHCN wanted to compile a history of manual changes, they would have to in effect create their own metddata files.”
Which they could do. But they dont, opting instead to cut effort and use statsitical methods that can apparently produce a 6C trend where none exists in the data. Yet they stick with the metadata based adjustments in the US. If the statistical methods are sufficient to polish crappy third world data into a gem of a temp estimate, why not apply them everywhere?
Does the answer lie near the fact that the USHCN adjustments, whatever the distribution of their +/- magnitudes is, add a strong, nearly hockey stick shaped warming trend into the final US temp estimate?
BTW, where is the similar plot of GHCN gridded temps, comparing the net effect of raw vs adjusted data on the global estimate? Put the ‘nearly symmetric distribution’ and other handwaving aside. What is the net effect of GHCN temp adjustments on GHCN temp estimates?
To Nick Stokes
Coonabarabran NSW is given as an example of an NOAA NEGATIVE adjustment, to offest the positive 6 deg per century rate over 50 years at Darwin.
Here is a summary of Coonabarabran from recent Australian Burau of Meteorology files:
st,064008,64 ,COONABARABRAN (NAMOI STREET) ,01/1879, ,-31.2712, 149.2714,GPS ,NSW, 505.0, 510.0,94728,1957,2007, 99, 99, *, 0, *, 0,#
In short, although people setlled Coonabarabran about 1870s, the weather station was judged unreliable for use by the BOM except for the period 1957 to now. (or 2007 as it shows above).
On the other hand, the BOM accept the Darwin record back to 1869, with gaps for events like bombing in WWII of a couple of months.
Also, see an explanation for Coonabarabran:
“A Notable Frost Hollow at Coonabarabran,
New South Wales
Blair Trewin
National Climate Centre, Australian Bureau of Meteorology, Melbourne, Victoria
E-mail: b.trewin@bom.gov.au
Parallel observations, taken for 28 months between July 2001 and October 2003 at two sites at
Coonabarabran, New South Wales, show that topography has a dramatic influence on minimum
temperatures at the two locations. Over the period of the study mean minimum temperatures at a valley
site were 4.9°C lower than those at a plateau site 6.6 km away and 133 metres higher in elevation, with
differences of up to 14.3°C occurring on individual nights.
The observed minimum temperature differences were greater in winter (mean difference 6.0°C) than in
summer (3.0°C). Strong relationships were found between the magnitude of the minimum temperature
difference at individual nights and wind speed and cloud amount at 0300 and 0600 (local time), with
the minimum temperature difference largely disappearing on nights when the wind speed exceeded 8
m s-1 at the plateau site.
There was also a marked tendency, particularly in winter, for the largest temperature differences to occur
on the coldest nights, with the 10th percentile minimum temperatures at the two sites differing by 7.6°C
during the period of overlap. This corresponds to a dramatic difference in the frost risk between the two
sites, with minima falling below 0°C on 196 occasions during the 28 months of the study at the valley
site, but not at all at the plateau site. For a 2°C threshold the figures are 282 and 14 days respectively.
Mean maximum temperatures at the plateau site were 1.0°C lower than those at the valley site, approximately
consistent with the environmental lapse rate. There is little seasonal or day-to-day variation, with the
standard deviation of the daily differences being only 0.6°C, compared with 3.7°C for minima.
The large differences in minimum temperature identified in this study reinforce the fact that minimum
temperatures are highly dependent on local topography. This has substantial implications for the mapping
of frost risk, and other related climatic variables, at resolutions finer than the spacing of the station network
in areas with significant local relief.”
One suspects that there was an event like a site change in the 1950s. There might not have been. The absence of such information is used with dubious intent by pseuso-scientists to cast doubt. Such information alsoexplains the benefit of understatding individual stations well before jamming them into supercomputers with dumb inputs.
NOAA is therefore wrong again to use rejected data and GG was scientifically slack for not checking his sources.
Yours was not a quality post, Nick. The first example I tried gets 0/10.
\ JJ (17:00:37) :\What is the net effect of GHCN temp adjustments on GHCN temp estimates?
See above: Nick Stokes (14:56:44) :
wobble (16:28:53) :
Well Willis disagrees with you. Now we’re back to 1st base, right?
Yes. But look at the Coonabarabran plot. It’s even jumpier than the Darwin plot, but heads downward. If there’s a case for saying Darwin is manual, even more so for Coona. But what malign hand would be pushing it down?
Nick Stokes, you seem to be arguing that it is ok if the Darwin record gets messed up, so long as the amount that it gets messed up is offset by adjustments (perhaps valid ones) for other stations.
You’ve certainly succeeded in making me very suspicious of the assumptions upon which this changepoint method is based. When I have time/funding, I’m now curious to take a painstaking look at the fundamental assumptions underpinning the adjustment paradigm. We need accurate records at the local level to figure out the complexities of natural climate variations. (See the works of Currie.)
Bill Illis @ 11:41:23
Nice graphs. Shows it all. Interestingly some small fudge factor ~+-.25C is applied every year superimposed over a large fixed fudge factor spanning 10 to 40 year intervals where each interval steps up a half degree.
I was a prolific programmer for 30 years. If I wrote something that produced crap out like that adjusted temperature record and let it get released for use I would have been mortified, horrified, wouldn’t sleep until I found the flaw, fixed it ASAP, offer my most sincere and abject apologies to each and every harmed user, and beg for forgiveness.
An eyeball glance at the raw data reveals just one anomaly near 1940 where the temperature drops half a degree and remains down half a degree for the next 60 years. Software should have easily caught and corrected that and then there’s nothing else to catch. 120 years of very consistent data with little if any overall trend either up or down. I understand in 1940 the station moved which explains the one-time drop. The adjustment, however it was done, introduced serious flaws into what was an almost pristine record.
Geoff Sherrington (17:08:48) :
There’s nothing to indicate that your frost hollow story involves the Coona weather station. The town is by the Castlereagh river, but in a fairly flat region.
I suspect the reason for the big change is a possible merger with data from the Siding Springs Observatory. But that’s just a guess.
Nick Stokes:
“That’s basically what Romanm’s plot does.”
No it doesnt. That attempts to deal with the temporal issue, but in no way addresses the spatial. Already at 0.2C with just the temporal – a third of the whole value of the alleged ‘global warming’. What happens when those sites get area weighted differentially? When they’re used to fill in missing data cells? Another third?
How much are the UHI and othe runaccounted for local anthropogenic effects? Another third?
Until you can produce a plot of final GHCN global temps that compares adjusted vs raw, you’re just handwaving.
Nick Stokes (17:14:45):
“”look at the Coonabarabran plot. It’s even jumpier than the Darwin plot, but heads downward. If there’s a case for saying Darwin is manual, even more so for Coona.””
Willis did quite a bit of work in an attempt to convince people that the Darwin adjustment is manual. He showed correlations to neighboring stations and described the distances involved with those further away.
You merely claim that Coonabarabran is “jumpier?”
Gee, who puts forth a better argument?
JJ (16:45:42) :
JJ, as I said before, what you are offering is not help. It is handwaving and objections. I will gladly accept your help, but to date, you haven’t offered any. I’m not “lashing out”, I’m trying to stem your flood of well-meaning but meaningless platitudes. Let me give you some examples to clarify what I’m trying to say.
Despite trying, I am unable to create a reference series for Darwin because I can’t find relevant well-correlated neighboring stations. I know that. You know that. Help would be a list of appropriate stations, which you have not provided. Instead you keep saying that “neighboring” means over 1,500 km., well beyond the known limit of correlation between temperature stations. That’s no help at all, as any correlation beyond that distance would be by chance.
Despite trying, I have not been able to use the GHCN algorithm to twist nearly identical records in different directions. Help would be a demonstration from you that it can be done. Your statement that it can be done is meaningless. That’s just handwaving, and it is both useless and irritating. If you think it can be done, show us that it can be done.
I say that the GHCN algorithm cannot adjust the trend beyond that of its inputs. Why do I say that? Because I’ve read the description and I’ve tried it. Also, because the point of the algorithm is to make the target station agree with the inputs, not exceed them. You claim the algorithm can adjust it beyond the trend of the inputs. Help would be a demonstration that what you claim is true. Mindlessly repeating “yes it can, yes it can” is not help.
Finally, you keep bringing up Steve McIntyre. You say
As someone who has posted extensively on Climate Audit and who corresponds regularly with Steve, I can tell you that’s a fragrant vase of excrement. What Steve does is try to replicate results, as I have done. Often, he is in the situation that I am in … he can’t replicate the results. What he does then is post the fact that he can’t replicate the results. As I have done. Help would be you posting some way to actually replicate the results. Saying I should be like Steve is no help at all.
And no, I’m not using the Hokey Team excuse of “you go and publish before we can talk about this”. I’m asking for help, I don’t give a rat’s fundamental orifice if you publish. However, help does not mean saying “I really really really really think you are wrong about X”. That does nothing at all. If you think I am wrong, don’t talk about how right you know you are and provide endless justifications and reasons. Instead, demonstrate how right you are, show us how it can be done, bring in the citations that support your position, post a method, list the stations, anything but your unending litany of objections.
w.
PS – In researching this article I found this interesting report:
So I’m not the only poor fool who can’t replicate the GHCN algorithm, despite your repeated claim that it can be done. However, I wish you the best of luck in your inquiries with GHCN, and I await your report on their response.
Nick Stokes (17:14:45):
Coonabarabran has no less than thirty stations within 1,200 km. which are candidates for the earliest adjustment. Darwin has two …
@JJ: You repeat your mistake of the first analysis, by applying your own criteria for suitability, rather than GHCNs. GHCN does not appear to have any specific distance limit. GHCN appears to only be concerned if the stations are in the same ‘region’ (climatalogically) . You cannot say that GHCN did not apply its standard method.
Actually JJ, if you took the smallest trouble of looking at a map of Oz, you would see it is rather difficult to move more than 1500km from Darwin and still be in the same climatic region – its a big country, but not that big.
You strike me as a proponent of AGW posing as a sceptic that is using the rhetorical technique of challenging the minutae of every assertion as a form of obfuscation. It is similar to another form of obfuscation used on blogs that runs like this “before you challenge my simple assertion A then you must read obscure and difficult to obtain text B in great detail”. In your case you are pushing Willis to do more work to “prove” a point he has, in fact, already made very effectively. The condescending tone is no doubt intended to give the impression of great learning whilst not actually demonstrating anything other than fallacious argument. The fact that you clearly have no idea just how big Australia is exposes you as an ignoramus.
I accuse you of being a troll, and until you post something of real value to this blog then I suggest everyone treats you as a troll.
Nick,
The raw data can be adjusted so that it shows the famous hockey stick on a rolling average temperature plot and that wouldn’t change the trend line histogram.
Clearly the Darwin Zero data was badly misadjusted and in this case results in a drastic positive slope change on the trend line. The best case you can make from the trend line histogram is that the adjustments are equally flawed in both positive and negative slope changes. That’s not much of a defense for a broken procedure – saying it’s equally flawed in both directions so the flaws cancel out. It hardly inspires confidence that it isn’t broken in other ways. In any case the trend line histogram won’t reveal a temporal distribution problem where the adjustments are excessively negative in earlier records and equally excessive positive adjustments in the later records.
The entire analysis used at Darwin is wrong-headed anyway. Given we know what todays temperatures are (we can just go out and measure them) we only need to know the rate of change of past temperatures to get an idea of future trends. Therefore we don’t need to adjust to find the absolute temperatures at any site. If a site has been moved, had its measurement kit changed or whatever, then the discontinuities can be detected and the rate of change between those discontinuities can still be calculated without resorting to back-filling data with complicated computer algorithms that clearly can’t be relied upon not to do “wierd stuff”.
The UHI is a different matter, but in each case it should be possible to measure the effect on a specific site by taking actual observation at the site and near to the site but outside the urbanised area. The only places this would not be possible would be in very large cities.
The reason we are having to fiddle with all this data is because there is no good data. All the Stephenson screens were designed to monitor weather, not climate. They were put in places that were easily accessible to the chap that had to read the thermometers that then became urbanised and affected by heavy traffic over the last hundred years. They then had all the instrumentation updated to electronic thermometers with remote reading. Unfortunately this has given climatologists the excuse to fiddle with the data in the most bizarre ways possible to support their ridiculous claims, then dress it up as science to difficult to understand for the layman. It’s not difficult to understand – any high school kid could do a better job.
Given that much of the record at Darwin is the only record for a large geographical land area doesn’t that give it greater weight in computing global average temperature? It surely should. Was Darwin cherry picked for manual adjustment due to its larger weighting? Again this wouldn’t show up in a trend line historgram of all stations in the worldwide record. The trend line histogram (I presume) treats all stations equally where in the global average calculation each station must be weighted for how much of the earth’s surface it represents.
Anyway, Nick Stokes, I would still expect there to be a larger bias in the trends if the adjustments were not being overly managed. The fact that they are so close to 0 is not comforting. Like I said before, if the adjustments are made because of faulty siting, equipment failure, and so forth, I think it is very unlikely that it would work out to be a practically nil contribution to the trend. That it does needs an explanation because I seriously doubt the failures in need of adjustment work out to .0175C of difference. It honestly begs the question, why adjust the data in the first place? Shouldn’t unadjusted and adjusted look basically the same? Do you know if they do, Mr. Stokes?
Dave Springer: BINGO! Just as Antarctica came to be represented by one station on the Palmer peninsula, Darwin too has high weighting in the gridding step. This is a promising alley for more sleuthing by concerned people like Willis and Ed Smith:
Which temperature stations have large impacts on the global temperature anomaly? Of those stations, how many have “adjustments” that tend to exaggerate GW? I suspect that the answers to these questions may be enlightening. Alas, like Fermat, I’m too busy to do the research myself, but I hope Willis and Ed can make some headway.
Recall the CRUTAR index from Africa
http://climateaudit.org/2009/07/19/christy-et-al-2009-surface-temperature-variations-in-east-africa/
http://wattsupwiththat.com/2009/07/18/out-of-africa-a-new-paper-by-christy-on-surface-temperature-issues/
Why Nick Stokes et al continue regurgitating the same old arguments, it’s the same old problems that plague the surface station network.
Nick Stokes,
Your response to my observation that the BOM declines to use pre-1950s data for Coonanbarabran is?
If you go to St Kilda market on a Sunday you can pick up many sets of thimbles that are useful for hiding peas. Classically, the magician used 3. You might need a few dozen.
It’s not science to answer a comment with many points with one speculative answer to one selected point.
Yours was a very poor post.
Nick Stokes
RE my post Geoff Sherrington (22:40:07) : on 20/12
This refers to a 7-year BOM study which BOM documentation shows that the old Darwin site and the Airport location have identical mean temperatures. I have said that there could be reporting errors, though I suspect none will make a difference, for the BOM continues to make the comparison available.
If you have access to more information than I do (I have very little – I have to search for it), then perhaps you can explain why it is a good idea to put a step change into the data at about the 1940 changeover.
Ooooh, new information. I love new information.
I just paid the outrageous sum of $29.95 for Peterson’s paper describing the GHCN homogenization method. This is CREATION OF HOMOGENEOUS COMPOSITE CLIMATOLOGICAL REFERENCE SERIES, THOMAS C. PETERSON AND DAVID R. EASTERLING, INTERNATIONAL JOURNAL OF CLIMATOLOGY, VOL. 14. 671-679 (1994).
Regarding reference series and distance from the station in question, the paper says:
Now, for Darwin, the hundred nearest stations includes stations out to 1,656 km from Darwin. Since I had already looked at the stations out to 1,500 km from Darwin, this did not add many possibilities. Here are the first difference calculations for the final few of the possible candidate stations per Peterson for the 1920 adjustment to Darwin:
PALMERVILLE = -0.39
BOULIA = 0.31
NULLAGINE (NULLAGINE POST OFFICE) = -0.01
COOKTOWN AMO = -0.22
PORT DOUGLAS POST OFFICE = -0.07
LOW ISLES = -0.35
HERBERTON POST OFFICE = -0.31
As you can see, none of these stations are suitable. In addition to having low correlations, the correlation in all of these is negative, and Peterson says stations with negative correlation can’t be used.
So that complete the demonstration. We cannot construct a reference series for the Darwin 1920 adjustment using the GHCN method and criteria. There’s only three candidate stations with positive correlations for that time, and they have a correlation of 0.06, 0.35, and 0.35 … far too low for a reference series which, according to Peterson, must have a correlation of 0.80 with Darwin.
JJ, you happy now?
JJ:
GHCN does not appear to have a numeric limit on the distance for ‘neighboring’
JJ:
GHCN can apprently use any station within GHCN’s definition of ‘region’
I think, I got the point: “Neighboring” and “region” for GHCN does mean: somewhere in the same climatic region.
Ok, let it be so. But this brings us immediatedly to another severe problem. As I understand it the GHCN method is the following: 6 stations in this “region2 are behaving the same way, i.e. being correlated >0,8. The data of 5 are used to “correct” the data of No. 6. In the case of thermometer faults, station movements, all is fine, and this method can give hints to explore the chronic of No 6 for such effects. For Darwin is this for the year 1939 or 1940.
But if the deviation at No. 6 is caused by different reaction on climatic changes at its specific location, the latter will be wiped out! Then we end up which a situation we call in German: Die Katze beisst sich in den Schwanz (The cat bites in its own tail; I dont know an appropriate english expression). We start with the definition of “climatic region”, but if a station shows deviant behaviour because of different local climate, it will be corrected and corrected and corrected again, until it follows the proposition of being in the same “region”.
Best regards from Germany and Austria (not Australia ;-))
Geoff Sherrington (20:16:46) :
Nick Stokes,
Your response to my observation that the BOM declines to use pre-1950s data for Coonanbarabran is?
None. I don’t know who’s right. All I’m pointing out is that the mechanics that produce the Darwin rise can also produce the Coona fall. And these are outliers. They are in the outer 1% or so.
Dave F (19:38:26) :
why adjust the data in the first place?
Good question. GHCN themselves say that for a lot of purposes, you might prefer the unadjusted data. They say the adjustments help with regional attribution.
But put it the other way around. What would you folk be saying if they made changes, updated equipment etc, and didn’t adjust.
3×2 (05:46:54) :
I’m still left with a couple of questions.
For those who say that individual stations don’t matter, the global stats will sort things out
If, when scaled globally, local “adjustments” have no real impact … why bother “adjusting” in the first place? Adjustments at the individual station level obviously do impact both regional and global results. Especially if the end result is in the 1/10th degree range.
Specifically for Darwin adjustments.
If you take the difference between the “winter” and “summer” means over the entire record (25/28 from v2.mean) as being about 3°C. Is a 2.25°C “adjustment”, in any direction, at any point, even physically possible?
I would expect so, in ’41 the station was moved from the old PO (destroyed by bombing shortly after) to the airport. In the dry season the prevailing wind at Darwin is southeasterly, the old PO was situated on the end of a peninsula so the southeasterlies blow over water onto the land. This is shown by the data, the record low at the airport is 10.4, the record at the PO was 13.4. The move was made because the PO site was no longer satisfactory due to overhanging trees, subsequent to ’41 the site was relocated several times on the airport.
Nick Stokes (23:24:59) :
Nick, I don’t know about Dave, but I have no problem with adjusting for known discontinuities. Such things as equipment changes and station moves definitely introduce false signals into the data.
However, the devil is in the details. My own feeling is that we should take the first difference of the data, and eliminate the one observation from the year of the discontinuity. That way, the trend of the data (which is all that we are interested in) is maintained, and we can average the first difference of the observations directly with other stations to get an average trend over a period of time.
The averaging, of course, introduces other difficulties … but that’s a subject for another thread. My preferred method also doesn’t deal with slow changes, like buildings and trees and paving around the station … but then the GHCN method doesn’t deal with those either.
w.
GHCN have to adjust the data because they have almost no sites that have not changed over the last 100 years. They have sites that have had new screens, they have sites that went from mercury to electronic thermometers, they have sites that were in small villages 100 years ago that find themselves in large cities today, they have sites that had horse and cart going by in 1900 and then three lanes of articulated trucks in 1980, they have sites that were near coastal lighthouses that moved to being near inland airports. All sorts of changes that affected 99% of all the sites. What is worse, from a GHCN point of view, is that the few sites that didn’t change much over 100 years show no warming at all! They should be held up as “gold standards” but GHCN prefers to sublimate them under a mountain of dodgy data from less reliable sites.
Now, if GHCN could reliably detect when the site changes occured, they could simply look for the trend in temperature between those changes when it can be assumed that no changes to the measurement of any kind occurred and therefore the relative temperature change over time could be considered accurate. The fact that they choose not to adopt this simple approach speaks volumes in itself – copying data from one site 1500km away to another site has no scientific validity at all, especially when it intoduces huge adjustments in the trend to the final output. Remember, it is the trend that actually matters, not the absolute temperature. We know today’s absolute temperature and that’s all we need to know if we know the trend, so an adjustment to an absolute temperature that makes a dramatic change in the observed trend is a very bad adjustment approach indeed.
UHI must also be considered, but this can be detected at each site by direct measurement within the UHI affected area and just outside. If the UHI is measured this way over one year then it canb be used to introduce a downard uncertainty in any measured warming trend.
Here is how this particular aspect of the fraud works. Lets say we have two sites that in 1900 were edge of town. By 1960 these towns have expanded and there is heavy traffic causing the urban heat island effect. We then decide that a site is so bad that we move one site out of this urban heat island effect, and the other remains where it is, thus affected by the urban heat island effect.
The algorithm used here is looking for discontinuities, so it will detect the discontinuity in the site that is moved. It will detect no discontinuity in the site that was not moved, but which is affected by urban heat island effect. This is because the urban heat island effect will grow gradually over time and will be indistinguishable from other climate effects.
The algorithm will detect that the site that has been moved has dubious data. It will then seek to correct the data with nearby sites that have not shown a discontinuity. If this is the site with the urban heat island effect, it will copy the data from that site to the moved site and smooth it into one continuous trend.
The impact of this is that rural sites will appear to show a warming trend (which is asctually due to the urban heat island effect of neighbouring sites). Since the site is now rural it will appear as if it can’t be affected by urban heat island effect, but in fact it will have been affected by the copying of data from precisely those sites that are affected by urban heat island effect.
This algorithm is biased in favour of trashing rural site data unaffected by urban heat island effect in favour of data that is affected by urban heat island effect. It has no credibility whatever.
Nick Stokes (17:36:22) :
“There’s nothing to indicate that your frost hollow story involves the Coona weather station. The town is by the Castlereagh river, but in a fairly flat region.”
Coona is a cold hole in winter. It’s surrounded by a ring of low hills apart from the path of the Castlereagh. I don’t doubt at all that it’s in a frost hollow, although the magnitude of the cooling surprised me. It also has the Warrumbungles immediately to the west.
You might be confusing it with Coonamble which is also on the Castlereagh and *is* dead flat. It’s only about 100km NW of Coona, so it’s easy to confuse them if you aren’t familiar with the area.
Nick,
Your point that Darwin is among a small percentage of outliers may be pointless in light of the fact that not all stations are equal in the impact they have on the global average. As it stands now Darwin is a high impact station where the adjusted data contains a very large and very false warming trend inserted.
If you could provide an example of a similarly high impact station where the adjusted data has a false cooling trend of similar magnitude (-0.6C/decade) then your point about the flaws in the adjusted data cancelling each other out** might be valid. Otherwise that dog doesn’t hunt.
**They still don’t cancel out completely. It’s already been accepted at face value that the adjustments result in an average bias of approximately 0.2C/century of additional warming. That is a significant number when we’re trying to dig out what, if any, human-caused warming has happened.
So far you’ve persuaded me that at least 0.2C of the last century’s warming isn’t man-made but rather is Mann-made, if you know what I mean, and I think you do.
Ryan Stephenson (02:45:27) :
I certainly agree with your approach.
The idea of using stations up to 1500 km or more away “to adjust” a site seems very “dodgy” at best. I do not know Australia, but the whole idea that weather in one location is similar to that in another does not pass the smell test. Within 100 miles of where I live we have oceans, lakes, the piedmont and sand hills. Five miles away ALWAYS gets more rain that I or the airport weather station down the street does. The idea might work OK on a flat plain but not on mountains and ocean front areas. And yes I do understand a significance test for correlations. As a test to determine if the station may have a problem that needs investigation – fine. As the basis for “adjustments” NO.
The whole thing reeks of lazy data handling techniques.
Nick Stokes (23:24:59) :
I am aware of that, my point was not that the data shouldn’t be adjusted, just that you are saying that it does not contribute much. If that were true, then wouldn’t there be a minute difference in the way unadjusted and adjusted data look plotted next to each other for the entire GHCN dataset? Is there a .0175C difference in the trend when you do that, which is what you and GG are saying in your histogram?
Willis Eschenbach (22:57:54) :
“”our only test based solely on distance was limiting the list of potential reference stations to the 100 stations nearest to each candidate station.””
Great work, Willis.
1. I think you’ve shown that the Peterson algorithm couldn’t have been followed. (Can you just tell us where you read about the 0.8 correlation requirement? Isn’t it possible that they accepted a correlation of 0.4 as long as it was the “highest correlated neighboring station?”)
2. I think you’ve also shown at least one flaw in the Peterson algorithm even if it is utilized objectively. Testing 100 stations for adequate correlation doesn’t seem reasonable. As you hinted at before, even if 1,000 adjustments are using correlations with a p-value = 0.05, then that implies 50 of those 1,000 correlations were a coincidence.
Overall, I think JJ did you a favor. He was definitely condescending, but he did push you to strengthen your argument quite a bit. However, I agree that it would be nice for him to help out with some of these efforts. He’s obviously smart enough. Unless he’s working on something of his own right now. JJ, do you care to share?
Nick Stokes, I appreciate some of the analysis you’ve done, but it seems that your entire argument boils down to the fact that Coonanbarabran is jumpier than Darwin.
All your histograms of trends are quite useless without being able to prove that Coonanbarabran was also manually adjusted. Sorry, but this can’t be done using a visual jumpiness test. Maybe we should start by looking at the nearest 100 stations and finding the most highly correlated.
@Willis: Have you tried taking the approach you/we suggest? Take the data between the discontinuities detected by the algorithm, then check the gradient of that data? Maybe it is difficult because there are only 10 or so datapoints and it is dependent on where you start and stop exactly, but it seems that the gradients between the discontinuities are about 0Celsius/Century compared to the gradient imposed by the corrections. Excel can plot a line of regression between the points, e.g. for the last 12 years we seem to have stable data and Excel can plot a line of regression between the last 12 points of the raw data and the gradient of this line can be derived. Sorry to dump the extra work on you but if you have the data tabulated already it should take you long.
That would really blow this whole thing out of the water. After all, what the algorithm itself is telling us is the site changed about three times after 1940 but was stable as a measuring site between those changes. So that is where the quality data is – the raw data between the site changes. Measure the gradient between the changes and its job done. It sure isn’t 6Celsius/century. So the algorithm will be shown to fail the sanity check badly.
Without having looked at this particular station specifically, a couple of points, just because A and B look similar doesn’t mean they will pass/fail the same statistical test.
Given an artificial cut off (e.g. p-value 0.01), it is possible that one station will “fail” the test (e.g p-value = 0.0098) and needs correction, and one that “looks” it like won’t (e.g. p-value = 0.011 (which means no correction).
The samething happens with the correlations. Even assuming that two stations that look a like both “fail”, there is no guarantee that they won’t be corrected differently. A and B are highly correlated. A is correlated to C with a value of 0.81. C is used to correct A. B has a correlation to C of a value of 0.79. C isn’t used to correct B. As C is likely highly correlated to whatever else is used to correct B, it is likely the correction is similar, but over a number of years with enough corrections, it is possible to start with two things that look very similar, but don’t end up being very similar.
The last issue is your given correlations. I’m not sure why 1915 is relevant, or giving the correlation over the whole rest of the life of the stations (e.g 1915-until the data ends) (which is what I presume you did as you didn’t give any years).
My reading of the work was that they were doing a “local” (in time) tests to do a “local” correction based on pretty “local” correlations (I don’t remeber the exact times frames, but 5 years sticks in my mind). In other words, if I’m going to correct something in 1945 (just an example), then I don’t need the correlation back to 1915. I do need a station that has a rather long data set (25 years sticks in my mind), but that I don’t think the that 25 years is even post-correction data (e.g. I could use a station that ran from 1940 to 1965 to correct an issue with another station with an issue starting in 1945).
There are issues with exactly reproducing what they did. For example, it wasn’t clear to me if they were doing correlations to find “good” stations based on the corrected data for the station or the uncorrected data for the station.
And you are right, they’ve AT BEST poorly defined “neighborhood” and what stations they weren’t able to find a “neighborhood” for and didn’t correct and which ones they did are unclear.
I generally don’t think they expected anybody to try and EXACTLY reproduce what they did when they published the method (the publication before this really exploded into the issue it has). They assumed that people would either simply take their corrected data and use it, OR take the raw data and come up with their own method.
I will point out that isn’t exactly rare. In biology, for example, papers rarely truely give all the details. They rarely state, for example, how solutions are stored, and what quality of water was used to make them.
I understand what you are trying to do, but it would be much more convincing if you developed a method based on the raw data that didn’t show warming. There were always be points of contention with any method (e.g. why use the time frame the did to do the tests, why use the p-value cut off they used to say a particluar set of data “failed” and need a correction, why use a correlation of 0.8, etc.).
If for a some set of stations, their method “fails” that isnt’ shocking and doesn’t change the total result.
A reasonable method that took the same data and showed that golbally warming isn’t happening would be a big deal.
Using sites 1500KM away?
This shows the difference in sites across the UK,
Ten years with the hottest months from the Met Office Station Data.
Order of maximum temperature
Oxford back to 1853
2006 1983 1995 1911 1921 1976 1868 1947 2003 1874
Stornaway back to 1873
1899 1947 1997 1880 1955 2006 1933 1955 1976 1901
Durham back to 1880
2006 1995 1887 1947 1983 1975 1989 1901 1940 1990
Sheffield back to 1883
2006 1975 1983 1995 1976 1947 1921 1934 1995 1887
Eskdalemuir back to 1911
2006 1947 1955 1983 1989 1995 1975 1976 1934 1940
temp (08:34:27) :
“”If for a some set of stations, their method “fails” that isnt’ shocking and doesn’t change the total result.””
I don’t think Willis was trying to show that their method failed. I think he was trying to show that they failed to use the method.
Yeah, but he hasn’t done that. One of his simple claims is that since one was corrected and the other wasn’t, and they look the same that is evidence that they didn’t use their method, but that isn’t true.
You actually have to run the method and see which if either fail the test that is described. If they all fail the test (again, fail meaning they need correction), then you have issues, and if none of them fail the test, then you have issues.
Once you’ve determine if (and where) they fail the test, then you can start to worry about identifying the relevent stations to do the correction.
To simply state that they didn’t because it doesn’t “look” they did, and without actually trying to carry out the steps they do describe doesn’t mean you’ve reached a valid conclusion.
temp (11:57:47) :
“”To simply state that they didn’t because it doesn’t “look” they did, and without actually trying to carry out the steps they do describe doesn’t mean you’ve reached a valid conclusion.””
Check is post at (22:57:54).
I think it completes the effort of trying to carry out the steps they do describe.
He tested the 100 closest stations (which is prescribed by the Peterson method) and found that none had a correlation adequate enough to make adjustments.
What more do you want him to do to prove that the method wasn’t objectively used?
Sorry, I meant, check HIS post at (22:57:54).
As I said originally, I am assuming he is giving correlations over the entire time series because he isn’t giving starting and ending years for producing the correlation, which isn’t what they did, I think.
If that is wrong, then fine, but it isn’t clear from what he has written.
temp (12:33:00) :
“”which isn’t what they did, I think.””
What do you think they did? Do you think they truncated time series in order to improve the correlation? If so, then many people agree with you, and you might be right.
So what exactly would you recommend? When calculating correlations of 100 neighboring stations to be used to test for a discontinuity in a specific year you will be required to perform thousands of correlation calculations by truncating each time series by one data point at a time.
Then, you’ll need to repeat that process in order to test for a discontinuity for each point of the station’s time series. 70 years worth of time series data discontinuity tests represents hundreds of thousands of calculations in order to test a single station record.
If you have a better solution for replication, then please share it.
Willis,
With vjones’s help and with the aid of EMSmith’s excellent documentation, I’ve been carrying out my own analysis of the NOAA GHCN data. My first step was to reproduce you excellent analysis for Darwin (unlike the Team who think that ‘there’s nothing to see here, move on’). I’ve therefore been applying the scientiic method and have attempted to falsify your analysis. I’m sorry (actually I’m glad) to say that I failed! I’ve reproduced your charts and results almost 100% and have documented my efforts on vjones blog ‘diggingintheclay‘. You can read the thread in which I reproduce your analysis by clicking on the link below.
Reproducing Willis Eschenbach’s WUWT Darwin analysis
As I’m sure you already know and appreciate science progresses by ‘standing on the shoulders of giants’ so I’ve taken the liberty of further extending you excellent analysis for Darwin to all the WMO stations in the NOAA GHCN dataset.
Specifically I’ve attempte dto answer the question posed by others on your original Darwin thread as to whether or not Darwin is a special case or not?
Well judge for yourself by clicking on the link below which documents my extension of your analysis to include the whole NOAA GHCN dataset.
Physically unjustifiable NOAA GHCN adjustments
The following is an excerpt from the thread
“In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data. As can be seen from the following “Cooling turned into warming” table (Table 1) below, which lists the Top 30 WMO station on the “cooling to warming” list, Darwin is ranked in only 26th place! The list is sorted by the absolute difference in the magnitude of the raw to adjusted slopes i.e. the list is ranked so that the worst case of “cooling” converted to significant “warming” comes first, followed by the next worse etc.
It’s clear from looking at the list that Darwin is certainly not “just a special case” and that in fact that there are many other cases of WMO stations where (as with Darwin) NOAA have performed physically unjustifiable adjustments to the raw data. As can been seen from Table 1 many of these adjustments result in trend slopes which are greater than the IPCC’s claimed 0.6 deg. C/century warming during the 20th century said by the IPCC to be caused by man’s emissions of CO2 through the burning of fossil fuels.
”
KevinUK
Gail Combs (05:44:57), you raise an interesting and often misunderstood point:
What is used to adjust the data is not the absolute value of the data. It is the change in the data. As you point out, someplace five miles away may get very different temperatures from where you are.
However, what is related is the change in temperature. If you are having a hot month, it is very likely that someplace five miles away is having a hot month as well. It is this relationship that is used for the adjustment.
However, that doesn’t mean that the adjustment is valid. As someone pointed out above, what it can do is re-institute a UHI signal on a station which has been moved.
Someone else pointed out that Darwin is an outlier. While this is true, such outliers should be a huge warning flag to whoever is using the algorithm. People often say “the exception proves the rule” without realizing that “prove” in this expression has the meaning used in the phrase “proving ground”, as in “testing ground”. The real meaning of the phrase is “the exception tests the rule”. In the current case, the rule seems to be failing the test …
wobble (06:53:46) :
Not according to the GHCN document, An Overview of the Global Historical Climatology Network Temperature Database, which says (emphasis mine):
There are many flaws in the Peterson algorithm, both theoretical and practical.
I’m kinda weird, in that I do all the scientific work that I do simply because I’m passionate about climate science. I have persevered and gone forward with my analysis not because of JJ, but in spite of him. These kinds of studies take time, as unlike Peterson and Hanson and their ilk, I have a regular job that does not involve climate science.
I don’t think JJ is a bad guy, I just get upset by constant cavilling with no corresponding productive input. Lead, follow, or get out of the way.
Ryan Stephenson (08:27:36) :
The problem is that the method you suggest assumes that there really were some kind of problems needing adjustment in those years. I find nothing, either in the record itself or in the station metadata, indicating any problem in those years. The Aussies make adjustments based (very loosely, it appears) on the metadata … but they don’t make a single adjustment in any of the years adjusted by GHCN.
temp (12:33:00) :
My apologies for the lack of clarity, temp, you raise an important point. For the 1920 step, I used the correlation for the period 1900-1940, or as much of that period as the station covered. There is nothing I could find in their documentation which describes what period they actually used.
Nor does their documentation specify a couple of other important things. One is whether the correlation is statistically significant. If we flip two coins, and then flip two other coins, they may come up all heads (correlation = 1.0 between the two sets of two). However, because of the length of the dataset (N=2), the correlation is not statistically significant. A ten year temperature dataset is generally not long enough to establish significance.
The other is the effect of autocorrelation on the significance. (Autocorrelation measures how much this year’s data resembles last year’s data.) Temperature records tend to have fairly high autocorrelation, a warm year is likely to be followed by a warm year and vice versa. As a result, we are more likely to get a spurious correlation between temperature series.
I chose a forty-year period because of those two factors, which make a shorter time period more likely to produce spurious correlations. I did not use the entire period of the overlap between two stations because the correlation on that is likely to be lower than on the 40 year period, and I wanted to give the algorithm every chance. However, I still found nothing resembling Darwin’s record in the “neighboring” stations.
Which brings me back to my point that nature is not homogeneous. It is freckled and fractured, a “ghost of rags and patches”. Trying to force it into a homogeneous strait-jacket can be done … but what we end up with, almost by definition, will be “un-natural”. Nature specializes in jumps and changes, in instant inversions and retroversions, and not in bland homogeneity. As the poet said:
Homogenize that and see how it all looks … not pretty.
Willis Eschenbach (14:52:53) :
“”according to the GHCN document,…the correlation between first-difference station time series and its reference time series must be 0.80 or greater””
Thanks! Again, great job.
@Willis: I think maybe you misunderstand me. I am not suggesting that the algorithm that detects the discontinuities was right or wrong. That doesn’t matter to me. What matters to me is if the adjustment that was made, automatically or manually, and whether that adjustment was reasonable. Up till now you have been asking “the Team” to justify their adjustments, and waiting…. an waiting….. But I have suggested an approach that will indicate if their adjustments were wrong or not without their help.
So lets assume the algorithm that detects the discontinuities is doing so “willy-nilly”. So what? The Teams algorithm tells us that between 1961 and 1980 there was no need for any adjustment – because the black line is perfectly flat here. This means that the algorithm is telling us that the site was perfectly stable during that time and that the temperature measurments were valid. So let us look at the raw data between those dates. We can pick out the specific years that are between the adjustments that were made – about 17 years altogether – then we can plot a line of regression between them. The gradient of that line of regression will give us the rate of change of temperature. I haven’t done the plot, but just looking at it suggests about 2Celsius per century during that period*. But the gradient of the adjusted data shows a gradient of 6Celsius per century. These two facts are in direct contradiction. The site is detected to have been apparently stable for 17 years, by the Teams own algorithm, so the measured rate of change during those 17 years should be valid, by the Teams own criteria. But their own adjustment method comes up with a rate of change three times higher than this. The adjustment method fails this simple sanity check. There is far more adjustment than is necessary.
*naturally the measured rate of change is not attributable to any specific source. Could be CO2, could be UHI, could be something else.
@Willis: OK, I tried to do it myself, but GISS/GHCN are both unable to deliver me the raw data 😉
I read the results from the graph you have shown of the raw data during about 1962 to 1979 and stuffed them into Excel, which was then used to add a linear line of regression. The gradient of that line came out at 3.5Celsius per century. Very high, certainly, but definitely not 6Celsius per century. The algorithm used to make the adjustment fails the sanity check – the adjustments used at Darwin are not reasonable.
Before anybody says “wow, 3.5Celsius per century- we’re all going to fry” remember this is data over only a 17 year period, it is preceeded by a long cooling period of about 1Celsius per century lasting some 50 years or more, and we aren’t able to indentify the source of the warming trend – it may not be CO2. I think if you did the same analysis of the full dataset using this approach then you would probably see that temperatures today are about the same as they were in 1880. Shame we can’t get hold of the raw data as it seems to have been, um, taken off line.
Phil. (23:26:41) :
This is shown by the data, the record low at the airport is 10.4, the record at the PO was 13.4.
Which data? I can’t find any 10.4 or 13.4 for Darwin in v2.mean.
My reply to, Willis Eschenbach (15:36:39).
Well, I’m pretty sure that they didn’t use a signficance measure or they’d certainly of mentioned that. It seems like they used a straight cut off of 0.8. You could certainly argue not using a p-value cut-off is a weakness.
I’m also pretty sure your 40 year period is likely to wrong. Remember in total, they are only looking for 25 years or so of data from the station (certainly not 40). You’ve linked the pdf to one of their new papers, but if you go back and look at the refs in that, I’m pretty sure that they talk about a 5 year window analysis, and at the average global level that has given us something that looks similar to what they’ve produced (not exactly).
No homogenization method is going to be perfect or “natural”, but the fact of the matter is that the data does need to be homogenized (though I will point out that if you simply take all of the data and average it and ignore everything you still get a nice bi-step warming this century, including a late 20th century warming trend, resulting in now being as warm as any other time this century).
For your point to be relevant:
1. You have to claim no use of the data is “good” and that we should essentially ignore all of the surface data.
or
2. You need to suggest a method that produces a more “natural” result that doesn’t show warming.
My reply to wobble (12:57:25).
If he REALLY wants the details about this particular station, I’d suggest he’d do what has already been suggested- that he contact GHCN and ask them what they did.
Ryan Stephenson (17:10:25)
“Shame we can’t get hold of the raw data as it seems to have been, um, taken off line”
If you are after the NOAA GHCN V2 data you can find it and download it from here.
NOAA GHCN V2 data
and it might be worth having a look at the links in this thread.
Reproducing Willis Eschenbach’s WUWT Darwin analysis
KevinUK
Ryan Stephenson (17:10:25)
If you (and anyone else) are interested in getting access to the NOAA GHCN data using a very friendly user interface that amongst other things enables you to easily select and then export raw and/or adjusted data for a selected WMO station to an Excel spreadsheet/CSV file, then click the link below to the TEKTemp system (as I’ve called it).
You’ll need to register first before you can login and use it, but its simple as you just need to supply a valid email address (to be used as your username) and a password on the TEKTemp registration page
The TEKTemp system
Note this system is currently running on my own test web server and so may not always be on-line. I hope to port it to a proper web hosting account shortly
KevinUK
KevinUK (13:23:59) :
In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.
And how many instances of turning warming into cooling?
temp (18:47:44) :
BWA-HA-HA-HA …
Sorry, temp, that just slipped out, couldn’t help it …
But heck, why not? I’ve tried this path many times without any joy, but this time might be different. I just sent the following to the address given on the NCDC web site for questions about GHCN, ncdc.info@noaa.gov .
So … we’ll see. I’m not holding my breath. I’ll report back with the answer. My bet is on the polite brushoff, but since we’re now in the post-Climategate era, who knows? I’d be happy to be surprised.
temp (18:44:00) :
Do you have a citation for this? Is it gridded, or simply an average? How is the averaging done?
For the record, I should say that I do think that the planet warmed in the 20th century … and the 19th century … and the 18th century …
I also think that we are no warmer than the earlier warm period in the 1930s-40s.
And the ride through the fun house begins … in reply to my email of questions for GHCN, I just got the following:
I’ve resent the email to the address in the automated reply, ncdc.orders …
I’m sorry this is a bit late, may not be relevant to Willis Eschenbachs work or may have already been covered. WUWT has been so busy lately it’s hard to keep abreast of things. I do feel however that it would of interest to Australian station researchers. The late John L. Daly examined Darwin station and here is what he found.
But there is something peculiar about Darwin, a tropical site. It shows an overall cooling. But that cooling was mostly done in a period of only 3 years between 1939 and 1942.
http://www.john-daly.com/darwin.htm
@temp 18:44.
“No homogenization method is going to be perfect or “natural”, but the fact of the matter is that the data does need to be homogenized ”
Unfortunately the homogenization DOES need to be perfect. We are looking to adjust data that only persists for about 15 years between significant site changes, and the underlying trend we are looking for during those periods is only 1Celsius per century – i.e. 0.15Celsius when the difference in annual means can be anything up to 1.5Celsius. So a very small error in the homogenisation process will kill the accuracy of that data.
So is the total homogenisation process perfect? Well the approach I have used to do a sanity check of the data at Darwin uses the Teams own algorithm to detect when the site is perceived to be “stable” and not requiring any homogenisation. The problem is that these periods are only about 15 years long, and during that time the data can be going up and down more or less randomly by 1.5Celsius. Fortunately, at Darwin between 1962 and 1979 the rate of rise is very steep. Therefore we can plot a line of regression between the points and even though there is considerable uncertainty in exactly where the line should run, we can see that it is certainly closer to 3.5Celsius/century rather than 6Celsius per century. No amount of fiddling will allow you to impose a 6Celsius per century rate of rise on the raw data between 1962 and 1979, during which time the Teams own algorithm tells you the site was stable.
So Darwin fails this simple sanity check – the algorithm as applied here is not working. Period. OK, so now you say “ah, but Darwin is unusual – the algorithm works in ‘usual’ cases”. Problem is – how can you prove that? In the “usual” cases the rate of rise in the adjusted data is much smaller. If I applied the same sanity check approach to a “normal” station I couldn’t say anything with any certainty because there would be so much uncertainty about where exactly the line of regression should be fitted, so I wouldn’t even try. So we can’t sanity check the algorithm in the “usual” cases – we have to take it on faith that the algorithm is working. Most mathematicians would throw out the algorithm at this point – they would say it fails one sanity check and those it might pass are highly disputable. They would take a look at the algorithm again and make sure it was doing something sensible with the Darwin data. They would use the Darwin data to expose the flaws in their algorithm.
“the fact of the matter is that the data does need to be homogenized ”
Well, why is there a need? We can see from the Teams own algorithm that the sites are really only stable for about 15 years before some massive adjustment is necessary. We can see that doing a line of regression to find the rate of change of the temperature during those 15 year periods is problematic because there isn’t enough data in 15 years to be sure that the gradient we get is correct. But is replacing the data with data from another station a valid method of getting a long, unbiased trend?
Well remember what we have just said. The sites are changing every 15 years according to the Team own algorithm. But the algorithm looks for correlation between several sites in order to backfill dubious data in the site under inspection. But how can the algorithm reliably do that? If all the sites are likely to be changing every 15 years, how can you reliably correlate one site to another? Remember that both the site under investigation and the sites used for homogenisation are changing very often – if this wasn’t the case then you could just throw out the data from the sites that had changed and stick with the sites that show no site changes. So the algorithm is probably backfilling data between all the sites in a given region, taking data from site 1 and giving it to site 2 which gives data to site 3 and also to site 4 in some crazy difficult to follow method that might very well result in some circular changes being made (unless of course they are using the method “all sites are created equal, but some are more equal than others”!). The data may match over some specific 15 year period to an extent, but is that sufficiently reliable? Is it any more reliable than fitting a line of regression to 15 years of stable data? Well the algorithm doesn’t seem to look for perfect correlation anyway, and the correlation it is looking for is based on the annual changes in the means i.e. a weather signal not a climate signal. This of course sounds rather dubious as a process, and then we have the sanity check peformed above which shows us that the algorithm is known to fail anyway.
There is another underlying problem with the homogenisation process. It tends to be biased in favour of stations that suffer the worst UHI effect. Most sites at the beginning of the 19th century would have been edge of town locations with low-rise building, little tarmac and no heavy traffic. During the 20th century all these factors would have changed contributing to significant UHI (which is why simply averaging all the data only proves there was warming of the measurment kit, not that the wamring was specifically related to CO2). Thos sites that were moved during the 20th century were almost certainly moved from urban areas (where they got in the way of development) to rural areas. However, the Teams algorithm automatically detects these sites because of the discontinuities in the readings caused by the site change, and seeks to replace the data from elsewhere. Where is it most likely to get that data? From those sites that have not changed much. Those are sites that were on the edge of time, have seen massive development around them, but then have not been moved. The site then suffers from UHI – but the Teams algorithm can’t detect that because it can only see sudden changes in the means either side of data from a given year. So the algorithm detects these urbanised sites as “good” sites and then uses these to replace the data at the “bad” rural sites. This is exactly the opposite of what logic would tell you should be the case.
Fundamentally Willis has shown that the algorithm is definitely wrong, that the output of the algorithm cannot be relied upon in any cases and that the underlying logic of the algorithm is faulty. We can go on to say that the high frequency of site changes and the unknown precise impact of UHI at each site makes it impossible to rely on the surface temperature measurements as a means of detecting climate change of 1Celsius per century reliably and should be abandoned. The best we can do is to learn from these mistakes and set up true climate monitoring stations to give us data reliably for the next 100 years unimpacted by UHI or site changes.
Nick Stokes (20:34:00) :
“KevinUK (13:23:59) :
In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.
And how many instances of turning warming into cooling”
Nick why don’t you visit my thread on ”diggingintheclay‘ and you’ll find out?
The link to the thread on ‘Physically unjustifiable NOAA GHCN adjustments’ is here
As you’ll see I found 216 instances in which NOAA’s GHCN adjustments resulted in ‘warming being tirned into cooling’ and I list the Top 30 worse cases. The worst case is Mayo Airport, Yukon, Canada. As with Darwin and many other stations there is no physically justifiable reason as to why its raw data needs to be adjusted. It is an almost complete set of raw data spanning 1925 to 1990 with (according to NOAA’s GHCN station inventory data), no station moves and no duplicate series. Its raw data trend looks relative ‘flat’ but is in fact slightly positive at +1.3 deg. C/century. For some reason NOAA feels the need to adjust this raw data so that all data prior to 1983 is increased using a ‘staircase’ type step function (see the thread for the charts) similar to Darwin (and Edson – see thread) except in this case the adjustments result in a strong cooling trend of -4.5 deg. C/century.
I’ve checked the raw data for Mayo Airport on the Environment Canada web site here and it shows that raw data is available from 1984 to 2009 and it all looks fine with occasional missing data so why did it ‘drop out’ of NOAA’s adjustment analysis after 1983?
Perhaps you’d like to have a look at some of the other stations on my two lists? It’s easy to extract the data for individual WMO stations using my TEKTemp admin system (see earlier post for link).
KevinUK
KevinUK (02:46:24) :
I loaded Mayo A data from the Climate Explorer data into Excel, plotted the adjustments and see what you mean by ’staircase’.
Jesus… there must be some kind of ‘statistical explanation’ – or whatever the hell scientists call it – to substantiate why the algorithm does not introduce spurious trends, and does, in fact, qualitatively improve on the ‘original’ data. And no – that the adjustment trends largely cancel themselves out over whole dataset is not a sufficient explanation, I think.
Actually, would it be possible and not too difficult to group and at least average the trends (or the resulting anomalies) somehow – without weighting them and not within grid but, say, by some of the countries?
“Do you have a citation for this? Is it gridded, or simply an average? How is the averaging done?”
No reference. I did it myself. Took 10 minutes. It is an easy thing to do, and I doubt it is publishable in of to itself, but we are working on doing something with it (essentially taking an opposite direction with respect to constructing global temp trends).
If you don’t believe me, do it yourself, or ask somebody that you trust to do it (ask a regular contributor to this blog to do it). Doing the simple average is really really easy.
For averaging w/ no gridding, I simply took the values for all the stations availible at any time point and averaged them.
Gridding adds layers of compexitity, but I did it as simply as I could.
With the completely raw data just averaged, I don’t get the the late 1930’s to be especially warm. The other “warm” period in this century shifts by about 30 years or so. I contribute this mostly to changes in the distribution of stations. In early years, they are mostly European and N. American. Post-WWII, more stations from other parts of the world come in, and you see warming (there might be a urban heat island and/or green house gas effect contributing). From there, the average decreases pretty substantially, but then it goes back up.
Ryan Stephenson (02:35:00)- I just want to point out that did I offer another option. That is to simply throw out all of the surface data.
Generally thought, I disagree with you. Through different data analysis methods people routinely pull good information out of data that has serious flaws.
Possibly, this homogenization method isn’t the best and has issues, which is why I suggested from the beginning that something that would generate real interest would be a “better” homogenzation method that doesn’t show global warming.
I spotted another logical flaw in these homogenisation approaches.
The temperature data is only measured to the nearest 0.1Celsius. In order to detect a correlation between two sites, you would have to allow variations of +/-0.1Celsius to allow for variations that are purely due to the random uncertainty in the measurement due to the low resolution of thermometers. This means that to allow correlation between two sites we must allow discrepancies of (at least) 0.1Celsius that occur between the two sets of data.
However, the climate signal we are normally looking for (not at Darwin, which seems to be a special case!) is of the order of 0.1Celsius per decade. So if we use a correlation algorithm which says “oh, if the sites don’t correlate right down to 0.1Celsius we don’t care” and we use this to correct for sites that have moved every ten years then we are permitting uncertainties in the correlation which are as big as the alleged climate change signal that we are looking for.
That isn’t mathematically sound.
Probably the correlation looks okay, because the charts go up and down in the right places, but the precision of the correlation can’t possibly be sufficient to reliably extract a climate trend signal from the merged data.
KevinUK (02:46:24) : Read your site. That is actually good work, though I’m not sure you conclusions are right.
You seem to see exactly about what you’d expect. “Corrections” lead to as much warming as cooling.
I don’t think for example, you can claim there is no reason for the GHCN to correct the data in the manner they have. There are legitimate issues with stations where you’d expect there to be artificial induced warming and cooling, and if those cases can be identified, then a correction should be made.
You could make the argument over long periods of times, they should cancel out (which is why I thought it would be interesting to just look at absolute averages), which is what your results seem to suggest.
Janis B (05:51:10) :]]
“Actually, would it be possible and not too difficult to group and at least average the trends (or the resulting anomalies) somehow – without weighting them and not within grid but, say, by some of the countries”
I’m working on this right now. I’ll be following up with a further serie sof threads on digginthclay over the next few days. You’ll be very surprised by the number of WMO stations that drop out of the NOAA GHCN analysis in the last two decades or so.
If you arent aware of them I’ll be reproducing the analyses of Giorgio Gillestri and RomanM very shortly. I had hoped GG would be prepared to extend his interesting analysis but it seems he comes from the Eric Steig school of the scientific method where you give up on your analysis once you’ve reached your pre-conceived conclusion. Pity, but I’m happy to pick up where he left off and do a somewhat more detailed analysis which will include diaggregating his analysis into NH/SH and high, mid and low latitudes in each hemisphere.
KevinUK
Willis Eschenbach (22:29:16), you do realize it is Dec. 23rd?
Wait until the 3rd week in Jan., and then send a simple e-mail saying you can’t figure out how the corrected the data with respect to whatever station, and any input/information they can give you beyond the very general details in published reports would be greatly appreciated. Information related specifically to whatever information would be appreciated.
“Information related specifically to whatever information would be appreciated.”
should read:
Information related specifically to whatever station would be appreciated.
temp (07:28:55)
Thank you for taking the time to look at my NOAA GHCN analysis on digginintheclay. You seem to have not quite got the message I’ve tried to convey.
NOAA has made adjustments to at least 400 WMO stations that are just not physically justified. Have a look at Edson and at Mayo Airport again. Why should such pretty much reasonable, non-problematic raw data be adjusted into oblivion? I’ve analysed several other WMO stations on both lists now and they all get the staircase ‘step function’ treatment. It doesnt matter a jot to me whether or not in the end these all cancel out. According to Giorgio Gilestris anlys they dont. The fact is they are just not physically justifiable and thats what matters to me.
Even if they do by chance cancel out on a global basis, they won’t necessarily cancel out on a regional basis as Willis’s original Darwin thread demonstrates. It ony takes a small number of these in a given region e.g. Northern Australia, New Zealand, Siberia etc for these non-physically justifiable adjustments to make a large difference to claimed warming trends in certain regions of the world.
My next analysis is going to look into this very point. Given Edson, Alberta and Mayo, Yukon it looks like Canada might be a good place to start.
KevinUK
temp (08:41:11) :
What? It’s December 23? My gosh, how could I have missed that? Now the GHCN folks won’t be in the office for a week or so, and my email will get all stale and faded from sitting in their inbox for a week, and they’ll refuse to answer it because it’s so old …
This is great. You object to me not writing GHCN. Now I write, and noooo, that’s not good enough for you. You bitch that I haven’t done it at the right time, or included the magic words, or said “mother may I”, or something.
Look, temp, if you don’t think my timing or my exact phraseology will achieve the desired result, I strongly suggest that you write GHCN yourself.
Here’s what I’d do if I were you, temp. I’d wait until the 3rd week in Jan., and then send a simple e-mail saying you can’t figure out how they corrected the data with respect to whatever station, and any input/information they can give you beyond the very general details in published reports would be greatly appreciated.
I advise that method because I understand that the “be vague” approach works much better than telling them exactly what information you are requesting.
Make sure you tell them, however, that “Information related specifically to whatever information would be appreciated”, because a friend of mine says you’ll get better results that way.
Bear in mind that my email will still be in their system at that time, albeit somewhat stale and faded, and the race is on … we’ll see which one (if either) of us gets any joy from GHCN first.
I love all the good folks who are too tired or bored or busy or uninterested to get up off of their dead keesters and actually do something, but are all too happy to sit on the sidelines and give me the benefit of their vast experience by telling me that I’m doing it oh so wrong …
temp (07:07:13) :
On the surface, that sounds like a good idea, but how is the satellite data converted to actual temperature? Isn’t it calibrated against the surface record?
Willis Eschenbach – a question for you sir. Questions – Have the Met Office commented on your analysis? Have they released their workings and reasons? Have they said anything at all?
Ignore that – didnt read the posts. Lets see if GHCN reply and if so what they say
KevinUK (02:46:24) :
Nick Stokes (20:34:00) :
“KevinUK (13:23:59) :
In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.
And how many instances of turning warming into cooling”
Nick why don’t you visit my thread on ”diggingintheclay‘ and you’ll find out?
That’s exactly what’s wrong with this “analysis”. A process subject to apparently random fluctuation, which can be beaten up into a conspiracy with selection of results.“194 instances of WMO stations where “cooling” has been turned into “warming””! And if you want to find that 216 went the other way, as you’d expect with random movement, has to be discovered from the website.
Willis did the same with his “smoking gun”. “a regular stepped pyramid climbing to heaven” . This echoes far and wide. The observation that it is an extreme case from a large sample, and pyramids are almost as likely to go to hell, comes much later, but which time Willis is saying that all he was complaining about was some apparent difficulty with squaring with the Peterson algorithm in Darwin. 1920.
Nick Stokes (13:14:00) :
Say what? I reported what I found. What did you expect me to do? I thought then, and I think now, that the Darwin adjustment was not done by the claimed method. I think that for a number of reasons listed above, which I guess you didn’t read, including the one you mentioned – because I can’t find any stations that are suitable for the adjustment by the claimed method. That’s not “some apparent difficulty”, it is a complete inability to replicate their results.
Nor has anyone else been able to find such stations, including yourself. When you do, or when you can explain how the Darwin adjustment was done using their claimed method, you will have something other than your fantasies to be complaining about.
I don’t care if the pyramids go up or down. I do care if they are bogus. I do care if the listed procedures weren’t used, no matter which way the pyramids go.
And “almost as likely” is meaningless when the claimed adjustments add up to a significant part of the claimed global warming. Nor does “almost as likely” mean a damn thing when the results are used to make regional claims.
I got into this from investigating Professor Karlen’s claims about the region used by the IPCC to represent “Northern Australia”. The unadjusted trend for this large region is 0.06C per century, in other words, no trend at all. The adjusted trend for this region is 0.57C per century, which coincidentally agrees with the global trend … convenient, huh?
In other words, the entire trend for this area comes from the GHCN adjustment. Perhaps you find that immaterial, since the adjustments are “almost as likely” to go down as up.
I don’t.
Finally, you excuse it as “an extreme case from a large sample”. However, unless you are claiming that Darwin Zero really, truly did warm at 6C per century, or that it was adjusted to a local trend of 6C per century, I fear you are left with showing just why it is so extreme. The fact that it is so extreme is not an explanation, it is something that needs explanation.
Willis Eschenbach (15:05:03) :
Let me remind you again of your “smoking gun”:
They’ve just added a huge artificial totally imaginary trend to the last half of the raw data! Now it looks like the IPCC diagram in Figure 1, all right … but a six degree per century trend? And in the shape of a regular stepped pyramid climbing to heaven? What’s up with that?
Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.
Doesn’t sound like a complaint about how to apply an algorithm. It sounds like you’re saying that someone’s deliberately creating a stairway to heaven. And I’m sure that’s what, say, Megan McArdle was conveying when she ran your plot under the heading “Was data faked?”.
If you apply any break recognition adjustment algorithm to 6000+ instances, you’ll get some false positives and other odd effects. And if you look hard enough, you’ll find some where their effect accumulates. So, I don’t know if Darwin really warmed at 0.6 C/decade over that late 20C. Nor whether Coonabarabran really cooled at such a great rate. I think it is more likely that the algorithm overreacted to something in the data. That doesn’t make it a useless algorithm – just one to be interpreted with care, at some suitable level of aggregation. Which is pretty much what Peterson’s paper says.
[REPLY – Peterson, Parker peeked a poke of packed parking lots. ~ Evan]
Nick Stokes (13:14:00) :
“KevinUK (13:23:59) : In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data.”
And how many instances of turning warming into cooling?
That’s exactly what’s wrong with this “analysis”. A process subject to apparently random fluctuation, which can be beaten up into a conspiracy with selection of results.“194 instances of WMO stations where “cooling” has been turned into “warming””! And if you want to find that 216 went the other way, as you’d expect with random movement, has to be discovered from the website.
Nick Stokes you are not being honest. Maybe this is not deliberate. Maybe you are not being honest with yourself.
Just like justice not only should be done, but it must seem to have been done, so also it is with science. It demands openness and transparency.
That there are 194 instances of WMO stations where “cooling” has been turned into “warming”, is a finding of fact. That this is purely random in nature and thus will cancell out is an assumption.
When a huge amount of stations are dropped over 20 years, coinciding with increase in trends, this is a finding of fact. That this has nothing to do with the increase in trends is conjecture.
That Global Warming has proceeded in step with the march of thermometers southwards, is an observation, (fact). That because the temperatures are anomalies, and thus an anomaly at 20N for example will be identical to an anomaly at 60N is an assumption. At best it is sloppy science.
Whether the assumptions are true or not, we just dont know. Even before, because data, methods and programs were refused to be divulged, there was suspicion that things might not be ok.
After the climate-gate emails, we just cant take the scientists at their word. And that is not the way science is done anyway.
When people like Willis Eschenbach and others take an enormous amount of effort to look into the matter, their efforts should be applauded, not attacked in the way you have. If discrepancies are found, they should either be explained or corrected.
Are you not interested in checking, auditing, quality control and the truth being out?
Because if you are not I will be far less polite with you. I have no patience with crooks. I would like to deal with them in good old kiwi fashion where I can. And you seem to be siding with them for no good reason.
Nick Stokes (16:03:01) :
Are you just pretending not to get it, or do you really not get it? I don’t think the algorithm was used on Darwin. I might be wrong, but that’s what I think. So no, it was not a complaint about “how to apply an algorithm”, that’s your fantasy. It was a claim that the cited algorithm wasn’t applied.
You say “I don’t know if Darwin really warmed at 0.6C/decade over that late 20th century”. Hey, I don’t know if Darwin really cooled by 3.7C per day, or if there are really fairies in the back garden … but since we have absolutely no evidence for either of those temperature claims, and we do have evidence that neither temperature claim is true, you’re wandering around in “what if” rather than sticking to the science.
You say the algorithm “overreacted” … we’re paying rooms full of scientists to design an algorithm, on the basis of which we are told we must spend billions of dollars to avoid imaginary catastrophe, and your claim is that it “overreacted”? So would you agree that the claim we should spend billions based on this algorithm might possibly be an “overreaction” as well?
However, I note that you have given up your claim that none of this matters because the algorithm is “almost as likely” to overreact on the plus side as on the minus side, which I guess is some progress …
Finally, whether the data was faked by someone manipulating the Darwin data individually or by someone manipulating all of the data en bloc, the Darwin results are indeed fake – Darwin did not do what the GHCN says it did.
“the UK Met Office (which works closely with the CRU and relies heavily on its product) announced a three-year project to re-examine 160 years of temperature data, signalling its own lack of confidence in its CRU-based temperature record.”
Nick Stokes (16:03:01) :
“”So, I don’t know if Darwin really warmed at 0.6 C/decade over that late 20C. Nor whether Coonabarabran really cooled at such a great rate. I think it is more likely that the algorithm overreacted to something in the data.””
Nick Stokes, Willis’ efforts have convinced me that the algorithm wasn’t used for Darwin.
If you really believe that the algorithm was used at Coonabarabran, then we have a problem. We have a problem because that would imply that the cooling adjustment at Coonabarabran was appropriate while many of us believe that the warming adjustment at Darwin is not.
If you believe that the algorithm wasN’T used at Coonabarabran, then we still have a problem. We have a problem because that would imply that Darwin isn’t the only case of the algorithm isn’t being applied – this really raises serious questions about possible false claims regarding non-US temperature adjustments.
wobble (18:30:48) :
As I said, I believe that all the non-US data was computed with the same algorithm. All Willis has persuaded me of is that there may have been some change to the algorithm since 1997. And yes, if so there is a documentation fault.
Willis Eschenbach (16:27:12) :
Willis, the Petersen paper included with the data made this perfectly clear:
And no, I do think it’s important that the bias of the effect is small. That’s what P is saying – occasionally enormous effects locally, little effect on global trends (which is what really affects the fate of the billions, not Darwin 1920). He lays out what the difficulties are and when to use it.
Incidentally, I doubt that Peterson had roomfuls of scientists. The whole staff of CRU could fit into a smallish room, and I doubt that GHCN is much better resourced.
Nick Stokes (19:55:03) :
Yes. He says to use it regionally. Which is why I pointed out that it totally changes the regional picture for Northern Australia. Why should an algorithm that claims to merely make the outliers look more like the average radically change the average? To believe that, we’d have to assume that most of the erroneous data erroneously shows cooling. Perhaps you could comment about the change that GHCN makes to the Northern Australia record?
CRU is part of a university. GHCN is a project of the NCDC, which is part of the US Government. NCDC have a staff of 133 including support staff. CRU has a staff of 25 including support staff. Please do your homework before making this kind of claim. In addition to making yourself look foolish, you are making me do your homework.
Willis,
You showed pictures from the AR4 indicating an upward gradient in Northern Australia, which you apparently blame on homogenization. But you showed no evidence that they used GHCN homogenised data. The graphs indicate that their data source was CRU. Now, as you say, CRU use data equivalent to GHCN raw data. But there’s no evidence that they used GHCN homogenization. In fact, the posted CRU data shows no great rise.
I have used this graph and two others at “Digging in the clay” to look at the behaviour of the algorithm. I can say that the algorithm has several bugs due to sloppy programming, and these bugs are the footprint of a automatic adjustment – the adjustment was not done by hand. The bugs are as follows:-
1] Incorrectly detecting data for adjustment that does not require adjustment, i.e the algorithm detects discontinuities where it is not clear they exist. (e.g. Darwin 1980)
2] Correctly detected adjustments which are then made in the wrong direction, e.g. the algorithm should shift the data down by amount “x” but then shifts it up by the same amount. (e.g. Darwin 1930 – 1940 and Darwin 1920 – 1930)
3] Adjustments of unreliable data or data that is too short in duration to be used reliably – which results in rather bizarre adjustments. (Edson, Alberta)
4] Adjustments of single years which are then not applied to all following years (e.g. Darwin 1890 and Darwin 1905 – suggests the algorithm just didn’t like those two readings for some reason!)
5] Failure to detect some obvious (and known) discontinuities that actually require adjustment (e.g. about 1940 in Darwin)
6] Applying adjustments up to two years too early or up to two years too late based on incorrect detection of the date at which the discontinuity occured. (e.g. Darwin 1963 – two data points just before the adjusted data at that point seem to have been “left behind” )
7] Errors in the magnitude of the correction applied. (e.g. Darwin 1963 -1980 looks like 0.5Celsius shift was needed but 0.7Celsius shift applied)
[8] The algorithm does not shift the adjusted data such that the most recent years are shown as coincident in temperature. (e.g. Darwin 1995 adjustment tells us that the temperature measured by the modern electronic Stevenson screen is 2.5Celsius out and the measurement made in 1880 using a thermometer bolted on the side of the post office was much more accurate)
The algorithm is a complete mess as exposed by these graphs. There is every reason to believe that the bugs in the algorithm have made the same erroneous adjustments with all the data. There is no reason to believe that some of the bugs are not biased towards a particular temperature trend outcome. The adjusted data is not reliable over small regions and therefore could not be used to correlate to tree-ring data or satellite data.
I should perhaps add two further conclusions from the study of the three graphs
A] The adjustments are indeed unjustified and result in data that is actually far less reliable than the underlying data they were supposed to correct.
B] The adjustments are roughly equally likely to cause donward trends in temperature, so there is no real evidence of deliberate malign intent in the adjustments made. It is simply seriously bad programming, although some of the bugs may be biased towards a particular direction of adjustment, perhaps as a result of confirmational bias on the part of the programmer.
Nick Stokes (23:24:59) :
So your answer can be named another “Hide the decline?”
How about a straightforward answer, where there will be no criticism of a “don’t know”. Even if Giss have altered their public presentation of data imcluding GHCN corrections recently, can you be sure that they do not continue to use adjusted data in compiling global patterns?
Second question. Do you have daily or monthly data for the Darwin airport versus Post Office comparison that went from Feb 1941 to January 1942?
Nick Stokes (19:55:03) :
“”All Willis has persuaded me of is that there may have been some change to the algorithm since 1997. And yes, if so there is a documentation fault.””
A documentation fault? Are you suggesting that the Peterson paper isn’t the proper algorithm to use for recreation? Everyone on here was telling Willis to dig into Peterson. Were they wrong?
“”As I said, I believe that all the non-US data was computed with the same algorithm. “”
Nick Stokes, I think Willis has done an adequate job attempting to recreate the Darwin adjustment. If you think it’s possible to recreate the objective use of an algorithm then please show us your work. (And don’t forget to tell us which algorithm you use.)
Ryan Stephenson (02:12:36):
“”I can say that the algorithm has several bugs due to sloppy programming, and these bugs are the footprint of a automatic adjustment – the adjustment was not done by hand.””
If the programming being used isn’t properly employing the algorithm, then the algorithm isn’t being used.
So it would be accurate to state, “The algorithm wasn’t used objectively.”
Geoff Sherrington (03:15:31) :
There’s no reason to believe GISS ever used GHCN adjustments, and I think it is very unlikely. They have their own method, which actually makes very little adjustment to Darwin.
I don’t have that Darwin data.
Well, dang-a-lang. To my immense surprise, I got a before-Christmas present, an email from Tom Peterson of GHCN, one of the authors of the GHCN homogenization method.. I’m usually loath to post emails, but this one was so genteel and was all business so I hope that Dr. Peterson will forgive me for posting it here.
From the sound of it, they were not happy with the algorithm which was used to homogenize the temperature including Darwin, and are replacing it with a new algorithm due out in the spring.
So, I didn’t get a single one of my questions answered, but in a separate email Dr. Peterson said that he would send me the old code when he returns from the holidays. He also sent me a number of papers on the subject, all but one of which I already had, but I greatly appreciate his thoroughness.
I must warmly commend Dr. Peterson for his openness and his willingness to send the (now obsolete) code. He also says that they will be releasing the new code concurrent with the release of the new GHCN series. When the old code arrives I’ll let y’all know, and maybe the questions will be answered at last.
So while we still have not definitively established whether Darwin was adjusted manually, or whether it was just wildly mis-adjusted by a bad algorithm and overlooked by shoddy quality control, it looks possible that we may get an answer when the code arrives.
Given Dr. Peterson’s very quick response, I’d say that Climategate has had a very salubrious effect on the climate science community. I say this because of Peterson’s totally different response given in the CRU emails. In response to Phil Jones saying
Peterson replied
As a result of having read that response, I didn’t bother to write. Live and learn, it appears that both Dr. Peterson and I have learned something from all of this.
I’m very interested to see what the new algorithm makes of Darwin. Heck, it might even make the obvious 1940 correction that the previous correction didn’t make.
Again, I want to extol the actions of Dr. Peterson. They are in the finest tradition of scientific transparency, a tradition which has been sadly lacking in the climate science community for many years. This is a very welcome development in the field. In particular I call attention to his last paragraph.
Best Christmas wishes to everyone,
w.
REPLY: That’s some Christmas gift, nice to see such an inquiry handled professionally. – Anthony
Willis,
A very interesting response. Could you please post a list of the papers that he sent?
Thanks.
Wow. This is fascinating. Dr. Peterson is definitely handling this correctly. Who knows, if rest of the climate warming community starts to handle issues in this manner, then they may begin converting quite a few skeptics.
I’ll be excited to see the new results.
Nick Stokes (22:19:48) :
Glad to. They are:
First Difference Method: Maximizing station density for the calculation of long-term global temperature chance, Peterson et al., JGR 1998
Creation of homogeneous composite climatological reference series, Peterson & Easterling, IJoC, 1993
A new method for detecting undocumented discontinuities in climatological time series, Easterling & Peterson, IJoC, 1994
Homogeneity adjustments of in situ atmospheric climate data: a review, Peterson et al., IJoC, 1998
Historical African Data, Peterson et al., BAMS, 1997
Best Xmas wishes,
w.
Thanks, Willis,
and a Merry Xmas to you