by Willis Eschenbach
People keep saying “Yes, the Climategate scientists behaved badly. But that doesn’t mean the data is bad. That doesn’t mean the earth is not warming.”
Darwin Airport – by Dominic Perrin via Panoramio
Let me start with the second objection first. The earth has generally been warming since the Little Ice Age, around 1650. There is general agreement that the earth has warmed since then. See e.g. Akasofu . Climategate doesn’t affect that.
The second question, the integrity of the data, is different. People say “Yes, they destroyed emails, and hid from Freedom of information Acts, and messed with proxies, and fought to keep other scientists’ papers out of the journals … but that doesn’t affect the data, the data is still good.” Which sounds reasonable.
There are three main global temperature datasets. One is at the CRU, Climate Research Unit of the University of East Anglia, where we’ve been trying to get access to the raw numbers. One is at NOAA/GHCN, the Global Historical Climate Network. The final one is at NASA/GISS, the Goddard Institute for Space Studies. The three groups take raw data, and they “homogenize” it to remove things like when a station was moved to a warmer location and there’s a 2C jump in the temperature. The three global temperature records are usually called CRU, GISS, and GHCN. Both GISS and CRU, however, get almost all of their raw data from GHCN. All three produce very similar global historical temperature records from the raw data.
So I’m still on my multi-year quest to understand the climate data. You never know where this data chase will lead. This time, it has ended me up in Australia. I got to thinking about Professor Wibjorn Karlen’s statement about Australia that I quoted here:
Another example is Australia. NASA [GHCN] only presents 3 stations covering the period 1897-1992. What kind of data is the IPCC Australia diagram based on?
If any trend it is a slight cooling. However, if a shorter period (1949-2005) is used, the temperature has increased substantially. The Australians have many stations and have published more detailed maps of changes and trends.
The folks at CRU told Wibjorn that he was just plain wrong. Here’s what they said is right, the record that Wibjorn was talking about, Fig. 9.12 in the UN IPCC Fourth Assessment Report, showing Northern Australia:
Figure 1. Temperature trends and model results in Northern Australia. Black line is observations (From Fig. 9.12 from the UN IPCC Fourth Annual Report). Covers the area from 110E to 155E, and from 30S to 11S. Based on the CRU land temperature.) Data from the CRU.
One of the things that was revealed in the released CRU emails is that the CRU basically uses the Global Historical Climate Network (GHCN) dataset for its raw data. So I looked at the GHCN dataset. There, I find three stations in North Australia as Wibjorn had said, and nine stations in all of Australia, that cover the period 1900-2000. Here is the average of the GHCN unadjusted data for those three Northern stations, from AIS:
Figure 2. GHCN Raw Data, All 100-yr stations in IPCC area above.
So once again Wibjorn is correct, this looks nothing like the corresponding IPCC temperature record for Australia. But it’s too soon to tell. Professor Karlen is only showing 3 stations. Three is not a lot of stations, but that’s all of the century-long Australian records we have in the IPCC specified region. OK, we’ve seen the longest stations record, so lets throw more records into the mix. Here’s every station in the UN IPCC specified region which contains temperature records that extend up to the year 2000 no matter when they started, which is 30 stations.
Figure 3. GHCN Raw Data, All stations extending to 2000 in IPCC area above.
Still no similarity with IPCC. So I looked at every station in the area. That’s 222 stations. Here’s that result:
Figure 4. GHCN Raw Data, All stations extending to 2000 in IPCC area above.
So you can see why Wibjorn was concerned. This looks nothing like the UN IPCC data, which came from the CRU, which was based on the GHCN data. Why the difference?
The answer is, these graphs all use the raw GHCN data. But the IPCC uses the “adjusted” data. GHCN adjusts the data to remove what it calls “inhomogeneities”. So on a whim I thought I’d take a look at the first station on the list, Darwin Airport, so I could see what an inhomogeneity might look like when it was at home. And I could find out how large the GHCN adjustment for Darwin inhomogeneities was.
First, what is an “inhomogeneity”? I can do no better than quote from GHCN:
Most long-term climate stations have undergone changes that make a time series of their observations inhomogeneous. There are many causes for the discontinuities, including changes in instruments, shelters, the environment around the shelter, the location of the station, the time of observation, and the method used to calculate mean temperature. Often several of these occur at the same time, as is often the case with the introduction of automatic weather stations that is occurring in many parts of the world. Before one can reliably use such climate data for analysis of longterm climate change, adjustments are needed to compensate for the nonclimatic discontinuities.
That makes sense. The raw data will have jumps from station moves and the like. We don’t want to think it’s warming just because the thermometer was moved to a warmer location. Unpleasant as it may seem, we have to adjust for those as best we can.
I always like to start with the rawest data, so I can understand the adjustments. At Darwin there are five separate individual station records that are combined to make up the final Darwin record. These are the individual records of stations in the area, which are numbered from zero to four:
Figure 5. Five individual temperature records for Darwin, plus station count (green line). This raw data is downloaded from GISS, but GISS use the GHCN raw data as the starting point for their analysis.
Darwin does have a few advantages over other stations with multiple records. There is a continuous record from 1941 to the present (Station 1). There is also a continuous record covering a century. finally, the stations are in very close agreement over the entire period of the record. In fact, where there are multiple stations in operation they are so close that you can’t see the records behind Station Zero.
This is an ideal station, because it also illustrates many of the problems with the raw temperature station data.
- There is no one record that covers the whole period.
- The shortest record is only nine years long.
- There are gaps of a month and more in almost all of the records.
- It looks like there are problems with the data at around 1941.
- Most of the datasets are missing months.
- For most of the period there are few nearby stations.
- There is no one year covered by all five records.
- The temperature dropped over a six year period, from a high in 1936 to a low in 1941. The station did move in 1941 … but what happened in the previous six years?
In resolving station records, it’s a judgment call. First off, you have to decide if what you are looking at needs any changes at all. In Darwin’s case, it’s a close call. The record seems to be screwed up around 1941, but not in the year of the move.
Also, although the 1941 temperature shift seems large, I see a similar sized shift from 1992 to 1999. Looking at the whole picture, I think I’d vote to leave it as it is, that’s always the best option when you don’t have other evidence. First do no harm.
However, there’s a case to be made for adjusting it, particularly given the 1941 station move. If I decided to adjust Darwin, I’d do it like this:
Figure 6 A possible adjustment for Darwin. Black line shows the total amount of the adjustment, on the right scale, and shows the timing of the change.
I shifted the pre-1941 data down by about 0.6C. We end up with little change end to end in my “adjusted” data (shown in red), it’s neither warming nor cooling. However, it reduces the apparent cooling in the raw data. Post-1941, where the other records overlap, they are very close, so I wouldn’t adjust them in any way. Why should we adjust those, they all show exactly the same thing.
OK, so that’s how I’d homogenize the data if I had to, but I vote against adjusting it at all. It only changes one station record (Darwin Zero), and the rest are left untouched.
Then I went to look at what happens when the GHCN removes the “in-homogeneities” to “adjust” the data. Of the five raw datasets, the GHCN discards two, likely because they are short and duplicate existing longer records. The three remaining records are first “homogenized” and then averaged to give the “GHCN Adjusted” temperature record for Darwin.
To my great surprise, here’s what I found. To explain the full effect, I am showing this with both datasets starting at the same point (rather than ending at the same point as they are often shown).
Figure 7. GHCN homogeneity adjustments to Darwin Airport combined record
YIKES! Before getting homogenized, temperatures in Darwin were falling at 0.7 Celcius per century … but after the homogenization, they were warming at 1.2 Celcius per century. And the adjustment that they made was over two degrees per century … when those guys “adjust”, they don’t mess around. And the adjustment is an odd shape, with the adjustment first going stepwise, then climbing roughly to stop at 2.4C.
Of course, that led me to look at exactly how the GHCN “adjusts” the temperature data. Here’s what they say
GHCN temperature data include two different datasets: the original data and a homogeneity- adjusted dataset. All homogeneity testing was done on annual time series. The homogeneity- adjustment technique used two steps.
The first step was creating a homogeneous reference series for each station (Peterson and Easterling 1994). Building a completely homogeneous reference series using data with unknown inhomogeneities may be impossible, but we used several techniques to minimize any potential inhomogeneities in the reference series.
…
In creating each year’s first difference reference series, we used the five most highly correlated neighboring stations that had enough data to accurately model the candidate station.
…
The final technique we used to minimize inhomogeneities in the reference series used the mean of the central three values (of the five neighboring station values) to create the first difference reference series.
Fair enough, that all sounds good. They pick five neighboring stations, and average them. Then they compare the average to the station in question. If it looks wonky compared to the average of the reference five, they check any historical records for changes, and if necessary, they homogenize the poor data mercilessly. I have some problems with what they do to homogenize it, but that’s how they identify the inhomogeneous stations.
OK … but given the scarcity of stations in Australia, I wondered how they would find five “neighboring stations” in 1941 …
So I looked it up. The nearest station that covers the year 1941 is 500 km away from Darwin. Not only is it 500 km away, it is the only station within 750 km of Darwin that covers the 1941 time period. (It’s also a pub, Daly Waters Pub to be exact, but hey, it’s Australia, good on ya.) So there simply aren’t five stations to make a “reference series” out of to check the 1936-1941 drop at Darwin.
Intrigued by the curious shape of the average of the homogenized Darwin records, I then went to see how they had homogenized each of the individual station records. What made up that strange average shown in Fig. 7? I started at zero with the earliest record. Here is Station Zero at Darwin, showing the raw and the homogenized versions.
Figure 8 Darwin Zero Homogeneity Adjustments. Black line shows amount and timing of adjustments.
Yikes again, double yikes! What on earth justifies that adjustment? How can they do that? We have five different records covering Darwin from 1941 on. They all agree almost exactly. Why adjust them at all? They’ve just added a huge artificial totally imaginary trend to the last half of the raw data! Now it looks like the IPCC diagram in Figure 1, all right … but a six degree per century trend? And in the shape of a regular stepped pyramid climbing to heaven? What’s up with that?
Those, dear friends, are the clumsy fingerprints of someone messing with the data Egyptian style … they are indisputable evidence that the “homogenized” data has been changed to fit someone’s preconceptions about whether the earth is warming.
One thing is clear from this. People who say that “Climategate was only about scientists behaving badly, but the data is OK” are wrong. At least one part of the data is bad, too. The Smoking Gun for that statement is at Darwin Zero.
So once again, I’m left with an unsolved mystery. How and why did the GHCN “adjust” Darwin’s historical temperature to show radical warming? Why did they adjust it stepwise? Do Phil Jones and the CRU folks use the “adjusted” or the raw GHCN dataset? My guess is the adjusted one since it shows warming, but of course we still don’t know … because despite all of this, the CRU still hasn’t released the list of data that they actually use, just the station list.
Another odd fact, the GHCN adjusted Station 1 to match Darwin Zero’s strange adjustment, but they left Station 2 (which covers much of the same period, and as per Fig. 5 is in excellent agreement with Station Zero and Station 1) totally untouched. They only homogenized two of the three. Then they averaged them.
That way, you get an average that looks kinda real, I guess, it “hides the decline”.
Oh, and for what it’s worth, care to know the way that GISS deals with this problem? Well, they only use the Darwin data after 1963, a fine way of neatly avoiding the question … and also a fine way to throw away all of the inconveniently colder data prior to 1941. It’s likely a better choice than the GHCN monstrosity, but it’s a hard one to justify.
Now, I want to be clear here. The blatantly bogus GHCN adjustment for this one station does NOT mean that the earth is not warming. It also does NOT mean that the three records (CRU, GISS, and GHCN) are generally wrong either. This may be an isolated incident, we don’t know. But every time the data gets revised and homogenized, the trends keep increasing. Now GISS does their own adjustments. However, as they keep telling us, they get the same answer as GHCN gets … which makes their numbers suspicious as well.
And CRU? Who knows what they use? We’re still waiting on that one, no data yet …
What this does show is that there is at least one temperature station where the trend has been artificially increased to give a false warming where the raw data shows cooling. In addition, the average raw data for Northern Australia is quite different from the adjusted, so there must be a number of … mmm … let me say “interesting” adjustments in Northern Australia other than just Darwin.
And with the Latin saying “Falsus in unum, falsus in omis” (false in one, false in all) as our guide, until all of the station “adjustments” are examined, adjustments of CRU, GHCN, and GISS alike, we can’t trust anyone using homogenized numbers.
Regards to all, keep fighting the good fight,
w.
FURTHER READING:
My previous post on this subject.
The late and much missed John Daly, irrepressible as always.
More on Darwin history, it wasn’t Stevenson Screens.
NOTE: Figures 7 and 8 updated to fix a typo in the titles. 8:30PM PST 12/8 – Anthony
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
carrot eater (06:56:49):
You certainly have ample time to post here pretty much non-stop, and on other sites too. So I have a proposal for you.
With plenty of time to criticize Willis Eschenbach for his unpaid, amateur scientist’s efforts [keeping in mind that Willis follows in the footsteps of Pascal, Einstein Pasteur, Semmelweiss and numerous other unpaid amateur scientists], why don’t you use the time you spend endlessly trying to find fault with Willis, and write your own article instead? I think Anthony would be happy to post it.
See, it’s easy to take constant potshots from the sidelines at someone like Willis, who wrote about his findings, posted them, and answers questions and criticisms in a straightforward manner [unlike those pushing the current AGW malarky that pretends to follow the scientific method].
When Willis makes a mistake, he corrects it. Everyone makes mistakes. But when the devious CRU, Michael Mann and the IPCC make major errors that negate their conclusions, and intentionally fabricate data, unlike honest amateur scientists they go running for cover and never correct their errors or answer critics. Because, of course, their errors were intentional.
So instead of endlessly trying to find fault with Mr Eschenbach, why don’t you write your own article for WUWT, and see what it’s like for your belief system to be deconstructed?
You certainly have plenty of time to comment here and on other blogs [where you label scientific skeptics “deniers”], so your protestations of not having the time are questionable at best:
“When I have time, I’ll have to look at your findings for the early times. Or rather, just do it myself from scratch. I won’t have time for such a thing anytime soon, though.”
I look forward to poking holes in any AGW conjecture you can come up with. If you’ve got the cojones to write your own article.
Steve Short:
In terms of what difference the GHCN homogenisation makes (keeping in mind that GISS does its own thing altogether): why don’t we both, for ourselves, make plots of a simple average of all data, as raw and adjusted anomaly vs time? It is less good than doing a proper spatial average, but using v2.mean and v2.mean_adj, it should be doable with a half day’s effort. If you want a starting point, the page at gg has some code from gg, Nick and Steven van Heuven, in three different languages; the first two do something a bit different, but at least have the file input in there.
Smokey: There is nothing in particular that I’d like to write an article on. It just bothers me when somebody makes an accusation of fraud based on so little. If you do that, you’re sticking your neck out, and you shouldn’t be surprised if people are critical.
carrot eater (07:01:59) :
carrot eater, you’re talking to the wrong guy. The GHCN took them to be independent measurements. The GHCN adjusted each of them separately, and adjusted them differently.
I’m just trying to understand what the GHCN have done, and they treat them as independent measurements. So if you have a problem with that, I suggest that you take it up with GHCN, and not with me.
carrot eater (06:36:44) :
Geoff Sherrington (17:50:21) :
Look, I’m used to more precise expression than you are. When I say I did NOT refer to the Reference Climate Stations set, I did not. There’s no point in you rebutting by saying as you did “I’m pretty sure you have. You quoted directly from a file that listed the adjustments that were made for the purpose of that reference set – ‘high quality’, or whatever they call it” I simply did not. I quoted a bit from Torok & Nicholls on post Geoff Sherrington (16:27:47) : on 14.12 . T & N published some corrections for a particular exercise, but my statement did not rest on whther it was the RCS or not. I did not even use the term RCS. I quoted T&N in the sense that I do not know, and I suspect that you do not know, if those correction were used, for how long, on which stations, whether they were later replaced by others. So any putative reference I might have made to the RCS was not to make a point about its properties, but as an example of residual uncertainty. Indeed, we hear little about the RCS these days, as if it was abandoined because it was a wrong idea.
Then you admit that Darwin was atypical and I gave you Broome, with similar suspect patterns. You say that one exception does not prove a point, nor does two. But I could continue on, giving you more and more stations where the earliest data I can find (pre 1993) are essentially flat and horizontal, with the GISS adjusted being a strong warming trend. How many counter-examples does it take to shoot down a proposition? In some cases, merely one. How many Australian stations were removed from GISS calculations since 1993?
You are still horribly confused in what you write about assuming this and assuming that. So am I. Why don’t you start with a clean sheet and answer these questions one by one. You have seen them before, above:
1. Australia’s BOM sends data to USA & British bodies for incorporation in global sets. Which authorities are primary recipients?
2. How do you know that these are the necessary and sufficient bodies for the purposes of this thread? Can you confirm it independently? e.g. does NOAA and GHCN both get data or does it go to WMO?
3. Which geographic set of Australian information is currently sent? Is if the RCS set, or a different one?
4. Are the data aggregated into monthly before sending?
5. Are the data as currenly used by the USA folks available in a file that can be accessed today, or have there been a number of files, some of which might have been updated in Australia – or not updated?
6. What is a site for all Australian data so far digitised, in primordial, raw form?
I do not know where you work so I do not know if you can answer these. Maybe Nick Stokes (ditto).
Re gg’s quasi-symmetrical distribution graph, the key is in the choice of stations. I am starting to show you, one by one, stations that have a strong artificial adjustment. I don’t recall seeing any Australian stations that have a strong negative correction. So I presume that the symmetry arises from not including stations that I can show you that are positively corrected but not in the set making the graph.
It would be good science to keep on going after constructing that graph, picking a number of stations that are not on the list used, to see if the conclusion hold for excluded stations as well as included stations. This is called verification and it is almost a mandatory step. But you can’t do that until you clarify the several questions above, because withpout answers to those questions you are flying in the dark. Like Gavin not knowing that Butterworth is in Malaysia?????
Geoff
1 – Ian Butterworth is the name of the BOM scientist who’s report on Darwin and the adjustments needed due to changes in location, geography or other such factors contained in the meta data I am referencing in that paragraph.
2 – The whole of Darwin Airport is on a peninsula that it is surrounded on 3 sides by the sea, on these 3 sides within the airport boundaries the furthest away point from the sea is around 4km on all 3 sides, and the closest is around 1-2 km on 2 sides and 5-6 km on the 3rd side. Unless Google Maps is lying.
3 – Darwin post office is listed as having closed in 1942 by BOM, and I think it likely they’d have noticed at some point in the 50 years between that data and the date that the Darwin Zero plot ends if someone was still operating a weather station at the Post Office site. Unless you’re suggesting the Darwin Zero data was being complied for 50 years by some rogue scientist not connected with BOM who only submitted their data direct to GHCN, and then managed to get GHCN to accept their data and not reference the source in their comprehensive list of where their data originates? is this your contention?
4 – Darwin Regional Office is listed as only supplying temperature data for 6 years from 1967-73, so it’s unlikely to have been used at all in relation to Darwin Airport’s Homogonisation, and it’s location is therefore irrelevant.
5 – I’d suggest you firstly look up the definition of ‘moderating factor’, and secondly produce any evidence you might have to back up the idea that the ocean doesn’t act as a moderating factor on climate, and then consider emailing it to every meteorologist on the planet because you’ll have just disproved one of the most widely accepted scientific paradigms in history. This is schoolboy geography stuff.
6 – Any pilot who decided to use the GHCN data to tell them the air temperatures when they’re about to land rather than the actual raw data feed from the monitoring station would deserve to be sacked on the spot. At best the GHCN data would be at least a month out of data, and it’s also clearly not designed for the purpose you’re attempting to give it. The GHCN data’s purpose is to allow climate modellers to represent the average temperature and temperature change over time for a region as part of a global dataset that gives a best estimate for global average temperature change over time. That is all it is for, it is not designed to produce an accurate picture of what it actually happening at any one individual location, and it’s definately not designed to be used by pilots.
7 – What am I making up as I go along?
Where I am hypothesising is over whether Darwin Zero was the dataset originally used by either GHCN or possibly the Australians as their reference series for Darwin. It could be something else, but as it’s clearly not an actual station in it’s own right, I’m struggling to think what else it could be, and as I mention there are other indicators that support this hypothesis.
I’m also hypothesising about which of the surrounding stations might have been used to produce a reference dataset, but the point I was trying to make was that there actually were datasets in the surrounding area of over 20 years length that could be used to produce a reference series, as well as that these reference series are likely to have produced the increased warming due to being mostly landlocked.
In my original and second post I hypothesised 2 potential justifiable methods that could have been used to homogonise the darwin data, the first method would only be justifiable for areas where datapoints were much scarcer than this, so probably wouldn’t be used here, the second method is pretty much the method outlined in the various papers, so I’d stick by that as being the most likely reason for the adjustments post 1950ish.
8 – I am certain that Darwin Zero is not a monitoring station, as evidenced by it not being on any BOM list of monitoring stations that have ever existed in Australia.
I am certain that the adjustments to the data pre 1942 are justified by the information contained in the metadata as described in the BOM report by Ian Butterworth. Notably in addition to the move from the Post Office to the Airport in 1942, the metadata describes the Post Office site as having trees shading the Post Office site in the 1930’s, which would be an excellent explanation for the drop in temperatures preceding the move to the airport. Butterworth also examines a years worth of concurrent data from both sites, concluding that the Post Office produces significantly higher readings than the airport site.
I’m also certain that expecting a report that’s described as being ‘An Overview’ to perfectly describe the methodology used to produce the homogonised data for every single climate station in the global records is neither sensible nor realistic.
(hypothesis)
If anything Darwin’s likely to be covered by the phrase in the overview…
‘…and those stations we could reliably adjust to make them homogeneous.’
which is basically slang for using the most accurate method possible for producing the most reliable dataset possible in areas where there’s not enough data to use the homogonisation techniques outlined in the overview… eg adjusting the data based on the metadata for the early period, and possibly using the less exacting Australian method of Homogonising the data for later periods using surrounding stations with at least 10 years records, and data sets with a correlation of at least 0.7 as outlined by Torok and Nicholls in their 1995 paper on the subject, or the (Plummer et al., 1995) method described in the Australian section of the 1998 review paper (both linked below), or some updated version of these.
http://134.178.63.141/amm/docs/1996/torok.pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.122.1131&rep=rep1&type=pdf
Well I am a bit puzzed by the big brown spot in Fig 6 in the 1995 paper.
I believe that most of that is achieved by the homoginisation of
The town was moved in the 1940’s 12 km west and 62 mts uphill.
from Old Halls Creek (which you can see is in a depression) to New Halls Creek
That’s how come the homogenised version ended up like this
Figure 8 in this paper should have set off alarm bells.
http://reg.bom.gov.au/amm/docs/2004/dellamarta.pdf
Yet Jones compares it to Broome?
In fact He compares Kalgoorlie to Perth as well.
Crikey! Anyone that lives out here knows that the climate changes rapidly once you leave the coast. Indeed Mullewa has more in common climate wise with Meekatharra (450km away)then Geraldton (100km away)
Willis,
I’ve just checked the raw list of all 7280 monitoring stations that make up the GHCN network available directly from the GCHN website at the link below, as well as the full list of stations actually used by GISS, and there is only one station at Darwin listed as being used by either within their actual global dataset – Darwin Airport.
As far as I can see the data you’re highlighting in this article is only found in the section marked ‘Raw GHCN data + USHCN corrections’, and is just that, the raw data files used by GHCN to compile their original global temperature record.
So, let;s be clear about this, it’s not GHCN or GISS who’ve cocked up here, it is you who has misunderstood and misrepresented what these records actually represent, and how GHCN and GISS have used them.
There isn’t and never has been an actual station called Darwin Zero, and it’s clear that all 5 of the Darwin records listed in the raw data files represent different versions of the original records and amalgamated records produced during the original GHCN process prior to the intial 1992 release of the GHCN record vs 1.
GHCN make this data available specifically to be open about where the data has come from, and to enable additional checks to be made in future about the assumptions made when the original amalgamated records were created.
I don’t and probably never will know exactly what is represented by the Homoginised graph of Darwin Zero records you highlight in fig 7, but given that it is not an actual station record, and that the non-adjusted data for it is obviously a working version of the amalgamation process, it would be reasonable to assume as I have that the homogonised data represented some version of the working process of developing the initial homogonised data for the final Darwin Airport record. As such, it potentially provides an interesting insite into some of the inner workings of the process, but seeing as nobody yet has come along with any actual factual information about what this data represents, it’s a wee bit early to be calling it a smoking gun, or using it to imply that the scientists involved are guilty of scientific dishonesty.
if there’s any dishonesty going on in this article, it would be from yourself, but I’ll give you the benefit of the doubt and call it an honest mistake if you’re prepared to add in a disclaimer to this article to recognise that Darwin Zero is not and never was an actual station, and that you’ve made an honest mistake with this article.
Appologies, I forgot to add the links to the full list of stations used by GISS and GHCN in their networks, showing only one station record for Darwin being used…
GISS : http://data.giss.nasa.gov/gistemp/station_data/v2.temperature.inv.txt
GHCN : ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.inv
It is not often that I go to this site:
http://data.giss.nasa.gov/gistemp/station_data/
Therefore, I was surprised today to read there
“Note to prior users: We no longer include data adjusted by GHCN and have renamed the middle option (old name: prior to homogeneity adjustment).”
The options are now –
raw GHCN data+USHCN corrections
after combining sources at the same location
after homogeneity adjustment
Is there a recent story behind this? It has taken all the fun from those extreme examples we seem to be discussing. KNMI is also cutting them out, as far as I can tell.
Returning to the central theme, Gavin asks above “7 – What am I making up as I go along?”. Answer – about 90 percent. You have a pathological urge to take part statements out of context and to guess instead of proving. That’s why I put in an occasional trap like Butterworth. To see if you have read the paper. I see it’s now “Ian Butterworth” so maybe you’ve been googling as well. Worthwhile effort, commendable.
The best way that I can see for you to get out of the quicksand is to answer these earlier questions, a bit enlarged now, with references:
1. Australia’s BOM sends data to USA & British bodies for incorporation in global sets. Which authorities are primary recipients?
2. How do you know that these are the necessary and sufficient bodies for the purposes of this thread? Can you confirm it independently? e.g. does NOAA and GHCN both get data or does it go to WMO?
3. Which geographic set of Australian information is currently sent? Is if the RCS set, or a different one?
4. Are the data aggregated into monthly before sending? Is is Tmax and Tmin, or just Tmean, or is it a longer record now that there is almost continuous sampling at many stations”
5. Are the data as currenly used by the USA folks available in a file that can be accessed today, or have there been a number of files, some of which might have been updated in Australia – or not updated?
6. Name a web site or other public source for all Australian data so far digitised, in primordial, raw form?
Gavin Andrews (08:15:31) :
Obviously, you haven’t looked at the data. If what you claim is true, perhaps you can explain why GHCN did not use a single unified Darwin Airport record in making their adjustments? Instead, they adjusted the various records (Darwin Zero, Darwin 1, etc) separately, and applied separate and different adjustments, adjustments made at different times, to each record. If they are just “different versions of the original records”, why would they be adjusted differently? And if they are all the same record, why would they not be combined before being adjusted? And if they are all the same record, why would they not be combined after they were adjusted?
Next, if these records were “produced during the original GHCN process prior to the intial 1992 release of the GHCN record vs 1.”, perhaps you can explain why two of these records extend past 1992?
Next, if these records are all just copies of each other, why are their monthly averages different by up to three tenths of a degree? And why do Darwin Zero, Darwin 1, and Darwin 2 disagree ninety percent of the time?
Next, since you claim the only station involved is “Darwin Airport”, how do you explain the start of Darwin Zero in 1882? Are you claiming there was an airport at Darwin in 1882?
Finally, you say these records appear in “the section marked ‘Raw GHCN data + USHCN corrections’”. The GHCN has no section with that title, so what does that have to do with GHCN?
You close with:
You say I’m being dishonest (simply because you happen to disagree with me) but you’re prepared to give me “the benefit of the doubt” about my honesty if I recant and agree that the sun moves around the earth??
Gavin, that’s just too precious, it appears you really think that the “benefit of your doubt” makes the slightest difference to my honesty … lay off the personal attacks and you’ll get more traction. Concentrate on the science, calling someone a liar is just a cheap debating tactic and people see right through it.
I may well be wrong, Gavin, I’ve been wrong many times before, but I’m an honest man. Your baseless insinuations that I am lying don’t touch my honesty, but they reflect very poorly on your character. Unfortunately, that kind of vile personal attack against the bearer of bad news is becoming all too common from AGW supporters, and it reeks of desperation … understandable desperation, I suppose, but unpleasant nonetheless.
[snip – policy – invalid email address]
Willis,
With vjones’s help and with the aid of EMSmith’s excellent documentation, I’ve been carrying out my own analysis of the NOAA GHCN data. My first step was to reproduce you excellent analysis for Darwin (unlike the Team who think that ‘there’s nothing to see here, move on’). I’ve therefore been applying the scientiic method and have attempted to falsify your analysis. I’m sorry (actually I’m glad) to say that I failed! I’ve reproduced your charts and results almost 100% and have documented my efforts on vjones blog ‘diggingintheclay‘. You can read the thread in which I reproduce your analysis by clicking on the link below.
Reproducing Willis Eschenbach’s WUWT Darwin analysis
As I’m sure you already know and appreciate science progresses by ‘standing on the shoulders of giants’ so I’ve taken the liberty of further extending you excellent analysis for Darwin to all the WMO stations in the NOAA GHCN dataset.
Specifically I’ve attempte dto answer the question posed by others on your original Darwin thread as to whether or not Darwin is a special case or not?
Well judge for yourself by clicking on the link below which documents my extension of your analysis to include the whole NOAA GHCN dataset.
Physically unjustifiable NOAA GHCN adjustments
The following is an excerpt from the thread
“In total, I have found 194 instances of WMO stations where “cooling” has been turned into “warming” by virtue of the adjustments made by NOAA to the raw data. As can be seen from the following “Cooling turned into warming” table (Table 1) below, which lists the Top 30 WMO station on the “cooling to warming” list, Darwin is ranked in only 26th place! The list is sorted by the absolute difference in the magnitude of the raw to adjusted slopes i.e. the list is ranked so that the worst case of “cooling” converted to significant “warming” comes first, followed by the next worse etc.
It’s clear from looking at the list that Darwin is certainly not “just a special case” and that in fact that there are many other cases of WMO stations where (as with Darwin) NOAA have performed physically unjustifiable adjustments to the raw data. As can been seen from Table 1 many of these adjustments result in trend slopes which are greater than the IPCC’s claimed 0.6 deg. C/century warming during the 20th century said by the IPCC to be caused by man’s emissions of CO2 through the burning of fossil fuels.
”
KevinUK
KevinUK, well played and many thanks.
w.
The other possibility of the spiking data could be two or more ‘cooks’ thinking a bit more salt is necessary and unaware that someone else was adding salt too.
You could have more than one ‘true believer’ cooking the data unaware of a compatriot.
As a teacher you know something is wrong when your students can see it without your help.
I’m just your average Joe Layman, having never looked at this climate debate before. So I’m a bit out of my element here, but I have a question.
I stumbled upon this website and read the initial piece above by Eschenbach and through some of the posts here, but not all (I just don’t have enough time or understanding to get it all–so my question may have already been addressed.
The analysis given by Eschenbach appears pretty impressive. But a little more Googling led me to “Are the CRU data ‘suspect’? An objective Assessment” at http://www.realclimate.org. It appears to quash the argument here by plotting random series from World Monthly Surface Station Climatology against adjusted CRU data. The result is no substantial difference in ‘raw’ and ‘adjusted.’
All these different datasets I see mentioned on various climate sites are confusing to me–I’m not real sure what’s really ‘raw’ and what’s not.
But, anyway, what is the response to the RealClimate apparent ‘objective’ refutation of ‘bad’ data?
My thanks to anyone who answers.
Mike
Murf (15:23:20) :
Mike, you’re gonna have to ask someone else about that. I’ve been censored so many times on RealClimate for asking simple scientific questions that I refuse to up their visitor count. There’s a peer-reviewed account of one of these at this link. My experience is that if RC says it is so … it ain’t. In the meantime, if they were on fire I wouldn’t piss on them to put it out. They flat-out lie about their censorship policy, which in my book is despicable.
For a view of what Gavin the Unreliable calls “no substantial difference” take a look at WUWT here. There’s plenty of other examples. If Gavin claims to have selected “random” examples, you can be sure that they aren’t.
However, the most telling question is this. If, as you say RealClimate claims, there is “no substantial difference in ‘raw’ and ‘adjusted’ data” … then why on earth would they spend thousands of man-hours to adjust it?
Get back to us when you have considered that for a while.
The problem is that nobody knows whether the data is bad or not. Nobody is sure what is raw and what is not. The data is in a huge muddle, of which Darwin is only one example among many. What the Aussies use for “raw” data differs from what GHCN uses, which differs from what CRU uses, which differs from what GISS uses. Heck, the Aussies themselves have three versions of “raw” Darwin data. Go figure …
Gavin Schmidt of RealClimate is a computer modeler for the GISS dataset. As such, he is hardly an unbiased commenter. There are a host of problems with the “adjustments” done by GISS. See Chiefio’s site for an introduction to some of the issues. Gavin thinks GISS is quality science … however, they have been very unwilling to reveal what they are actually doing to the data. In addition, they make (IIRC) five separate adjustments to the data, and then ignore what that does to the error estimates. Coincidence? You be the judge.
Thanks for your questions,
w.
Willis,
Thanks for the response. I’ll take a look at the stuff you suggest and study on your points.
I guess the only question I still have at the moment is what exactly is your take on the World Monthly Surface Station Climatology data? Do you have any particular reason to think it’s not good data overall, other than there just generally seems to be a big data muddle. I’m wondering since, apparently, one can download the data to see for oneself what it shows ( I’m a bit puzzled why RealClimate takes a random sample rather than just all the data, but they seem to give some sort of rationale for it).
I guess, though, If I’m getting your point, it is that your examination of the Aussie (and other) particular data series call into question all the climate data. Again, I’m confused about datasets. How is the Aussie data related to the World Monthly Surface Station Climatology data, or is it?
I think I’m a little confused as to your general point: is it that’s there is in fact no reliable raw data to analyze or is it that if you take what raw data there is and don’t adjust it then any warming trend is much reduced or eliminated?
I hope these questions aren’t too dumb. You’ve probably already answered them, but it’s taking me a while to get there, there’s so much stuff here.
Oh yes, I think the rationale the RealClimate guys give for the need for data adjustment is so that individual stations can be properly compared over time. But, they argue, those adjustments over many stations don’t create any bias in the overall trends. That seems like a reasonable argument in principle.
Thanks,
M
Murf (20:05:19) :
I’ve never been able to find enough metadata in the WMSSC archives to come to any kind of conclusion. Might be there, but I haven’t found it.
You can download a host of data. WMSSC data, GISS data, GHCN data, CRU data, plus data from the Aussies and the individual met services. Trouble is, they’re all different.
Can’t help you there. The WMSSC data is listed as:
941200 1882-1993 DARWIN AIRPORT
Unfortunately, the Aussie data does not show any dataset that ends in 1993 … so what is the relationship? Anyone’s guess.
And sadly, this is all too typical. There is no authoritative, agreed upon data anywhere. Every group has its own, and they are all different.
I think I’m a little confused as to your general point: is it that’s there is in fact no reliable raw data to analyze or is it that if you take what raw data there is and don’t adjust it then any warming trend is much reduced or eliminated?
The only dumb questions are the ones you don’t ask, because then you don’t get any smarter …
Nonsense, at least for the GISS/USHCN adjustments. Overall, they add a distinct and quite large warming trend. Might be valid and justified … or it might not. See WUWT here for more details.
But it’s just like RC to wave their hands and say “nothing to see here, folks, move along now” …
w.
Re Murf,
If I can chip in here, I have been sending more Darwin material to Willis than he can handle, because it is so mixed up. One Australian climatologist wrote a while back that maximum and minimum temperatures are commonly taken each day, but that about 100 different ways were used to arrive at an average temperature from them.
So, when you write “Oh yes, I think the rationale the RealClimate guys give for the need for data adjustment is so that individual stations can be properly compared over time,” I have to ask you in return, “compared with what?” AFAICS, a series of temperatures taken way back to the 1860s are likely to be mostly correct. They might have larger error bars than recent equipment allows, but that’s no a reason to infer a bias and try to correct it.
You can only really compare a temperature record with itself and note events like dropping and breaking a thermometer, or a recalibation that gives a slightly different answer. It is BIAS that is the root of the global warming problem, or rather, imaginary bias and its imaginary correction.
The adjustments to the negative that “balance” the adjustments to the positive do not stand close scrutiny. One example offered to me recently involved a cooling correction from about 1900, but the responsible collection agency had already voluntarily deleted the pre-1950 data as unacceptable. The post- 1950 data showed a gentle warming. I supect that a station by station analysis of the “balanced” claim would result in destruction of the hypothesis. I’ve seem too many warming adjustments and seldom a cooling adjustment. Besides, for many stations, nobody seems to know which initial data were used for the performance of adjustments because the meta data are too sparse.
Willis and Geoff,
Thanks again for the replies.
I don’t think I’m sophisticated enough in this to ask any more questions or make any replies myself to your questions to me.
I’ll continue to read over your comments and suggested materials, but here’s my working plan to see if I can just establish one or two reasonable facts on my own:
(1) see if I can actually download the World Monthly Surface Station Climatology ‘raw’ data as the RealClimate guys claim is possible and, if it is,
(2) see if I can plot it myself (I don’t know why I couldn’t, but maybe there’s some special knowledge involved in how to properly use the data–I’ll have to see), and, if I can, take a look at what it shows.
Does that sound reasonable? It seems to me that narrows things down to a more managable size. If that data shows little warming, RC’s foundational case for warming goes out the window, it seems to me.
If it does show serious warming, then the question is the quality of that data (possibly vis-a-vis the other datasets mentioned above).
Since RC claims this WMSSC data is very highly correlated with the adjusted data, I don’t see any reason to mess with the adjusted data at at all. But maybe I’m missing something here, so correct me if I’m off base.
I’ll let you know what happens.
M
Murf,
It’s not so hard to download the data. I can do it, so can you.
One quick caution – it you are going to use an early Excel, it has more than 65,000 rows when unzipped and overflows so it’s best to open in another program. I can get there via Word, have not tried Access, but it would most likely work. But maybe you are more modern.
The NOAA data for Darwin start with WMO station #94120, then go to slightly different numbers about 5 times. Some of the years overlap, some are similar, some are different.
So here is a deeply philosophical question. If it is necessary to make an adjustment to the data supplied from another country, and that string of data has missing values, what is the proper procedure for in filling “adjusted missing values”? To me, there is one answer. Guesswork. There is one right answer. Leave the series separate and treat them as entities without combining them.
Here is an amusing little “difference” graph, where I have taken Darwin’s BOM online monthly data and calculated annual averages. Then, I have taken the efforts of the BOM at other times, plus the efforts of various adjusters, and subtracted their values from BOM on-line. Keep in mind that I used only one of the Giss options from several, merely the first one that came along so no cherry picking. They are all supposed to be identical in an ideal world.
(Note that I have tailed down when the data stop before 2009 or start after 1885)
If you can make sense of this, you are a better man than I.
http://i260.photobucket.com/albums/ii14/sherro_2008/Darwindifferencespaghetti.jpg?t=1261996491
Geoff,
I’ll have to come back to your graph a little later–I’m not following it right off. (I think I missed the BOM acronym somewhere–what is that?)
Right now I’m looking at the WMSSC temp data (ds570.0) I’ve downloaded.
I see there’s lots of missing data in any given station series. at least outside the U.S. This, I guess, relates to your comments above. Clearly there’s no ‘right’ answer to fill in the data. I guess I’d maybe want to take an average from a couple of years on either side where that’s possible, as an approximation, in order not to throw so many records out.
The most significant thing I notice right off: It looks like the stations are pretty much in large urban areas. Isn’t it well known and accepted that urban areas are warming due to the inherent characteristics of their being developed areas (and not due to greenhouse gases)?
If so, I don’t know that this data is worth doing anything with; it would just probably show warming that virtually nobody disagrees with. Do you think that’s right?
If it is, it would seem RC is misleading when using it to support global warming in their article: “Are the CRU data ‘suspect’? An objective Assessment.” Seems like they would note the urban characteristics of the records. Maybe the stations are not actually in the cities?
Perplexing! I must be missing something.
M
I just ran across this site that has a graph of satellite temp data:
http://www.drroyspencer.com/latest-global-temperatures/
I read somewhere along the way that satellite data doesn’t have the urban heat problem. Does anybody know if that’s right?
What do people here think of the graph? Do you think the data is good?
It seems to show warming in the latest decade but no reason to believe it’s a long term upward trend of a magnitude leading to disaster. That’s how I would view it, anyway.
This guy Roy Spencer and John Christy seem to be pretty reasonable and reputable people. Does anybody know what either side thinks of them?
M
Murf,
You ask “Perplexing! I must be missing something.”
To the contrary. You are catching on fast. You should read earlier posts by Anthony and those noted in the margin of Climate Audit, for background.
“BOM” is Bureau of Meteorology, Australia, who have collected and compiled the one record from which all the others on the graph are derived. My fault, I made it up for you in a hurry late at night and it lacks labels. Willis has better examples at the top of his articles. They say the same thing. Geoff.
I’m now at the Goddard Institute site for Surface Temperature Analysis. I thought I might see what I could do with that data for just rural areas.
It seems odd; I only see data for stations on a one-at-a-time basis. Surely you don’t have to get the data one little chunk at a time.
Am I missing it somewhere? Does anybody here have the complete Station data?
M