In honor of Google’s latest diversity kerfuffle, I continue with my diversity initiative on WUWT with a guest post by Nick Stokes.~ctm
By Nick Stokes,
There is an often expressed belief at WUWT that temperature data is manipulated or fabricated by the providers. This persists despite the fact that, for example, the 2015 GWPF investigation went nowhere, and the earlier BEST investigation ended up complementing the main data sources. In this post, I would like to walk through the process whereby, in Australia, the raw station data is immediately posted on line, then aggregated by month, submitted via CLIMAT forms to WMO, then transferred to the GHCN monthly unadjusted global dataset. This can then be used directly in computing a global anomaly average. The main providers insert a homogenization step, the merits of which I don’t propose to canvass here. The essential points that you can compute the average without that step, and the results are little different.
The accusations of data corruption got a workout with the recent kerfuffle over a low temperature reading on a very cold morning at Goulburn, NSW in July, so I’ll start with the Bureau of Meteorology online automatic weather station data. I counted recently a total of 712 such stations, for which data is posted online every half hour, within ten minutes of being measured. You can find the data by states – here is NSW. You can find other states from the bar at the top, under “latest observations”. Here is a map of the stations in NSW in this table:
For context, I have marked with green the stations of Goulburn and Thredbo top which had temperatures of below -10C flagged on that very cold morning in July. On that BoM table, you can see stations listed like this (switching now to Victoria):
I switched because I am now following a post from Moyhu here, and I want a GHCN station which I could follow through. But it is the same format for all stations. This data is from 4 December 2016, and I have highlighted in green the min/max data that will flow through (unchanged except for possible quality control flagging) to GHCN unadjusted. It shows for Melbourne Airport, the most recent temperature (22.4) at 7pm, various other data, and then the min and max, along with time recorded. The min is incomplete; it showed the latest 7pm temperature, but would no doubt be lower by 9am the next day, which is the cut-off. The max probably wouldn’t change. You can see the headings by linking to the page here.
If you click on the station name, it brings up a full table of the half-hourly readings for the last three days, in this style:
Apologies for jumping forward to now (7 Aug), but I didn’t record this back in December. It shows the headings relevant to the above too; the top line is present (a few minutes ago), going back. Now you can see that this has to be automated; no-one is hovering over this stream of data with an eraser. If you click on the “Recent months”, it brings up the following table (an extract here, and we’re back in Dec 2016):
That was taken at the same time (just after 7pm, 4 Dec), and you’ll see that it shows the minimum attributed to Sunday 4th (before 9am), at 9.1, but not yet the max. If you look below that table you’ll see a list of the last 13 months linked, for which you can bring up the complete table. Here is what that Dec 2016 table now looks like:
…
The max of 31.7 is there; the min went down to 15.7. The other data hasn’t changed. Further down on that page, as it appears now, are the summary statistics for the month:
At the end of Dec 2016, that was transmitted to the WMO as a CLIMAT form, which you can see summarized at the Ogimet site
…
You can see that the min and max are transmitted unchanged. The mean of the two has also been calculated and is marked in brown. If you want further authenticity, that site will show you the code that the met office transmitted.
Finally, the CLIMAT form is transcribed into the GHCN unadjusted file, which you can see here. It’s a big file, and you have to gunzip and untar. You can also get a file for max and min. Then you have a text file, which, if you search for 501948660002016TAVG (which includes the Melb code) you see this line:
There is the 19.5 (multiplied by 100, as GHCN does). The other numbers will appear in the GHCN TMAX and TMIN files.
You can even go through to the adjusted file, and, guess, what, it is still unchanged. That is because homogenization rarely modifies recent data. But older data may be. GHCN unadjusted does not change, except if the source notifies an error. There are quality controls, which don’t change numbers, but may flag them.
There have been endless articles at WUWT about individual site adjustments, but no-one has tried to calculate the whole picture of the effect of adjustment. With the unadjusted vs adjusted files, it is possible to do that. I have been calculating a global anomaly every month, using the unadjusted GHCN data with ERSST. The June result is here; there is an overview page here, with links to the methods and code. This post compares the result of unadjusted vs adjusted GHCN; the difference is small. Here from it is a plot from 1900 to start 2015 showing TempLS (my program) unadjusted (blue) vs adjusted (green) and GISS (brown), 12 month running man. It’s an active plot, so you can see more details at the linked site.
If you want more convenient access to the station data, I have a portal page here. The heading line looks like this:
The BoM AWS link takes you to this page, listing all station names with links to their current month data page. BoM also posts the metadata for all their stations, and that link takes you to this page, which lists all stations (not just AWS, and including closed stations) with links to metadata. The GHCN Stations button links to this page, which links to the NOAA summary page for each GHCN station by name, or if you click the radio buttons, to station annual data in various formats.
Summary
I have shown, for Australia (BoM) at least, that you can follow the unadjusted temperature data right through from within a few minutes of measurement to its incorporation into the global unadjusted GHCN, which is then homogenized for global averages. Of course, I can only show one example of how it goes through without change, but the path is there, and transparent. Those who are inclined to doubt should try to find cases where it is modified.
The entire point of all this anal temperature focus is that alarmists try to draw a line between reportedly increasing temperatures and global warming. Since this theory is fundamentally flawed it’s all just a bunch of sound and fury signifying nothing.
And now for something completely different.
The fundamental premise of the atmospheric radiative greenhouse theory is that the earth without an atmosphere is 33 C colder than with. (255 K cold, 288 K warm)
This is incorrect.
Any object in the path of the sun’s expanding photosphere, 1,368 W/m^2 at the earth’s average orbital distance, i.e. the moon, the international space station, satellites, will be exposed to an equivalent radiative temperature of 394 K, 121 C, 250 F. That’s hot, sort of.
So, what would the earth be like without an atmosphere?
At 394 K the oceans would boil away removing the giga-tons of pressure that keeps the molten core in place which then pushes through the thin ocean floor flooding the surface with dark magma, reducing albedo and increasing BB emissivity. With no atmosphere, a steady rain of meteorites would pulverize the surface to dust same as the moon. No clouds, no vegetation, no snow, no ice and a completely different albedo, certainly not the current 30%. The earth would be much like the moon with a similar albedo (0.12) and large swings in surface temperature from lit to dark sides. An albedo of 0.12 produces a radiative balance of 1,203 W/m^2 and an equilibrium temperature of 382 K, 109 C, 228 F. The naked earth would be hotter by 94 C not colder by 33 C.
The earth’s atmosphere and albedo do not keep the earth warm, they keep it cool.
http://writerbeat.com/articles/14306-Greenhouse—We-don-t-need-no-stinkin-greenhouse-Warning-science-ahead-
http://writerbeat.com/articles/15582-To-be-33C-or-not-to-be-33C
http://writerbeat.com/articles/16255-Atmospheric-Layers-and-Thermodynamic-Ping-Pong
Nick
There are lots of sceptics here who will be only too keen to find cases in which the apparently transparent process you have highlighted has been modified. This is an ideal opportunity for them to do so.
(I think this might be a short thread…)
I am not sure how many people here believed that temperature data was being intentionally manipulated by providers. It would be extremely difficult to hide such an effort. Indeed, the problem with automatic stations in Australia was discovered as soon as it was cold enough to reveal the problem. If that was intentional, it was really stupid It is far more likely that it was a technical mistake that is being corrected immediately.
The real manipulation is taking place in climate studies, where the raw data is adjusted, not in the raw data.
One of the big issues with trusting the data is the adjustment of past temperatures, not of current temperatures.
I beg your pardon for my English, but it seem to me that this post relate to the news about limiting temperature below -10°C in the Goulburn weather station.
I look at the link you cited and I was not able to see any “lower temperature” under -10°C.
I’ve I misunderstood the problem, read the wrong data or what?
and what about the amount of concrete and bitumen at Melbourne Airport, Olympic Park or Laverton say since 1900. All the BoM temperature anomaly is revealing is the steady increase in the UHI in areas increasingly built up then projecting that as being global. There is a relationship to CO2 emissions no doubt but its not the ‘greenhouse effect’. Nick’s piece misses the point methinks, which is that the BoM data set is not fit for purpose to contribute to a proper global temperature trend. Ditto the sea surface set but for different reasons. I do not trust BoM for that reason. There is no way such an outfit can plead ‘duuhh – we didn’t think of that’. Lets say that the BoM stations are fine for the local temperature, telling it like it is but that does not make them fit for the global anomaly (essentially a trend parameter) purpose.
Leave BoM aside and think of the potential site location issues around the planet. Only a select set of sites could be considered as fit for the global anomaly task. That BoM and its counterparts do not use only properly sited stations goes directly to their credibility. Add to that this fetish for sexed up science communications as feedstock to the msm ( exemplar bar none in Oz being Peter Hannam at ‘F&$k Facts’ aka Fairfax) and this is all ready about the credibility of witnesses. Spare us the pet expert testimony.
Thankyou. Nice to have “how it’s done” resources like this available. It would be interesting to see how many of the sources internationally offer such transparent access to the end to end data stream.
As Nick Stokes wrote, the problem is not the present temperatures but the historical temperatures. For example the latest GISS version 2016 shows about 0.5 Celcius more warming from 1880 to 2016 than the previous version. It is simply too much. The homogenization of raw temperature data and calculating averages is not that difficult. The Americans went to the Moon in 1969. Now the history of GISS temperature versions show that they are not capable to calculate the average temperature of the Earth. What is going on? Why should I rely on the latest GISS version when all the versions so far have been wrong?
Great! System normal! We are in an interstadial warm period and by golly it should be warmer with little ticks up and down during this peak period that mean nothing compared to the big swings. Check. The Earth should be greening. Check. The oceans should be in net discharge mode with sea level slowly rising. Check. The climate should on average be wetter than when it is in a stadial period. Check. Flora and fauna should be flourishing. Check. People should not be complaining about this wonderful warming climate. Oops.
People should understand that there might be slight differences (likely much less than a degree F or C) between temperature data processes which show overall warming anyway, because those slight variations just don’t matter in the long run. What we should be worried about are the people who DO think this warm period is not supposed to happen and is something we should try to fix!!!
Lordy.
The problem, Nick, is that this does not address all the possible points of data correction or corruption. I want to believe the best about the integrity of those in charge of the data (I wont even mention points of physical corruption such as the Urban Heat Island effect or proximity to airports), but when there are literally billions of dollars on the line it is difficult to do so.
I stopped believing in AGW and the integrity of those who do so on the day I saw the infamous commercial from the U.K. where the teacher blew up the heads of non-believing children. My physical and moral revulsion to what I saw caused me to question a lot of things that day.
But for the sake of argument, let us assume there is no data manipulation, correction or corruption. Let us assume that it is truly warmer today then the at any point since we began keeping records. Let us also assume that there is no need for upper or lower error bounds. Let us even further assume that CO2 is the primary cause of AGW. If we implement the Paris Treaty, will this temperature go down? If we switch to 100% renewables will this temperature go down? If we take away every car, shut down all the airlines, remove all the AC units, implement a 1 child per family policy, make everyone become vegans, and eliminate capitalism will the temperature go down?
And if it does go down, is it proof that we were suffering from AGW?
Will we go to a magical norm, where it is neither colder nor warmer than normal? Where there is no temperature change from year to year, decade to decade or century to century?
Yeah, you have no idea.
Class diversity. Discrimination between individuals based on the color of their skin, their sex (i.e. male or female), etc. is logically and by definition racist, sexist, etc. A bigoted (i.e. sanctimonious hypocrisy) philosophy normalized by the Pro-Choice Church and its progressive (i.e. monotonic) liberal (i.e. divergent) acolytes.
Still, denying individual dignity is a far cry from denying evolution of human life from conception. I wonder how many people actually believe in spontaneous human conception; in social justice adventures that commit mass a-bortion and force global refugee crises.
“I have been calculating a global anomaly every month, ”
I suppose if it pays the bills , why not.
Thanks, and very welcome.
With this post you have put clear blue water between yourselves and Ars Technica, Real Climate and the Guardian, none of whom would ever feature an article by a known skeptic, all of whom are mired in a morass of censorship of opposing views and endless cries of ‘fossil fuel funded denier’.
Thanks also to Nick for continuing to comment and participate in the face of sometimes very unfortunate and rude replies. His patience is commendable. His presence enriches the site greatly, and while I often differ from him, I always read his comments with interest.
A very useful post, Nick, with useful links…..thanks!
There is so much data available out there and that’s perhaps a large part of the problem. Once upon a time, we’d have been content to record it, use it for historical comparisons and identify extreme events and suchlike to fill a column in a newspaper and comment on the unpredictable nature of weather. Now, thanks to computers, we build models and make extrapolations and projections as if we actually understand what the underlying processes are. Hubris, I’m afraid.
Airport stations are placed to measure conditions at the runway — hopefully a worst case so we can accurately asses the risk to flight based on density altitude and wait for a cooler time of day if necessary.
The Surface Stations project documented anomolies in many other stations.
If we can use one pristine station for carbon dioxide, why not use one pristine station for temperature.
The more we talk about process, the less we talk about outcomes, and the costs/benefits of our actions.
OK Nick, the temperatures are fine. Things are warming up a little bit. They might get as warm as they were in the MWP. Then again, they might cool down, as they did in the Little Ice Age, or heaven forfend, enough to put us into another cold period, which has been the dominant ‘climate’ of the last few hundreds of thousands of years, and not that friendly to life as we know it
There is an implicit thought here that if you can convince us your temperature series are valid, all the other claims of the CAGW advocates are valid.
Put simply, if a few degrees of temperature rise make folks so unhappy, why do northerners vacation in the Caribbean and retire to Florida. We are going to completely cycle our infrastructure in less than 100 years anyhow, and we will naturally cope with temperature, sea level, and other changes in the natural course of things.
Back to my airport. We tend to wonder that the change of life that is supped to accrue as the temperature varies a few tenths of a degree. The adibatic lapse rate is 4 deg/1000 feet, so I can cool 8 degrees
by climbing 2000 feet. Folks who live on a hill live in a different climate. I live in the temperate zone. I can change my average by moving north or south a few hundred miles. Assertion that we’ll not be able to adapt to temperatures changing makes no sense. The flora and fauna adapt along with us. We grow different plants in different regions. The plant mix has changed here as the temperature has cooled over several decades, probably a local effect.. .
So I’ll stipulate your damn temperatures, and call BS on the implied conclusions.
Thank you Nick for this, and Charles. This policy is focussing the debate rather well. Anthony has always had an open invitation to proponents of looming disaster to submit articles, but either direct invitation has been the best way to go or the time is right for this.
Well if the unadjusted is truly that close in all land based stations, the data keepers have have shot themselves in the PR foot for no gain. It has long been my contention that if we are facing an existential threat, that a hundred scattered stations around the globe would be, without adjustment, sufficient an early warning system. We would see if we are departing seriously from 0.6C/ century (we aren’t) relatively quickly. Hundredths of a degree adjustments are expensive foolishness and raise scepticism that can’t be good for morale or sober policymaking.
Paul Homewood found Paraguay and Ecuador traces grandly adjusted for GHCN and a Capetown raw record that even had the familiar wiggles of the global pause – further proof that a) you don’t need so many hands in making the broth and that b) for disaster warning, you don’t need many stations at all.
The fact that 70% of the globe is water does offer up a playground for fiddling that hasn’t been ignored, too. Since the satellite temperatures over land are frequently argued to be close to surface station results, logic would cry out for this record to be the source for ocean temperatures. Do the rejected or neglected buoys validate satellite readings?
Similarly for SLR. If the fear is we are going to be struggling with a 3m rise, why are we running down to the sea with micrometers?The tide guages, whether relative sea levels vary up or down because of crustal movements are more than adequate to give us long lead time to deal with it. It presently seems steady at ~2mm/yr. The adjustment is only to 3mm with allowances for ocean basin volumetric changes from isostasy which makes the silly situation of official sea level standing above the actual water surface.
“Of course, I can only show one example”
Nick,
There is no “of course” in climate science. One example doesn’t prove anything, either.
Andrew
This was a well thought out article that demonstrates there is no massive conspiracy among (at least) Australian temperature monitors. However, it misses a related problem – sampling errors. Let’s grant the history record back to 1900 for NSW as shown in your figure. I have a difficult time believing the border between South Australia and the Northern Territories has as accurate a record. Similar for Siberia, the Northwest Territory of Canada, Mongolia, central portions of the Sahara Desert, interior rain forest in Brazil, central Greenland, the South Pole, and so on. In fact I would suggest that we have an excellent historical record for land area within 200 miles of the ocean or major sea (Mediterranean, Caribbean, etc) but not much else. I doubt the South Atlantic, South Pacific, South Indian or any of the Pacific is well sampled before 1950. So when we compare global averages to the past we limit the past to coastal land. I could accept that a sampling is an accurate measure of the whole except that the sampling needs to be unbiased. Since our historical sample is biased to places where people lived I don’t think we can compare to the past. Maybe I’m wrong, I’m open to correction. So in that vein – The Pacific Ocean is as large as the 6 occupied continents put together, and had very little shipping across it in 1900. How well do we know the temperature of the Pacific Ocean in 1900?
“Adjusted data” is an oxymoron. Scientists who introduce their own fallible judgment into “improving” measurements are corrupting the entire process and rendering it useless.
If data is no good, the only option is to throw it out. Trying to unscramble the egg just creates a mess and ruins breakfast.
Gary……you are naive to think that reasoned, logical methodology has any merit in the race to obtain portions of a Trillion Dollar plus industry.
I get a little bit of a kick out of all the people and money spent on the minutia involved with climate on both sides of the fence. It is all a shell game and most have lost sight of the pea.
A very recent WUWT Post on ‘Dodgy Greenhouse Data’ is an example. For decades the amount of CO2 and Greenhouse gasses have been used a propaganda to convince the public of CAGW (50% of public doesn’t know the meaning of individual letters), yet the veracity and collection method of GHG’s is completely unverified.
OT, but ctm did open with a reference to Google’s ‘diversity kerfuffle’: CNS ran a much more sympathetic story, which I think would be a lot more accurate than the Daily News version. The Daily News version reads like a totally PC distortion.
The CNS version is at http://tinyurl.com/yckx66ro
The Google boss’s statement confirmed that the engineer’s comments were accurate.
The “adjustments” are, indeed, not particularly large. However, they have a subtle goal: to make the series appear to be less variable and more in line with the AGW hypothesis.
:large
So, while advocates can legitimately feign dismay to criticism on the grounds that linear trends drawn through the data have actually decreased with adjustments, it is a sleight of hand – the adjustments are erasing the features that show the warming is not actually strongly correlated with CO2 concentration.
This reminds me of a “Chain of Custody” whereby there are a series of hand-offs of evidence and each must follow proper protocols (including sign off). So at least THIS data seems to be aggregated from each station and collected properly. What is missing is each station’s formal practice, then any down line attempt to use this data including the methods, justifications, and processes used to change it or derive answers from it. At any point corruption of clean data can occur (either on purpose or by accident).
Strict CoC is only necessary in fields where corruption and mistakes are deemed likely to influencing the outcome- by corruption I include the misuse of statistical methods and bad analysis which is may just be sloppiness. It is the fault of the practitioners in the field of “Climate Science” that no one believes them anymore. All data is eyed with great skepticism.
To fix this, practitioners in “Climate Science” will have to be more careful with their data and analysis. They need to understand and separate facts from the model derived outputs commonly used in place of facts. They need to answer questions from skeptics with respectful language. They need to possess a more open mind where uncertainty allows a conversation to occur, not be shut down. In short, they need to transform their practice from a self-contained religion into a normal science based one.
Nick
Thank ŷou for posting this. I do admire your integrity and wish there were more people like you at this site which otherwise can often be a bit of an echo chamber.
It’s not often we agree on things but as you know I am sceptical about the widespread corruption of raw data. My main point of contact is with the met office and I find it faintly ridiculous that as soon as I leave there the scientists such as Richard betts rub their hands and say ‘ right he’s gone, now we can get on with doctoring the data.’
Having investigated historic temperatures for many years I do however have a problem with comparing like for like between now and previous eras.
When you realise instruments were hauled to observers by mule or that observers were often untrained, used different times of day to take temperatures, used a variety of screens, that a mercury reading may be different from an alcohol one on a thermometer, that readings were often added in days after the event, that instruments were moved or urbanisation crept up on them, then it becomes very difficult to say with certainty that the same place has become notably warmer in the last 100 , 500 or 1000 years.
Certainly in my neck of the words we can trace the ups and downs of temperatures over the centuries and I would find it very difficult, based on evidence such as tree elevations, known crop elevations, harvesting dates eyc, to say that the climate was warmer today than at various points in the past. Similarly, glacier advances and retreats in such places as the alps can be followed with some certainty due to the works of such people as Le Roy Ladurie.
So all in all I do not think anything remarkable is happening and certainly do not believe that weather is as extreme as it has been at periods over the last 1000 Years.
Keep up the good work but let us put all this into the context of the Holocene, rather than just the last few decades
All the best
Tonyb
Another problem, and this has to do with confirmation bias. Adjustments are sought that tend to produce results favoring the AGW hypothesis. They are not sought if they would tend not to do so.
If you go looking for something, you will generally find it. These corrections to BOM figures may be insignificant, but how many other such corrections could legitimately be found if the principals were actually looking for them?
So, while adjustments favoring the outcome one way may be legitimate on their own, they are not being counterbalanced by unfavorable ones, and as a whole are thereby not legitimate.