Summary of GHCN Adjustment-Model Effects on Temperature Data
Guest essay by John Goetz
As the debate over whether or not this year will be the hottest year ever burns on, it is worth revisiting a large part of the data used to make this determination: GHCN v3.
The charts in this post use the dataset downloaded at approximately 2:00 PM on 9/23/2015 from the GHCN FTP Site.
The monthly GHCN V3 temperature record that is used by GISS undergoes an adjustment process after quality-control checks are done. The adjustments that are done are described at a high-level here.
The adjustments are somewhat controversial, because they take presumably raw and accurate data, run it through one or more mathematical models, and produce an estimate of what the temperature might have been given a set of conditions. For example, the time of observation adjustment (TOB) takes a raw data point at, say 7 AM, and produces an estimate of what the temperature might have been at midnight. The skill of that model is nearly impossible to determine on a monthly basis, but it is unlikely to be consistently producing a result that is accurate to the 1/100th degree that is stored in the record.
A simple case in point. The Berlin-Tempel station (GHCN ID 61710384000) began reporting temperatures in January, 1701 and continues to report them today. Through December, 1705 it was the only station in the GHCN record reporting temperatures. Forty-eight of the sixty possible months during that time period reported an unflagged (passed quality-control checks) raw average temperature, and the remaining 12 months reported no temperature. Every one of those 48 months was estimated downward by the adjustment models exactly 0.14 C. In January, 1706 a second station was added to the network – De Bilt (GHCN ID 63306260000). For the next 37 years it reported a valid temperature every month and in most of those months it was the only GHCN station reporting a temperature. The temperature for each one of those months was estimated downward by exactly 0.03 C.
Is it possible that the models skillfully estimated the “correct” temperature at those two stations over the course of forty plus years using just two constants? Anything is possible, but it is highly unlikely.
How Much Raw Data is Available?
The following chart shows the amount of data that is available in the GHCN record for every month from January, 1700 to the present. The y-axis is the number of stations reporting data, so any point on the curve represents the number of measurements reported in the given month. In the chart, the green curve represents the number of raw, unflagged measurements and the purple curve represents the number of estimated measurements. The difference between the green and purple curves represents the number of raw measurements that are not changed by the adjustment models, meaning the difference between the estimated value and raw value is zero. The blue curve at the bottom represents the measurements where an unflagged raw value was discarded by the adjustment models and replaced with an invalid value (represented by -9999). The count of discarded raw data (blue curve) is not included in the total count represented by the green curve.
The second chart shows the same data as the first, but the start date is set to January 1, 1880. This is the start date for GISS analysis.
How Much of the Data is Modeled?
In the remainder of this post, “raw data” refers to data that passed the quality-control tests (unflagged). Flagged data is discarded by the models and replaced with an invalid value (-9999).
In the next chart the purple curve represents the percentage of measurements that are estimated (estimated / raw). The blue curve represents the percentage of discarded measurements relative to the raw measurements that were not discarded (discarded / raw). Prior to 1935, approximately 80% of the raw data was changed to an estimate, and from 1935 to 1990 there was a steady decline to about 40% of the data being estimated. In 1990 there was an upward spike to about 55%, followed by a steady decline to the present 30%. The blue curve at the bottom shows that approximately 7% to 8% of the raw data was discarded by the adjustment models, with the exception of a recent spike to 20%. (Yes, the two curves combine oddly enough to look like a silhouette of Homer Simpson on his back snoring.)
The next chart shows the estimate percentages broken out by rural and non-rural (suburban and urban) stations. For most of the record, non-rural stations were estimated more frequently than rural stations. However, over the past 18 years they have had temperatures estimated at approximately the same rate.
The fifth chart shows the average change to the raw value due to the models replacing it with an estimated value. There are two curves shown in the chart. The red curve is the average change when not including measurements where the estimated value was equal to the raw value. It is possible, however, that the adjustment models will produce an estimated value of zero. The blue curve considers this possibility and represents all measurements, including those with no difference between the raw and estimated values. The trend lines for both are shown in the plot, and it is interesting to note that the slopes for both are nearly identical.
What About the Discarded Data?
Recall that the first two charts showed the number of raw measurements that were removed by the adjustment models (blue curve on both charts). No flags were present in the estimated data to indicate why the raw data were removed. The purple curve in the following chart shows the anomaly of the removed data in degrees C * 100 (1951 – 1980 baseline period). There is a slight upward trend from 1880 through 1948, a large jump upward from 1949 through 1950, and a moderate downward trend from 1951 to present. The blue curve is the number of measurements that were discarded by the models. Caution should be used in over-analyzing this particular chart because no gridding was done in calculating the anomaly, and prior to 1892 only a handful of measurements are represented by that data.
Conclusion
Overall, from 1880 to the present, approximately 66% of the temperature data in the adjusted GHCN temperature data consists of estimated values produced by adjustment models, while 34% of the data are raw values retained from direct measurements. The rural split is 60% estimated, 40% retained. The non-rural split is 68% estimated, 32% retained. Total non-rural measurements outpace rural measurements by a factor of 3x.
The estimates produced by NOAA for the GHNC data introduce a warming trend of approximately a quarter degree C per century. Those estimates are produced at a slightly higher rate for non-rural stations than rural stations over most of the record. During the first 60 years of the record measurements were estimated at a rate of about 75%, with the rate gradually dropping to 40% in the early 1990s, followed by a brief spike in the rate before resuming the drop to its present level.
Approximately 7% of the raw data is discarded. If this data were included as-is in the final record it would likely introduce a warming component from 1880 to 1950, followed by a cooling component from 1951 to the present.
Epilogue
The amount of estimation and its effects change over time. This is due to the addition of newer data that lengthens time series used as input to the adjustment models. The following chart shows the percentage of measurements that are estimated (purple curves) and percentage of discarded measurements. The darker curves are generated from the data set as of 9/23/2015 (data is complete through 8/2015). The lighter curves are generated from the data set as of 6/27/2014 (data is complete through 5/2014). Clearly, fewer measurements were estimated in the current data set than in the past data set. However, more measurements from the early part of the record were discarded in the current data set.
A chart showing the average change to the raw data is not shown, because an overlay is virtually indistinguishable. However, the slope of the estimated data trend produced by the current data set is slightly greater than the past data set (0.0204 versus 0.0195). The reason that the slope of 0.0204 differs from the slope in the fifth chart above (blue curve) is that the comparison end month is May, 2014, whereas the chart above ends with August, 2015.
Note: the title was changed to better reflect the thrust of the article, the original title is now a sub headline. The guest essay line was also added shortly after publication, and a featured image added as the guest author did not provide these normal elements of publication at WUWT – Anthony
With all the adjustments/estimates/modeling over the decades, the temp error ranges must look like that of the multiple computer model runs for future temperatures versus CO2. Ahh Sooo now I see why those model runs appear okay to climastrologists, because if the temp “data” error bands were overlaid onto the model runs then they would overlap!
It’s mostly claimed that the adjustments to raw data add more cooling to the data than warmth, I can’t find a graph to match that assertion. Why are they saying this?
Because what the adjustment does is ‘cool’ the past to make it appear as though the slope of the increase is steeper. Their ‘adjustments’ are anchored in the present, so to make apparent global warming fit their models they have to decrease past temperatures to make is appear as though warming has occurred at the rate they predict.
Is there a graph showing this? Like a single a graph, not fifteen
oh flow- this article:
http://wattsupwiththat.com/2015/06/04/noaancdcs-new-pause-buster-paper-a-laughable-attempt-to-create-warming-by-adjusting-past-data/
Figure 4 show the annual adjustments.
The totally bogus adjustments warm recent decades but cool more distant decades. On balance there is cooling, but it’s all bogus in the interests of promoting the Warmunistas’ agenda.
Whether BEST, HadCRU, GISS or NOAA, they are all rent-seeking, greedy, self-serving, trough-feeding swine, enemies of humanity who are destroying the good name of science built up over centuries.
Lady Gaiagaia Ahem, other provide you with wonderful bacon,chops and ham what dear friend has “Swine” done to you to speak of them in such a disparaging manner. Heavens I’m just besides myself, snacking here on sausage.
michael 🙂
All this talk of global average temperature adjusted since the 1800’s is just proof evident that the science behind “climastrology” is shonky!
So 66 percent of the temperature data are estimates and that is a big problem. It is an even bigger problem because the data only covers approximately 15 percent of the Earth’s surface. There are virtually no stations for some 85 percent of the oceans or continents.
Tim: A huge problem is that the oceans are not warming. If 2/3 of the Earth’s surface is not warming at all then the land would have to warm by 6C to reach the catastrophic 2C global average.
A truly shocking statistic that most of the reported temperatures are fictitious and we all know that the global warming models are much hotter than reality.
thanks, this required a lot of work
The only way to check if the adjustments are correct is to pick a certain time and place for a certain station adjustment, take your time machine, go to that time and place and measure better and compare. The pesky experiment pseudo scientists hate so much. Since it’s not possible, I invoke Newton’s flaming laser sword.
Although there may ways you can quite rightly mock climate ‘science ‘ you have to give them credit where it is due and that fact that they have managed to build so much on so poor a foundation of ‘better than nothing ‘ does show some skill.
It is therefore a real; shame that they cannot show the same skill or effort when it comes to following good scientific practice in there work . although to be fair if they did then they may get the ‘wrong type of right results ‘ which would be no good at all for their careers and certainly not met any political ‘needs’.
John Goetz or Nick Stokes.
One thing that does look intriguing is the “sawtooth” pattern in chart 5, the one showing the average adjustments over time.
Are they monthly average values?
If they were averaged annually the sawtooth pattern might disappear.
To me it looks like most of the adjustments are for Time of Observation (ToB) bias, considering how many of the GHCN-M stations are in the United States which tended to use volunteers before they got automated.
The x-axis increment of each chart is monthly.
I noticed that, too. I think it probably as you say, monthly averages.
I believe it may illustrate something that Walter Dnes showed in a post here last year (second graph) though only for USHCN:
http://wattsupwiththat.com/2014/08/23/ushcn-monthly-temperature-adjustments/
In recent years, the months December through April are adjusted much more than the other months are. I suppose it is only coincidence that the reigning CO2-temperature hypothesis says winter should warm faster than summer does… but I digress. 🙂
I also am very reluctant to believe that there is a conspiracy, at least a conscious one, to adjust the data in a particular direction, and I can see why some of the people who work on this (e.g. the BEST guys) get a bit upset at the tone of the comments, and may as a result refuse to engage with the doubters. Please, folks, try to discuss these things on topic, at least.
However, essays like those from John Goetz here are not without value, because they remind us of the frequency and the volume of adjustments and estimates that are made – and this has consequences for our understanding of the headlines we see. For me, it just goes to underline the very large general uncertainty there is about temperature trends (though I accept that the keepers of the adjustments do believe that they are reducing that uncertainty) and pushes me towards the satellite observations. Sure, there are adjustments and estimates there too, but they are of a different (and, importantly, much more readily auditable) nature to those in the surface record.
To summarise the comments so far:
A Comment: ‘66% adjustments – the recent past always warmer and the longer-term past always cooler. Suspicious.’
SC/NS: ‘Adjustments have to be made, and they’re consistent because they’re scientific.’
A Comment: ‘Consistent, yes. But agenda-driven/on message – present warmer, past cooler.’
SC/NS: ‘Can you substantiate that?’
A Comment: ‘Yes – look at X, Y, Z.’
SC/NS: ‘I can’t look at such stuff, it’s rubbish.’
A Comment: ‘Well it’s not as rubbish as your adjusted temperature estimates!!’
SC/NS: ‘Adjustments have to be made, and they’re consistent because they’re scientific.’
[…]
Am I missing something?
NB – didn’t James Hansen include an extra 30 years’ worth of ‘data’ for Antarctica, 30 years before the first met station opened there? Wasn’t the single station he used outside the Antarctic circle and, oddly enough, adjusted to show the whole South Polar region as 2 degrees cooler than the present?
Happy to see this presentation but disappointed in the layout and some of the assertions/assumptions.
Has great potential to be redone and represented more clearly, not by me, I am a critic and suffer from lack of follow through.
Pleased to see Nick and Zeke commenting here. They often come on board when their particular viewpoints are challenged. .Usually in Nick’s case to say forget that, look at this.
We seem to be missing Mosher who has been commenting at depth on a similar issue of the record accuracy at J Curry’s > 100 matches.
His claim?
“That is why you can for example pick 110 PRISTINE sites in the US
(CRN) and predict the rest of the country: Including
100s of other pristine sites ( RCRN) and 1000’s of “bad” sites.
What’s it tell you when you can start with 60 samples and get
one time series… then add 300 and get the same,,, then add
3000 and get the same…. then add 30000 and get the same?
whats that tell you about sampling?
Whats it tell you when you can pick 5000 and then predict any
other 5000 or 10000?”
I replied What it tells anyone with common sense is that all the sites have been linked to each other by an algorithm and are no longer individual raw or individual site modified data but data that has been homogenized to fit in with every adjacent site.
This is not something to be proud of.
This is scientifically very, very wrong.
Any true set of recordings makes allowance for the fact that temperatures very from minute to minute from site to site and that due to known weather variations sites do not have to match each other in step.
What Mosher alludes to is pure chicanery.
Any set of sample sites that agrees this perfectly means they are not real temperature recordings anyway in any form.
It is the Cowtan and Way Kriging experience over again.
You must be able to pick sites that do not agree with each other in any sample.
That is what weather temperature, measurement is all about.
When you link everything to each other so they all move in step whatever sample you take you do not have real measurements.
Try it on the raw data Steven. See if they all move the same way whether you use 2 or 50,000.
I will guarantee they don’t.
Take your modified data and prove they all link perfectly.
I guarantee they do as well. I have your word for it.
And what do you call your data?
Well not data anymore.
Sorry for cross threading but the absolute ability for any sample of the data to agree with all other samples of the data means the 66% of values are estimated is incorrect.100% of the data is estimated and homogenised.
As Nick Stokes said September 24, 2015 at 6:34 pm
“The public expects that people who actually know what they are doing will give the best estimate they can of global temperature. And they do. That involves detecting and correcting inhomogeneities.”
Ie none of the data we get is real it is all correlated and homogenized as per Mosher’s stunning observation.
Surely people here and elsewhere can follow up on this admission of data being so modified that all real variations have been removed.
Sums up the ‘conspiracy’ nearly perfectly..
As she sings…
“Everyone’s a super-hero”
“Everyone’s a Captain Kirk”
With orders to identify, classify etc”
“Floating in the summer sky – something’s out there
“Somethings here from somewhere else”
99 minsters meet, worry worry super scurry
“The President is on the line…
Just brilliant, and she’s very pretty.
While I can accept adjusting individual data points in a well documented manner, and using them for local functions; I have a problem with using the local adjustments in a larger dataset unless those specfic adjustments were applied universally. The averaging functions of the large data sets should incorporate the rationale for the adjustments in their error calculations.
In other words, the adjusted dataset is weather, the raw data is climate.
One thing that is missing from this study is the number of Sites that now have decades of “Estimated” data which never existed. There was no Raw Data, the Estimated data has been added to lengthen the historic records or fill in where data was just missing.
Of course the estimated data is lower than average temperatures for that area.
I did have code looking for that occurrence but none was found. USHCN does do infilling, but the GHCN adjustment process does not use the USHCN infilled data.
One of the real problems here is spatial coverage, and station drop out (there has been a substantial decline in stations in recent years)..
The plot shows how little global coverage there was in the 1880s. Even today, only about 2500 stations are used. Can one truly ‘estimate’ global temperatures from just a few hundred or even 3000 stations. The globe is a big place and the spatial coverage is poor. This is not simply a number issue, but a density issue. It is not a question that in 1880 there were say about 300 stations equally positioned throughout the globe, and today there are about 2,500 equally distributed. There are very large tracts of planet Earth where there are none, of just a handful of stations.
The temperature record has now become so horribly bastardised (for a number of reasons) that we are now left at reviewing the veracity and probative value of the adjustments/homogenisation rather than the underlying data itself.
Personally, I consider the spatial coverage to be so poor and the margins of error so high that we cannot say anything of value about global temperatures.
Probably all that can be said about temperatures is a generalisation that there is much year to year variability and that the 1880s, 1930s and today are all warm periods but due to limitations of the data (and its true error margins), we are unable to say on a global basis whether today is warmer than the 1880s or the 1930s, but as far as the US is concerned it is likely cooler today than it was in the 1930s.
Obviously we can say that it is today warmer than it was at the lows of the LIA, and we are unable to demonstrate form the thermometer record an increase in the rate of warming in the modern period, over and above the rate of warming which was seen in the circa 1880s to 1900s and the circa 1920s to 1940s (indeed I recall that Phil Jones of CRU acknowledged that there was no statistical difference in the rates of warming during these 3 periods).
Of course, the thermometer record can at best tell us something about temperature, but not about the cause of any rise in temperature, and the land temperature record cannot tell us anything about changes in energy.
The whole thing is simply too course to be useful, it has little scientific use.
Let me say first of all I’m taking it that the adjustments to temperature readings taken between 20 and 120 years ago are made in good faith. However if it was put to jury that these people had told politicians that the world was warming and that great sacrifices would have to be made to stop future catastrophes The politicians had taken them at their word and brought in policies to combat climate change that resulted in an increase of energy, coal miners to be thrown out of work, industries decamping to more energy friendly nations and were generally frowned upon by the voters.
Would it not seem reasonable that their jobs and livelihoods depended upon their being correct in their diagnosis and prognosis and it was then shown that they had systematically changed previous data to exaggerate the warming in the 20th century that they were doing this to save their faces and jobs?
I’m waiting for the class action suit.
I note that Nick Stokes has not replied to Lee {9/24@11.30}.
I am confused. I would like Mr. Stokes (if he has not retired hurt) to explain: “The organizations who estimate the global temperature do so on the basis of estimates of the temperatures of regions. Not of thermometers. Thermometer readings are part of the evidence.”
If thermometer readings are only “part of the evidence” what are the other parts? And how can these contribute to any estimate of temperature if they are not based on measurement? And if they are based on measurement how are those measurements derived without thermometer readings?
By the way, I understand the need for TOBs when estimating Tmean as (Tmax +Tmin)/2 but for at least ten years it has been possible to measure Tmean accurately as a continuous function over any 24 hour period and more than one station that I have seen is doing this.
Extract of report into BoM-
‘The Forum noted that the extent to which the development of the ACORN-SAT dataset from
the raw data could be automated was likely to be limited, and that the process might better be
described as a supervised process in which the roles of metadata and other information required
some level of expertise and operator intervention. ‘
http://www.bom.gov.au/climate/change/acorn-sat/documents/2015_TAF_report.pdf
So “operator intervention” -AKA best guess.
John Goetz, can you please just provide the trend of the data which is still Raw, not adjusted.
This is the temperature data WE should use and start promoting as the real land temperature record. There is an issue with respect to gridding a reduced database, but it will be the truth.
It is clear that the adjusted data is “wink, wink, nudge, nudge” used by the climate scientists to keep their theory alive. Maybe it is not a conspiracy, but there is a lot of winking going on whenever a new Karl et al 2015 paper comes along with a new adjustment increasing proposal.
The point is “what is the real temperature increase. Is this theory true or not”. This is what is important. John’s data above shows that 0.4C of the trend is just an artifact of the winking process.
Throw out the adjusted temperatures. Facts are more important.
When all is said and done, changing the data does not change the temperature of what is being measured. Again, the ocean temperatures at an around the ARGO buoys have not suddenly jumped. The ARGO Buoy thermometers have not suddenly lost their accuracy.
All you do what you tamper with the data is to increase the difference between the data and reality.
If CAGW were real, they wouldn’t have to adjust the data to prove warming, and 2, since 1998 if CAGW were real, we wouldn’t be still arguing about it. Any reasonable person could see the results. I think we have seen the results and CAGW is unreasonable. It didn’t and isn’t happening. When the Temps have fallen out of the lowest projection, how is it reasonable to think that co2 controls temperature?
To estimate 66% from a decline of 0.2% to 0.1% coverage of the planets surface is a disgrace for anybody that say it is better than satellite data.
Could you imagine the uproar if the satellite data only covered the tropics and polar zones, but the rest of the world was estimated?
There is little doubt this data has deteriorated from bad to worse and now mainly relies on confirmation bias modeled temperatures for majority of the data set.
When proper data was used and not adjusted for models or infilling from land to ocean surface, the GISS in particularly resembled something a bit more realistic.
The corrected data shows 0.4 c artifact especially since 2001 and another artifact by the shifting the anomalies between the 1940’s and 1980. This shift now corrected resembles the ERSST global surface temperatures between the 1940’s and 1980. There is now still a 0.8 c difference between the early 1900’s and recent period (like HADCRUT), whereas before GISS had suddenly been dishonestly changed to ~1.3 c difference.
http://i772.photobucket.com/albums/yy8/SciMattG/GISS-corrected2_zpssymskhge.png
The real problem is not manipulated data, but a lack of adequate data.
There are just not enough data to estimate what they are trying to estimate.
Using estimates instead of actual data produces smoothing. The histogram of estimated values is different from the histogram of actual values. There is also the problem of bias – systematic over-estimation or under-estimation.
Lack of samples has been certainly part of the problem, but they have deliberately reduced stations numbers because they thought it was adequate enough. The less samples are used the easier it is to introduce bias when they are changed to somewhere else or sample numbers are changed. HADCRUT4 did this in the last version by introducing an extra 100+ stations in the NH and 400+ in the SH. Why reduce thousands of stations and then add hundreds? No other reason other than to show a warming bias and this change reflected that between the versions 3&4. Cooling numerous stations significantly over the earliest decades to cause a warming trend instead of a overall cooling one, is not caused by lack of adequate data.
Using estimates only produces smoothing depending how it is done. If it is like GISS, ocean surface temperatures help smooth the data because of their slow response. Infilling from land temperatures over ocean surface reduce the smoothing and increase warm bias during warm periods. Estimates don’t cool past station data and warm recent data, that is due to bias human adjustments.
The best method is not to use estimates at all and only use quality stations that don’t need estimating. If the station data are missing for any month then the method should be done like HADCRUT does it. They emit it while keeping the rest of the stations unaffected by it.
How about a post that shows the temperature history of only non-adjusted data, only adjusted data, and only discarded data?
To dismiss all adjustments to raw data is to throw out the baby with the bathwater. However, it seems to me that we have two massive problems with the adjustments made to the global temperature record:
1. The direction of the adjustments is very heavily skewed in one direction creating an obvious appearance of bias (whether or not such bias is real). The limited transparency concerning the basis of the adjustments and the use of invalid statistical methods in some cases has only made matters worse.
2. The magnitude of the adjustments is very large. Almost as large as the warming signal being studied.
These simple facts made the surface temperature record untrustworthy to the point of uselessness.
Policy decisions which involve economic impacts measured in trillions of dollars can only reasonably be made based on the far more reliable satellite data record.
John
‘The skill of that model is nearly impossible to determine on a monthly basis, but it is unlikely to be consistently producing a result that is accurate to the 1/100th degree that is stored in the record.”
WRONG.
1. The skill of the model has been tested many times and the results are published.
2. NOBODY claims ACCURACY to 1/100th of a degree. That is NOT TRUE.
Its like this
Suppose I weigh you with a scale that has a precision of 1 lb
200, 201, 200 are the measurements I record
Now, I ask you to PREDICT or ESTIMATE what a PERFECT scale would record
The best prediction is 200.333333333333333333333333333333333333333.
That DOES NOT MEAN I am claiming to KNOW your weigh to with 1/100th or a degree or whatever
it means THIS
IF you recorded the weigh with a perfect scale the estimate of 200 1/3 pounds would MINIMIZE the error
of prediction.
get it.. So when we adjust for tobs and say 74.76478 F, we are saying THAT ESTIMATE minimizes
the error of PREDICTION
That is why in the tests of TOBS models the error of prediction is recorded.. Its typically around 0.25F
Mosher,
Your example sums up “Climate Science” practice very well. If your scale is accurate to 1 pound, the average of 200, 201, and 200 is 200, and you simply do not have any more information than that to report. Anything past the decimal point is unknown, and reporting such numbers is false, nothing more, nothing less. Historical temperature records are not accurate enough to discuss any digits past the decimal point, but you guys claim to know the average temp of the Earth in 1880 to .00 accuracy, or at least, you permit the media to publish such rubbish.
You could use the scalpel on the outlier, 201, and call it 200.
What? You cannot call it anything other 200. An instrument accurate to pounds cannot report any information of fractions of pounds, how fundamental is this to science???