E.M. Smith over at the blog Musings from the Chiefio earlier this month posted an analysis comparing versions 1 and 3 of the GHCN (Global Historical Climate Network) data set. WUWT readers may remember a discussion about GHCN version 3 here. He described why the GHCN data set is important:
There are folks who will assert that there are several sets of data, each independent and each showing the same thing, warming on the order of 1/2 C to 1 C. The Hadley CRUtemp, NASA GIStemp, and NCDC. Yet each of these is, in reality, a ‘variation on a theme’ in the processing done to the single global data set, the GHCN. If that data has an inherent bias in it, by accident or by design, that bias will be reflected in each of the products that do variations on how to adjust that data for various things like population growth ( UHI or Urban Heat Island effect) or for the frequent loss of data in some areas (or loss of whole masses of thermometer records, sometimes the majority all at once).
He goes on to discuss the relative lack of methodological analysis and discussion of the data set and the socio-economic consequences of relying on it. He then poses an interesting question:
What if “the story” of Global Warming were in fact, just that? A story? Based on a set of data that are not “fit for purpose” and simply, despite the best efforts possible, can not be “cleaned up enough” to remove shifts of trend and “warming” from data set changes, of a size sufficient to account for all “Global Warming”; yet known not to be caused by Carbon Dioxide, but rather by the way in which the data are gathered and tabulated?…
…Suppose there were a simple way to view a historical change of the data that is of the same scale as the reputed “Global Warming” but was clearly caused simply by changes of processing of that data.
Suppose this were demonstrable for the GHCN data on which all of NCDC, GISS with GIStemp, and Hadley CRU with HadCRUT depend? Suppose the nature of the change were such that it is highly likely to escape complete removal in the kinds of processing done by those temperature series processing programs?….
He then discusses how to examine the question:
…we will look at how the data change between Version 1 and Version 3 by using the same method on both sets of data. As the Version 1 data end in 1990, the Version 3 data will also be truncated at that point in time. In this way we will be looking at the same period of time, for the same GHCN data set. Just two different versions with somewhat different thermometer records being in and out, of each. Basically, these are supposedly the same places and the same history, so any changes are a result of the thermometer selection done on the set and the differences in how the data were processed or adjusted. The expectation would be that they ought to show fairly similar trends of warming or cooling for any given place. To the extent the two sets diverge, it argues for data processing being the factor we are measuring, not real changes in the global climate..The method used is a variation on a Peer Reviewed method called “First Differences”…
…The code I used to make these audit graphs avoid making splice artifacts in the creation of the “anomaly records” for each thermometer history. Any given thermometer is compared only to itself, so there is little opportunity for a splice artifact in making the anomalies. It then averages those anomalies together for variable sized regions….
What Is Found
What is found is a degree of “shift” of the input data of roughly the same order of scale as the reputed Global Warming.
The inevitable conclusion of this is that we are depending on the various climate codes to be nearly 100% perfect in removing this warming shift, of being insensitive to it, for the assertions about global warming to be real.
Simple changes of composition of the GHCN data set between Version 1 and Version 3 can account for the observed “Global Warming”; and the assertion that those biases in the adjustments are valid, or are adequately removed via the various codes are just that: Assertions….
Smith then walks the reader through a series of comparisons, both global and regional and comes to the conclusion:
Looking at the GHCN data set as it stands today, I’d hold it “not fit for purpose” even just for forecasting crop planting weather. I certainly would not play “Bet The Economy” on it. I also would not bet my reputation and my career on the infallibility of a handful of Global Warming researchers whose income depends on finding global warming; and on a similar handful of computer programmers who’s code has not been benchmarked nor subjected to a validation suite. If we can do it for a new aspirin, can’t we do it for the U.S. Economy writ large?
The article is somewhat technical but well worth the read and can be found here.
h/t to commenters aashfield, Ian W, and rilfeld
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
@davidmhoffer:
Averaging temperatures is just silly. Yet “it’s what they do” in “climate science”.
I occasionally point out the silliness in it (that “intrinsic” link above goes through the scientific philosophy bankruptcy of the notion of averaging temperatures in depth) and I occasionally point out that looking for changes in energy content or heat via temperatures alone is something even a freshman in chemistry would laugh at. But it doesn’t seem to ‘click’ with most folks.
http://chiefio.wordpress.com/2011/07/01/intrinsic-extrinsic-intensive-extensive/
FWIW most folks confuse heat and temperature anyway. Only engineers, chemists and the odd physicist seem to ‘get it’ and even they often go ahead and ‘average temperatures’ anyway.
THE classic error in calorimetry is to screw around with the thermometers and change or move them in the middle of the run; yet we are doing a large “calorimetry” measurement on the earth while constantly changing the thermometers by huge percentages. Yet “climate science” doesn’t care…
( I sometimes wonder why all the Chemists are not up in arms about that point… then again, when doing calorimetry in college chem classes, many of the chem students didn’t seem to ‘get it’ either…)
Then pointing out that averaging a bunch of temperatures does NOT get rid of systematic error, only random error, only got me lambasted for weeks by folks asserting I was an idiot for not being quite happy to do such a thing as clearly any such average was subject to the law of large numbers and would have ever increasing precision. (That the precision is false precision and that the presence of a systematic error can not be removed even though the precision of the average can be known to ever greater degree being seen as a red flag to more insults.) Attempting, then, to point out that most of the errors in the data were not “random error” but more systematic just caused more abuse. Attempting to point out that measuring one thing a dozen times or with a dozen different thermometers can remove random error but that measuring a dozen things with a dozen thermometers just has an error in each reading that may well be systematic and you simply can not assume that averaging them removes it; well, talk about red flags and bulls…
(The simplest example of ‘systematic’ error would be something like: If folks reading the Liquid In Glass thermometers and recording data in whole degrees F tended to just report the last whole degree the meniscus crossed, then you get a ‘low bias’ in all the readings. Averaging them together does not restore those missing fractional portions.)
All that is even before we get to your points about energy flux not being linear with temperature.
Heck, take one place that cools by 1 C from +0.5 C to -0.5 C AND gets a foot of snow. Average that with some desert area that goes up by 1 C and has no water. What on God’s Earth does that say about heat flux? Nothing. You have TONS of frozen water worth of heat being ignored. Ignoring enthalpy is a Very Bad Thing. (We won’t even talk about rising air and moving heat with non-correlated temperature changes… adiabatic rate anyone? 😉
But the “debate” seems firmly rooted in the bankrupt notion of “average temperature” even though it has no philosophical basis in reality, confounds heat with temperature, does not conform with the physics of heat flux (as you point out) and is frankly just silly. As near as I can tell, it is only because “temperature is what we measure” and folks do not understand that averaging is used to hide information that is “in the way” of seeing something else and is NOT an easy tool to use and is completely meaningless when used to average temperatures as they are an intrinsic property…
But pointing that out just causes the average person to glaze over and causes “climate scientists” and Warming Believers to toss rocks and flames at you. So I only do it on special occasions 😉 Most of the time I behave myself and pretend that ‘an average of temperatures’ is a sane thing to do… though I’ll usually sneak in a small disclaimer about it…
It really is an “Angels and Pins” argument, in terms of physics… (Which is why I mostly show faults in the method and data biases rather than try to find a ‘better way to calculate the Global Average Temperature’…. I’m not fond of trying to show one way of counting Angles and measuring Pinheads is better than some other one…;-)
There is no warming trend in the raw data. Two adjustments, SHAP and TOBS are responsible for creating the trend.
You can see the consequence of each individual adjustment in the graph below, direct from the source, so don’t take my word for it. Basically SHAP (station homogeneity) and TOBS (time of observation) each account for about half the trend and neither have much effect until 1950.
http://www.ncdc.noaa.gov/img/climate/research/ushcn/ts.ushcn_anom25_diffs_pg.gif
Global warming is anthropogenic alrighty but it’s done with a computer not an SUV.
For anyone who is not already familiar with Chiefio’s blog, I highly recommend it! It is always entertaining, no matter what the topic of the day may be, and his clarity and attention to detail rivals Misters MacIntyre, Eschenbach, Montford and Watts to name but a few. Although some of the science (especially the statistics) is beyond my comprehension on most of these sites, I can appreciate that ‘loose tooth’ feeling of something not being ‘right’ and the obsessive tendency to worry the tooth until it comes loose and reveals its secrets. To be able to do that and explain it clearly is a gift.
I also greatly appreciate the magnanimity with which he handles comments, having read his blog for some time (I found it through WUWT) I have almost always found him to be polite, light hearted and respectful, rare qualities these days.
Before I sound too much like lead cheerleader, I wish to offer my thanks to all the people who operate/ maintain/ help out with/ comment on these blogs, without them we would all be relying on mainstream media and I think we are all aware how that would go.
Kudos to you all gentlemen and ladies, your dedication to the truth is laudable and fully deserving of recognition.
Tony
p.s. I know I missed out Jo Nova, Jeff Conlon and many others, my apologies. Also, one of my personal favourites (and the one who started me looking down this road), John Brignell at numberwatch who doesn’t post very regularly due, I believe, to ill health but is a fascinating read on the perils of epidemiology.
Aha! Back to 1740; I wondered what your start date was.
I want to determine the trends in V1 and V3 from 1900 (to 1990). There are two reasons for this.
1. The long-term trend estimates done by GISS, UK Met Office etc are usually from no earlier than 1900. I want to compare your V1 and V3 with them for the same (or roughly the same) time period.
2. The pre-1900 data gets sparser and sparser. Adjustments are bound to have a more obvious effect the further back in time you go. GISS and UK Met Office don’t offer trend estimates from before 1900 because the data is uncertain.
I bet that there will be litle difference between V1 and V3 for the global trend from 1900. This will accord with pretty much everything I’ve read on the subject. I don’t think trend estimates using surface data starting from the 18th century are going to be in any way reliable. Does anyone else do this?
if the satellite warming trends since 1979 are correct,then surface warming during the same time should be significantly less,because moist convection amplifies the warming with height.- Roy Spencer
@KR
Why is Roy Spencer wrong?
Quinn the Eskimo – The SPPI paper you referenced contains data from 48 sites, chosen one per contiguous US state, meaning Texas and Rhode Island have the _same_ influence on the results. The word “weight” does not appear anywhere in the document, there is no accounting for the size of the regions represented.
It’s therefore unsurprising that their results differ from area-weighted estimates – a comparison of apples and oranges.
In regards to those UHI issues, I would point to the Berkeley work, http://berkeleyearth.org/pdf/berkeley-earth-uhi.pdf:
(emphasis added)
A note on SB equation and T^4. Given an approx. earth surface temperature variation of +/- 41C, in K that is a variation of 273K+/- 15%. 0.85^4= 0.522 and 1.15^4= 1.749 In other words, 3.5 times the heat flux at maximum than at minimum. Averaging temperature would not seem to be conducive to accurate results.
Just pick the raw data from the 50 sites which have the longest continuous data.
The problem is one doesn’t even know if the NCDC raw data really is the raw data.
If they have decided to adjust the records to help the global warming case, then they have also been mucking around in the raw temperature database as well.
So, nobody is really working with the raw unadjusted records.
DR – “…if the satellite warming trends since 1979 are correct,then surface warming during the same time should be significantly less…”
That’s a good point. I will note that satellite data is itself highly adjusted, based upon modelling of microwave emission from various points in the atmosphere, and has itself had numerous corrections (the UAH data initially showed cooling, not warming, until certain errors were noted and corrected). I suspect more corrections for things like diurnal drift will be applied in the future. There’s a good discussion of this in Thorne et al 2010 (http://onlinelibrary.wiley.com/doi/10.1002/wcc.80/abstract). Personally (opinion only here), although analyses such as Fu et al 2004 (www.nature.com/nature/journal/v429/n6987/abs/nature02524.html) have some statistical issues, I suspect the possibility of tropospheric temperatures being contaminated by uncorrected stratospheric signal are correct.
The radiosonde data shows trends since 1960 of 0.13C/decade surface, 0.16C/decade mid-tropospheric (http://www.ncdc.noaa.gov/sotc/upper-air/2010/13), incidentally, which is consistent with that tropospheric amplification by moist convection. There is also discussion on that page of uncorrected stratospheric influence in _both_ satellite sets.
KR: “urban warming does not unduly bias estimates of recent global temperature change.”
NASA: “Summer land surface temperature of cities in the Northeast were an average of 7 °C to 9 °C (13°F to 16 °F) warmer than surrounding rural areas over a three year period, the new research shows. The complex phenomenon that drives up temperatures is called the urban heat island effect.”
http://www.nasa.gov/topics/earth/features/heat-island-sprawl.html
“In July 2008, the central high building regions of Beijing have a monthly mean Tskin above 308 K (Fig. 1c). This is significantly higher than the surrounding non-urban regions where the cropland-dominated landscape has Tskin values in the 302-304 K range. The forests north of Beijing have Tskin as low as 298–300 K.”
http://www.met.sjsu.edu/~jin/paper/Tskin-UHI-jclimate-finalaccepted.pdf
If BEST can’t find UHI, the methodology or data or corrections or all three are broken.
Bill Illis,
If you think GHCN raw data isn’t really raw, try ISH, GSOD, WDSSC, etc. They will give you substantially the same results. Or take the Berkeley dataset and use only non-GHCN data and see what the results are. Here are three useful articles for reference:
http://wattsupwiththat.com/2010/07/13/calculating-global-temperature/
http://rankexploits.com/musings/2011/comparing-land-temperature-reconstructions-revisited/
http://judithcurry.com/2012/02/18/new-version-of-the-berkeley-earth-surface-temperature-data-set/
We’ve been around this so many times now, posted so many different codes by Jeff Id/Roman, Mosher, Chad, Nick Stokes, myself, Tamino, etc. If you take the raw data with no adjustments, you will get global results similar to NOAA/NASA. If you use non-GHCN data, you will get similar results to NOAA/NASA. If you use only rural stations (using any objective urbanity proxy, be it nightlights, impermeable surface area, MODIS, population density, population growth, etc.), you will get global results similar to NOAA/NASA. If you don’t trust me (and, like any good skeptic even if you do trust me you should go out and verify it yourself), I’d strongly suggest downloading the data sets (available here: http://berkeleyearth.org/source-files/ ) and create your own analysis. Its pretty trivially simple to convert the data into anomalies and to a simple gridding.
Is the v1 data available at GHCN? I can only see v2 and v3.
“The hardest thing to remove from science in order to obtain ‘pure science’ is the human element.”
E.M. Smith is a ‘pure’ scientist. He has given you the ‘pure’ results of his search and efforts. The rest is up to you. (Thanks again EM!)
Zeke Hausfather says:
June 22, 2012 at 7:42 am
“We’ve been around this so many times now, posted so many different codes by Jeff Id/Roman, Mosher, Chad, Nick Stokes, myself, Tamino, etc…”
____________________
That certainly clears up the whole thing for me.
Why does the actual temp even matter?
No need to nitpick… massaged or not, whatever temps we’re seeing are unprecedented and leave no doubt of anthropogenic origins, plunging beloved Mother Earth into catastrophe.
We must be made to pay.
/s
(just in case)
sunshinehours1 – Yes, urban areas are much warmer than rural areas. But what folks don’t seem to recognize is that the trend measurements shown by NASA/GISS, HadCRUT, NOAA, etc., are of temperature anomalies, not absolute temperatures. Changes in temperature from the long term average for each particular area.
So if the city is 8°C warmer than the nearby farmland, but has always been 8°C warmer – then that difference in temperature has exactly zero effect on the anomalies.
The only possible effect would come from the growth of urban areas – and quite frankly the percentage of the globe that has changed urbanization (although large in absolute terms) is quite small in percent area of the planet. And hence has very very little effect on temperature anomalies.
If you can take the raw data and get (substantially, as Zeke says) the same results why bother adjusting it?
KR says:
” if the city is 8°C warmer than the nearby farmland, but has always been 8°C warmer – then that difference in temperature has exactly zero effect on the anomalies.”
Given the increase in population density, decrease in green space over time, and increasing energy density, use, and specifically air conditioning, is it reasonable to assume the UHI differential would stay constant?
KR-
Are most reporting thermometers located in urban or rural areas?
Most folks don’t seem to realize….?
David Falkner,
Raw and adjusted global temps are close (~5% difference or so, depending on the timeframe), but adjustments can make a much larger difference on a regional level. They are also needed in many cases for regional climatological analysis. For example, I was working on a project the other week examining Chicago temperature trends. I kept getting some weird results where summer max temperatures were constantly above 40C prior to 1940 or so and almost never that high after, until I realized that the station I was examining had been moved from an urban rooftop to an airport around 1940. There simply aren’t enough “pristine” stations that have not had a move (or many moves), instrument change, time of observation change, etc. over the past century to do long-term analysis without trying to correct for these biases.
KR and others. Enough with the obfuscation!
What happened was a physical reality — it got warm, it got cold. Contemporary individuals tried to measure this using various instruments. There is one set of raw data. Two different processings of the data differ by an amount similar to the effect supposedly shown by the data. Global warming may exist. We may have caused it.
CO2 may be evil. What EM has said is that it is difficult to reasonably base public policy on an effect supposedly “proven” by a specific manipulation of this data set.
The criticism here ranges from reasonable scientific argument to ad hominem idiocy, but to me most is off point.
Why should we base massive changes in public policy on these data sets?
rilfeld says:
June 22, 2012 at 9:56 am
“Why should we base massive changes in public policy on these data sets?”
This is very simple rilfeld.
The left-wing, “green” climate scientists need money (lots of it!) and fame. Scaring people about climate gives them both. That’s all there is to it. They could care less about the public…
rilfeld says:
June 22, 2012 at 9:56 am
….
_____________
Your points are valid and well taken.
As far as ad hominem idiocy, I am guilty as charged.
When it became apparent long ago that reason and logic and valid scientific method held no sway with “the forces arrayed against us”, then nothing was left, but to stand on the side of the road and jeer at the emperor with no clothes.
With an eye to recent revelations about the data collection and classifications of free citizens as undertaken by our US govt. (and doubtless, same for our foreign friends, too), I can assume that my machinations have put me on some sort of list as being dangerous and a threat. The end, for those at whom I throw stones, has boundless justification of means.
KR at 6:45 am
The adjusted NCDC data don’t show any meaningful difference between urban and rural either. It’s the validity of the upward adjustment to the rural trend that results in that agreement that is in question.
As for Long’s selection criteria, as he explains it the overlay of the 5 degree by 5 degree grid is roughly 1 per state, except in the northeast. Long’s choice of one rural and urban per state for simplicity would overweight the northeast.
The Best study compares rural stations to all stations, not rural to urban as in Long, so the contrast would be muted in Best and heightened in Long. Also, Best has a much, much larger sample size, being global to Long’s lower 48.
Best is 1950 to 2010, Long is 1900 to 200x.
It is a curiosity, to say the least, that the UHI, which is an indisputable physical reality, is asserted to have no meaningful effect in temperature time series in which urban areas are substantially overrepresented. If Long is to be believed, in the contiguous US, the curiosity is explained by adjustments which quintuple the rural warming trend and lower the urban trend, bringing them into contrived agreement. In light of this, the multitude of other credibility problems afflicting these time series, the GHCN v1 to v3 differences discussed by Chiefio, and the USHCN v1 to v3 differences, I am a long way from being convinced the instrumental record has sufficient validity and reliability to support the AGW policy agenda.
Regards,
mfo says:
June 21, 2012 at 7:19 pm
A very interesting and thorough analysis by EM. When considering the accuracy of instruments I like the well known example used for pocket calculators:
0.000,0002 X 0.000,0002 = ?
[Reply: I hate you for making me go get a calculator to do it. ~dbs, mod.]
[PS: kidding about the hating part☺. Interesting!]
[PPS: It also works with .0000002 X .0000002 = ?
Also works for .0002 x .0002 (or any other number to that many decimal places) simply because the answer has more digits than the average calculator can…er…. calculate. .0002 x .002 just fits!
I would think it likely, KR, that the increase in the urban temperatures weren’t instantaneous but happened over time and may continue to do so. The average they are measured against is unmoving and obviously predates some or all of the warming. Your thoughts are wrong.