Contribution of USHCN and GISS bias in long-term temperature records for a well-sited rural weather station

Guest post by David W. Schnare, Esq. Ph.D.

When Phil Jones suggested that if folks didn’t like his surface temperature reconstructions, then perhaps they should do their own, he was right. The SPPI analysis of rural versus urban trends demonstrates the nature of the overall problem. It does not, however, go into sufficient detail. A close examination of the data suggests three areas needing address. Two involve the adjustments made by NCDC (NOAA) and by GISS (NASA). Each made their own adjustments and typically these are serial, the GISS done on top of the NCDC. The third problem is organic to the raw data and has been highlighted by Anthony Watts in his Surface Stations project. That involves the “micro-climate” biases in the raw data.

As Watts points out, while there are far too many biased weather station locations, there remain some properly sited ones. Examination of the data representing those stations provides a clean basis by which to demonstrate the peculiarities in the adjustments made by NCDC and GISS.

One such station is Dale Enterprise, Virginia. The Weather Bureau has reported raw observations and summary monthly and annual data from this station since 1891 through the present, a 119 year record. From 1892 to 2008, there are only 9 months of missing data during this 1,404 month period, a missing data rate of less than 0.64 percent. The analysis below interpolates for this missing data by using an average of the 10 years surrounding the missing value, rather than basing any back-filling from other sites. This correction method minimizes the inherent uncertainties associated with other sites for which there is not micro-climate guarantee of unbiased data.

The site itself is in a field on a farm, well away from buildings or hard surfaces. The original thermometer remains at the site as a back-up to the electronic temperature sensor that was installed in 1994.

The Dale Enterprise station site is situated in the rolling hills east of the Shenandoah Valley, more than a mile from the nearest suburban style subdivision and over three miles from the center of the nearest “urban” development, Harrisonburg, Virginia, a town of 44,000 population.

Other than the shift to an electronic sensor in 1994, and the need to fill in the 9 months of missing reports, there is no reason to adjust the raw temperature data as reported by the Weather Bureau.

Here is a plot of the raw data from the Dale Enterprise station.

There may be a step-wise drop in reported temperature in the post-1994 period. Virginia does not provide other rural stations that operated electronic sensors over a meaningful period before and after the equipment change at Dale Enterprise, nor is there publicly available data comparing the thermometer and electronic sensor data for this station. Comparison with urban stations introduces a potentially large warm bias over the 20 year period from 1984 to 2004. This is especially true in Virginia as most such urban sites are typically at airports where aircraft equipment in use and the pace of operations changed dramatically over this period.

Notably, neither NCDC nor GISS adjusts for this equipment change. Thus, any bias due to the 1994 equipment change remains in the record for the original data as well as the NCDC and GISS adjusted data.

The NCDC adjustment

Although many have focused on the changes GISS made from the NCDC data, the NCDC “homogenization” is equally interesting, and as shown in this example, far more difficult to understand.

NCDC takes the originally reported data and adjusts it into a data set that becomes a part of the United States Historical Climatology Network (USHCN). Most researchers, including GISS and the East Anglia University Climate Research Center (CRU) begin with the USHCN data set. Figure 2 documents the changes NCDC made to the original observations and suggests why, perhaps, one ought begin with the original data.

The red line in the graph shows the changes made in the original data. Considering the location of the Dale Enterprise station and the lack of micro-climate bias, one has to wonder why NCDC would make any adjustment whatever. The shape of the red delta line indicates these are not adjustments made for purposes of correcting missing data, or for any obvious other bias. Indeed, with the exception of 1998 and 1999, NCDC adjusts the original data in every year! [Note, when a 62 year old Ph.D. scientist uses an exclamation point, their statement is rather to be taken with some extraordinary attention.]

This graphic makes clear the need to “push the reset button” on the USHCN. Based on this station, alone, one can argue the USHCN data set is inappropriate for use as a starting point for other investigators, and fails to earn the self-applied moniker as a “high quality data set.”

The GISS Adjustment

GISS states that their adjustments reflect corrections for the urban heat island bias in station records. In theory, they adjust stations based on the night time luminosity of the area within which the station is located. This broad-brush approach appears to have failed with regard to the Dale Enterprise station. There is no credible basis for adjusting station data with no micro-climate bias conditions and located on a farm more than a mile from the nearest suburban community, more than three miles from a town and more than 80 miles from a population center of greater than 50,000, the standard definition of a city. Harrisonburg, the nearest town, has a single large industrial operation, a quarry, and is home to a medium sized (but hard drinking) university (James Madison University). Without question, the students at JMU have never learned to turn the lights out at night. Based on personal experience, I’m not sure most of them even go to bed at night. This raises the potential for a luminosity error we might call the “hard drinking, hard partying, college kids” bias. Whether it is possible to correct for that in the luminosity calculations I leave to others. In any case, the lay out of the town is traditional small town America, dominated by single family homes and two and three story buildings. The true urban core of the town is approximately six square blocks and other than the grain tower, there are fewer than ten buildings taller than five stories. Even within this “urban core” there are numerous parks. The rest of the town is quarter-acre and half-acre residential, except for the University, which has copious previous open ground (for when the student union and the bars are closed).

Despite the lack of a basis for suggesting the Dale Enterprise weather station is biased by urban heat island conditions, GISS has adjusted the station data as shown below. Note, this is an adjustment to the USHCN data set. I show this adjustment as it discloses the basic nature of the adjustments, rather than their effect on the actual temperature data.

While only the USHCN and GISS data are plotted, the graph includes the (blue) trend line of the unadjusted actual temperatures.

The GISS adjustments to the USHCN data at Dale Enterprise follow a well recognized pattern. GISS pulls the early part of the record down and mimics the most recent USHCN records, thus imposing an artificial warming bias. Comparison of the trend lines is somewhat difficult to see in the graphic. The trends for the original data, the USHCN data and the GISS data are: 0.24,

-0.32, and 0.43 degrees C. per Century, respectively.

If one presumes the USHCN data reflect a “high quality data set”, then the GISS adjustment does more than produce a faster rate of warming, it actually reverses the sign of the trend of this “high quality” data. Notably, compared to the true temperature record, the GISS trend doubles the actual observed warming.

This data presentation constitutes only the beginning analysis of Virginia temperature records. The Center for Environmental Stewardship of the Thomas Jefferson Institute for Public Policy plans to examine the entire data record for rural Virginia in order to identify which rural stations can serve as the basis for estimating long-term temperature trends, whether local or global. Only a similar effort nationwide can produce a true “high quality” data set upon which the scientific community can rely, whether for use in modeling or to assess the contribution of human activities to climate change.

David W. Schnare, Esq. Ph.D.

Director

Center for Environmental Stewardship

Thomas Jefferson Institute for Public Policy

Springfield Virginia

===================================

UPDATE: readers might be interested in the writeup NOAA did on this station back in 2002 here (PDF, second story). I point this out because initially NCDC tried to block the surfacestations project saying that I would compromise “observer privacy” by taking photos of the stations. Of course I took them to task on it when we found personally descriptive stories like the one referenced above and they relented. – Anthony

Get notified when a new post is published.
Subscribe today!
0 0 votes
Article Rating
196 Comments
Inline Feedbacks
View all comments
February 28, 2010 6:15 am


Nick Stokes (14:32:28) :
Jim, the old thermometers could tell you the peak in a 24hr period.

Do tell; I think this is painfully obvious to all but a novice in this field.
A reference would be appreciated; not for me, but for the novices …


But when they are read has an effect.

Please explain this, for the novices; this isn’t RC.
.
.

Tim Channon
February 28, 2010 7:09 am

What I think they are getting at…
At 4pm you read the max thermometer and then reset it.
The following day is colder.
At 4pm the next day you read the thermometer _which is showing a max from 4pm the previous day, the same as when you reset it. Yesterday was hotter at 4PM than the peak today.

February 28, 2010 8:06 am


Nick Stokes (14:32:28) :

That meant that on a hot day, the peak would be counted twice, with a genuine peak in mid-afternoon say, and the temp just after the reading (and re-setting) being the peak for the next day.

Oh, okay. I was hoping for a little more supporting logic and rationale (a ‘value added post’ rather than regurgitation of texts) but I can see its the old “hot day bleed-over effect” bleeding from one day’s noted observation to the next.
Let’s look at the case where … each day has same profile for ‘warming’ in the diurnal cycle (each day) AND the fixed point in time where the reading was care fully chosen. The result: Equal _peak_ temperatures for adjacent days on our min-max thermo … again if things were *perfect* (no chaotic weather).
I’ll grant you: adjacent days could see equal peaks.
This falls apart as far as ‘gathering data’ is concerned in the real world though –
Consider a long time series (one where the “law of large numbers” would apply), with a rigidly adhered-to sampling schedule (or even a random about a centroid point in time), with each time period representing a ‘bin’, where recording the min and max for that bin period results in systematic skewing of data (computed averages) to one side or the other?
EVEN IF that theoretical point for reading the temperature in the afternoon could be chosen at the inflection point where the temperature is at its peak, there would be steadily decreasing (or increasing) sets of ‘peak’ temperature throughout the year … where does this skew the results given the LLN ( Probability: Law of large numbers ) .
Correct me where I am wrong, but please do it with reference to sampling theory, noted observations in nature or demonstration by experiment.
.
.

February 28, 2010 8:15 am


steven mosher (21:34:44) :

Karl’s TOBs adjustment is an empirical model with data held out for verification as I recall. been a couple years since I read it

Hide the recline; a fall-back position (“It’s not who votes that counts. It’s who counts the votes.” – As seen in Diebold ad parody) …
.
.

February 28, 2010 8:45 am


Kevin Kilty (17:17:30) :
I would just like to add one more comment to what Nick Stokes said above. The NCDC data set comes from COOP stations (at least that is my interpretation) and we do not know the daily cycle at these stations. All we have is a time series of max-min values. This is why I am pretty skeptical of the TOB bias correction that NCDC applies.

WHEN would I (were I a volunteer reader) read (and record) my min max thermo, gee, in the evening, after getting home (were it not in the morning) and that has worked out to sometime after 7 PM local, which is well after the assumed (and not in dispute) peak temp of the day.
Quoting from Aguado and Burt “Understanding Weather & Climate” 3rd edition, pg 81, “the warmest period of the day … is sometime in the early afternoon, usually between 2:00 and 4:00 PM.”
So, the assertion above by some of the min/max reading in late afternoon (an assumption, perhaps a gross assumption “the assumption of scale”) would result in a peak less than the peak of the actual peak that day …
Also, I think COMMON SENSE (stemming from in situ observation of the data) would prevail after awhile, after discussion amongst peers, ag agents, et al – these volunteers didn’t just fall off a turnip truck and start recording temperature, they have/had an interest in weather, and parametrically (measurement of parameters, like temperature).
.
.

Kevin Kilty
February 28, 2010 10:26 am

steven mosher (21:34:44) :
Kevin Kilty (17:17:30) :
Karl’s TOBs adjustment is an empirical model with data held out for verification as I recall. been a couple years since I read it

Your memory is correct, but the period of study was still just a few year period in the 1960s and I have wondered about how well this makes the adjustment for all time periods.
Anthony: Your comments tacked onto mine have shown me this TOB bias correction is more insane than I thought. I wonder if we can find someone to help out who has cross disciplinary work in both psychology and climatology? What’s more, there are other problems beyond TOB — like making adjustments out of order and smearing error all over via homogenization.

Tim Channon (07:09:06) : and _Jim:

Yes you guys have this correct now. Maybe I could have explained this better, but the concept is not all that easy to explain. The correction NCDC tries to apply to this is even harder to visualize — read Tom Karl’s paper. ”A Model to Estimate the Time of Observation Bias Associated with Monthly Mean Maximum, Minimum, and Mean Temperatures.” by Karl, Williams, et al.1986, Journal of Climate and Applied Meteorology 15: 145-160.
actually a good place to start is Donald G. Baker, 1975, Effect of Observation Time on Mean Temperature Estimation, J. Applied Meteorology, 14, 471-476.

_Jim (08:45:43) :
Kevin Kilty (17:17:30) :
I would just like to add one more comment to what Nick Stokes said above. The NCDC data set comes from COOP stations (at least that is my interpretation) and we do not know the daily cycle at these stations. All we have is a time series of max-min values. This is why I am pretty skeptical of the TOB bias correction that NCDC applies.

WHEN would I (were I a volunteer reader) read (and record) my min max thermo, gee, in the evening, after getting home (were it not in the morning) and that has worked out to sometime after 7 PM local, which is well after the assumed (and not in dispute) peak temp of the day.
Quoting from Aguado and Burt “Understanding Weather & Climate” 3rd edition, pg 81, “the warmest period of the day … is sometime in the early afternoon, usually between 2:00 and 4:00 PM.”
So, the assertion above by some of the min/max reading in late afternoon (an assumption, perhaps a gross assumption “the assumption of scale”) would result in a peak less than the peak of the actual peak that day …

Have a look at Tom Channon’s posting. He is dean-on about the reason and sign of the bias. The thermometers catch the max-min values of a 24 hour period, but because of individual reading schedules, the 24-hour periods do not coincide among all stations.
Better yet look at Donald Baker’s paper, which I referenced above in this post, and you’ll see how the examination of actual data shows the sign of the bias. It is very interesting.

Also, I think COMMON SENSE (stemming from in situ observation of the data) would prevail after awhile, after discussion amongst peers, ag agents, et al – these volunteers didn’t just fall off a turnip truck and start recording temperature, they have/had an interest in weather, and parametrically (measurement of parameters, like temperature).

I am not saying these people were ignorant in any way. However, they were collecting data for a very different purpose than the climate researchers are putting the data to now. The biases mattered not one jot to them, nor did anyone even imagine how this bias could enter long-run records at the time. The new use for this data makes the bias important.

Kevin Kilty
February 28, 2010 10:29 am

Sorry Tim, “Tim Channon” not Tom Channon in the previous post.

February 28, 2010 11:02 am

Thank you for a substantive post Brian.

_Jim (09:50:56) : “Are we…measuring increased convective activity vis-a-vis higher reported satellite temperatures…?”
“…MSU’s aboard those sats are also going to see the result of convective activity, i.e., precipitation in its varied forms, which are more reflective of temperature seen at altitude and not the boundary layer or ground.”
Brian Dodge (15:59:08) : Yes we do see increased convective activity highly correlated with SST.

Well, not arguing that point, but rather: Is the satellite AMSU measuring the increased tropospheric temperature as a result of the release of latent heat during during the processes that result in precipitation (wholesale condensation, release of latent heat, etc)?


The powerful advance in technology of the multiple spectral channel data from the Aqua and Terra Atmospheric Infrared Sounder (AIRS) and Advanced Microwave Sounding Unit (AMSU) is that we see the temperature from the bottom up, and can tell which signal is coming from where. The algorithms to sort it all out aren’t trivial, and

Amen.
Quoting Shakespeare “ay, there’s the rub”; the tuning, the tweaking, the adjustments of coefficients and factors (and perhaps powers) in those non-trivial algorithms …
.
.

steven mosher
February 28, 2010 9:09 pm

_Jim (08:06:40) :
You can see how TOBS works by going over to CA.
climateaudit.org
search on TOBS
find the thread named TOBS
Look at the comments.
Find my comment commenting on JerryBs work.
Download the data.
Analyze for yourself.
The data is there. the work is simple.
we’ve been over this ground before.

steven mosher
February 28, 2010 9:11 pm

Kevin Kilty (10:26:59) :
For a long time back in the day I kinda lobbied for people to have another look at TOBS. Not because it doesnt make sense.. in theory.

steven mosher
February 28, 2010 9:15 pm

Nick Stokes (21:45:44) :
I was doing some background reading on TOBS the other day and ran across an approach that di the adjustment WITHOUT ANY METADATA.
But my old brain is failing and I can’t recall the paper. Anyways, I thought that would be a cool thing to look at. Not much of a hint on where to look, sorry

Eric Gamberg
February 28, 2010 11:04 pm

” _Jim (05:50:21) :
Eric Gamberg (17:57:08) :
I think you miss the point that to be able to use data from a given station to determine
Not at all Eric; levity, (excessive or unseemly frivolity) yes, miss the point, no.”
I still think you are tollously missing the point. Certainly only humorous to you.

Eric Gamberg
February 28, 2010 11:23 pm

” steven mosher (21:11:25) :
Kevin Kilty (10:26:59) :
For a long time back in the day I kinda lobbied for people to have another look at TOBS. Not because it doesnt make sense.. in theory.

Given that most sites in the NH exhibit a bigger max-min in the Spring than in the Fall and that there is a daily temp rise in the Spring and fall in the Fall (!!! who knew?), TOB adj should easily be a function of the day of the year for a given lat. and long.

E.M.Smith
Editor
March 1, 2010 12:16 am

Kevin Kilty (07:51:41) : One does not have to do a lot of research to figure that things have gone wrong–for example doing homogenization before UHI corrections is simply wrong and this is what NCDC does.
It is also what GIStemp does. “Homogenization” is done in STEP1 and UHI is done in STEP2… I had not thought about it, but you are quite right. That is a very silly order of action…
John F. Hultquist (09:20:08) : Okay, that’s not fair. You had me laughing. Then this “not humor” bit. Now I feel like I just laughed at a funeral.
I think of it more like laughing at one of those “funniest videos” films where someone “not so bright” managed to mangle various body parts, repeatedly… You don’t want to laugh. You know someone was injured. But when they launch off the ramp on a bike into a mud pit with gators and hit the 4 x 4 next to the ramp, then spin into the mud, landing ON the bike… well, it’s just a bit hard not to snicker… but it’s wrong 😉
So while they did not intend their actions to be humor, it’s perfectly OK to laugh in one of those “What WERE you thinking?” kind of ways… after all, the alternative is weeping uncontrollably…

D. Patterson
March 1, 2010 2:37 am

steven mosher (21:11:25) :
Kevin Kilty (10:26:59) :
For a long time back in the day I kinda lobbied for people to have another look at TOBS. Not because it doesnt make sense.. in theory.

Steve, I would argue that you don’t even need to look at TOBS at all to conclude the Daily Summary air temperatures are too erroneous and unreliable to use for a determination of a Global mean air temperature (assuming for the sake of argument such a global mean can exist).
Using only the 24 hour minimum and maximum air temperatures, with fixed or variable observation periods, is non-representative of the true average daily air temperature. The TOBS adjustments no matter how well accomplished can only compound the already egregious errors in the daily observation of only the two extremes.
This problem is easily illustrated by simply taking stations with more than 24 observations of air temperatures, 36 or more are better, and computing the simple averages for all air temperature observations in the daily period and then the typical synoptic subsets of the same observations for the 1-Hourly, 3-Hourly, 6-Hourly, and once daily MIN-MAX-MEAN datasets. The different methods typically yield differences in daily average air temperatures up to 1F or more before any TOBS or other adjustments can be applied. TOBS simply adds to the errors and the unrealities when used with only a single daily observation of MIN-MAX-MEAN.

1DandyTroll
March 1, 2010 8:11 am

How much, if any, does the luminosity change when street lamp design is changed to lessen light pollution, especially in the upwards direction, and also when switching to more energy efficient but more environmental hazardous lamps?
Is the negative change equally dramatic the additive change is when they first light up a newly built 50 km freeway?

March 1, 2010 5:17 pm


Eric Gamberg (23:04:42) :

I still think you are tollously missing the point. Certainly only humorous to you.

Eric, on the slimmest of evidence, on the basis of a single post evidently used as a ‘proxy’ for the sum of my life (and endeavors, discoveries) to date, you arrive at this conclusion … so, I have to ask: “Climate Science much?”
A little trick I picked up from Mosher (A/K/A Moshpiy); a YouTube video, since Klaus sez (sic) it better:
[youtube=http://www.youtube.com/watch?v=d-Yrg9xNSS0&hl=en_US&fs=1&]
.
.

March 1, 2010 5:26 pm


steven mosher (21:09:05) :
You can see how TOBS works by going over to CA.
climateaudit.org
search on TOBS
find the thread named TOBS
Look at the comments.
Find my comment commenting on JerryBs work.
Download the data.
Analyze for yourself.
The data is there. the work is simple.
we’ve been over this ground before.

Then … there must be a theorem worked up; other than the RC-style ‘go find it in the references’ answer you just gave.
Something based on ‘sampling’ or binning theory, min-max peak detect concepts? C’mon … you’re a systems analyst kinda guy …
Any accommodation for instrumentation response (over time; as when a a min/max thermo is reset, how much time does it take to repeat if any?)
How about the recognition by field personnel of this facet, leading to self-directed resetting of (some) instruments? (Argument getting thin at this point)
.
.

March 1, 2010 5:51 pm

Not sure this took 1st time … try again …


steven mosher (21:09:05) :
You can see how TOBS works by going over to CA.
climateaudit.org
search on TOBS
find the thread named TOBS
Look at the comments.
Find my comment commenting on JerryBs work.
Download the data.
Analyze for yourself.
The data is there. the work is simple.
we’ve been over this ground before.

Then … there must be a theorem worked up; other than the RC-style ‘go find it in the references’ answer you just gave.
Something based on ‘sampling’ or binning theory, min-max peak detect concepts? C’mon … you’re a systems analyst kinda guy …
Any accommodation for instrumentation response (over time; as when a a min/max thermo is reset, how much time does it take to repeat if any?)
How about the recognition by field personnel of this facet, leading to self-directed resetting of (some) instruments? (Argument getting thin at this point)
.
.

Mike Rankin
March 1, 2010 8:50 pm

After reading this post and some comments I did some investigation of my own. JamesS at 16:35:58 on 02/26/2010 gave a link to
http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.html
I used the link and selected Virginia on the drop down menu and clicked on “Map Sites”. This shows the cluster of surface stations in VA and brings up a listing on the right of the stations. I clicked on Dale Enterprise and the map brought up a bubble of Dale Enterprise showing key info plus offering options. I clicked on “Get Daily Data”. This brought up a page with numerous further opportunities to get information. I scrolled down to the bottom of the page and found means to obtain a CSV file of data. I clicked on boxes for Temperature min and Temperature max as well as the data quality flags for each. I next clicked on the submit button. After a few seconds I was taken to a page showing a link for the file containing the data. Clicking on this brought up a form that allowed me to select “Save File”. I did so and clicked “OK”. This copied the file to my Download folder (XP). I started a session in open office calc and opened the downloaded file.
The file had daily records for max and min. According to some information on other pages, this data is not adjusted … yet. I imagine that adjusting daily data taken in integer degrees F would not be easy. Over the past three days I have spent a lot of effort to examine the data up close and personal. The raw data certainly shows major weather patterns constantly changing. I programmed in open office to re-arrange the data into yearly sheets.
I subsequently downloaded a file of the monthly data for Dale Enterprise by using the link provided by David Schnare at (20:29:56) : 02/26/2010. I selected “Raw GHCN data + USHCN corrections” and entered “Dale Enterprise” for the station. I clicked on the link and clicked on the link “Download monthly data as text”. I imported the data into open office. I programmed open office to give an equivalent of the “Raw GHCN data + USHCN corrections” from the daily data. I programmed open office to give a month by month difference. There appears to be substantial pattern in the differences. I found a huge difference in the data for Jan 1904. I would share the difference report but have no means of presenting it.

March 11, 2010 5:44 pm

For those interested, here are the papers on how the GISS surface temperature analysis was originally designed and how – and why – raw station data are adjusted:
http://pubs.giss.nasa.gov/docs/1987/1987_Hansen_Lebedeff.pdf
http://pubs.giss.nasa.gov/docs/1999/1999_Hansen_etal.pdf
http://pubs.giss.nasa.gov/docs/2001/2001_Hansen_etal.pdf and
http://data.giss.nasa.gov/gistemp/updates/
Makes for dense reading, but if you want to tell the Climate Change wheat from the chaff, you gotta RTFR!

1 6 7 8