My UCAR slideshow and station quality ratings online

With 33% of the USHCN weather station network now surveyed, the site quality rating is now applied, see the USHCN Station Master List file in HTML and XLS format.

The rating system for site quality was borrowed verbatim from the new Climate Reference Network being put into operation by NCDC and NOAA to ensure quality data. Their siting criteria can be found here.

I welcome input on this work in progress. The site rating will now be a running total in the spreadsheet and always available online as new stations are added to the survey. What is important to note is that the majority of stations that have a rating of 4 are MMTS/Nimbus equipped stations, which according to NCDC’s MMS equipment lists, make up 71% of the USHCN network. It appears that cable issues with the electronic sensors have forced them closer to buildings, roads, etc because NOAA COOP managers don’t often have the budget, time, or tools to trench under roads, sidewalks etc to reach the site where Stevenson Screens once stood. While this isn’t always the case, a pattern is emerging.


For background, see this first: Conference presentation given at CIRES/UCAR on 8/29/07 describing this project and the methods used to assign station site quality ratings, along with examples of many site issues seen thus far.

Click to view the slideshow I presented at UCAR

Immediately after the conference, a senior official at NCDC requested a copy of the above slide show, which I provided to him on CDROM. After receiving it, in a follow up email he inquired as to distribution rights which I granted within NCDC and NOAA for the purpose of review. That was last week. Thus far no issues have been raised with the presentation content. Since no issues were raised at the conference or in the two weeks afterwards (two weeks as of today) I have decided to release it publicly.

Note that of the 33% surveyed, only 13% meet the CRN site criteria (Rating of 1 and 2)for an acceptable location to accurately measure long term climate change free of localized influences.

The CRN site rating system is described here:

Climate Reference Network Rating Guide – Class 1 and 2 are considered best, 5 is the worst.

Class 1 – Flat and horizontal ground surrounded by a clear surface with a slope below 1/3 (<19deg). Grass/low vegetation ground cover 3 degrees.

Class 2 – Same as Class 1 with the following differences. Surrounding Vegetation 5deg.

Class 3 (error 1C) – Same as Class 2, except no artificial heating sources within 10 meters.

Class 4 (error >= 2C) – Artificial heating sources <10 meters.

Class 5 (error >= 5C) – Temperature sensor located next to/above an artificial heating source, such a building, roof top, parking lot, or concrete surface.”

0 0 votes
Article Rating
Newest Most Voted
Inline Feedbacks
View all comments
Douglas Hoyt
September 12, 2007 12:44 pm

Although Woodstock VA gets a rating of 2, it has only been in its present location for a little over one year. Perhaps it would be helpful to add another column after the rating to list number of years the site has been in exactly the same location.

September 12, 2007 12:49 pm

I inferred this from your text, but could you be explicit in your post as to whether 5 is the worst or the best?

September 12, 2007 12:56 pm

Anthony, I looked at your presentation. It is really an excellent bit of work – congratulations. I presume the next step is to sort the temperature data from these sensors into these five categories to see what correlations might exist between the category and the mean and standard deviations of the measured historic temperature changes.

September 12, 2007 1:20 pm

Wow, I didn’t expect this. Only 4% meet all specifications for a good station.

Anthony Watts
September 12, 2007 1:24 pm

GTTofAK, actually it’s a total of 13% as ratings of 1 and 2 would be considered acceptable as I understand it.

George M
September 12, 2007 1:51 pm

Great presentation. You let the pictures speak for themselves. I think the success of this effort can be assured by continuing to do just that. Present the information, photographs, etc., while doing the associated research off-line and not making any controversial unsubstantiated claims. Save the conclusions for when the majority of the data is in, and then do not extrapolate beyond the obvious. I suspect your audience was stunned with what they saw, and are even now making lots of phone calls to the pictured sites asking “What the….?” The downside of this is they have now been warned of what is coming, and the spin doctors are revving up in preparation for the end game. The wheels are beginning to fall off the AGW bandwagon, and given tha amounts of money involved, it is not going to be a pretty sight.
Side note to Coyote: 1 is best, 5 is worst.

Evan Jones
September 12, 2007 2:58 pm

HOly CaNOle, Rev!

September 12, 2007 4:37 pm

That “1” is best and “5” worst is clear from the site criteria. Where does it say that anything over a “2”
is not acceptable?

September 12, 2007 4:53 pm

At some point we need to see the average temperature increase for stations in each class, eg the average temp increase for 1, for 1 +2, for 1 +2 +3, etc.

Anthony Watts
September 12, 2007 4:58 pm

BCL, That is a valid question, but perhaps you aren’t familiar with the concept of “signal to noise ratio”.
In any measurement environment, there will always be a certain amount of noise in the measurement, whether you are measuring temperature, voltage, sound, or water flow.
The object of any properly designed measurement system is to have a signal to noise ratio that is far lower than the least significant digit or resolution of measurement. In this case with temperature, while the readings done by observers are at 0.1°C resolution when the observer makes the reading, they are rounded by the observer in recording it into the data logbook which is then sent to NCDC. Thus the resolution of the USHCN data is 1°C with no tenths.
Therefore, as any scientist or engineer knows (or should know), you want the noise component (error) to be LOWER than the least significant digit of the measurement, the resolution of 1°C in this case.
Since Class 1 and 2 have less than 1°C errors by this scheme, they would be considered acceptable. Class 3, 4, and 5 have errors (noise) that is equal to or greater than the 1°C resolution of the temperature measurement, and thus may mask or bias upwards or downwards, the signal that is trying to be extracted (the trend over time). So far the trend value that has been put forth for the worldwide surface temperature record is a positive 0.6°C, but given that we have a measurement system that may have many stations with errors greater than that value it calls the accuracy into question.
If you go look at the new NCDC approved CRN climate monitoring sites, with photos of them visible in this document:
…you’ll see that they are Class 1 and 2 stations. The CRN was setup correctly, and I would trust the data it produces. Unfortunately, there hardly is any data yet and the CRN network is scheduled for completion in 2008.

September 12, 2007 5:11 pm

Is Waldo hiding inside the Class 3 through 5 error bars? (and subsequent fitting done on such data? Is Waldo a sort of curve fitting Mahout – e.g. able to fit an elephant?)

September 12, 2007 5:55 pm

Where in the official siting criteria does it talk of “signal to noise” ratio?
In other words, is the criteria of acceptability that of the NOAA itself, or YOUR criteria (perhaps backed by “Any Scientist or Engineer”) applied to their stations?
(Put aside what the criteria SHOULD be. Who knows? Maybe you’re right about that.)
By the way, you WERE right about the deleting of site information. I had conflated “location” with “Address”. I had to take that post down.

Evan Jones
September 12, 2007 9:21 pm

Noise? Noise?
Yes, let’s make some!

Chris Kaiser
September 12, 2007 9:56 pm

It appears that you have a large enough sample size now to do some meta analysis. I.e., what is the trend of just the class 1 and 2 sites? I think that is the $64,000 question.

Evan Jones
September 12, 2007 11:04 pm

Rate of change in 1s and 2s vs. that of 4s and 5s.
That is the gold speck. Not the levels. The comparative rates of change.
Will 4/5 the results jibe with the rate of exurban creep which has been devouring the rural stations for a century? Will they match the adjustment numbers of the CRN? One would expect so.
Will this swamp the alleged 0.6+ C increase? It’s a distinct possibility. If so, that cuts the legs out from under the entire premise.
And the code. Let’s not forget that. From what I figure, it seems that the GISS adjustments are pretty much ass-backwards. Talk about your double-biases!
Not to mention that little Titanium Oxide V. Calcium Carbonate deal–that looks like the equivalent of a Level 4 violation on the remaining Stevenson stations, right off the top. And I’ll hazard a guess that the above chart doesn’t even take that into account.
Brrr. I think I need a vicuna coat!

September 13, 2007 6:22 am

So now you’re saying that the new CRN acceptability guidelines are also implicit? They don’t actually STATE that only class 1 and 2 stations are “acceptable”?
It makes a big difference for you to claim that the stations fail to meet THEIR OWN standards, and to claim that they fail to meet YOUR standards (and this whether or not YOUR standard makes sense). I have trouble with you reading a standard of “acceptability” into documents where such a standard is not explicitly set forth.

September 13, 2007 11:44 am

BCL, think about it this way.
According to the CRN, a class 5 site has a potential error of 5 degrees. Imagine reprogramming the MMTS device to only read out whole temperatures that were divisible by 5. (i.e. when it was 60, 61, or 62.4, it would read out “60.”) If someone told you that using such a device they measured a 0.6 degree rise in average temperature, would you believe it?

September 13, 2007 12:54 pm

Congrats Anthony,
With 33% of the data in, your audit has demonstrated that most of the sites do not meet the CRN standards, and therefore are not suitable to be included in Climate Change analyses. I think that if that was indeed the goal, you’ve succeeded with only 33% of the site surveyed.
So my question now, is what is the purpose for continuing the site surveys? I don’t expect the further sites to change the distrubtion significantly.
May I humbly offer, in a proactive way, the raison d’etre to complete the survey is to help the CRN identify those sites which are candidates as”gold standard” for climate change analysis? Such sites would be in your categories 1 & 2, but with additional criteria, that have come out of the experience of performing site surveys.
Using Walhalla as an example, Sites that are rural, have historical imagery and/or other data to verify land use hasn’t changed significantly over the last 50-60 years, have a good record any instrument change or relocation, etc… I think from your experience you can come up with a good list of additional criteria to be imposed on the “gold standard” sites.
Also, proactively, you could suggest to the NOAA, additional things that need to be done to maintain the “gold standard” site list such as performing annual site checkups to verify compliance.
All in all you’ve done a wonder!
ps, did you get the Walhalla aerial images?

Larry Grimm
September 13, 2007 2:05 pm

Bigcitylib: I wholly concur with Anthony’s comments. You cannot get meaningful data from ANY scientific measurement if the signal to noise ratio is poor. It really is impossible.
There is a lot more involved with obtaining good data such as counting statistics, measurement techniques and sample size. However, what Anthony is doing is just looking at the measuring instrument. This is usually one of the first things a decent scientist should look at to determine if he/she is going to have a successful data collection.
There are two types of noise to ratio problems: intrinsic and extrinsic.
Scientists will explicitly tell what device they used for the measurement because an engineered/manufactured device will have its signal to noise ratio in the device’s specifications, which can be referenced by other scientists. This is an example of an intrinsic factor that must be considered in designing the data collection. Intrinisic factors are generally the easiest to control.
However as Anthony shows in his pictures, there are also extrinsic factors that can infuence the data collection, e.g. the thermometer sits next to an air conditioner. Extrinsic factors can be difficult to control. It is one of the reasons so many “correlation” studies are worthless because not all the extrinsic factors have been accounted for.
If I allowed the error factors within my work, as seen in Class 3-5 sites, I would get creamed by the regulators, my boss and anyone who reviewed my work.
Let me pose the question back to you – would you trust the data from a class 4/5 site? How about looking at it another way – would you want to measure your child’s temperature with a thermometer that may read 5 degrees high or low? And then make grave decisions from the (faulty) data? Oiks!

Evan Jones
September 13, 2007 4:17 pm

One would want to test it. The data has been terribly manhandled, and it all needs a good going over.
Naturally, a rating-by-rating evaluation of the raw data may prove helpful as a guideline.
Unless we get the data right and adjusted as well as we openly can, we won’t be getting anywhere on this issue.
P.S., I’ve looked in on your site and I note: a.) You identify yourself, and b.) You seem to tolerate adverse opinion pretty well; yes, you can dish it out but you can take it.

Robert in Calgary
September 13, 2007 6:38 pm

I’ll add my congratulations to Anthony here and to Steve at Climate Audit for the valuable work you’re both doing. As someone who has seen BCL at Canadian political blogs, BCL always misses the basic premise. (sigh) I’ll leave it at that before I become too impolite.

September 13, 2007 8:51 pm

RE: Posted by: Anthony Watts | September 13, 2007 10:48 AM
I am a scientifically degreed, high technology professional with over 25 years of experience. Anthony’s description of SNR is one of the best I’ve seen in quite a while. I wholeheartedly concur.

Evan Jones
September 14, 2007 3:27 am

Consider what’s going on here.
The gummint is replacing possibly well sited old Steve stations with the Type-4 sited Nimbus variety.
In goes the bad air out goes the good.
The big deal here is that so many of these these are RECENT changes. The implications for the recent historical “trend” is obvious.
I think the thing about this that really throws me for a loop here is the lack of decimal points.
Not POINT ONE degree to the warm, ONE degree. Not POINT TWO degrees, TWO degrees. Not POINT FIVE degrees, FIVE degrees!
And not an up-down variance, but primarily a bias to the warm! (If the “noise” is all one tone, in one direction it may be easier to filter it out, right?)
And the “significant difference” we’re supposed to be shutting down the engine of the world on account of is, what, a measley POINT 6 or 7 degrees!
Say, WHAT?
THAT margin of error doesn’t even feed the bulldog for one of my half-assed historical models!
“It is an outrage. I shall tell everybody.”

September 14, 2007 3:36 am

Anthony and others, you may be right about signal to noise ratios and etc., or someone like Jay Lawrimore may be right when he says that all these deviations from perfect siting are accounted for and corrected. Here we can argue about what makes an acceptable station.
But this is an entirely different argument from saying that according to CRN guidelines, or according to the NOAA’s OWN guidelines, these stations are not acceptable. If you can show that the stations break established guidelines, than it doesn’t MATTER what those guidelines are,you’ve got an immediate check-mate as it were.
Insofar as Anthony is making this second argument, he is not entitled to it until he can point to some CRN/NOAA guidelines that these stations break such that they are rendered unacceptable according to these VERY SAME guidelines.
Thus far he has been unable to do so, and I suspect he will not be able to from NOAA documents, as their apparent attitude to low quality stations is to upgrade them when they are moved (can’t find the quote as yet).

Anthony Watts
September 14, 2007 5:37 am

BCL thanks for the clarification of what your argument is about, it made it easier to understand what you were saying.
Here then is a document that shows a clear statement of proper siting for the US COOP network, of which USHCN is a subset.
This link has been on my “resources” page on since June.
Note the statement:
“The sensor should be at least 100 feet from any paved or concrete surface.”
That last line about “should be 100 feet from” corresponds to the distance boundary of a CRN Class 2 station which is 30 meters. 30 m = 98.4251 ft. (rounded up to 100 feet).
From the CRN Spec: “Class 2 – Same as Class 1 with the following differences. Surrounding Vegetation less than 25 centimeters. No artificial heating sources within 30m.”
The 100′ standard NOAA/NWS published for proper siting in their COOP (and USHCN) network is rooted in the need to obtain a signal to noise ratio that is lower than the least significant digit of measurement.
So from the NOAA/NWS coop spec, it is clear that 100 feet/30 meters is the exclusion zone for potentially biasing elements near a thermometer. That makes class 1 and 2 station sites “proper” by that published standard and Class 3, 4, 5 falling in the “not proper” category. Or put another way, “acceptable and unacceptable”.

September 14, 2007 6:32 am

Anthony, the 33% mark is an awesome milestone. Of course, some in the blogosphere will claim that the volunteers simply cherry-picked the worst one third of the USCHN stations!

September 14, 2007 6:58 am

While clear standards for rating the stations and careful scrutiny are required, this nitpicking on whose guidelines are violated is looking like an attempt to distract from the main point that there has been significant failure to minimize site biases at many locations. Use any standard you want, use multiple standards. Just be clear about what they are, how they’re applied, and what their strengths and weaknesses are. Rating scales are subjective at the core, anyway.

Chris D
September 14, 2007 7:00 am

I would say that qualifies as an “immediate check-mate”.

steven mosher
September 14, 2007 12:15 pm

Fool’s mate, chris D

September 14, 2007 12:20 pm

Anthony, what I find more interesting is this statement from the siting criteria you link to:
“The selection of a USCRN instrument site will be the result of a balance
between competing demands, such as those highlighted above and an assessment of the “quality
of measurements” guidelines outlined below.
There will be many sites that are less than ideal. The USCRN will use the classification scheme
below to document the “meteorological measurements representativity” at each site.”
If class 3 4 and 5, the “less than ideal” sites, are not acceptable, then there would be no need of giving information pertaining to them. This seems to pretty clearly indicate the USCRN considers 3, 4s, and 5s to be capable of providing usable information (with corrections, obviously).
In fact, you might say that all you’ve discovered is that indeed many sites are less than ideal.
I won’t beat this dead horse much longer, but will probably write something up on it over the weekend.

Jeff Wood
September 14, 2007 2:00 pm

BCL, you are very carefully not getting the point.
The standards associated with these stations were not set by Anthony, Steve or anyone here: they were set by CRN/NOAA.
It follows that the records returned by Class 1 stations are not comparable to say, Class 5.
If you will kindly look at the bloody photos, and the associated graphs, perhaps it will come to you that you have been had.

Anthony Watts
September 14, 2007 2:19 pm

“This seems to pretty clearly indicate the USCRN considers 3, 4s, and 5s to be capable of providing usable information (with corrections, obviously). ”
Here we have BCL advocating for sloppy science. Wow, BCL, that’s really over the top. Show us then, just where are those CRN correction methods for microsite biases, and then be sure to show us all those CRN sites that are 3, 4, and 5. The onus is on you now.
You’ll write whatever you want to write to prove the point you want to make, but, you’ll be wrong, just like before. Defending sloppy data gathering isn’t a defensible position. Why do you think CRN was built in the first place?
Using BCL’s logic lets play a thought experiment or two:
Voting: Don’t worry about ballot boxes stuffed with random ballots, we’ll use sophisticated computer programs to mathematically weed out the errors afterwards to get the true count.
Your water bill: We’ll leave all the water main leaks unfixed and then apply the total water draw to your bill, then “adjust” it later based on equations that will reveal the true water use at your home.
Your electric bill: “Don’t worry about the neighbor who leaves the lights on 24/7 and runs the a/c 24/7, we’ll average all power use in your neighborhood grid, homogenize it and then give you an mathematically adjusted bill. In fact, we only need one meter for the neighborhood grid now.
The grocery store: Some of our meat and vegetable scales aren’t calibrated, some read high, but don’t worry about that, we’ll just average the good ones and bad ones and use that figure to adjust the weight for the filet mignon and portobello mushrooms you just bought. We have an equation for that to ensure you get an accurate weight.
BCL your agenda is showing

Larry Sheldon
September 14, 2007 2:52 pm

I am a nobody. A reasonably well-read nobody, but a nobody none the less.
But maybe I can help “Bigcitylib” with an analogy with which she might be more familiar.
Consider a relatively quiet room where somebody is trying to read a report into the record of some proceeding.
Some people (who it happens don’t want the record to reflect the report, but that is irrelevant here) at measured intervals start to shout and scream (and perhaps blast air horns or something). The result of what is in the resulting recording is at risk of being in error–it will be unreliable and any reliance on it will yield even more errors.
Kinda like standing on the shoulder of a highway with the assignment of counting the cars in the center lane when the near lane has a steady stream of large trucks in it.

September 14, 2007 3:40 pm

Anthony (and Steve McI),
Congratulations on what you’re doing.
I don’t know what you’ll finally come up with when all is said and done. But you’ve accomplished one thing for sure – you’re holding a significant element of the scientific community to a rigorous audit.
They will now need to defend their AGW “thesis” before a very public panel of astute statistians and computer data analyzers. (And all you Masters and PhD candidates thought you had it tough with your theses!). A cleansing breath of fresh air in an environment where the AGW High Priests are demanding: “don’t look at our numbers – you just gotta trust us to decide your futures in your own best interests based on our unique, and opaque, system of data analysis”.
The bottom line – the AGW High Priests might be right on one issue – there might really be climate warming. It’s just that their database can’t prove it since it’s contaminated. And you hold out the possibility that there is no warming.
But also, it calls into question the credibility of the entire AGW scientific lobby’s claim to objectivity.
Warning – the Powers-That-Be will be all over you guys.
But you’re going to get a SCREAMING LOT OF PRESS AND WEB EXPOSURE!!!! Thought the Y2K revelation got traction? Wait a few weeks! I pity you guys – you’re going to get no rest.

Evan Jones
September 14, 2007 7:45 pm

>>”This seems to pretty clearly indicate the USCRN considers 3, 4s, and 5s to be capable of providing usable information (with corrections, obviously). “>>
Fine. Do it both ways:
1.)with good stations and
2.) with station adjustments. With a very close eye out as to WHEN those violations occurred.
It make a good cross-check. If the numbers don’t jibe then one might be able to isolate the problem.

September 14, 2007 7:50 pm

One point your missing is how these sites affect each other. Sites are used to adjust other sites that have biases and discontinuites. Since no one had documented the site quality, low quality sites were, due to their preponderance, were most likely being used to adjust higher quality sites thereby degrading those sites and making the entire calculation suspect. The 4’s and 5’s may be useful but only if they are known and used appropriately which they couldn’t have been since no one had done a survey.

September 14, 2007 9:04 pm

your comment about “voting” has actually is proposed every four years, usually in urban areas where certain agreived groups don’t vote in proportion to their population. Advocates for these agreived groups have been demanding for years that the undervote be estimated and added to the vote totals.
That’s why they call it “political” science!

steven mosher
September 15, 2007 10:38 am

I’ve posted this before, but it bears repeating.
The effects of microclimate issues are well established. I’ll give some links later. One of the first studies done with the CRN was a CRN to ASOS comparison; essentailly, since future sites in the CRN are expected to last 50-100 years they need to be calibrated against past sites.
So, a CRN site and a fairly pristine ASOS site were selected and a side by side study was conducted
If you don’t want to wade through the whole paper, I’ll reproduce the last Paragraph.
“At the Asheville site, the effect of siting difference
between the ASOS and CRN led to a ∆Tlocal effect of
about 0.25o C, much larger than the ∆Tshield effect (about
-0.1o C). This local warming effect, caused by the heat
from the airport runway and parking lots next to the
ASOS site, was found to be strongly modulated by
wind direction, solar radiation, and cloud type and
height. Siting effect can vary with different locations
and regions as well. This term, undoubtedly, needs to
be taken into account in the bias analysis if two
instruments of interest are separated by a significant
So, this isnt Anthony, or SteveMc, or the Mosh pit, this is NOAA. Calibrating the CRN. Calibrating it against an ASOS, which probably ranks as a class 1 or class 2.. AND EVEN HERE they find a measureable bias. So, in fact the 1-5 ranking scale may not be stringent enough.
Anthony, I’m wondering if any ASOS can be a class 1.
Start here. Then everything else makes sense

September 16, 2007 4:45 am

Am waiting for a response from CRN before I post re ANthony’s mis-use of their guidelines. Meanwhile, here’s an update on how Anthony and Steve’s work is panning out:

Evan Jones
September 16, 2007 8:53 am

BCL I don’t see your problem.
It seems to me we got a great field-test here.
I don’t think you’ll find a lot of folks around here who are not in favor of continual open evaluation
All we have to (as has been previuosly suggested by others) is measure Temperature Levels and [Rates of Change Vs Violation accumulation] of the 5 ratings.
That will provide hard data either supporting or disputing CRN evaluations.
Results to be open and available–as opposed to being filed in the Lost Ark under “Worthy Eyes Only”.
Heck, the Rev has been trying to get the ear of NCDC, anyway. The CRN should embrace the effort.
So, Where’s the misuse?

Anthony Watts
September 16, 2007 1:41 pm

Misuse? This is your baileywick, your words, Big City Lib, and nothing I could say would change that, since your mind has been made up on that since you started this thread. Again you are making up things to suit your own purpose. Your history on that hasn’t been too good.
So responding to your points BCL, is a pointless exercise at this time. Have your fun, but be careful.
While you are at it BCL, do a writeup on why Hansen switched the data around on GISS a couple of days ago with no notice. Now THATS misuse.

Kristen Byrnes
September 17, 2007 6:11 pm

It seems like some one jumped to conclusions. After a closer look at JohnV’s data it seems that the unadjusted USHCN data and GISS adjusted data is what is matching. That means that GISS did not adjust out any of the errors caused by UHI, Microsite effects, station moves, changes in equipment and so on.

%d bloggers like this:
Verified by MonsterInsights