Quality Control of pre-1948 Cooperative Observer Network Data

I stumbled across this paper “Quality Control of pre-1948 Cooperative Observer Network Data” on the AMS website while looking for something else. I know the work of many of the people who authored this paper, and thus I have reason to believe it is well done for the issues they were attempting to address. However, one thing that was not addressed, and really needs to be, is a site bias timeline. While it will be difficult to go back and find photos of weather station siting for the entire network, there does exist one resource already in NCDC’s hand that could be utilized for the task: B44 forms. These are site surveys, completed by USWB, and later NWS personell that set up these COOP stations. Each B44 includes a site sketch to scale, such as this one from a station in California:

Putting together these B44 forms over the life of a station would enable the creation of a timeline that shows building and road proximity biases that may exist, which are just as important as catching the errors that creep into the human transcription of the data.

Unfortunately, the B44 forms are not available to the public because of NCDC’s privacy concerns for observers. I can appreciate that. However that just doesn’t hold up anymore because NCDC provides all the tools needed on their Metadata MMS website now to identify the location of the observer, including a Google image map and accurate lat/lon, plus on their “free data” section of the NCDC website, you can download the B91 station originals. These are the daily record of observations done in the observer’s own hand, and have the name of the observer written on them in most cases. They are the source of the US climate data and the focus of the QC paper named above.

A sample of the B91 Record of Climatological Observations Form – click for larger image

Here is another B91 form, done by a private observer I happen to know in Livermore, CA His name is listed at the bottom. Put his last name, “Livermore”, and “weather” together in this Google search, and you get all the details you need to proceed. You can even see his picture courtesy of the NWS COOP awards page here.

It is now an easy task to locate anyone and any address given a name, Google, public records, and mapping programs like Google Earth. Since the “privacy issue” no longer exists due to this volume of info available on individual observers, perhaps it is time for NCDC to free the B44 forms.

Here are some excerpts from the paper:

The National Weather Service’s (NWS) cooperative observer network (COOP) is the core climate network of the U.S. In operation since the late 19th century, it consists primarily of volunteer observers using standard equipment provided by the NWS. The typical suite of elements, observed daily, include precipitation, maximum temperature, minimum temperature, snowfall, and snow depth. Some stations report only precipitation variables. A few stations observe other variables such as pan evaporation and soil temperature.

These observations were routinely digitized beginning with 1948. Although there have been occasional projects to retroactively digitize selected data (e.g. Kunkel, et.al. 1998), most pre-1948 observations remained available only on paper or microfiche. This
has recently changed. The U.S. Congress has provided funding to the National Climatic Data Center (NCDC) for the Climate Database Modernization Program (CDMP 2001). The goal of CDMP is to convert data only available in hard-copy form to computerized formats.

The pre-1948 COOP data was one of the first data sets chosen for this conversion.
The authors have undertaken a project to quality control this data set. This paper describes the QC procedures and discusses certain aspects of the data set.

And also:

There are a number of potential sources of errors in the data set. Some primary examples include observer errors in reading the instruments, observer errors in writing the observations on the form, liquid separation in the thermometers, and legibility of the forms. There are also a number of potential issues with continuity of the data for each station due to changes in instrumentation, observing practices, and exposure.

The primary purpose of the QC for this project was to identify the largest errors in individual values, particularly those that might affect analyses of extreme events.

About these ads

12 thoughts on “Quality Control of pre-1948 Cooperative Observer Network Data

  1. “Since the “privacy issue” no longer exists due to this volume of info available on individual observers, perhaps it is time for NCDC to free the B44 forms.”

    Anthony,
    Perhaps a FIOA request from someone (not you) would help things along? Steve seems well versed at these, though not in this country, but I’m sure there are those among us that could help frame such a request.

    JimB

  2. Please , no more snow. I suffer from ADD and the snow is so distracting that I can’t concentrate on reading the articles. Thank you.

  3. Janet,
    If you shrink your screen from side to side and eliminate the borders, the snow all but dissapears.

    Jim

  4. Anthony,
    A note for general interest. When considering precipitation records keep in mind that readings at most stations are done at 7AM local. Any rain falling after that time gets recorded the next next morning. When transitioning from one month to the next this may be the difference that determines whether there’s been a record drought or record rainfall. The Altoona B-91 is a good example. The 0.04 rainfall entered on the 1st may have actually fallen in April. ASOS recorded data is generally for midnight standard time. In most cases this wouldn’t make a difference but in this hyped up world – who knows?

  5. No more snow indeed – Houston got a pretty good dusting last night. http://www.chron.com for pics – first snow in 4 years.

    In other news Al Gore was sighted at the Houston Hobby airport waiting for a connecting flight to somewhere..

  6. It seems that the quality control portion might be a very hard task to perform due to the possibility of the forms having been lost by agencies using them for study and number crunching. I can only imagine the embarrassment for some agencies that have been very vocal about the amount of warming if the original data doesn’t agree with the “corrected” data that they are using to expand their programs. Times are bad and getting worse I have an idea that the people paying for this will take a dim view if there has been any chicanery in the process. I think some of our overzealous advocates are going to be in trouble especially if they are incharge of keeping the records.

    Bill Derryberry

  7. @Janet Rocha
    Disable Javascript to get rid of the snow. IMHO the easiest way is to get Firefox (if you don’t already), and then use the NoScript add-on to enable/disable it on a per-site basis.

  8. Bill in Vigo (06:05:34) :
    “It seems that the quality control portion might be a very hard task to perform due to the possibility of the forms having been lost by agencies using them for study and number crunching.”

    I was under the impression that NCDC’s QC was performed early-on after receipt and before being released for any other purpose. And unless the old memory is failing more than I am aware of (a distinct possibility), errors found for Air Weather Service Observations were in my day returned to the originating site for final correction. Perhaps someone here has a different memory.

    As a comment on the supplied examples, I fail to see that any really significant QC is possible at NCDC.

    Further, I was under the distinct impression that “original” records were only maintained for a short time at NCDC these days (5-10 years) due primarily to lack of storage space.

    I remember several decades ago hearing that NCDC had 23 warehouses filled with original weather records. Unfortunately, I no longer remember the source of that revelation.

  9. I’m aware of the NCDC gaffe last year whereby they restricted the access to the metadata in the name of “privacy concerns”. This may well be the same sort of issue, however the federal law may not be easily dismissed because the data is available elsewhere. That doesn’t preclude making redacted versions of the forms available. But bear in mind that the are harsh penalties for a bureaucrat that improperly releases e.g a social security number. The Privacy Act has teeth for a reason, and provides for criminal penalties:

    “Any officer or employee of an agency, who by virtue of his employment or official position, has possession of, or access to, agency records which contain individually identifiable information the disclosure of which is prohibited by this section or by rules or regulations established thereunder, and who knowing that disclosure of the specific material is so prohibited, willfully discloses the material in any manner to any person or agency not entitled to receive it, shall be guilty of a misdemeanor and fined not more than $5,000.” 5 U.S.C. § 552a(i)(1).

Comments are closed.