FTP access for Climate Reference Network Data

The CRN station at Mauna Loa, HI – I’m guessing the concrete pad was needed due to the cinders not being a stable anchoring medium.  So this station gets a CRN 5 rating. OTOH, as some readers point out, it is probably cooler than the cinders.

A couple weeks ago I promised that I’d post the ftp address where CRN data can be downloaded when it was made available to me, so here it is. Have at it lads.

Please read the details on the file format and caveats related to the data file timestamps.

NCDC provides ftp access to data from the Climate Reference Network (CRN) via anonymous ftp at the address

ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/hourly01 

(you can open this with most browsers)

here is a sample data file:

ftp://ftp.ncdc.noaa.gov/pub/data/uscrn/products/hourly01/ftpsample.txt

That directory contains a separate subdirectory for each year that the ftp data have been available, starting with 2005.  Each such subdirectory contains a collection of ASCII files named according to the following convention:

     CRNFFTT-YYYYMMDDHHmm.txt

where

       FF = 2-digit file frequency, in minutes (currently always 60)

       TT = 2-character file format number (currently always 01)

     YYYY = 4-digit year

       MM = 2-digit month (01=Jan, …, 12=Dec)

       DD = 2-digit day of month

       HH = 2-digit hour of day

       mm = 2-digit minute

Each one of these files contains CRN data that were loaded into the CRN database during the period of time lasting FF minutes and ending at UTC time YYYYMMDDHHmm.  The 2-character sequence TT indicates the file format number.  It is currently always 01.  If at some point in the future for format for CRN ftp data changes, this number will be changed.

Each file contains fixed-column formatted lines, each of which represents one hour’s observation data from one station.  The format of each line is as follows:

    cols 1 — 5 [5 chars] WBANNO

The station WBAN number.

    cols 7 — 12 [6 chars] COOPNO

The station COOP number.

    cols 14 — 21 [8 chars] UTC_DATE

The UTC date of the observation

    cols 23 — 26 [4 chars] UTC_TIME

The UTC time of the observation.

    cols 28 — 33 [6 chars] CRX_VN

The version number of the station datalogger program that was in

effect at the time of the observation.

    cols 35 — 41 [7 chars] LONGITUDE

Station longitude.

    cols 43 — 49 [7 chars] LATITUDE

Station latitude.

    cols 51 — 57 [7 chars] T_CALC

Average temperature, in degrees C, during the last 5 minutes of the hour.  Note: CRN stations have multiple co-located temperature sensors that record independent measurements.  This T_CALC value is a single temperature number that is calculated from the multiple independent measurements.

    cols 59 — 65 [7 chars] P_CALC

Total amount of precipitation, in mm, recorded during the hour.

Note: CRN stations have multiple independent measures of precipitation; this P_CALC value is a single precipitation number that is calculated from the multiple independent measurements.

    cols 67 — 72 [6 chars] SOLARAD

Average solar radiation, in watts/meter^2, for the hour.

    cols 74 — 74 [1 chars] SOLARAD_FLAG

QC flag for solar radiation.  0 means good, 2 means suspect, 3 means erroneous.

    cols 76 — 82 [7 chars] SUR_TEMP

Average surface temperature, in degrees C, for the hour.

    cols 84 — 84 [1 chars] SUR_TEMP_FLAG

QC flag for surface temperature.  0 means good, 2 means suspect, 3 means erroneous.

IMPORTANT NOTES:

1. All fields are separated from adjacent fields by at least one

    space.

2. Leading zeros are omitted.

3. Missing data are indicated by the value -9999 (or -9999.0).

4. The T_CALC and P_CALC fields do not have QC flags associated with them because they are values that are derived from the raw data (which is not included in these files).  When the raw data is flagged as erroneous or suspect, these derived values are not calculated — they are reported as missing instead.  So the T_CALC and P_CALC fields may be assumed to always be good (unflagged) data, except when they are reported as -9999.0, which indicates a missing value.

5. The files are named according to WHEN THE DATA WERE LOADED INTO THE CRN DATABASE AT NCDC, not according to observation times. Data from CRN stations generally arrives at NCDC and is loaded into the CRN database steadily throughout the hour, but sometimes an observation shows up several hours later, if there was a temporary problem with its transmission.  In some cases, if there is a more long-term problem with data transmissions from a CRN station, data from that station will be transferred manually via PDA, which can happen months after the data were

collected.  In any case, no matter how the data arrives at NCDC, it will show up in an ftp file that is named according to the time at which it was loaded into the CRN database.

This means that in general each .txt file will contain one hour’s worth of data, most of it for the previous hour.  There will occasionally be observations in a file that are from more than one hour in the past, however.  There are also sometimes hours during which no data is loaded at all (for example when the CRN database is down for maintenance for several hours on Sunday nights), which means that the .txt files for this period will be empty.  This is normal.  These periods will be followed by one or more hours in which the .txt files will be larger than average because the system is loading data that that came in over more than a one-hour period.