The "great dying of thermometers" – helping GISS find the undead thermometers, complete with code

Guest post by Walter Dnes.

There was phenomenon known as “the great dying”, where most of the Canadian stations being used for GISS world monthly temperatures disappeared, at least according to GISS.

In reality, many of these “dead” stations are still there, putting out data every month.  This

post is about finding additional Canadian monthly mean temperatures and anomalies from the Environment Canada website.

First, some administrivia…

– I’m a retired Environment Canada employee.  I do not speak for

Environment Canada.  This is all done in my capacity as a private

citizen.

– the following is all based on publicly available data from the

Environment Canada website as of late July, 2011.

– there are 2 versions of the code and data and scripts.  Each

clisum?.zip file is approximately 5 megabytes.  You will only need

one.  Both zipfiles contain BASIC source code, some scripts, and a

subdirectory with over 300 data files.  The  BASIC code is generic

enough that it should run under most BASIC versions on most platforms.

– The linux files are the definitive version.  I did a translation to

MS-DOS because most people still use Windows.  I haven’t been able to

fully test the DOS version, but I hope it works.  All file and

directory names comply with the old DOS 8.3 filename spec, for maximum

backwards compatability

– The files in clisumd.zip assume an MS-DOS or Windows environment.

This includes old-fashioned DOS, as well as a Windows DOS box command

prompt.  The files all have MS-DOS end-of-line, and the .BAT files

are written to be executed under COMMAND.COM or CMD.EXE.  I don’t

have a Windows or DOS machine, so I can’t be sure the BASIC code is

correct for QBASIC, or whatever version you may have.  Be sure to

edit the .BAT files to replace the command “bas” with whatever BASIC

interpreter/compiler you’re actually using.  You may need to tweak

some of the BASIC programs for minor syntax differences.  Click here

clisumd.zip to download the MS-DOS files if you run

DOS or Windows.

– The files in clisumu.zip assume a unix/linux/bsd environment.  They

all have unix end-of-line, and the scripts are written to be

executed under bash.  The BASIC programs were written for the “bas”

BASIC interpreter.  “bas” can be installed using your linux distro’s

standard install comand (e.g. “apt-get” in Debian/Ubuntu and

derivatives; “emerge” in Gentoo).  If your distro doesn’t have “bas”,

see http://www.moria.de/~michael/bas/ to download the source tarball

and build it manually.  Click here clisumu.zip to download the unix/linux/bsd files if you run any of unix/linux/bsd.

– there are links to various free BASIC interpreters and compilers at

http://www.thefreecountry.com/compilers/basic.shtml

Getting code and data

The first step is to download clisumd.zip (for Windows/DOS users) or clisumu.zip (for unix/linux/bsd users) and unzip it.  The result is a directory called either clisumd or clisumd.  Inside that directory are 13 or 14 files and a subdirectory “csvfiles” with over 300 CSV data files.

The next step is to download a monthly climate summary in text format.  With javascript enabled, go to webpage:

http://climate.weatheroffice.gc.ca/prods_servs/cdn_climate_summary_e.html

Select the desired month/year, Povince=”All”, Format=”Plain Text”. You should see something like this screenshot:

Once you get to this point, click on “Submit”

Save the resulting text webpage to a textfile in the clisumu or clisumd directory.  Since I used May 2011 data I named the textfile cs201105.txt. I also downloaded June as cs201106.txt.  You’ll want to download the latest month every month.  The data is generally available 7 to 10 days after the end of the month.

************************************************************************

** The following portion only needs to be run once for initial setup  **

** You do not have to do the next portion, including downloading 300+ **

** data files.  I’ve done it already and included the output in the   **

** zipfiles.  The following instructions are documentation for the    **

** sake of the scientific method, in case anybody wants to duplicate  **

** this the hard way.  The most likely use is in the case of manual   **

** over-rides, which I’ve found one case for so far.  There may be    **

** be other cases.                                                    **

************************************************************************

Creating a list of candidate stations with normals data

=======================================================

The next step is to create a subset file containing only sites with data in the most recent normals.  We actually want 1951-1980 normals for comparison to GISS.  Stations with current normals are candidates for having 1951-1980 normals, and their data will be downloaded.

We need to pick out only the lines with values in the “D” (Departure from normal) field, and copy only relevant data fields to a subset file. The program subset.bas is launched by the script “subset” in linux, or the batch file “subset.bat” in DOS.  The script/batchfile sets up the name of the input and output files as environment variables before launching subset.bas.

The program “subset.bas” scans for lines with column 64 == “.”.  This signals the presence of some temperature normals data for the period 1971-2000.  For only those lines, the climate ID, monthly mean temp, count of days with missing mean temp, and station name are extracted and written to a second file.  In the case of May 2011, subset.txt has

monthly mean temperatures for 313 sites which have normals for 1971-2000 to compute anomalies against.  In this example, I’ve called the output file “subset5.txt” to remind me that it’s for May.

The DOS batch file is invoked as…

subset

and the bash script is invoked as…

./subset

Because this only needs to be run once, I hardcoded the filenames into the batch/script files.

Downloading monthly data in CSV format

======================================

Unlike the 1961-1990 and 1971-2000 normals, the 1951-1980 Canadian climate normals do not appear to be on the web.  But since the monthly data is available online for downloading, we can do the calculations ourselves, after downloading the monthly data.  Here is how the data was downloaded…

We search by station name.  The first line in subset5.txt is…

“1012055”,9.5,17,48.829,-124.052,”LAKE COWICHAN”

The climate data advanced search page is at…

http://www.climate.weatheroffice.gc.ca/advanceSearch/searchHistoricData_e.html

Use the “Search by Station Name:” menu as shown in the screenshot:

Enter the name, or a portion thereof, as shown in the red rectangle.  Note that upper/lower case doesn’t matter, and spaces are ignored.  Thus “lakecow” matches “LAKE COWICHAN”.  Then click on the “Search” button as shown by the red arrow in the

screenshot.  Alternately, you can press “Enter” on your keyboard.  This takes you to the search results page as shown in the screenshot:

We run into a problem here… there are two stations named “LAKE COWICHAN”, which does happen on occasion.  It’s not until you actually select a station that you find out if you’ve got the right one.  To select monthly data, you must first select “Monthly” in the drop-down menu under “Data Interval”, and then click on the “Go” button

corresponding to the station you want.  You’ll get a display similar to the screenshot:

I’ve highlighted a couple of areas.  At the upper left is the climate ID in a red rectangle.  This must match the climate ID at the beginning of the line in the subset file, unless you’re doing a manual over-ride (more about this later).

The red arrow at the bottom right corner points to the link for downloading the data in CSV format.  I right-clicked on that link and saved the file to the csvfiles directory.  My convention is to name the file after the climate ID.  Thus, this one would be “1012055.csv”.  Note that this is merely a label for convenience only.  The files could be assigned any legal filenames, and the BASIC programs would still work, because they read the climate ID from data inside the csv data files.

Rinse/lather/repeat the above for all 300+ lines in the subset file. Now you know why you don’t want to repeat this yourself.

Now for the manual over-ride example.  Look in the climate summary file cs201106.txt.  Station “WINNIPEG RICHARDSON AWOS” with climate ID “5023226” has a mean monthly temperature, but it does not have normals data.  Searching for “winnipeg” in the climate data advanced search page yields several Winnipeg sites.  If you click “Go” on “WINNIPEG RICHARDSON AWOS” you’ll see that it’s located at 49 55’N and 97 14’W and

elevation 238.7 m.  Go back to the Winnipeg search results page, select “Monthly” and click “Go” for “WINNIPEG RICHARDSON INT’L A”.  You’ll notice that it’s located at 49 55’N and 97 14’W and elevation 238.7 m. They’re at EXACTLY the same location.  Why the split reporting, I don’t know.  Anyhow, I downloaded the CSV monthly data with filename

“5023222.csv” to the csvfiles directory.  Then I opened it with a text editor, and changed the 6th line from

“Climate Identifier”,”5023222″

to

“Climate Identifier”,”5023226″

This causes the BASIC programs to treat the monthly data as belonging to the AWOS site when computing monthly normals.  Thus we will get monthly temperature anomalies versus 1951-1980 for the AWOS site, even though it’s relatively new.

Calculating the monthly normals

===============================

The normals batch/script file needs to be run only when the contents of the csvfiles subdirectory change.  This includes individual files being added, deleted, or edited.

The program normals.bas opens a CSV file for input, and the normals file in append mode.  It then calculates the normal temperature for one station, appends one line of data and exits.  It is called serially by a FOR loop in the normals shell script or normals.bat batchfile, once for each file in the csvfiles directory.  Since lines are always being appended to normals.txt, the script deletes the normals file before starting the loop.  This starts off with a clean slate.  The script then sets the name of the normals file, and the value of the normals start and end years, and then loops through all the files in the csvfiles

directory that match the spec “*.csv”.  The file is invoked in unix/linux/bsd as…

./normals

and in a DOS box (including Windows) as…

normals

Because of limitations in the DOS FOR command, normals.bat has a couple of extra steps…

1) The bash FOR command sorts filenames when evaluating “*.csv”, which results in the file normals.txt being in sorted order.  The DOS FOR command doesn’t do this.  The workaround is to write output to a scratch file (normals.000) and sort that file to normals.txt at the end.

2) The bash FOR command accepts multiple commands in a DO/DONE block.

The DOS FOR command doesn’t do this.  It has to be a “one-line-wonder”. The workaround is to make the one command a CALL to a 2nd DOS batch file, namely normals2.bat.  normals2.bat has the multiple commands to execute.

Note that normals and normals.bat set STARTYR=1951 and ENDYR=1980. This is because the immediate goal of this project is to generate normals to compare against GISS, which happens to use 1951..1980 as its base period.  There’s nothing preventing anybody from using 1961..1990, or any other random base period for that matter.

The output format for the normals file is… Columns 1-10  The 7-character climate ID in quotes, followed by comma

This is followed by 12 repetitions (1 for each month) of…

SNN.N,NNN,

Where “SNN.N” is the monthly mean temp, with a minus sign if needed,

“NNN” is the number of years of data for that month

************************************************************************

** This finishes the portion that only needs to be run once for       **

** initial setup.  The following is run every month after downloading **

** the latest monthly climate summary from Environment Canada.        **

************************************************************************

Calculating the monthly temperature anomalies

=============================================

Because the monthly data will be calculated using different filenames and months, the anomaly batch/script files accept parameters.  The first parameter is the month as a number from 1 (January) to 12 (December). The second parameter is the name of the monthly climate summary file that you’ve downloaded from the Environment Canada website.  Note that the program *ALWAYS WRITES TO THE SAME OUTPUT FILE NAMES*.  If you want to keep anomaly output files, and not have them overwritten, rename them before doing the next run of anomaly files.  I’ve included 2 sample

monthly climate summaries, cs201105.txt for May 2011, and cs201106.txt for June 2011.  An example invocation for June 2011 data is, in DOS…

anomaly 6 cs201106.txt

and in bash

./anomaly 6 cs201106.txt

There are 2 output files.  anomaly0.txt has output for every station with a monthly mean temperature in the climate summary, and a line in the normals file.  anomaly1.txt only has those lines for which…

a) the month’s summary data shows zero days missing, and

b) there are 30 years of data for this month in the normals

This is intended as a filter to list only the best data for consideration.  You can relax the criteria if you wish, by modifying anomaly.bas.

An example line from the anomaly outputs is…

“1021830”, 15.1, 0, 15.0, 30,  0.1,49.717,-124.900,”COMOX A”

The fields are…

Climate ID

Actual monthly mean temperature this month

Number of missing days this month

Normal monthly mean temperature

Number of years of normals data for this month

Temperature anomaly versus the normal value

Station latitude

Station longitude

Station name

This output is suitable for importing into a spreadsheet, and

especially into a GIS program for plotting.

***********************************************************************