An update on my Climate Reference Network visualization project

Readers may recall that when Peter Gleick first sent out the stolen Heartland board documents and the fake document to give the story “legs”, there was a mention in there of a project that I asked Heartland to help me fund. They found a private donor who was interested in my idea, and agreed to fund it. The amount of funding I received, based on the plan I submitted, was $44K, which when compared to many scientific papers and projects that get funded, is a proverbial drop in the bucket. For example, Michael Mann got a boatload of money: Mann’s $1.8 million Malaria grant – “where do we ask for a refund’? to study something outside his normal area of expertise – mosquitoes and malaria.

And as you recall, when the Heartland documents were made public by the “publish first, ask questions later” tabloid reporters at the Guardian and other news outlets, there was much wailing and gnashing of teeth by the usual suspects, claiming that somehow I’m in the employ of big oil, big coal, big tobacco, yada yada yada. One of the worst offenders was local eco-activist and Chico State University professor Mark Stemen who wrote to me with similar ludicrous charges, calling me a “Koch whore” and adding that “I’m making sure that everyone in town knows it” followed by posting a series of smears on his Facebook page. Stemen’s abusive smearing was typical of the over the top unprofessional and adolescent behavior that was directed at me for daring to do this project related to making climate data more available to the public.

Today, I’m going to offer my first update on it, so that everyone can see for themselves just how terrible and evil it is in the scheme of all things climate.

The goal of this project is to provide a publicly accessible one-on-one live comparison of temperatures between GHCN and other hourly reporting stations from the older surface network, to the new Climate Reference Network (CRN). The impetus was the heat wave in Texas last year, where I noticed that while there were a number of record setting high temperatures, many of them were higher than temperatures seen in the CRN. This suggested to me that UHI and siting effects play a role in elevating such temperatures. Unfortunately at that time there was no easy way to offer such visual comparisons, and I thought there should be, hence my idea that I asked Heartland to help me find a funding source for.

Here’s how Heartland Described my project in one of the stolen documents:

Weather Stations Project

Every few months, weathermen report that a temperature record – either high or low – has been broken somewhere in the U.S. This is not surprising, since weather is highly variable and reliable instrument records date back less than 100 years old. Regrettably, news of these broken records is often used by environmental extremists as evidence that human emissions are causing either global warming or the more ambiguous “climate change.”

Anthony Watts, a meteorologist who hosts WattsUpwithThat.com, one of the most popular and influential science blogs in the world, has documented that many of the

temperature stations relied on by weathermen are compromised by heat radiating from nearby buildings, machines, or paved surfaces. It is not uncommon for these stations to over-state temperatures by 3 or 4 degrees or more, enough to set spurious records.

Because of Watts’ past work exposing flaws in the current network of temperature stations (work that The Heartland Institute supported and promoted), the National Aeronautics and Atmospheric Administration (NOAA), the government agency responsible for maintaining temperature stations in the U.S., has designated a new network of higher-quality temperature stations that meet its citing specifications. Unfortunately, NOAA doesn’t widely publicize data from this new network, and puts raw data in spreadsheets buried on one of its Web sites.

Anthony Watts proposes to create a new Web site devoted to accessing the new temperature data from NOAA’s web site and converting them into easy-to-understand graphs that can be easily found and understood by weathermen and the general interested public. Watts has deep expertise in Web site design generally and is well-known and highly regarded by weathermen and meteorologists everywhere. The new site will be promoted heavily at  WattsUpwithThat.com. Heartland has agreed to help Anthony raise $88,000 for the project in 2011.  The Anonymous Donor has already pledged $44,000. We’ll seek to raise the balance.

Terrible huh?

Since it wasn’t clear in the Heartland description above what data I was going to use, for those who don’t know, here’s some background on the Climate Reference Network, this entry from Wikipedia:

The US Climate Reference Network (USCRN) Program was begun in 2002 and remains under construction with an expected completion date in late 2008. It has the long-term commitment of the Department of Commerce (DOC) and the National Oceanic and Atmospheric Administration (NOAA). This network will be maintained and modernized during the life of the program.

This is a program that will collect and analyze the highest quality climate data possible. Research based on these data will directly support near and long term policy and decision plans made by senior government and business leaders.

The research community, government agencies, and private businesses have identified significant shortcomings in understanding and examining long-term climate trends and change over the U.S. and surrounding regions. Some of these shortcomings are due to the lack of adequate documentation of operations and changes regarding the existing and earlier observing networks, the observing sites, and the instrumentation over the life of the network. These include inadequate overlapping observations when new instruments were installed and not using well-maintained, calibrated high-quality instruments. These factors increase the level of uncertainty when government and business decision-makers are considering long-range strategic policies and plans. Never before have people been so aware of the impact of the environment and climate variability and change on the quality of life and the economic health of a nation, its citizens, and the population of the world. This project will serve as a model for establishing similar networks in other countries.

Basically, the CRN is NCDC’s response to their realizations of problems in the existing climate observing network, something that I’m long since identified in my own surfacestations.org work, including a peer reviewed paper on the subject. Plus, my conclusions about the problems with the surface network in the USA were backed up by an investigation done by the US General Accounting Office. An example of the kind of problems the many surfacestations.org volunteers found are illustrated well by the fact that some stations used specifically to measure climate, were in the most ridiculous locations, like this USHCN station in downtown Ardmore, Oklahoma:

Ardmore USHCN station , MMTS temperature sensor, January 2009

While NCDC has gone to great lengths to defend the quality of the USHCN network and the highly adjusted data it produces, their actions of closing them (after we pointed out numerous train wrecks like Ardmore) speak far louder than written words and peer reviewed publications:

Ardmore USHCN station , MMTS temperature sensor removed, March 2011

The new CRN has none of these problems. Sites were meticulously chosen, temperature sensors have triple redundancy, there’s a QC process for data collection, and most important, since the data is not measured once a day (as observers did with high and low readings) but is done continuously, there’s no need for the Time Of Observation Adjustment, which we know adds significantly to the overall temperature trend of data used for climate study.

Note that the TOBS adjustment (in black) amounts to the lions share.

Here’s how NCDC describes the CRN:

The U.S. Climate Reference Network (USCRN) consists of 114 stations developed, deployed, managed, and maintained by the National Oceanic and Atmospheric Administration (NOAA) in the continental United States for the express purpose of detecting the national signal of climate change. The vision of the USCRN program is to maintain a sustainable high-quality climate observation network that 50 years from now can with the highest degree of confidence answer the question: How has the climate of the nation changed over the past 50 years? These stations were designed with climate science in mind. Three independent measurements of temperature and precipitation are made at each station, insuring continuity of record and maintenance of well-calibrated and highly accurate observations. The stations are placed in pristine environments expected to be free of development for many decades. Stations are monitored and maintained to high standards, and are calibrated on an annual basis. In addition to temperature and precipitation, these stations also measure solar radiation, surface skin temperature, and surface winds, and are being expanded to include triplicate measurements of soil moisture and soil temperature at five depths, as well as atmospheric relative humidity. Experimental stations have been located in Alaska since 2002 and Hawaii since 2005, providing network experience in polar and tropical regions. Deployment of a complete 29 station USCRN network into Alaska began in 2009. This project is managed by NOAA’s National Climatic Data Center and operated in partnership with NOAA’s Atmospheric Turbulence and Diffusion Division.

More here. Given the advanced way it is measured, there’s no need to adjust the CRN station data whatsoever.

In addition, many USHCN stations are being modernized and added to the CRN network as I’ve described in detail here: What the modernized USHCN will look like.

Overall I’m pleased with that CRN project and the USHCN modernization, and I endorse it. But little of the data from it is finding its way into the public realm, and I aim to change that.

The first job was to arrange for and to program data ingestion. Initially it looked like the project was designated to be done with an Internet based FTP fetching, which can be fraught with problems related to network delays, timeouts, server load, etc. Fortunately it was discovered that the entire CRN data set was delivered on an hourly basis via one of NOAA’s satellite feeds. This was a godsend, because I worried about NCDC trying to cut off web access (like they did once before) when they decided they didn’t like what I was doing or if my bandwidth usages got to be too high. Similar issues plagued Steve McIntyre when he went to fetch a large amount of data once, the Gavinator of GISS blocked him. So unless NOAA/NCDC decides to pull the CRN data from the sat feed that services NWS WSFO’s nationwide, there should not be any access issues and no complaining over bandwidth and server loads.

Using the funds provided with the help of Heartland’s private donor, I hired a specialist programmer familiar with NOAA systems to trap and convert the NOAA sat feed data to look like any other hourly station (like ASOS hourly stations at airports etc) so that we’d be able to start the visualization and comparison process. This is just one phase of the project before it is ready for public consumption. When finished, there will be a website free and open to the public that will allow tracking and visualization of temperatures from the CRN right alongside that of the regular surface network

I’m happy to report that ingest programming is now completed and in the Alpha stage of testing. I have some “first light” imagery and data produced by the custom ingest system to share.

Below are some screen captures of the first output.

Click the images above to enlarge them.

As you can see for yourself, the project is pure evil.  I’m sure people like Professor Mark Stemen will be just shocked at what the *Koch Brothers/Big Oil hath wrought. /sarc

I’ll have updates when I have news from my programmer that he has completed the next phase of programming.

* For the record there’s no funding from either in this project, nor is any US taxpayer money being used, and anyone who says otherwise is a damned liar.