An update on my Climate Reference Network visualization project

Solar powered USCRN Station in Montrose, Colorado, 2006. Note the siting well away from any influences and the triple redundant air aspirated temperature sensors. The data is sent via satellite uplink (grey cone antenna) Photo: NOAA/NCDC

Readers may recall that when Peter Gleick first sent out the stolen Heartland board documents and the fake document to give the story “legs”, there was a mention in there of a project that I asked Heartland to help me fund. They found a private donor who was interested in my idea, and agreed to fund it. The amount of funding I received, based on the plan I submitted, was $44K, which when compared to many scientific papers and projects that get funded, is a proverbial drop in the bucket. For example, Michael Mann got a boatload of money: Mann’s $1.8 million Malaria grant – “where do we ask for a refund’? to study something outside his normal area of expertise – mosquitoes and malaria.

And as you recall, when the Heartland documents were made public by the “publish first, ask questions later” tabloid reporters at the Guardian and other news outlets, there was much wailing and gnashing of teeth by the usual suspects, claiming that somehow I’m in the employ of big oil, big coal, big tobacco, yada yada yada. One of the worst offenders was local eco-activist and Chico State University professor Mark Stemen who wrote to me with similar ludicrous charges, calling me a “Koch whore” and adding that “I’m making sure that everyone in town knows it” followed by posting a series of smears on his Facebook page. Stemen’s abusive smearing was typical of the over the top unprofessional and adolescent behavior that was directed at me for daring to do this project related to making climate data more available to the public.

Today, I’m going to offer my first update on it, so that everyone can see for themselves just how terrible and evil it is in the scheme of all things climate.

The goal of this project is to provide a publicly accessible one-on-one live comparison of temperatures between GHCN and other hourly reporting stations from the older surface network, to the new Climate Reference Network (CRN). The impetus was the heat wave in Texas last year, where I noticed that while there were a number of record setting high temperatures, many of them were higher than temperatures seen in the CRN. This suggested to me that UHI and siting effects play a role in elevating such temperatures. Unfortunately at that time there was no easy way to offer such visual comparisons, and I thought there should be, hence my idea that I asked Heartland to help me find a funding source for.

Here’s how Heartland Described my project in one of the stolen documents:

Weather Stations Project

Every few months, weathermen report that a temperature record – either high or low – has been broken somewhere in the U.S. This is not surprising, since weather is highly variable and reliable instrument records date back less than 100 years old. Regrettably, news of these broken records is often used by environmental extremists as evidence that human emissions are causing either global warming or the more ambiguous “climate change.”

Anthony Watts, a meteorologist who hosts WattsUpwithThat.com, one of the most popular and influential science blogs in the world, has documented that many of the

temperature stations relied on by weathermen are compromised by heat radiating from nearby buildings, machines, or paved surfaces. It is not uncommon for these stations to over-state temperatures by 3 or 4 degrees or more, enough to set spurious records.

Because of Watts’ past work exposing flaws in the current network of temperature stations (work that The Heartland Institute supported and promoted), the National Aeronautics and Atmospheric Administration (NOAA), the government agency responsible for maintaining temperature stations in the U.S., has designated a new network of higher-quality temperature stations that meet its citing specifications. Unfortunately, NOAA doesn’t widely publicize data from this new network, and puts raw data in spreadsheets buried on one of its Web sites.

Anthony Watts proposes to create a new Web site devoted to accessing the new temperature data from NOAA’s web site and converting them into easy-to-understand graphs that can be easily found and understood by weathermen and the general interested public. Watts has deep expertise in Web site design generally and is well-known and highly regarded by weathermen and meteorologists everywhere. The new site will be promoted heavily at  WattsUpwithThat.com. Heartland has agreed to help Anthony raise $88,000 for the project in 2011.  The Anonymous Donor has already pledged $44,000. We’ll seek to raise the balance.

Terrible huh?

Since it wasn’t clear in the Heartland description above what data I was going to use, for those who don’t know, here’s some background on the Climate Reference Network, this entry from Wikipedia:

The US Climate Reference Network (USCRN) Program was begun in 2002 and remains under construction with an expected completion date in late 2008. It has the long-term commitment of the Department of Commerce (DOC) and the National Oceanic and Atmospheric Administration (NOAA). This network will be maintained and modernized during the life of the program.

This is a program that will collect and analyze the highest quality climate data possible. Research based on these data will directly support near and long term policy and decision plans made by senior government and business leaders.

The research community, government agencies, and private businesses have identified significant shortcomings in understanding and examining long-term climate trends and change over the U.S. and surrounding regions. Some of these shortcomings are due to the lack of adequate documentation of operations and changes regarding the existing and earlier observing networks, the observing sites, and the instrumentation over the life of the network. These include inadequate overlapping observations when new instruments were installed and not using well-maintained, calibrated high-quality instruments. These factors increase the level of uncertainty when government and business decision-makers are considering long-range strategic policies and plans. Never before have people been so aware of the impact of the environment and climate variability and change on the quality of life and the economic health of a nation, its citizens, and the population of the world. This project will serve as a model for establishing similar networks in other countries.

Basically, the CRN is NCDC’s response to their realizations of problems in the existing climate observing network, something that I’m long since identified in my own surfacestations.org work, including a peer reviewed paper on the subject. Plus, my conclusions about the problems with the surface network in the USA were backed up by an investigation done by the US General Accounting Office. An example of the kind of problems the many surfacestations.org volunteers found are illustrated well by the fact that some stations used specifically to measure climate, were in the most ridiculous locations, like this USHCN station in downtown Ardmore, Oklahoma:

Ardmore USHCN station , MMTS temperature sensor, January 2009

While NCDC has gone to great lengths to defend the quality of the USHCN network and the highly adjusted data it produces, their actions of closing them (after we pointed out numerous train wrecks like Ardmore) speak far louder than written words and peer reviewed publications:

Ardmore USHCN station , MMTS temperature sensor removed, March 2011

The new CRN has none of these problems. Sites were meticulously chosen, temperature sensors have triple redundancy, there’s a QC process for data collection, and most important, since the data is not measured once a day (as observers did with high and low readings) but is done continuously, there’s no need for the Time Of Observation Adjustment, which we know adds significantly to the overall temperature trend of data used for climate study.

Note that the TOBS adjustment (in black) amounts to the lions share.

Here’s how NCDC describes the CRN:

The U.S. Climate Reference Network (USCRN) consists of 114 stations developed, deployed, managed, and maintained by the National Oceanic and Atmospheric Administration (NOAA) in the continental United States for the express purpose of detecting the national signal of climate change. The vision of the USCRN program is to maintain a sustainable high-quality climate observation network that 50 years from now can with the highest degree of confidence answer the question: How has the climate of the nation changed over the past 50 years? These stations were designed with climate science in mind. Three independent measurements of temperature and precipitation are made at each station, insuring continuity of record and maintenance of well-calibrated and highly accurate observations. The stations are placed in pristine environments expected to be free of development for many decades. Stations are monitored and maintained to high standards, and are calibrated on an annual basis. In addition to temperature and precipitation, these stations also measure solar radiation, surface skin temperature, and surface winds, and are being expanded to include triplicate measurements of soil moisture and soil temperature at five depths, as well as atmospheric relative humidity. Experimental stations have been located in Alaska since 2002 and Hawaii since 2005, providing network experience in polar and tropical regions. Deployment of a complete 29 station USCRN network into Alaska began in 2009. This project is managed by NOAA’s National Climatic Data Center and operated in partnership with NOAA’s Atmospheric Turbulence and Diffusion Division.

More here. Given the advanced way it is measured, there’s no need to adjust the CRN station data whatsoever.

In addition, many USHCN stations are being modernized and added to the CRN network as I’ve described in detail here: What the modernized USHCN will look like.

Overall I’m pleased with that CRN project and the USHCN modernization, and I endorse it. But little of the data from it is finding its way into the public realm, and I aim to change that.

The first job was to arrange for and to program data ingestion. Initially it looked like the project was designated to be done with an Internet based FTP fetching, which can be fraught with problems related to network delays, timeouts, server load, etc. Fortunately it was discovered that the entire CRN data set was delivered on an hourly basis via one of NOAA’s satellite feeds. This was a godsend, because I worried about NCDC trying to cut off web access (like they did once before) when they decided they didn’t like what I was doing or if my bandwidth usages got to be too high. Similar issues plagued Steve McIntyre when he went to fetch a large amount of data once, the Gavinator of GISS blocked him. So unless NOAA/NCDC decides to pull the CRN data from the sat feed that services NWS WSFO’s nationwide, there should not be any access issues and no complaining over bandwidth and server loads.

Using the funds provided with the help of Heartland’s private donor, I hired a specialist programmer familiar with NOAA systems to trap and convert the NOAA sat feed data to look like any other hourly station (like ASOS hourly stations at airports etc) so that we’d be able to start the visualization and comparison process. This is just one phase of the project before it is ready for public consumption. When finished, there will be a website free and open to the public that will allow tracking and visualization of temperatures from the CRN right alongside that of the regular surface network

I’m happy to report that ingest programming is now completed and in the Alpha stage of testing. I have some “first light” imagery and data produced by the custom ingest system to share.

Below are some screen captures of the first output.

Click the images above to enlarge them.

As you can see for yourself, the project is pure evil.  I’m sure people like Professor Mark Stemen will be just shocked at what the *Koch Brothers/Big Oil hath wrought. /sarc

I’ll have updates when I have news from my programmer that he has completed the next phase of programming.

* For the record there’s no funding from either in this project, nor is any US taxpayer money being used, and anyone who says otherwise is a damned liar.
Advertisements

  Subscribe  
newest oldest most voted
Notify of
pat

Good for you. A watch out for any heat records around Chico State. Seems a lot of hot air radiates therefrom.

I’ve had an R package for ingesting CRN for sometime. It will be interesting to compare the two approaches.
REPLY: one time scrapes of data for R analysis is one thing, sustained production on a 24/7 basis is something else entirely – Anthony

Beginning with your Chico Fan, and ending with all of the rest, this tendency towards smearing is the thing that will destroy your opponents. Congratulations on being able to do science on a pittance, instead of Stalinism on the budget of a medium-sized country.

F. Ross

“…For example, Michael Mann goat a boatload of money:…”
[+emphasis]
He possibly is caprine but did you mean “got”
REPLY: Fixed, thanks – A

Cris

Anthony,
This looks very cool. Is there any reason they couldn’t encrypt the CRN data on the sat feed?
REPLY: oh they could, but this is public domain data, made freely available like any other NOAA data. We would collectively have Karl and Peterson’s head on a stick if they pulled a stunt like that. When they tried it the first time in 2007, they had to recant. You can’t make “public data” non public once the cat is out of the bag. – Anthony

Excellent work Anthony.

dyrewulf

*dramatic pose* You bad bad man! (lol) Keep up the good, errr…. bad? work!

john in cheshire

It seems the white car to the left, in the background, hasn’t moved in two years; it is abandoned?

REPLY:
it belongs to the city, that’s city hall, and the designated parking space. – Anthony

DirkH

Well, but it is evil when it destroys their freedom to adjust. It strengthens the reactionary notion of an objective reality and works against temperature relativism and post-normal science. /sarc

I hope the missing Ardmore sensor wasn’t just mounted on the roof of the building.
REPLY: No, it was moved to outskirts of town at a fire station, and made precipitation only. They no longer record temperature. – Anthony

Mike McMillan

Real pretty so far. Are they measuring CO2 locally?
REPLY: No CO2 monitors – A

Wow !!!
Really outstanding work !!
Let’s hope it leads to much better data !

Latitude

Anthony…not enough people can tell you thank you….
…so thank you again!

TomRude

Excellent work!
[ snip -while I think Dr. Stemen is really, really, childish and wrong, that’s a bit over the top – Anthony]

kMc2

How very clear and helpful. Many thanks to Anthony, and to the Heartland Institute. Bravos all round!

Crispin in Johannesburg

My compliments Anthony. I love to see real numbers made accessible, especially publically funded ones. The presentation is really top drawer. In my current field (emissions testing) I fight the same battle – no access to the raw data that has been mish-mashed to hide the poor performance of favoured technologies. This is inspirational work. Naturally we all look forward to the ‘first light’ of your national ‘scope.

Thanks for all you do. I mean that.

Ian S

Really excellent work. I look forward to visiting this new site on a regular basis once completed.

David, UK

Thank you Anthony. /sinc
(That means I was being sincere!)

Paul Homewood’s website ‘Notalotofpeopleknowthat’ has found a similar rewriring of the historical record in South America: http://notalotofpeopleknowthat.wordpress.com/2012/04/03/adjustmentgatesouth-america/#more-1058
He’s calling it ‘Adjustmentgate’. Until now it appeared that the GHCN record was only being fiddled in the Arctic. I’ve documented some of it here: http://endisnighnot.blogspot.co.uk/#!/2012/03/giss-strange-anomalies.html

LazyTeenager

DirkH says
Well, but it is evil when it destroys their freedom to adjust. It strengthens the reactionary notion of an objective reality
————
I am sure DirkH that they would prefer not to have to do any adjustments at all because it is a lot of very hard work.
And since they spent a truck load of money to improve the climate network to avoid doing adjustments I think there is plenty of evidence to support the notion they would prefer to not do adjustments at all.
And since they spent a truck load of money on satellites to provide global temperature measurements that were independent of the historical records, that is more evidence that they are anxious not rely on the historical records.
I am keen to have better data cos that’s one less excuse climate skeptics can use to twiddle their thumbs.

MrX

Good work! I always say persistence is the key to success. Glad to see this going forward, though I have no idea how 40K-80K is enough (putting aside the ridiculous claims by the alarmist that it is too much).

Joe Prins

From a total layman and interested in the whole settled debate: Great stuff. sir.
Just added some fling funds.

cui bono

Anthony, thanks for the preview of this new work funded by the oil-funded planet-destroying baby-bayoneting denier conspiracy. It looks typically impressive! And for $88,000. That’s less than 5% of a Mann.
Why are University professors sometimes the most pathetic, juvenile moroons on the planet?

Don Bennett

Good job, Anthony. I’m looking forward to the final product.

Alexander K

Congrats and thanks!
Real science!

John F. Hultquist

john in cheshire says:
April 7, 2012 at 1:30 pm
It seems the white car to the left, in the background, hasn’t moved in two years; it is abandoned?

Also, someone photoshopped in a yellow truck, added clouds to the sky, and shortened the shadows just to make it look real.
As Anthony said, it does seem to be the same city car in the same spot. Ardmore seems to be saving money. They kept the leaning water hydrant too. I believe the applicable word is parsimonious.
I note, also, in the first photo (Montrose, Colorado) the solar power. Appropriate technology at the appropriate scale. Nice.
———————–
Anthony,
Thanks for the update. Seems a great idea is coming together.
John

Jaypan

Remarkable how all the ‘sceptics’ work hard to make data and methods public, not the ones in duty and paid by tax dollars.
The other question is what qualifies this smeary professor to teach young people …

Mike McMillan says:
April 7, 2012 at 1:45 pm

Real pretty so far. Are they measuring CO2 locally?
REPLY: No CO2 monitors – A

One thing they don’t measure which I think is a real pity, is atmospheric pressure. In addition to providing evidence of increased (or decreased!) storm strength, pressure data would help establish long term changes in weather patterns and might even provide information about total water vapor over time.
Fortunately, air pressure is about the easiest climate attribute to measure accurately and we are very confident in the quality of data from old instruments.
It’s sort of a tossup what I would prefer more – a grandfather’s clock or a mercury barometer. I think my wife would go for whatever is quieter. 🙂

DocMartyn

do you by any chance have a white blue-eyed Turkish Angora?

u.k.(us)

john in cheshire says:
April 7, 2012 at 1:30 pm
It seems the white car to the left, in the background, hasn’t moved in two years; it is abandoned?
================
Look at the lengths of the shadow cast by the fire hydrant, it confirms the seasons.

“in Texas last year, where I noticed that while there where a number of record setting high”
I think that should “were” not “where”.
Also Anthony, an outstanding effort and we finally get some real temperature information instead of the usual doctored and manipulated mess we have had to put up with for so long. Like a previous commenter stated, they can no longer fudge the facts with their own interpretations. Time well spent and an outstanding effort.
REPLY: Fixed the spelling, and please save the kudos for later, what you see above is just crude output. – Anthony

Ivan

Antony,
you seems to have missed Roy Spencer’s bombshell; by reanalyzing the US 48 temperature data, he corrected for the warming bias of socioeconomic development and found that instead of 0.20 C per decade, the US has warmed negligibly only 0.01 or 0.029 C since 1970s, depending upon some variations in methodology.
Moreover, Spencer himself now concedes (although I warned him two years ago to this possibility, after reading a paper by E. Long) that it is quite possible that the UAH data (showing 0.20 C per decade since 1979) has a large warming bias, possibly associated with the transition from one to another NOAA satellite in 1995. This is potentially the biggest climate news in years, and maybe the end of the entire “global warming” narrative.
http://www.drroyspencer.com/2012/04/new-u-s-population-adjusted-temperature-dataset-pdat-1973-2012/
REPLY: No, didn’t miss it, was planning on running it today…just been busy, and I have to ask him
Anthony

Ivan

Anthony,
it seems that you have missed Roy Spencer’s bombshell; by reanalyzing the US 48 temperature data, he corrected them for the warming bias created by the socioeconomic development and found that instead of 0.20 C per decade, the US has warmed negligibly, only 0.01 or 0.029 C since 1970s, (depending upon some variations in methodology).
Moreover, Spencer himself now concedes (although I warned him two years ago to this possibility, after reading a paper by E. Long) that it is quite possible that the UAH data (showing 0.20 C per decade since 1979) has a large warming bias, possibly associated with the transition from one to another NOAA satellite in 1995. This is potentially the biggest climate news in years, and maybe the end of the entire “global warming” narrative.
http://www.drroyspencer.com/2012/04/new-u-s-population-adjusted-temperature-dataset-pdat-1973-2012/
REPLY: The WUWT search box is your friend, see http://wattsupwiththat.com/2012/03/30/spencer-shows-compelling-evidence-of-uhi-in-crutem3-data/
– Anthony

RoyFOMR

A publically-funded employee Mark Stemen feels secure enough to call a local bill-payer a “Koch Whore’; nice play on words there Mark; I just love professorial innuendo. Not!
“I’m making sure that everyone in town knows it”
What everyone in town now knows is not something YOU, your students and employers should be entirely happy about.

Ivan

Anthony,
the article you linked is the old one; I am talking about the two completely new posts published in the last two days with even more astonishing results!
REPLY: Ah, well then I stand corrected – Anthony

graphicconception

“It seems the white car to the left, in the background, hasn’t moved in two years;”
Quite so – but they have moved the badge on the door …

polistra

It occurs to me that Anthony has actually created a university department of climate science.
Teaching, research, publication, grants from foundations, managing techs and student assistants.
Two differences from the usual: (1) It’s outside the walls of a university. (2) It’s honest and scientific.

Tom E

Anthony, WUWT has gone from the last blog I check to a position in front of the Wall Street Journal. Thanks for taking Climate Change to a level that a retired engineer can understand.

The issue with the TOB adjustment isn’t whether it is required, there is no question that it is required, but the size of the adjustment.
Basically Karl came up with an adjustment algorithm with fairly weak empirical basis that even he concedes could be out by 25%.
All we can really say is that is that TOB and the adjustment introduces a large uncertainty into the critical 1960s to 1990s period temperature record in the USA. As for TOB in the rest of the world, who knows?

DirkH

LazyTeenager says:
April 7, 2012 at 2:05 pm
“I am sure DirkH that they would prefer not to have to do any adjustments at all because it is a lot of very hard work.”
Yeah, fudging numbers is hard work, sure, you seem to know a lot about work.
“And since they spent a truck load of money to improve the climate network to avoid doing adjustments I think there is plenty of evidence to support the notion they would prefer to not do adjustments at all.
And since they spent a truck load of money on satellites to provide global temperature measurements that were independent of the historical records, that is more evidence that they are anxious not rely on the historical records.”
They also spend a truckload of money on non-government servers set up with help of Holdren to be able to communicate in covert means in an attempt to evade FOIA laws.

jjthoms

Good stuff!
However just a couple of things to consider please.
Use Z time (as shown on your current o/p) and include local offset/time somewhere obvious
More importantly please use a universally accepted date format yyyy mm dd hh mm ss.
And continue with the deg celsius output (configurable if you like to degF)!

ntesdorf

Congratulations on the best value in scientific outcome for $88,000 in the history of Science. It will curb the ability of the forces of evil to adjust and manipulate. They will just have to report the results and that will be so unproductive for their scam. Objective reality overtakes temperature relativism and post-normal science.
REPLY: Thanks but, I was only given half of it, and the other half is no sure thing thanks to sliminess by Peter Gleick and his helpers – Anthony

Steve C

Not sure about “terrible” or “evil”, but I’ve lived through times when the kids might have called it “bad” or “wicked” …
Seriously, that’s datamashing on an impressive scale – nice work, and that Mk.I output looks pretty good already. And although you’re bullish about receiving no government funding – hang on. Comprehensive, intelligible public access to data collected at public expense? Sounds like something they ought to be supporting.

On first screen shot – little box…maybe a C to F conversion?
Thank you Mr Watts .. Heartland and friends.

mfo

What extraordinary language from a middle aged assoc professor, living in a modest suburban house in a respectable part of Chico with a responsibility for teaching young people.
One of his students said, “He wants to turn environmentalism into a quasi-religion. He’s very opinionated. On controversial issues he will only display his side. I expected a sober look at possible solutions to enviro problems. However, there is almost none of that. It is mostly marginal enviro issues used as justification for his own set of ethics”.
http://www.ratemyprofessors.com/ShowRatings.jsp?tid=379580
What would his employer and students think, not to mention Terri, Beverly and Edgar.
The man’s a minnow in the academic world and envious of your success. Good luck with this great project, the website looks extremely interesting.

ROM

Why Oh Why does a private citizen have to do this??
It is a terrible indictment and displays a level of complete incompetence and a paranoia on the part of the so called “scientists” who supposedly run the national weather and climate data networks that a private citizen, on an absolute pittance, probably a good deal less than they each get in annual pay, feels that as the information the public has paid for but which the scientists who run the Climate reference Network have failed to ensure the public has ready access to, he has to develop and then place on the net, vital public data on weather and climate for all to see and analyse.
And secondly he can only do this with the help and assistance of private funders with no assistance at all from the national science funding bodies for what is likely to be a very significant source of publicly available climate and weather data.
And he has to do all of this seemingly without any co-operation or even assistance from those same so called scientists who run the same “Climate Reference Network”.
The credibility of both the so called climate warming scientists and the national science funding bodies has just gone down another half dozen notches.
And climate warming scientists wonder and agonise over why the public’s trust in scientists and science is starting to go into free fall..
REPLY: For the record, they do make it available here: http://www.ncdc.noaa.gov/crn/observations.htm , but have no means for easy side by side comparison of the new more reliable data with the problematic old network data, which is the reason CRN was built in the first place. – Anthony

David Kleppinger

Isn’t NOAA the National Oceanic and Atmospheric Administration? I don’t believe a “National Aeronautic and Atmospheric Administration” exists. I think someone got NOAA and NASA mixed up.

John Blake

Post sounds a wee bit apologetic, even defensive– not acceptable in confronting AGW extremists. Big Government shills such as Hansen, Mann, Romm, Schmidt, Trenberth et al. should be seeking indulgences from WUWT.

Babsy

Ric Werme says:
April 7, 2012 at 2:26 pm
Barometric pressure observations *may* be available from the local airport if ASOS is available there.