by Anthony Watts
There has been a lot of buzz about the Menne et al 2010 paper “On the reliability of the U.S. Surface Temperature Record” which is NCDC’s response to the surfacestations.org project. One paid blogger even erroneously trumpeted the “death of UHI” which is humorous, because the project was a study about station siting issues, not UHI. Anybody who owns a car with a dashboard thermometer who commutes from country to city can tell you about UHI.
There’s also claims of this paper being a “death blow” to the surfacestations project. I’m sure in some circles, they believe that to be true. However, it is very important to point out that the Menne et al 2010 paper was based on an early version of the surfacestations.org data, at 43% of the network surveyed. The dataset that Dr. Menne used was not quality controlled, and contained errors both in station identification and rating, and was never intended for analysis. I had posted it to direct volunteers to so they could keep track of what stations had been surveyed to eliminate repetitive efforts. When I discovered people were doing ad hoc analysis with it, I stopped updating it.
Our current dataset at 87% of the USHCN surveyed has been quality controlled.
There’s quite a backstory to all this.
In the summer, Dr. Menne had been inviting me to co-author with him, and our team reciprocated with an offer to join us also, and we had an agreement in principle for participation, but I asked for a formal letter of invitation, and they refused, which seems very odd to me. The only thing they would provide was a receipt for my new data (at 80%) and an offer to “look into” archiving my station photographs with their existing database. They made it pretty clear that I’d have no significant role other than that of data provider. We also invited Dr. Menne to participate in our paper, but he declined.
The appearance of the Menne et al 2010 paper was a bit of a surprise, since I had been offered collaboration by NCDC’s director in the fall. In typed letter on 9/22/09 Tom Karl wrote to me:
“We at NOAA/NCDC seek a way forward to cooperate with you, and are interested in joint scientific inquiry. When more or better information is available, we will reanalyze and compare and contrast the results.”
“If working together cooperatively is of interest to you, please let us know.”
I discussed it with Dr. Pielke Sr. and the rest of the team, which took some time since not all were available due to travel and other obligations. It was decided to reply to NCDC on a collaboration offer.
On November 10th, 2009, I sent a reply letter via Federal Express to Mr. Karl, advising him that we would like to collaborate, and offered to include NCDC in our paper.. In that letter I also reiterated my concerns about use of the preliminary surfacestation data (43% surveyed) that they had, and spelled out very specific reasons why I didn’t think the results would be representative nor useful.
We all waited, but there was no reply from NCDC to our reply to offer of collaboration by Mr. Karl from his last letter. Not even a “thank you, but no”.
Then we discovered that Dr. Menne’s group had submitted a paper to JGR Atmospheres using my preliminary data and it was in press. This was a shock to me since I was told it was normal procedure for the person who gathered the primary data the paper was based on to have some input in the review process by the journal.
NCDC uses data from one of the largest volunteer organization in the world, the NOAA Cooperative Observer Network. Yet NCDC director Karl, by not bothering to reply to our letter about an offer he initiated, and by the journal not giving me any review process opportunity, extends what Dr. Roger Pielke Senior calls “professional discourtesy” to my own volunteers and my team’s work. See his weblog on the subject:
Professional Discourtesy By The National Climate Data Center On The Menne Et Al 2010 paper
I will point out that Dr. Menne provided thanks to me and the surfacestations volunteers in the Menne et al 2010 paper, and I hear through word of mouth, also in a recent verbal presentation. For that I thank him. He has been gracious in his communications with me, but I think he’s also having to answer to the organization for which he works and that limited his ability to meet some of my requests, like a simple letter of invitation.
Political issues aside, the appearance of the Menne et al 2010 paper does not stop the surfacestations project nor the work I’m doing with the Pielke research group to produce a peer reviewed paper of our own. It does illustrate though that some people have been in a rush to get results. Texas state Climatologist John Nielsen-Gammon suggested way back at 33% of the network surveyed that we had a statistically large enough sample to produce an analysis. I begged to differ then, at 43%, and yes even at 70% when I wrote my booklet “Is the US Surface Temperature Record Reliable?, which contained no temperature analysis, only a census of stations by rating.
The problem is known as the “low hanging fruit problem”. You see this project was done on an ad hoc basis, with no specific roadmap on which stations to acquire. This was necessitated by the social networking (blogging) Dr. Pielke and I employed early in the project to get volunteers. What we ended up getting was a lumpy and poorly spatially distributed dataset because early volunteers would get the stations closest to them, often near or within cities.
The urban stations were well represented in the early dataset, but the rural ones, where we believed the best siting existed, were poorly represented. So naturally, any sort of study early on even with a “significant sample size” would be biased towards urban stations. We also had a distribution problem within CONUS, with much of the great plains and upper midwest not being well represented.
This is why I’ve been continuing to collect what some might consider an unusually large sample size, now at 87%. We’ve learned that there are so few well sited stations, the ones that meet the CRN1/CRN2 criteria (or NOAA’s 100 foot rule for COOPS) are just 10% of the whole network. See our current census:
When you have such a small percentage of well sited stations, it is obviously important to get a large sample size, which is exactly what I’ve done. Preliminary temperature analysis done by the Pielke group of the the data at 87% surveyed looks quite a bit different now than when at 43%.
It has been said by NCDC in Menne et al “On the reliability of the U.S. surface temperature record” (in press) and in the June 2009 “Talking Points: related to “Is the U.S. Surface Temperature Record Reliable?” that station siting errors do not matter. However, I believe the way NCDC conducted the analysis gives a false impression because of the homogenization process used. As many readers know, the FILNET algorithm blends a lot of the data together to infill missing data. This means temperature data from both well sited and poorly sited stations gets combined to infill missing data. The theory is that it all averages out, but when you see that 90% of the USHCN network doesn’t meet even the old NOAA 100 foot rule for COOPS, you realize this may not be the case.
Here’s a way to visualize the homogenization/FILNET process. Think of it like measuring water pollution. Here’s a simple visual table of CRN station quality ratings and what they might look like as water pollution turbidity levels, rated as 1 to 5 from best to worst turbidity:
In homogenization the data is weighted against the nearby neighbors within a radius. And so a station might start out as a “1” data wise, might end up getting polluted with the data of nearby stations and end up as a new value, say weighted at “2.5”. Even single stations can affect many other stations in the GISS and NOAA data homogenization methods carried out on US surface temperature data here and here.
In the map above, applying a homogenization smoothing, weighting stations by distance nearby the stations with question marks, what would you imagine the values (of turbidity) of them would be? And, how close would these two values be for the east coast station in question and the west coast station in question? Each would be closer to a smoothed center average value based on the neighboring stations.
Essentially, in my opinion, NCDC is comparing homogenized data to homogenized data, and thus there would not likely be any large difference between “good” and “bad” stations in that data. All the differences have been smoothed out by homogenization (pollution) from neighboring stations!
The best way to compare the effect of siting between groups of stations is to use the “raw” data, before it has passed through the multitude of adjustments that NCDC performs. However NCDC is apparently using homogenized data. So instead of comparing apples and oranges (poor sited -vs- well sited stations) they essentially just compare apples (Granny Smith -vs- Golden delicious) of which there is little visual difference beyond a slight color change.
We saw this demonstrated in the ghost authored Talking Points Memo issued by NCDC in June 09 in this graph:
Referencing the above graph, Steve McIntyre suggested in his essay on the subject:
The red graphic for the “full data set” had, using the preferred terminology of climate science, a “remarkable similarity” to the NOAA 48 data set that I’d previously compared to the corresponding GISS data set here (which showed a strong trend of NOAA relative to GISS). Here’s a replot of that data – there are some key telltales evidencing that this has a common provenance to the red series in the Talking Points graphic.
When I looked at SHAP and FILNET adjustments a couple of years ago, one of my principal objections to these methods was that they adjusted “good” stations. After FILNET adjustment, stations looked a lot more similar than they did before. I’ll bet that the new USHCN adjustments have a similar effect and that the Talking Points memo compares adjusted versions of “good” stations to the overall average.
There’s references in the new Menne et al 2010 paper to the new USHCN2 algorithm and we’ve been told how it is supposed to be better. While it does catch undocumented station moves that USHCN 1 did not, it still adjusts data at USHCN stations in odd ways, such as this station in rural Wisconsin, and that is the crux of the problem.
Or this one in Lincoln, IL at the local NWS office where they took great effort to have it well sited.
Thanks to Mike McMillan for the graphs comparing USHCN1 and USHCN2 data
Notice the clear tendency in the graphs comparing USHCN1 to USHCN2 to cool off the early record and leave the current levels near recently reported levels or to increase them. The net result is either reduced cooling or enhanced warming not found in the raw data.
As for the Menne et all 2010 paper itself, I’m rather disturbed by their use of preliminary data at 43%, especially since I warned them that the dataset they had lifted from my website (placed for volunteers to track what had been surveyed, never intended for analysis) had not been quality controlled at the time. Plus there are really not enough good stations with enough spatial distribution at that sample size. They used it anyway, and amazingly, conducted their own secondary survey of those stations, comparing it to my non-quality controlled data, implying that my 43% data wasn’t up to par. Well of course it wasn’t! I told them about it and why it wasn’t. We had to resurvey and re-rate a number of stations from early in the project.
This came about only because it took many volunteers some time to learn how to properly ID them. Even some small towns have 2-3 COOP stations nearby, and only one of them is “USHCN”. There’s no flag in the NCDC metadatabase that says “USHCN”, in fact many volunteers were not even aware of their own station status. Nobody ever bothered to tell them. You’d think if their stations were part of a special subset, somebody at NOAA/NCDC would notify the COOP volunteer so they would have a higher diligence level?
If doing an independent stations survey was important enough for NCDC to do to compare to my 43% data now for their paper, why didn’t they just do it in the first place?
I have one final note of interest on the station data, specifically the issue of MMTS thermometers and their tendency to be sited closer to building due to cabling issues.
Menne et al 2010 mentioned a “counterintuitive” cooling trend in some portions of the data. Interestingly enough, former California State Climatologist James Goodridge did an independent analysis ( I wasn’t involved in data crunchng, it was a sole effort on his part) of COOP stations in California that had gone through modernization, switching from Stevenson Screens with mercury LIG thermometers to MMTS electronic thermometers. He sifted through about 500 COOPs in California and chose stations that had at least 60 years of uninterrupted data, because as we know, a station move can cause all sorts of issues. He used the “raw” data from these stations as opposed to adjusted data.
He writes:
Hi Anthony,
I found 58 temperature station in California with data for 1949 to 2008 and where the thermometers had been changed to MMTS and the earlier parts were liquid in glass. The average for the earlier part was 59.17°F and the MMTS fraction averaged 60.07°F.
Jim
A 0.9F (0.5C) warmer offset due to modernization is significant, yet NCDC insists that the MMTS units are tested at about 0.05C cooler. I believe they add this adjustment into the final data. Our experience shows the exact opposite should be done and with a greater magnitude.
I hope to have this California study published here on WUWT with Jim soon.
I realize all of this isn’t a complete rebuttal to Menne et al 2010, but I want to save that option for more detail for the possibility of placing a comment in The Journal of Geophysical Research.
When our paper with the most current data is completed (and hopefully accepted in a journal), we’ll let peer reviewed science do the comparison on data and methods, and we’ll see how it works out. Could I be wrong? I’m prepared for that possibility. But everything I’ve seen so far tells me I’m on the right track.
If doing a stations survey was important enough for NCDC to do to compare to my data now for their paper, why didn’t they just do it in the first place?
We currently have 87% of the network surveyed (1067 stations out of 1221), and it is quality controlled and checked. I feel that we have enough of the better and urban stations to solve the “low hanging fruit” problem of the earlier portion of the project. Data at 87% looks a lot different than data at 43%.
The paper I’m writing with Dr. Pielke and others will make use of this better data, and we also use a different procedure for analysis than what NCDC used.
You just know that whatever you, (Anthony), & Pielke Snr produce, it’s going to get a hostile review, AND it will be the people whos papers you were not allowed to review.
My prediction and I’m usually nervous about predicting the future. 😉
DaveE.
i am in houston texas, i notice on my car thermometer that when i drive in from the suburbs where i live to town about 15 miles there is a 5-6 degrees rise in temperature. heck just getting off interstate 45 and on to fm 1488 and driving into the state forest a distance of less than a mile usually results in a drop of 2 degrees. A few weeks ago i drove across san antoino from basically east to west on interstate 10 the temperature in the country was 44 degrees F when i got to down town it was 49 and then as i passed through then out to the west of town it dropped back to 44. So yes heat island effect is a very big deal and i can see it every day.
Anthony;
Thank you for finding time in what seems to be an almost man killing schedule to post this. I have read the Menne paper, though I only had time for a quick run through, and I hadn’t really seen any really obvious flaws. I still came away with the nagging thought that this looks way to on the nose to be real. From my experience the trap that most amateur numbers chefs fall into when trying to cook the books is not being able to resist the impulse to make it look to good. It’s a lot easier to forget to look behind the curtain when the appearance of unnatural perfection isn’t there to trigger suspicion.
The little game they played with the invitation to participate is interesting, one might even say exceedingly strange. It’s hard not to think of it in terms of the overriding arrogance demonstrated in the CRUtape emails.
At any rate the clarifications you offered make me feel better about trusting my instincts on these matters. The record of the little man in the back of my head continues to improve. He’s actually on an impressive unbeaten streak in this CAGW fiasco.
In regard to your post’s title, my first sentence, and the incredible number of irons you seem to have in the fire at the moment, I hope you’ll remember to set aside a little quality time for yourself to recharge. The work you are doing and have done here is incredibly important, but in the end “this to shall pass” and I would be terribly saddened if I dropped in here one day and found out your title had been contradicted.
REPLY: Thank you, most sincerely. – Anthony
Menne et al 2010 looks like it uses USHCN v 2, which we know has been heavily ‘adjusted’ from the v 1 data set. The adjustment took place subsequent to Tom Peterson’s examination of the siting problems (Peterson 2006).
While Dr Peterson’s paper compared good and bad sites with good sites that had been homogenized with bad sites (thus producing the unsurprising result that they looked similar), they now had a handle on the issues surfacestations.org might bring up. They might very well have taken those issues into account in generating the v 2 adjustment algorithm.
The paper speaks of two data sets, ‘adjusted and ‘unadjusted.’ In this case, the adjustment referred to is ‘homogenization,’ not the adjustments in USHCN v 2, which I believe is their ‘unadjusted’ dataset. If the v 2 changes account for siting problems, we may be comparing similar sets, just as Peterson 2006 did.
A second concern is that the data sets weren’t compared directly.
The anomalies were interpolated to a quarter-degree lat-long grid, and then area-weighted to generate the total anomaly for the U.S.
This was done for the good stations, and done again for the poor stations, to generate the temperature curves. Problem here is that the good and poor stations have different locations on the lat-long grid, and the interpolated, area-weighted grid points could easily blend out to produce similar numbers. Whatever tendencies the grid had (similar or different) would be carried uniformly through the years plotted on the temperature chart.
That said, my main question is how this study would look using the USHCN v 1numbers.
Anthony,
Remember that good actions almost always hurt right now but lead to your long term betterment. Bad actions almost always feel good right now but lead to your long term determent.
Your actions have been very good. Thank you.
Mike Ramsey
There’s more, too. Menne made an even more fundamental process error.
But we’ll get back to that when the paper is completed.
I think the rush to judgement on your findings will be their undoing. Never good to play your hand so quickly, as they have done. One might call it “premature edification”.
The NCDC has clearly underestimated the consequences of their poor decision in regard to your mutual cooperation.
You offered your cooperation on an equal basis, they refused in a very unprofessional and impolite manner, which resembles nothing more but their utter arrogance cloaking poor standards of operation and massaging data in support of policy rather than objective and honest science.
They have known from the beginning they were going to lose the argument if honest and objective input would.
You have done your best to save them from losing their reputation.
They have decided otherwise.
So they deserve what’s coming to them.
And it will be served cold.
May the best team win.
Watts up with That!
“Rumours of my death…”
“…But everything I’ve seen so far tells me I’m on the right track.”
I have followed this project with some enthusiasm for a long time and have even tried to find one of these stations here in TX (not easy btw). Know that this project is a success no matter what the result is.
Go to the NCDC site and read the order of corrections applied to data. Correction for UHI is last. Now read Karl and Williams (1987) regarding the homogenization. In it you will find the admonition to do the homogenization last. The reason should be apparent to anyone. Doing homogenization with known, secular (slow-drift) errors in select portions of the data will smear these into the entire set of data.
Even if we were to agree that every separate correction that NCDC does is accurate, then doing them out of order is a problem. We do not agree that the individual corrections are necessarily correct.
The issues with corrections are a small subset of the universe of issues with the surface data–this could go on and on ad nauseum.
“There’s” seems to be mis-used here, whether intended as “There is” or “There has”. Both of the subjects are plural, so they should be “There are”.
REPLY: Fixed, I’m overly tired today, and whn tht hapnz I sturt to slurr my werds – Atony
Ish. 43% scientifically selected would of course be very representative. 43% ad hoc with easily discerned bias is of course “not so much”.
I truly respect NOAA and NCDC, while being more than occassionally frustrated with them. I think they’ve got bias, and it shows, but I really do believe their hearts are in the right place and they are at least trying to be fair, as they see it.
I’ve been very disappointed to discover just how large a role bias plays in science, mostly by way of the Climategate letters. Understand, I’m not entirely against “bias”, however bad a name it has. Usually what is referred to as “bias” is a result of long experience. It’s the gut reaction. It has a place in science, by all means –but that legitimate place is a starting point for investigation, not enforcing an ending point.
As an amateur (tho published in an academic journal) historian, I very much believe the idea that “If you’ve done the work, you get to have an opinion and indeed have a responsibility to share it for the benefit of the future”. . .but what I suspect and what I know I can prove are very distinct areas.
Anthony, Your withholding of the data is just as bad as CRU.
The only difference is you cannot be subject to FOIA!
Free the Data! Please.
REPLY: Oh please. It is common practice for researchers to release data in an SI when they publish the paper, and that’s exactly what I plan to do. – Anthony
Anthony,
It’s interesting that they chose to rush out first with essentially a defensive move before seeing your paper. Moving preemptively is a weaker position. Strategically it says that they believe your paper will be damaging to their agenda and difficult to attack. As frustrating as this cheap shot is, you should view it as confirmation that you’re on the right track and that they are worried. Too bad they can’t put the same energy into actually improving the network and their data methods in pursuit of accuracy. Sadly, the increased funding that supporting alarmism brings is a powerful incentive. Such incentives can cause not only blind-spots but biases and now, apparently, unprofessional behavior. Keep up the good work!
Anthony
Menne et al have demonstrated that a PhD does not make one a better person. Their mothers must be embarrassed.
I understand your reluctance to go to press with inadequate data. Having worked with large geologic and environmental databases in the past, good QA/QC is vital as is having an adequte data distribution. The optimum, having 100% of the data in faultless form is a mission seldom accomplished. If one is concerned that the data in inadequate, it most probably is.
Keep up the quality work.
Anthony: quick note on your pie chart up top. Shouldn’t the ranges be
CRN 1/2: <=1 deg C
CRN 3: 1 deg C to <2 deg C
CRN 4: 2 deg C to = 5 deg C.
I think that’s what you mean. The way you have it written can interpreted as CRN 3 should include 91% of the pie and with CRN 4 & 5 being subsets within it. After all, 6 deg C IS greater than 1 deg C.
No biggie and perhaps I’m interpreting it wrong. Just hoping to make sure the details are correct.
Keep up the good work.
“Our current dataset at has been quality controlled.”
Irony abounds whenever I write sentences with phrases like “quality control” in them as inevitably they need some themselves. The grammar of your above sentence needs a wee bit of tweaking or clarification of meaning. I can’t make out what the “at” is doing in there, either some word is missing after it or it’s spurious possibly due to an edit.
[:)] Loved the pic by the way! This is a grave topic after all!
REPLY: Meant to write “at 87%” thanks fixed -A
ShaneOfMelbourne (18:14:59) :
http://news.nationalgeographic.com/news/2008/01/08
http://dyinginhaiti.blogspot.com/2007/07/mud-pies-in-soleil.html0130-AP-haiti-eatin.html
Still laughing?
DaveE.
Smarmy is the adjective that comes to mind in reading the Menne abstract and his ostentatious thanks to Anthony for his assistance. Anthony, your courtesy is unfailing but I think you are too generous in this case. I agree with the sentiments of Ron de Haan above. The best team will win!
OT – in his State of the Union Address, Obama declared that global warming is undeniably true. A rumble of dissent followed this assertion. There is still so much work to accomplish in demolishing this scam, but there are seeds of hope thanks to the dedication of Anthony and his moderators, Steve M and Ross M, D’Aleo, E.M. Smith, Lucia, and others like James Delingpole and Terence Corcoran who are bringing the many facets of this scam to the attention of an increasingly appreciative public.
Even without sorting by “quality” of stations, there is no significant warming trend in the raw data. A strong PDO, but nothing alarming.
I’m still waiting for a shot at CRU’s raw data… although that will likely be manufactured, as Jones had suggested that was the only way to reproduce it.
It’s ridiculous to homogenize in the first place. It is bogus from the getgo. There is zero need for it; it can only corrupt the process. Just take the average of trends within each grid and then average the grids. That takes care of the station distribution issue then and there. (I think that’s how USHCN1 did it.)
And yes, there is indeed an “MMTS adjustment” of +0.05C/century in USHCN1.
And, yes, siting matters. I can’t say more on this yet, but all will be made clear.
It appears to me that all that Menne et al shows is that crappy sited and illogically adjusted sites produce a result that is similar to the “warming shown by GISS etc. We already knew that! GIGO. Crappy siting and adjustment is how they produced the original “record”.
David Alan Evans (19:09:45) :
Both or the two URLs have returned “Site not found” message.
ShaneOfMelbourne.
I’ve seen you elsewhere on Australian sites.
You bemoan that the temperature at 2 LaTrobe street, (which used to be semi rural), is now recording record temperatures!! It wasn’t surrounded by high rise buildings back when the previous records were set!
People like you sicken me! You have no compassion as far as I can see!
You deride Monckton about mud-pies. It’s not uniquely Haitian, it happens in Africa too!
Cheap electricity, clean water, encouraging local farming, they’re what we should be doing: not discouraging all these things which was what Copenhagen was really about.
DaveE.
Pat Frank (18:06:27) :
Anthony posted his data on the internet. Is there anything on the internet that you feel is not in the public domain?
I won’t argue that NCDC exhibited poor form regarding authorship, despite the generous acknowledgements. But when you spend your days accusing scientists around the world of everything from incompetence to conspiracy to commit fraud, one should expect very little in the way of professional courtesy. This is one of those occasions when I am simply staggered by the hypocrisy shown by Anthony, contributers to this blog, etc. (intentionally avoiding use of the “D” and “S” words lest I unintentionally offend someone).
Menne, et al. performed a very clear, concise analysis that, among other things, demonstrated poorly sited stations did not meaningfully deviate from good sites (or “pristine” sites if you include the comparison to the admittedly brief USCRN series). Now, given that the sample relies on 43% of non-QA’d data should we consider the science settled? Of course not. But the finding is significant and worthy of publication. This is how science and the peer review process works. At the very least, it provides an opportunity for Watts, et al. to reproduce the analysis using a larger sample size, arrive at alternative conclusions, and have those findings published.