The Berkeley Earth Surface Temperature project puts PR before peer review

UPDATE: see this new story

BEST: What I agree with and what I disagree with – plus a call for additional transparency to prevent “pal” review

=======================================================

Readers may recall this post last week where I complained about being put in a uncomfortable quandary by an author of a new paper. Despite that, I chose to honor the confidentiality request of the author Dr. Richard Muller, even though I knew that behind the scenes, they were planning a media blitz to MSM outlets. In the past few days I have been contacted by James Astill of the Economist, Ian Sample of the Guardian, and Leslie Kaufman of the New York Times. They have all contacted me regarding the release of papers from BEST today.

There’s only one problem: Not one of the BEST papers have completed peer review.

Nor has one has been published in a journal to my knowledge, nor is the one paper I’ve been asked to comment on in press at JGR, (where I was told it was submitted) yet BEST is making a “pre-peer review” media blitz.

One willing participant to this blitz, that I spent the last week corresponding with, is James Astill of The Economist, who presumably wrote the article below, but we can’t be sure since the Economist has not the integrity to put author names to articles:

The full article is here. Apparently, Astill has never heard of the UAH and RSS Global Temperature records, nor does he apparently know that all the surface temperature records come from one source, NCDC.

Now compare that headline and subtitle to this line in the article:

It will be interesting to see whether this makes it past the review process.

And, The Economist still doesn’t get it. The issue of “the world is warming” is not one that climate skeptics question, it is the magnitude and causes.

I was given a pre-release draft copy of one of the papers, related to my work as a courtesy. It contained several errors, some minor (such as getting the name of our paper wrong i.e. Fell et al in several places, plus a title that implied global rather than USA) some major enough to require revision (incorrect time period comparisons).

I made these errors known to all the players, including the journal editor, and the hapless Astill, who despite such concerns went ahead with BEST’s plan for a media blitz anyway. I was told by a BEST spokesperson that all of this was “coordinated to happen on October 20th”.

My response, penned days ago, went unheeded as far as I can tell, because I’ve received no response from Muller or the Journal author. Apparently, PR trumps the scientific process now, no need to do that pesky peer review, no need to address the errors with those you ask for comments prior to publication, just get it to press.

This is sad, because I had very high hopes for this project as the methodology is looked very promising to get a better handle on station discontinuity issues with their “scalpel” method. Now it looks just like another rush to judgement, peer review be damned.

Below is my response along with the draft paper from BEST, since the cat is publicly out of the bag now, I am not bound by any confidentiality requests. Readers should note I have not seen any other papers (there may be up to 4, I don’t know the BEST website is down right now) except the one that concerns me.

My response as sent to all media outlets who sent requests for comment to me:

===========================================================

In contradiction to normal scientific method and protocol, I have been asked to provide public commentary to a mass media outlet (The Economist) on this new paper. The lead author,  Dr. Richard Muller has released me from a previous request of confidentiality on the matter in a written communication on 10/14/2011. 10/15/2011 at 4:07PM PST in an email.  The paper in question is:

Earth Atmospheric Land Surface Temperature and  Station Quality [Tentative title, may have changed] by Muller et al 2011, submitted to the AGU JGR Atmospheres Journal, which apparently has neither completed peer review on the paper nor has it been accepted for publication by JGR.

Since the paper has not completed peer review yet, it would be inappropriate for me to publicly comment on the conclusions, especially in light of a basic procedural error that has been discovered in the methodology that will likely require a rework of the data and calculations, and thus the conclusions may also change. The methodology however does require comment.

The problem has to do with the time period of the data used, a time period which is inconsistent with two prior papers cited as this Muller et al paper being in agreement with. They are:

Fall et al (2011), Analysis of the impacts of station exposure on the U.S. Historical Climatology Network temperatures and temperature trends J. Geophys. Res.

http://pielkeclimatesci.files.wordpress.com/2011/07/r-367.pdf

and

Menne et al  (2010), On the reliability of the U.S. surface temperature record, J. Geophys. Res.

Both papers listed above (and cited by Muller et al) do an analysis over a thirty year time period while the Muller et al paper uses data for comparison from 1950 – 2010 as stated on lines 142-143:

“We calculated the mean temperature from 1950 to the present for each of these sites, and subtracted the mean of the poor sites from the OK sites.”

I see this as a basic failure in understanding the limitations of the siting survey we conducted on the USHCN, rendering the Muller et al paper conclusions highly uncertain, if not erroneous.

There is simply no way siting quality can be established as static for that long. The USHCN survey was based on photographs and site surveys starting in of 2007, plus historical metadata. Since the siting of COOP stations change as volunteers move, die, or discontinue their service, we know the record of siting stability to be tenuous over time. This is why we tracked only from 1979 and excluded stations whose locations were unknown prior to 2002. 1979 represented the practical limit of which we assumed we could reasonably ascertain siting conditions by our survey.

We felt that the further back the station siting changes occurred, the more uncertainty was introduced into the analysis, thus we limited meaningful comparisons of temperature data to siting quality to thirty years, starting in 1979.

Our ratings from surfacestations.org are assumed to be valid for the 1979 – 2008 period, but with Muller et all doing analysis from 1950, it renders the station survey data moot since neither Menne et al nor Fall et al made any claim of the station survey data being representative prior to 1979. The comparisons made in Muller et al are inappropriate because they are outside of the bounds of our station siting quality data set.

Also, by using a 60 year period, Muller et al spans two 30 year climate normals periods, thus further complicating the analysis. Both Menne et al and Fall et al spanned only one.

Because of the long time periods involved in Muller et al analysis, and because both Menne et al and Fall et al made no claims of knowing anything about siting quality prior to 1979, I consider the paper fatally flawed as it now stands, and thus I recommend it be removed from publication consideration by JGR until such time that it can be reworked.

For me to comment on the conclusions of Muller et al would be inappropriate until this time period error is corrected and the analysis reworked for time scale appropriate comparisons.

The Berkeley Earth Surface Temperature analysis methodology is new, and may yield some new and potentially important results on siting effects once the appropriate time period comparisons are made. I welcome the BEST effort provided that appropriate time periods are used that match our work. But, by using time period mismatched comparisons, it becomes clear that the Muller et al paper in its current form lost the opportunity for a meaningful comparison.

As I was invited by The Economist to comment publicly, I would recommend rejecting Muller et al in the current form and suggest that it be resubmitted with meaningful and appropriate 30 year comparisons for the same time periods used by the Menne et al and Fall et al cited papers. I would be happy to review the paper again at that time.

I also believe it would be premature and inappropriate to have a news article highlighting the conclusions of this paper until such time meaningful data comparisons are produced and the paper passes peer review. Given the new techniques from BEST, there may be much to gain from a rework of the analysis limited to identical thirty year periods used in Menne et al and Fall et al.

Thank you for your consideration, I hope that the information I have provided will be helpful in determining the best course of action on this paper.

Best Regards,

Anthony Watts

cc list: James Astill, The Economist, Dr. Joost DeGouw, JGR Atmospheres editor, Richard A. Muller, Leslie Kaufman, Ian Sample

===========================================================

Despite my concerns, The Economist author James Astill told me that “the issue is important” and decided to forge ahead, and presumably produced the article above.

Here is the copy of the paper I was provided by Richard Muller. I don’t know if they have addressed my concerns or not, since I was not given any follow up drafts of the paper.

BEST_Station_Quality (PDF 1.2 MB)

I assume the journalists that are part of the media blitz have the same copy.

I urge readers to read it in entirety and to comment on it, because as Dr. Muller wrote to me:

I know that is prior to acceptance, but in the tradition that I grew up in (under Nobel Laureate Luis Alvarez) we always widely distributed “preprints” of papers prior to their publication or even submission.  That guaranteed a much wider peer review than we obtained from mere referees.

Please keep it confidential until we post it ourselves.

They want it widely reviewed. Now that The Economist has published on it, it is public knowledge.

There might be useful and interesting work here done by BEST, but I find it troubling that they can’t wait for science to do its work and run the peer review process first. Is their work so important, so earth shattering, that they can’t be bothered to run the gauntlet like other scientists? This is post normal science at its absolute worst.

In my opinion, this is a very, very, bad move by BEST. I look forward to seeing what changes might be made in peer review should these papers be accepted and published.

==============================================================

UPDATE: Judith Curry, who was co-author to some of these papers, has a post on it here

Also I know that I’ll be critcized for my position on this, since I said back in March that I would accept their findings whatever they were, but that was when I expected them to do science per the scientific process.

When BEST approached me, I was told they were doing science by the regular process, and that would include peer review. Now it appears they have circumvented the scientific process in favor of PR.

For those wishing to criticize me on that point, please note this caveat in my response above:

The Berkeley Earth Surface Temperature analysis methodology is new, and may yield some new and potentially important results on siting effects once the appropriate time period comparisons are made. I welcome the BEST effort provided that appropriate time periods are used that match our work. But, by using time period mismatched comparisons, it becomes clear that the Muller et al paper in its current form lost the opportunity for a meaningful comparison.

…

Given the new techniques from BEST, there may be much to gain from a rework of the analysis limited to identical thirty year periods used in Menne et al and Fall et al.

My issue has to do with the lost opportunity of finding something new, the findings may agree, or they may be different if run on the same time periods. I think it is a fair question to ask since my peer reviewed paper (Fall et al) and NOAA’s (Menne et al) paper both used 30 year periods.

If BEST can run their comparison on the 30 year period for which our data is valid, instead of 60 years, as stated before, I’ll be happy to accept the results, whatever they are. I’m only asking for the correct time period to be used. Normally things like this are addressed in peer review, but BEST has blown that chance by taking it public first before such things can be addressed.

As for the other papers supposedly being released today, I have not seen them, so I can’t comment on them. There may be good and useful work here, but it is a pity they could not wait for the scientific process to decide that.

================================================================

UPDATE2: 12:08 PM BEST has sent out their press release, below:

Here is the reply I got from Dr. Muller

==========================================================

UPDATE 5: The Guardian writer Ian Samples writes in this article:

The Berkeley Earth project has been attacked by some climate bloggers, who point out that one of the funders is linked to Koch Industries, a company Greenpeace called a “financial kingpin of climate science denial“.

Reader AK writes at Judth Curry’s blog: