Guest Post by Willis Eschenbach
With altogether far too much fanfare for my taste, the Berkeley Earth Surface Temperature (BEST) project has not released its preliminary results.
Or at least I can’t find them. I just wanted the month-by-month data that their hotrod new computer program spits out at the end of its run. The results they’re all hot and bothered about.
But despite releasing a massive database, 39,000 stations, along with the code in Matlab (which does me no good at all), I can’t find anywhere their freakin’ results. You know, the actual results of their work? The monthly average global temperature, the stuff that they mangled to produce things like their PR graph shown in Figure 1:
Figure 1. Purports to show that the BEST temperature record, and all the others as well, are all going “in one direction”, nowhere but up. I’m sure you remember the Climategate mantra, “hide the decline”? Keep that in mind as we proceed.
So … that will show those shifty skeptics, even BEST says it’s warming nonstop, evidence is right there before your eyes.
What’s not to like? How can you argue with that? The science is in.
Since I couldn’t get their results, I did the next best thing, and digitized their results. Even then, I was frunstrated. As far as I could tell, they never showed their actual results. The closest I found is in Figure 1 of their paper here:
Figure 2. Figure 1 of BEST’s “Decadal Variations” paper. Everybody’s going up, up, up, although you can’t really see what anyone is doing.
I blew that Figure up, and digitized it. Sixty years, 720 data points, boooring. Plus I hate it that they’ve smoothed the data, that makes it useless for statistical work. But it could have given us an idea of what’s going on in each of the records … if they hadn’t printed them atop one another in confusing colors. Enough of the spaghetti graphs already, you mad scientist persons, they show nothing! Figure 3 shows the BEST dataset along with the other datasets, this time displaced from each other so that we can actually see what’s happening:
Figure 3. The BEST land-only temperature record, compared to other surface and satellite land-only temperature records. 12 month moving average data, sadly. Note the decline.
[UPDATE: An alert reader noticed what I did not, that this is a subset of the BEST dataset that does not contain the stations used by the other groups (NOAA, etc). He points out that the full dataset is again different, in that it in fact rises more than the partial dataset. I have updated the figure and struck out some text to include that.
However, this doesn’t fix the questions. The post 1998 record from all of the BEST data is much more poorly correlated with the current records (~0.65) than prior to 1998 (~0.90). So this does not verify or validate the current groups datasets.
Hmmm … that gives a very different picture than Figure 1. Even with the bizarre 12-month moving average, the BEST record is clearly the outlier since 1998. You would think that in the modern era, the BEST would agree more closely with the other records. And indeed, from about 1975 to 1998 they were moving in something like lockstep.
But both before and after that time period, the BEST results are a clear outlier. And since 1998, BEST has been in a slow decline … funny how that didn’t show up in Figure 1. Yes, I know, a ten-year moving average shouldn’t show anything within five years from the end of the dataset. And I’m sure folks will argue that it’s just coincidence that they chose that exact smoothing length, and that it was the chance selection of colors that jumbled up the spaghetti graph so it’s unreadable … but y’know, after a while “coincidence” wears thin. I’m going with a more nuanced explanation, that it was a “deliberately unconscious choice to hide the decline”, although certainly you are welcome to stick to the story that it’s all just an unfortunate chain of events …
CONCLUSIONS:
Conclusion 1. It is extremely sneaky to send a truncated, smoothed result like Figure 1 out to the media to announce your results. That’s advocacy disguised as science. They did it to make it look like the temperature was headed for the sky and that BEST agreed. Instead, BEST actually disagrees with the other datasets by claiming that over the last decade, land temperatures are dropping, not staying stable or rising as per the other datasets. Using a graph that didn’t show that is … curious. As Gollum would say … “Oooooh, tricksy”. Including you, Judith. Figure 1 was nothing but “hide the decline” PR spin. Bad scientists, no cookies.
Conclusion 1. The correlation between the old data points used by the current groups, and the new data used only by BEST, is quite poor after 1998. This is visible in the plots of both the partial and full BEST datasets. The reasons for this are not clear, but it provides no support for the current datasets.
Conclusion 2. First point. The raw terror point, the thought that has the AGW alarmists changing their shorts, is the dreaded 2°C rise that is forecast from CO2. That is supposed to be the mythical “tipping point”. Second point. If we look at the 10-year smoothed data in Figure 1, BEST says that in the last two centuries, the temperature has risen about two degrees.
Let me note that over that two-century time period there have been:
a) No known increase in extreme weather events.
b) No known increase in catastrophes (other than from increased populations and property in vulnerable areas).
c) No major costs, deaths or damage from sea level rise. And don’t bother me with Katrina. A Category 3 hurricane took down ancient poorly maintained levees on a city below sea level. Absent that, no problem.
d) No climate-related spread of various infectious diseases.
e) No known increase in droughts or floods.
f) No loss of Tuvalu or other coral atolls.
g) Actually, none of the horrendous outcomes or biblical plagues of frogs and the like which are supposed to accompany the Thermageddon™ of a two degree temperature rise occurred over the last two centuries. To the contrary, the increased warming seems to have been a net gain for most humans, animals, and plants. Nobody likes freezing their asterisk off, after all, and the warming has mostly been in extra-tropical winter nights. That’s the theory, at least, although the BEST data should be able to tell us more.
Conclusion 3. BEST has done the world a huge service by collating and collecting all the data in one place, and deserves credit for that.
But they have done the world a huge disservice by becoming media whores, by putting out a shabby imitation of science in Figure 1, and by making a host of claims before peer review is complete.
This last one astounds me, that they’ve done it before peer review is finished. Doug Keenan and William Briggs have both raised separate and cogent arguments that the BEST analysis contains flaws. That would make me nervous, they’re kinda heavyweights, although any man can be wrong … but no, the BEST folks are making a host of claims as though their paper has already passed peer review. It’s the same publicity circus that Muller put on for Congress. And what three-ring media circus would be complete without their own brand new personalized “hide the decline” poster?
Since they have held out for extreme transparency, or at least given lip-service to the idea, I would be very interested to find out the names of the reviewers.
Because certainly, one possible explanation of their brazen trumpeting of their results before the peer review process is finished is that the fix is in. Why else the confidence that the reviewers will not find fault with their work? It is extreme hubris at a minimum, which is reputed historically to have unpleasant sequelae involving wax and feathers …
w.
PS—The world is warming. It has been for centuries. Rather than saying anything about anthropogenic global warming, all the BEST dataset does is confirms that. How that’s gotten twisted into some supposed “victory” for the AGW crowd escapes me.
Willis, Nigel Calder has an explanation for Richard Muller’s PR blitz of the global-warming-is-real story. Calder compares Muller’s PR with Tito’s propaganda which fooled Churchill.
http://calderup.wordpress.com/2011/10/22/hoodwinked-by-berkeley-earth/
has anyone correlated the BEST 1800-2000 series to the North American tree-ring density series yet?
I understood that the 1899-1900’s tree-rings were pretty flat, yet the BEST data shows a whole bunch of swings.
Reading the comments back and forth has given me a headache! This whole thing is just a complete crock, the fact remains that we are living in a period of nice temperate weather that allows us to grow plenty of food with nice levels of CO2 to help us. The temperature is not rising exponentially as was believed by some and is within normal ranges. There are no hard facts that prove humans have anything to do with it. The predictions from their ridiculously expensive computers have not come to fruition, the sky is falling frighteners have not come about.
The trillions that have been wasted on pointless research could have been much better spent on making sure that when natural events do happen, we were ready for them.
Even if, by some miracle, it were proven tomorrow that it IS our fault there is nothing we can do about it without going back to the dark ages everywhere on the planet, and that certainly won’t happen and shouldn’t!
All that is happening is that wealth is being re-distributed but not in the way some people had wished for. As always the poor are getting poorer and the rich richer. The likes of big oil have invested heavily in renewable energy schemes, the governments adds green taxes onto everyone’s fuel bills pushing many more pensioners and low earning families into fuel poverty. Here in the UK (20% green fuel tax on top of the 5% normal tax) it has got to the point that people are having to decide whether to eat or heat! Where is this money going? Into the pockets of the landowners who rake off huge amounts of subsidies from their wind farms for very little energy in return. If it were not for these subsidies wind would be a non starter in the energy front, no-one with any business acumen would touch them with a barge pole!
Soon we will come up with a way to make cheap, clean energy, hey that’s a novel idea lets use some of the money wasted… on overgrown windmills that: create huge amounts of CO2 during manufacture, transporting and siting, you know the ones that are pointless because they have to be backed up all the time by conventional power stations which, as these power stations have to be ramped up and down constantly, create even more CO2 than letting them run at best efficiency, yes that’s right the ones that create pollution on a huge scale as they use massive amounts of minerals that have to be mined for…on funding this research instead!
But wait, that would mean those getting richer off the backs of so called green technologies would lose their latest money spinner, so that aint gonna happen anytime soon!
There is something odd going on around 1967 that shows up in the BEST chart, but not in others. Like a huge El-Nino happened, bigger than 1998. Would guess some bad data got through (like several values of 99999 were averaged into the monthlies, then averaged again?), or some other issue that should stick out like a sore thumb…
No time to look at the moment, but if anyone else is looking at it, do you see anything strange in the data? I’ll look tonight.
Sorry Don B but that’s really Godwin’s law at work. Shame on Nigel Calder, there are other and more relevant things to say about BEST, Mulle and PR.
My preliminary conclusions:
1) Yes, the fix is in, has been in all along. The “All BEST” curve is more alarmist after 1998 than the others, not showing the levelling-off shown by the others. This is “Hide the Decline”, part 2, (which has been playing all along, well before BEST) showing that academics WILL NOT ADMIT THEIR INCOMPETENCE.
2) Peer review is thoroughly broken, and complaining about their not following it “properly” means so very, very little. Put simply: peer review = consensus. Remember “don’t throw the baby out with the bathwater”? Well, like it or not Anthony, peer review is NOT the baby, any more than the consensus is. Truth is the baby, and peer review has been hiding it, encrusting it with layers of mud (false theory), for a generation at least. Most skeptics don’t seem able to understand, the whole regulatory system of science is suborned, broken. There is no law west of Dodge, and we are in the wild, wild west. THE FIX IS IN, THROUGHOUT THE SYSTEM. Only public transparency, not “back room review”, can save climate science.
What is with you people?!
Willis posted his work for open review. An error was found, he promptly agreed it was an error, and made changes to the post recognizing the error.
Compare this to what we’ve come to expect as normal in Climate Science™, where a paper slides through peer-pal review, gets published, gets torn apart and debunked on skeptic blogs, and no matter how glaring the errors, how horribly fundamentally flawed it is, neither the authors nor the publication will ever admit anything was wrong and the paper remains complete to be cited as Absolute Proof™ for many years to come.
Willis has acted honorably. Comments critiquing his work were posted without censorship, and he explicitly invites constructive criticism. If what happened here causes you to lose respect for him or you think his reputation has been harmed, that’s your problem.
If you think this post should be stricken from this blog, this mess swept under the rug, because otherwise it damages the reputation of this site, you’re deluded. The openness of this site, with the willingness to leave such “flaws” open to view, has long been a great feature of this site, arguably its best one, as such “flaws” are a natural part of an atmosphere of open and vibrant debate. I’m just going to write you off with the other cranks who think this site should stick exclusively to the science, as the global warming debate has long since departed from being merely about the science, and it’d be suicidal nonsense to cede the politics and the PR to the “other side” when those have long been their main weapons.
Deserving of special mention, we also have this from Benedetto Castelli on October 23, 2011 at 1:00 am:
As mentioned by others, the link to the analysis chart data was buried at the bottom of a page, and the file was a zip file, not a simple ascii file. The five-column file is Full_Database_Average_summary.txt. Willis specifically stated:
The requested month-by-month data is in Full_Database_Average_complete.txt. It has 12 columns. That makes two counts of spouting off about the wrong thing in the same comment, in the same sentence as well.
And where is the data for Willis’ Figure 2, the BEST Figure 1? Where is the data for those “missing” stations, especially for the randomly-selected records that went into that graph? Could you possibly hold off on your righteous indignation until that particular data is revealed?
I have a bunch of charts on the Berkeley Earth land temperatures (downloaded from the Analysis page yesterday). These are the monthly values and the moving average values using the full 39,000 site database (this is not the individual stations databses, just the monthly averages – I also corrected one clear error in one month).
The Monthly Berkeley land temperatures going from 1800 to May 2010.
http://img200.imageshack.us/img200/3230/berkeleymonthlylandanom.png
There is huge variability in this data. I am almost certain they have underestimated the uncertainty involved here (and I start to wonder how the NCDC, Crutemp and GISS can have much more stable temperatures from month to month when they are using a smaller dataset). Here is a scatter (rather than lines/columns) of the same data.
http://img35.imageshack.us/img35/8241/berkeleymonthlylandscat.png
Look how much variability there is versus just the 12 month mean. Even today it is +/- 0.6C and in the early 1800s, it was +/- 2.0C. Any rising trend has to be viewed in that light. Any kind of error or change in measurement techniques is going to produce a trend all by itself.
http://img854.imageshack.us/img854/8255/berkeleyvariancescatter.png
Let’s compare Berkeley to the NCDC and Crutemp3 (GISS doesn’t really have a published comparable value given it includes some ocean temperatures). They are pretty close, at least on a 12 month moving average basis.
http://img21.imageshack.us/img21/7583/berkeleyncdccrutemp3.png
Berkeley, however, is very different on a monthly basis and has a higher increasing trend than both Crutemp3 and NCDC. Crutemp3 first.
http://img822.imageshack.us/img822/7093/berkeleyvscrutemp3.png
Berkeley is much closer to the NCDC.
http://img28.imageshack.us/img28/4859/berkeleyvsncdc.png
Finally, some comments about the early 1800s. Darn, it must have very cold and it must have been very difficult to grow crops in regions where frost is a concern today. 1807 to 1820 was a full 2.0C lower than today. I note the year without summer, 1815, does not actually show up as a cold period in this data, other years around it are much colder.
http://img828.imageshack.us/img828/2748/berkeleymovavglandanom.png
The coldest month on record was January 1809 at -4.2C (really?) and the warmest month on record was March 1822 at +2.4C.
‘They’ do try and estimate the anthropogenic contribution at various points in time. You’ll find considerable discussion in the IPCC reports under the topic headings including the word “Attribution.”
You clearly need to get familiar with the IPCC reports or the literature, or even semi-popular blog posts that are strictly about the science (as opposed to the tribalist howling from too many participants). In short – for the first half of the 20th century, no. For the latter half, pretty much. There are caveats. Acquaint yourself.
http://www.ipcc.ch/publications_and_data/ar4/wg1/en/ch9.html
One point I have never seen brought up is what to do about a leap year. If you can be bothered collecting millions of pieces of data and then you can’t be bothered thinking about the moving average having more or less days in it seems lazy to me. In other words should always be a factor of 4 years to show any trend.
In other words proof is not needed because the evidence is right there before your eyes. I guess that would be your lying eyes.
“If you think this post should be stricken from this blog”
No, you can’t strike a post once it was published. That would be dishonest.
But Willis could add an update at the top thus:
“Update: I made a mistake, I retract this post, please don’t comment on it any more. Will publish a new post later.”
Willis, I really enjoy your posts — more than any of the authors that most here — but this is not one of your brighter moments. So many climate scientists don’t post their code. These guys do, and then you gripe about it because it’s not in the language of your choice? Give me a break.
From Jacob on October 23, 2011 at 7:52 am:
Why? The post was clearly updated, changes were made with the original text preserved. There remains important points made and questions raised that should be addressed. They do not require retraction.
LazyTeenager:
“There have been many many claims that the world is not warming.”
Actually, the claim is that the world has not warmed since 1998. BEST produced a model result that shows it warming since 1998 and one that shows it cooling since 1998. And both HadCrut3 and RSS still show no warming since 1998.
“These attempts have been directed at the American and British datasets. Climategate, you may remember that, was promoted as a way of discrediting Phil Jones.”
Actually, the discrediting of data sets mostly had to do with proxy data. None of that has changed. Michael Mann is still using data sets upside down and he is still using data sets that cannot be reproduced. And he is still hiding the fact that his proxy records cannot match the instrument records in the last 50 years.
“Well it seems that all of the data sets are all producing the same results when analyzed by different people.”
But they aren’t. Even BESTs own data sets don’t produce the same results. Their decadal variations data set shows significant cooling since 98 and their main data set shows significant warming. As to the question of warming in the twentieth century, there was never any real debate about that in any case. With regard to that issue, the debate is about attribuition. And BEST does nothing to adress that issue. Also, the BEST effort to isolate the UHI effect is a complete failure, and therefore their decision to use no UHI correction means that their results are greater in the direction of warming than they should be.
Bill: “There is huge variability in this data. I am almost certain they have underestimated the uncertainty involved here ”
Apparently they did their uncertainty estimates on smoothed data, which many think is a big problem. Also, the removal of outliers, or their weighting of them towards a modeled value is another way that they remove uncertainty. And it is another questionable technique.
Robert Willee … even the CRU break down their data on the main index page by hemisphere. RSS and UAH break down the data by Tropics, Lower 48, etc.
BEST posted one number for each year in readily available form. And they just posted the land data which makes things appear even worse.
And they used a 1950-1980 baseline which resulted in almost 1.5C wamring from the late 1970s on.
They chose DRAMA. Not science.
barry says:
October 23, 2011 at 7:40 am
You clearly need to get familiar with the IPCC reports or the literature, or even semi-popular blog posts that are strictly about the science (as opposed to the tribalist howling from too many participants). In short – for the first half of the 20th century, no. For the latter half, pretty much. There are caveats. Acquaint yourself.
People who follow this blog know that many of us have studied the IPCC reports. Personally I have studied all 800 pages of the Physics justification. As a physicist I give a D to the report. I will not repeat here my reasons as I have posted them on the blog over the years, but I can assure you that their “estimate” of CO2 guilt is entirely dependent on shaky computer models with shaky assumptions in physics and statistics and replacement of spaghetti graphs instead of errors.
i see a lot of snark and bile that somebody has to suck back up. is it entertaining? not so much.
I do wonder why BEST goes all the way back to 1800. I seems like cherry-picking as this includes the Dalton Minimum and the Mount Tambora eruption of 1815. I am surprised that the plot link posted by Bill Illis at
http://img200.imageshack.us/img200/3230/berkeleymonthlylandanom.png
shows that the coldest year was 1808 rather than 1816, which was the renowned “Year Without a Summer” Any speculation why 1808 shows colder than 1816?
Looking at the divergence between land and sea temperatures since 1980, doesn’t this argue strongly that something is influencing surface temperatures that is unlikely to be CO2, which is well mixed?
Otherwise, why the divergence after sea and land temperatures tracked together for the 100 years previous?
“How that’s gotten twisted into some supposed “victory” for the AGW crowd escapes me.”
Willis, I agree with Joel Shore and LazyTeenager here. This is not a victory for AGW per se. It’s a defeat to those who were trying to discredit the AGW theory by implying that the temperature data was tampered with and that the Earth probably wasn’t even warming. Maybe you didn’t think that way, but many of your fellow AGW critics did. There is still a large proportion of the US population (I believe of the order of 30%) who think that the Earth may not be warming. If these BEST results help to finally lay to rest the issue of whether the Earth is warming or not, i.e. if the large majority of AGW critics can all stop questioning the rise in temperature, then this will be an indirect victory for the AGW advocates. Why you may ask? Because trying to debate whether GW is man-made or not is next to impossible if the opposing side doesn’t even acknowledge the warming!
Cheers.
I am surprised that none of the analysis of temperatures looks at subtracting the trend in sea surface temperatures from the trend in land surface temperatures, to isolate that portion of the trend that is specific to land use.
REPLY: I agree, see this, Anthony
I have the world wide text files, I have the land file; but not just the sea temperature average.
Anyone know where it is?
Pretty much all sceptics accept that there has been warming since the LIA.
However many suspect that the recent warming has been exagerrated by incompetent ‘adjustments’ especially in relation to the UHI effect.
Berkely has dealt with the first point but not the second.
The suggestion that UHI effect is not significant because rural and urban sites allegedly warm at much the same rate is flawed and deeply unscientific.
The error lies in dealing with a supposed absolute value for the UHI effect such that urban sites are developed and warm up but rural sites do not, being somehow preserved in aspic.
What really matters is the incremental UHI effect over time and on average it will be much the same for rural and urban sites for two reasons:
i) When a society is developing such development will occur around both rural and urban sites simultaneously.
ii) A small change near a rural site will have a larger effect than a large change near an urban site.
So, it is not surprising that urban sites are not seen to warm faster than rural sites. That fact (if correct) tells us nothing about the UHI effect.
I would have expected the Berkeley group to have worked that out without my help.