Guest Post by Willis Eschenbach
With altogether far too much fanfare for my taste, the Berkeley Earth Surface Temperature (BEST) project has not released its preliminary results.
Or at least I can’t find them. I just wanted the month-by-month data that their hotrod new computer program spits out at the end of its run. The results they’re all hot and bothered about.
But despite releasing a massive database, 39,000 stations, along with the code in Matlab (which does me no good at all), I can’t find anywhere their freakin’ results. You know, the actual results of their work? The monthly average global temperature, the stuff that they mangled to produce things like their PR graph shown in Figure 1:
Figure 1. Purports to show that the BEST temperature record, and all the others as well, are all going “in one direction”, nowhere but up. I’m sure you remember the Climategate mantra, “hide the decline”? Keep that in mind as we proceed.
So … that will show those shifty skeptics, even BEST says it’s warming nonstop, evidence is right there before your eyes.
What’s not to like? How can you argue with that? The science is in.
Since I couldn’t get their results, I did the next best thing, and digitized their results. Even then, I was frunstrated. As far as I could tell, they never showed their actual results. The closest I found is in Figure 1 of their paper here:
Figure 2. Figure 1 of BEST’s “Decadal Variations” paper. Everybody’s going up, up, up, although you can’t really see what anyone is doing.
I blew that Figure up, and digitized it. Sixty years, 720 data points, boooring. Plus I hate it that they’ve smoothed the data, that makes it useless for statistical work. But it could have given us an idea of what’s going on in each of the records … if they hadn’t printed them atop one another in confusing colors. Enough of the spaghetti graphs already, you mad scientist persons, they show nothing! Figure 3 shows the BEST dataset along with the other datasets, this time displaced from each other so that we can actually see what’s happening:
Figure 3. The BEST land-only temperature record, compared to other surface and satellite land-only temperature records. 12 month moving average data, sadly. Note the decline.
[UPDATE: An alert reader noticed what I did not, that this is a subset of the BEST dataset that does not contain the stations used by the other groups (NOAA, etc). He points out that the full dataset is again different, in that it in fact rises more than the partial dataset. I have updated the figure and struck out some text to include that.
However, this doesn’t fix the questions. The post 1998 record from all of the BEST data is much more poorly correlated with the current records (~0.65) than prior to 1998 (~0.90). So this does not verify or validate the current groups datasets.
Hmmm … that gives a very different picture than Figure 1. Even with the bizarre 12-month moving average, the BEST record is clearly the outlier since 1998. You would think that in the modern era, the BEST would agree more closely with the other records. And indeed, from about 1975 to 1998 they were moving in something like lockstep.
But both before and after that time period, the BEST results are a clear outlier. And since 1998, BEST has been in a slow decline … funny how that didn’t show up in Figure 1. Yes, I know, a ten-year moving average shouldn’t show anything within five years from the end of the dataset. And I’m sure folks will argue that it’s just coincidence that they chose that exact smoothing length, and that it was the chance selection of colors that jumbled up the spaghetti graph so it’s unreadable … but y’know, after a while “coincidence” wears thin. I’m going with a more nuanced explanation, that it was a “deliberately unconscious choice to hide the decline”, although certainly you are welcome to stick to the story that it’s all just an unfortunate chain of events …
CONCLUSIONS:
Conclusion 1. It is extremely sneaky to send a truncated, smoothed result like Figure 1 out to the media to announce your results. That’s advocacy disguised as science. They did it to make it look like the temperature was headed for the sky and that BEST agreed. Instead, BEST actually disagrees with the other datasets by claiming that over the last decade, land temperatures are dropping, not staying stable or rising as per the other datasets. Using a graph that didn’t show that is … curious. As Gollum would say … “Oooooh, tricksy”. Including you, Judith. Figure 1 was nothing but “hide the decline” PR spin. Bad scientists, no cookies.
Conclusion 1. The correlation between the old data points used by the current groups, and the new data used only by BEST, is quite poor after 1998. This is visible in the plots of both the partial and full BEST datasets. The reasons for this are not clear, but it provides no support for the current datasets.
Conclusion 2. First point. The raw terror point, the thought that has the AGW alarmists changing their shorts, is the dreaded 2°C rise that is forecast from CO2. That is supposed to be the mythical “tipping point”. Second point. If we look at the 10-year smoothed data in Figure 1, BEST says that in the last two centuries, the temperature has risen about two degrees.
Let me note that over that two-century time period there have been:
a) No known increase in extreme weather events.
b) No known increase in catastrophes (other than from increased populations and property in vulnerable areas).
c) No major costs, deaths or damage from sea level rise. And don’t bother me with Katrina. A Category 3 hurricane took down ancient poorly maintained levees on a city below sea level. Absent that, no problem.
d) No climate-related spread of various infectious diseases.
e) No known increase in droughts or floods.
f) No loss of Tuvalu or other coral atolls.
g) Actually, none of the horrendous outcomes or biblical plagues of frogs and the like which are supposed to accompany the Thermageddon™ of a two degree temperature rise occurred over the last two centuries. To the contrary, the increased warming seems to have been a net gain for most humans, animals, and plants. Nobody likes freezing their asterisk off, after all, and the warming has mostly been in extra-tropical winter nights. That’s the theory, at least, although the BEST data should be able to tell us more.
Conclusion 3. BEST has done the world a huge service by collating and collecting all the data in one place, and deserves credit for that.
But they have done the world a huge disservice by becoming media whores, by putting out a shabby imitation of science in Figure 1, and by making a host of claims before peer review is complete.
This last one astounds me, that they’ve done it before peer review is finished. Doug Keenan and William Briggs have both raised separate and cogent arguments that the BEST analysis contains flaws. That would make me nervous, they’re kinda heavyweights, although any man can be wrong … but no, the BEST folks are making a host of claims as though their paper has already passed peer review. It’s the same publicity circus that Muller put on for Congress. And what three-ring media circus would be complete without their own brand new personalized “hide the decline” poster?
Since they have held out for extreme transparency, or at least given lip-service to the idea, I would be very interested to find out the names of the reviewers.
Because certainly, one possible explanation of their brazen trumpeting of their results before the peer review process is finished is that the fix is in. Why else the confidence that the reviewers will not find fault with their work? It is extreme hubris at a minimum, which is reputed historically to have unpleasant sequelae involving wax and feathers …
w.
PS—The world is warming. It has been for centuries. Rather than saying anything about anthropogenic global warming, all the BEST dataset does is confirms that. How that’s gotten twisted into some supposed “victory” for the AGW crowd escapes me.
Well done toto!
Anyway, it looks like we can have thorough confidence in the terrestrial temperature records. And we can add the satellite records as well.
One more piece of the AGW puzzle cemented happily in place.
I noted on Saturday in an English newspaper that a social study expects ‘energy deficient’ deaths (translation: rising electricity prices due to the conversion to wind power means that pelderly pensioners cannot afford to heat their homes) will increase by 2,700/year. Let’s hope that AGW is right after all so that they may live …
I agree. I think, we should be more reserved, with the BEST data-set. Let’s see what information is obtainable, from community analysis (and peer review), before rushing to sweeping judgments. Unfortunately, media misreporting headlines, will be sure to rile skeptics into over reaction. We can self inflict injury by spurious rejection. No matter its derivation, I find the plot interesting. Let’s see how much accuracy and precision remain after washing. GK
From Willis Eschenbach on October 22, 2011 at 8:47 pm:
I recall seeing the claim here and elsewhere that they were (deliberately?) dropping the stations that showed a cooling trend, biasing the overall record to increased warming. Now the graph of the “forgotten” stations shows a cooling trend, a “slow decline”.
Amazing coincidence, no?
toto says:
October 22, 2011 at 7:44 pm
Willis, you really outdid yourself this time.
The figure you plotted is not the actual Berkeley reconstruction of land temperatures. As explained in the article and in the very caption of the figure that you are showing, it is a partial reconstruction made using only data that was not used by the other records, for the specific purpose of this particular paper (analysing the effects of the AMO).
The real, full Berkeley reconstruction is plotted in Figures 5 and 8 of their first paper on their website – the one that actually describes their methods and results. Which apparently you have utterly failed to read.
As it turns out, the actual Berkeley reconstruction shows the exact opposite of what you are ranting about. Namely, the Berkeley reconstruction runs hotter than both GISS and HadCrut (but closely follows NOAA) in the last decade.
Obviously that didn’t prevent you from throwing copious accusations of dishonesty, based on nothing else than your own misunderstanding of papers that you either read casually or did not read at all.
I encourage all WUWT reader to check the papers by themselves and draw their own conclusions. Keep that in mind for the next time you see one of Willis’ rants.
________________
Wilis Says:
Thanks, toto. I encourage readers to do the same, nice catch. You are right, I was wrong … but the point is still valid. What they are showing is the new, never before seen BEST data.
So despite my acknowledged error, the question remains. Why do the stations we’ve never seen show a different result than the stations we have seen?
_________
So, is the end result that this post was a complete waste of time And that Willis’ reputation had taken a blow, because of apparent carelessness? And for what?
If Willis, or anyone else, wants to use the sort of invective of Willis’ original post, mercilessly attacking the BEST project, they’d better get their facts straight or the credibility of this entire blog will, rightly, fall into doubt. One can only imagine the folks at Real Climate, or wherever, licking their lips over this blunder.
Bad play, guys. And over such a relative side issue.
Sure, the freaking planet has warmed fractionally over the past 150 years or so. But so what?
Attribution, causation, intepretation of future implications, is where the action is.
Not a debate over whether there has been a 0.8 Celsius increase, or something slightly less, in global temps over the past century and a half. Where does that debate even lead? That the increase has only been 0.7Celsius? So what?
As Willis has shown, it is too early draw conclusions. I am willing to wait a month or so while the auditors dig through the data. I think it might be a good idea for others to do the same.
A 1991 paper published in Northwest Science *The Effect of Observation Time and Sampling Frequency on Mean Daily Maximum, Minimum and Average Temperature* at
https://research.wsulibs.wsu.edu:8443/xmlui/handle/2376/1631
provides cautions regarding sampling and averaging of daily temperature. From the abstract: *The use of long-term temperature data for climatic, ecohydrologic and other studies must be scrutinized carefully because of the average daily differences due to time of observation (TOB). . . The significance of our findings is that studies which require historical temperature records and where only small changes in temperature are expected, such as climate change modeling, will be difficult to verify. Also, mean daily temperature will change at locations when several readings are used to compute mean daily temperature rather than computing mean daily temperature from the dialy maximum and minimum temperatures.*
Re the over-used spaghetti graph, I offer the alternative handlebar tassel graph, after the accessory seen on kids’ tricycles.
Re Willis Eschenbach October 22, 2011 at 8:34 pm reply about BEST PR crowing, *Statistics of Extremes* by E. J. Gumbel provides this quotation: *Wenn der Hahn kraeht auf dem Mist, Aendert sich’s Wetter oder belibt wie’s ist.* See Section 6.3.
kadaka (KD Knoebel) says:
October 22, 2011 at 9:49 pm
Not a debate over whether there has been a 0.8 Celsius increase, or something slightly less, in global temps over the past century and a half. Where does that debate even lead? That the increase has only been 0.7Celsius? So what?
__________________________________________
As of right now the AGW movement has no real world data to support the claim for the CO2 effect. My guess is there is an active attempt to try and disprove by any means some of the anchor effects.
Temperature Rise, Sea Level Rise, Arctic Ice Melt are the ones that I think you will see a concerted effort to ” Help the Skeptics not to worry because here we show you it all real “. These are the primary points that the Skeptics have given them no quarter.
The Best study is the first wave to attempt to get some wiggle room by increasing the temperature by any amount. By removing UHI effects they can now proclaim that this little bit of extra heat must be AGW. They have to try something. Time is not on their side, every year the temp, sea level, or ice doesn’t match the ” Models ” they loose credibility with the public and it becomes much easier to become a Skeptic.
Willis – thanks for your efforts, but I am left with more questions than answers.
Does your Figure 3 use the data extracted from your Figure 2?
if so, then it is “randomly chosen from 30,964 sites that were not used by the other groups”. Does this mean that these sites are unreliable and therefore should not be used, or does it mean that the other groups should have used them? Or does it mean that these sites are somehow only appropriate for “analysing the effects of the AMO” and not for calculating global temperature?
The BEST data shows a marked temperature decline from about 2000, while ‘All BEST’ and the others show increase. Could this be an artefact of the severe drop in GHCN(?) station numbers in recent years, ie. the set of stations that the analyses all use? IOW could it be that there has been a bias in the dropped stations towards those not warming (no intention implied)?
Another big difference between BEST and the others is a big spike around 1968. Any explanation?
I am very concerned by William M Briggs’ comments (http://wmbriggs.com/blog/?p=4530): “The authors did [] some checking; e.g. they remove truly odd values (all zeros, etc.), but this cleaning appears minimal. They instead modeled temperature (as above) and checked the given observation against the model. Those observations that evinced large deviations from the model were then down-weighted and the model re-run. The potential for abuse here is obvious, and is the main reason for suspicion of the term “outlier.” If the data doesn’t fit the model, throw it out! In the end, you are left with only that data that fits, which—need I say it?—does not prove your model’s validity.” and “[The authors] say, “In this case we assess the overall ‘reliability’ of the record by measuring each record’s average level of agreement with the expected field [] at the same location.” At least reliability is used with scare quotes. Once again, this has the direct effect of moving the actual observations towards the direction of the model“. Aren’t these extraordinarily unsound practices? Wouldn’t the correct approach be to analyse the data without reference to the model, and only then compare results with the model?
TIA
A peer review process might have benefitted this article judging by the amount crossed-out content.
Is toto (the other one) part of Berkeley Earth? He seems qualified enough to use his real name. Me, I’m just another dog on the internet.
BEST’s Figure 1 shows an interesting spike in the 1967-68 range. For a 2000/30964 subsample that seems large.
p.3
The BEST temp plot from 1800 to present is presented as “proof” of special warming since 1950. You could also look at this as a pretty linear temp increase all the way from 1800 if you allow for some cycle-like variations which seem to change pattern about 1910. Before that the data is not conclusive. After that it could be a warm-cool-warm supplement to that linear trend. Can BEST really rule this out?
I fail to see why people complain about the early release. over and over again I see people quote the Spenser “study” that was “published” here. You dont like the results, dig through the math. Dont understand the code, study harder or write Muller.
Personally, I’m going to wait till the papers go through review before wasting any more time on it
Neil Jordan says:
October 22, 2011 at 10:59 pm
It is somewhat difficult to verify that quote, but here is another quote, spelled correctly, including an easily verifyable source:
Source: German proverbs, http://en.wikiquote.org/wiki/German_proverbs
Willis one reason that post 1998 and ALSO the early parts diverge is that there are many series that STOP before CRU common period AND many stations that start AFTER,
its kinda the point of Best. If you work with romans method as I have you’ll find this over and over again.
crosspatch says:
October 22, 2011 at 6:30 pm
“…what many people are not aware of (because the news really didn’t cover it) is that the communities East of New Orleans in Mississippi were extremely hard hit. Some towns were basically scraped off the face of the earth. ”
Sort of like 1969 eh? That one was an actual category 5. Not the overblown, after the fact tiny cat 5 swath of Andrew.
“There were no levees at that location, that was all wind and storm surge.”
and to think they had 36 years to remedy that problem after Camille.
What I’ve seen here is the author of the post who’s not able to find the data but still draws conclusions by eyeballing the (poorly digitized) wrong graph. And the whole story ends with ad hom and insults (“media whores”) to a team of respected scientists who, right or wrong they may be, did a real scientific work.
Luckly a McIntyre weights in to save the day and help download and read a simple 5 columns ascii file, which an average high school student could probably do.
Is this the best science blog or something has changed from 2008?
Is it right that the essence of trhe Berkely findings is that the rural and urban sites have been warming at a sinilar rate and so it is assumed that UHI effects are not a significant factor for determining the direction and rate of the temperature trend ?
Well, if so, how about the proposition that the incremental rate of nearby development is on average the same for both rural and urban sites ?
Wouldn’t that produce just such an outcome ?
During a period of development isn’t it just as likely that it will occur near a rural site as near an urban site ?
>>When we began our study, we felt that skeptics had raised legitimate
>>issues, and we didn’t know what we’d find.
Never trust any article or paper that starts off with trite comments like this. Seen it all too often before — we were all skeptics, but but the end of our research we were all true believers….
Yeah, we believe you.
.
Joel Shore says:
October 22, 2011 at 7:05 pm
Well, maybe if the “skeptic” crowd hadn’t spent so much time talking about how poor the siting of the stations was and making all these grandiose charges of data manipulation on the part of CRU and GISS, and so forth, it would not be viewed as such a big victory. One reaps what one sows.
I knew a man who took weather readings for the Met Office. When he got drunk he used to urinate in the rain gague!
Willis says
How that’s gotten twisted into some supposed “victory” for the AGW crowd escapes me.
———–
Well that’s pretty obvious. There have been many many claims that the world is not warming. Furthermore there has been a consistent campaign to discredit the thermometer record data sets and the scientist who collected and analyzed the data.
These attempts have been directed at the American and British datasets. Climategate, you may remember that, was promoted as a way of discrediting Phil Jones. People like Smokey are still insisting on interpreting some portions of these files as evidence of data manipulation.
Well it seems that all of the data sets are all producing the same results when analyzed by different people. So the BEST results are proof that all of those claims about data manipulation were lies.
This is why its called a victory.
Has anyone looked at the actual data that can be downloaded, the data.txt files? And I don’t mean plotted it with a program, but looked at it.
The BEST site states: “We mistakenly posted the wrong text data file (TMAX instead of TAVG). We apologize for any inconvenience or confusion this may have caused. The correct files are now in place and may be downloaded below.”
I got both versions. The first version, that was headed as Tmax has LOWER temperatures than the new version that is headed as Tavg. WTF? And the data still doesn’t make any sense. Like Chicago University Tavg in Dec 1982 of 13.3°C and Tavg in Jul and Aug 1983 of 13.1°C (Station ID 110766). Or Tmax in Feb 1984 of 3.5°C but Tavg of 7.8°C.
If they really used this data then it is no more than GIGO.
I note the continued lack of any attempt to plot error estimates on these graphics.
Whats Up With That ?
Come on – this is supposedly physical science ! Take a stab at couching your ‘math’ results in error bars please, even if it does make the crappy data look like total crap. ALL the errors that we know about. Science is suppose to describe reality. There are normally distributed data populations out there but you aren’t going to describe them the way you have been doing it.
You want to do the ‘BEST’ science ? Start collecting temp, RH and heat content with replicated random samples from a statistically valid sampling scheme with a coherent experimental design and equipment designed to test your hypotheses. There is no rush; earth will still be here and doing fine after we are shown the door.
All this thrashing around for over 20 years now and “climate scientists” still don’t get it. Start over and do it right or don’t bother. Billions and years wasted and the new data is the same CRAP as the old data, just newer. GIGO
I suggest the review process is very unlikely to leave this attribution statement unchanged:-
On the other hand, some of the long-term change in the AMO could be driven by natural variability, e.g. fluctuations in thermohaline flow. In that case the human component of global warming may be somewhat overestimated.
from
http://www.berkeleyearth.org/Resources/Berkeley_Earth_Decadal_Variations
Willis,
I’m a great fan of your work, but in this case, since you made a substantial mistake, it would be best to retract your post, cancel it, declare it null, and stop arguing about it.
Do a more profound and leisurely analysis, and if you have some meaningful and findings – post them later.
PS—The world is warming. It has been for centuries. Rather than saying anything about anthropogenic global warming, all the BEST dataset does is confirms that. How that’s gotten twisted into some supposed “victory” for the AGW crowd escapes me.
And there’s the crux of the AGW “science” right there. If it’s so clear cut how come they can’t actually tell us what part and how much of the warming carries the anthopogenic signal? Do they really believe that 100% of the warming is down to AGW? If so they aren’t anywhere near sufficiently competent to call themselves climate scientists or any other kind of scientist.