Following yesterday’s first look at emails from the 2 year effort to get NOAA to release emails from FOIA requests, where we learn that some scientists felt ‘Hit on the head with a hockey stick’ and that “The paleodata always got a lot more attention from the general public than it deserved.”.
There’s a great article today at SPPI today Michael Mann — the ghost of climate past which summarizes the whole hockey stick affair quite well. It draws heavily on a post from May of this year on PJ Media by Rand Simberg titled The Death of the Hockey Stick?
In that post I found this little nugget below from December 2009, one that I apparently missed in the furor immediately following Climategate1. I’m now correcting that oversight.
What is it? It’s the hockey stick recreated in Excel, using proxy data and instrumental data freely available on the web, something you can easily do at home. I figure the more people that explore this themselves using the easy to follow steps, the more people will understand what a statistical abuse the hockey stick is. The source of this tutorial is as surprising to me as it may be to you, and there’s no special software or secret Mannian Excel plugins needed to do the work yourself. I’ll let Rand Simberg explain.
From his May 2012 essay:
Ultimately, in addition to Mann’s claim for the dramatic recent uptick (which we are supposed to presume was a result of the late industrial revolution and equally dramatic increase in carbon dioxide into the atmosphere as a result of the liberation of carbon from burning long-buried fossil fuels), Keith Briffa of the Climate Research Unit (CRU) at the University of East Anglia in England controversially declared, based on Eurasian data, that the well-documented Medieval Warm Period (MWP), from around 950 to 1250 CE — the European Middle Ages — didn’t actually exist.
This claim was important, if not essential, to Mann’s thesis, because his initial formulation only went back to 1400, the beginning of the so-called Little Ice Age. Critics of the theory thus argued immediately upon its presentation that it shouldn’t be surprising that the earth was warming now, given that we are still coming out of it, and that the medieval warming in the absence of late Carolingian SUVs and coal plants argued that the climate naturally cycled, with no need to invoke Demon Carbon. That is to say, to the degree that the hockey stick has a blade in the twentieth century, it would have another a millennium ago.
The theory has continued to take blows over the years since it was first presented. About a decade ago, a paper was published by Willie Soon and Sallie Baliunis claiming that there was good evidence that both the (still extant) MWP and current warming were driven by solar activity rather than carbon emissions. But these initial attacks were beaten back by the climate mafia (as we now know from the leaked emails between Mann and his partners in crime in East Anglia from two and a half years ago). The real damage came when a retired Canadian mining engineer, Steve McIntyre, and a professor at the University of Guelph, Ross McKitrick, started digging into Mann’s methodology, and found flaws in both his statistical analysis and data interpretation, and published a paper describing them in Geophysical Research Letters in 2005. They showed that Mann’s methodology would generate a hockey stick almost independently of the data input, by feeding it spectral noise. Later, Internet satirist (and apparent statistician by day) Iowahawk provided a primer on how to create a hockey stick at home, using a standard spreadsheet program.
Yep, Iowahawk.
I liked this part of Iowahawk’s “how to” primer the best:
Is there anything wrong with this methodology? Not in principle. In fact there’s a lot to recommend it. There’s a strong reason to believe that high resolution proxy variables like tree rings and ice core o-18 are related to temperature. At the very least it’s a more mathematically rigorous approach than the earlier methods for climate reconstruction, which is probably why the hockey stick / AGW conclusion received a lot of endorsements from academic High Society (including the American Statistical Association).
The devil, as they say is in the details. In each of the steps there is some leeway for, shall we say, intervention. The early criticisms of Mann et al.’s analyses were confined to relatively minor points about the presence of autocorrelated errors, linear specification, etc. But a funny thing happened on the way to Copenhagen: a couple of Canadian researchers, McIntyre and McKitrick, found that when they ran simulations of “red noise” random principal components data into Mann’s reconstruction model, 99% of the time it produced the same hockey stick pattern. They attributed this to Mann’s method / time frame for selecting of principal components.
To illustrate the nature of that debate through the spreadsheet, try some of the following tests:
- Run step 3 through step 7, but only use the proxy data up through 1960 instead of 1980.
- Run step 5 through step 7, but only include the first 2 principal components in the regression.
- Run step 3 through step 7, but delete the ice core data from the proxy set.
- Run step 2 through step 7, but pick out a different proxy data set from NOAA.
Or combinations thereof. What you’ll find is that contrary to Mann’s assertion that the hockey stick is “robust,” you’ll find that the reconstructions tend to be sensitive to the data selection. M&M found, for example, that temperature reconstructions for the 1400s were higher or lower than today, depending on whether bristlecone pine tree rings were included in the proxies.
What the leaked emails reveal, among other things, is some of that bit of principal component sausage making. But more disturbing, they reveal that the actual data going into the reconstruction model — the instrumental temperature data and the proxy variables themselves — were rife for manipulation. In the laughable euphemism of Philip Jones, “value added homogenized data.” The data I provided here was the real, value added global temperature and proxy data, because Phil told me so. Trust me!
I urge readers to replicate it yourselves. Knowledge is power, especially first hand knowledge. Here’s all you need to do it. I’ll be happy to publish what you learn.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Anthony, the link to “Michael Mann — the ghost of climate past doesn’t work” – it should be http://sppiblog.org/news/michael-mann-the-ghost-of-climate-past.
Regards
Anthony, I’ve always wondered, why isn’t there a link for the SPPI site on your list of skeptic blogs? I often use WUWT as a base or springboard to visit other skeptic sites, but SPPI is not there.
Get Lost
By Rich Lowry
August 22, 2012 1:15 P.M.
So, as you might have heard, Michael Mann of Climategate infamy is threatening to sue us.
Mann is upset — very, very upset — with this Mark Steyn Corner post, which had the temerity to call Mann’s hockey stick “fraudulent.” The Steyn post was mild compared with other things that have been said about the notorious hockey stick, and, in fact, it fell considerably short of an item about Mann published elsewhere that Steyn quoted in his post.
So why threaten to sue us? I rather suspect it is because the Steyn post was savagely witty and stung poor Michael.
Possessing not an ounce of Steyn’s wit or eloquence, poor Michael didn’t try to engage him in a debate. He sent a laughably threatening letter and proceeded to write pathetically lame chest-thumping posts on his Facebook page. (Is it too much to ask that world-renowned climate scientists spend less time on Facebook?)
All of this is transparent nonsense, as our letter of response outlines.
In common polemical usage, “fraudulent” doesn’t mean honest-to-goodness criminal fraud. It means intellectually bogus and wrong. I consider Mann’s prospective lawsuit fraudulent. Uh-oh. I guess he now has another reason to sue us.
Usually, you don’t welcome a nuisance lawsuit, because it’s a nuisance. It consumes time. It costs money. But this is a different matter in light of one word: discovery.
If Mann sues us, the materials we will need to mount a full defense will be extremely wide-ranging. So if he files a complaint, we will be doing more than fighting a nuisance lawsuit; we will be embarking on a journalistic project of great interest to us and our readers.
And this is where you come in. If Mann goes through with it, we’re probably going to call on you to help fund our legal fight and our investigation of Mann through discovery. If it gets that far, we may eventually even want to hire a dedicated reporter to comb through the materials and regularly post stories on Mann.
My advice to poor Michael is to go away and bother someone else. If he doesn’t have the good sense to do that, we look forward to teaching him a thing or two about the law and about how free debate works in a free country.
He’s going to go to great trouble and expense to embark on a losing cause that will expose more of his methods and maneuverings to the world. In short, he risks making an ass of himself. But that hasn’t stopped him before.
— Rich Lowry is the editor of National Review.
But, but, where’s the corrected graph, you know, the one that uses all the correct methodology to not produce a hockey stick? Oh, it’s still a hockey stick?
Reblogged this on Climate Ponderings and commented:
My Favorite Defense of Mr Mann
“Mann’s work has been proven time and time again to not be fraudulent because it has been reproduced by others.”.
My Usual reply:
Why, Yes! It has been reproduced…BUT the question is – Why would you want to reproduce it?
Ha ha ha ha 🙂
Dear Anthony,
Some advertisement is causing your webpage’s main page not to load correctly at least in firefox. Only the first topic is presented, then you can read “ADVERTISEMENT” and then nothing else loads. Happened already 3 times to me.
Ed Scott says:
August 23, 2012 at 11:20 am
“Get Lost
By Rich Lowry”
It’ impolite not to link.
http://www.nationalreview.com/articles/314680/get-lost-rich-lowry
Andrew W,
But… but… Here is the corrected Mann chart. See? There is nothing unusual about current temps.
Here is another chart.
And here is a comparison of results using data that Mann threw out. Where’d Mann’s Hokey Stick go?
And here is the data that Mann hid in an ftp file labeled “censored“.
Here is Mann’s email from Tim Osborne.
And here is Prof Wegman’s comparison in Wegman’s testimony before Congress.
Here, Mann erases the MWP in order to make his hockey stick scarier.
And of course, the late, great John Daly totally deconstructs Mann’s chart here.
Which brings us to the present day.
Now tell the truth, Andrew. Would you still buy snake oil from that Mann?
It appears that Mann’s definition of “robust” is not robust.
Thanks smokey, your first link gives no source, your second link gives no source, your third link is broken, your forth link gives no source, your fifth link means nothing, your sixth link doesn’t actually show climate date at all, it’s just a random noise graph Wegman used to illustrate his argument, your seventh link shows no source, your eighth link is to large to address in its entirely, some bits are out dated others meaningless, your ninth link is I guess some sort of popularity contest, so meaningless with respect to the science.
Andrew W,
May I translate your comment? Thank you:
Translation.
U R welcome. ☺
Well what I want to know is whether I can make myself a Mann / McIntyre / McKitrick / Mouse / Iowahawk / whatever, Hockey stick from the numbers in the Mann Hattan Telephone book; mebbe that’s the Mad Hatter Telephone book; well either one.
I happen to be among the cognoscenti who are very familiar with the Sally Baliunas / Willy Wei Hock Soon paper referenced above. Read it as Soon as Willy and Sally published it.
What say you Iowahawk ? Your Excel spread sheet robust enough to Hockey Stickefy a Telephone directory ??
Andrew W,
There’s nothing quite like watching someone flail about in defense of Mann’s hockey stick.
This has been an otherwise humorless afternoon, so many thanks for your efforts.
Andrew,
You presume that there is a way to extract temperature data from these proxies and then send readers on a wild goose chase. The fact is, it does not really matter for this discussion. Mann’s methodology has been shown to be flawed given it’s sensitivity to minor input data differences. It’s not incumbent on me to show a “correct” methodology, even if one exists. Mann’s is still wrong and his conclusions worthless. You can’t change this with logical fallacies.
It’s all in how you SAY it…. roBUST! Emphasis on BUST!
I wonder a bit why it should be a problem that the proxies are correlated. In my opinien they should be if they really are good proxies for temperature and then some sort of average would do better.
I have played with SVD on the internet to learn more of what happens. http://metamerist.com/excanvas/example23a.htm
It takes a little work to format dataseries so that you can copy them into the calculator, but then it works fine. I doubt the results of such treatments.
If you have two series that goes more or less opposite, the first component will resemble the variation of them, the next component will be very small and only have the “noise”, but the first component only shows how they variate, not that they more or less cancel each other if averaged.
The crucial point is to select the number of components and their weights, and in there is the devil in all his might.
Andrew W says:
August 23, 2012 at 12:38 pm
Notice I didn’t include anything Andrew W says because this whole CAGWCF* issue hasn’t been about science from the beginning–so why reference other sources that aren’t about science?
Come clean, Andrew–where is YOUR SCIENCE?
But to the bigger issue, the question remains: Is Mann a criminal or a totally inept scientist?
My answer (and you all chime in now, you hear?) is that he’s both: he’s an inept scientist that has committed criminal behavior to cover his ineptness. And that makes him the worst type of “scientist” possible.
Sorta like a lot of other “climsci” people out there slopped on the trough of taxpayer funding.
*CAGWCF: Catastrophic Anthropogenic Global Warming Control Freaks, aka “watermelons”.
Another brilliant post.
Btw, Lucia had a whole series of posts recently creating hockey sticks aplenty out of sticky-back plastic (UK joke) and statistical quirks. It turned out to be so easy Josh cartooned the process:
http://rankexploits.com/musings/2012/gergis-by-josh-haiku-by-lucia/
I urge readers to read what the National Academy of Sciences said. And to read the numerous other studies in scientific journals which found the same relationship with temperature.
See! It’s replicable! What’s not to like?
“using proxy data and instrumental data freely available on the web, something you can easily do at home”
Available where? Which of it has been jiggered in which ways? After following the web log for a while, it’s all a very confusing mess, from my POV. Perhaps if I wanted to set aside all other interests.
Iowahawk is hilarious. OT, but here’s a great blog entry of his recounting a tale of the Reverend John St. Edwards, Lord Plaintiff of Durham, and the Reverend Albert des Gores II, Earl Carbonet of Greenhouse…
http://iowahawk.typepad.com/iowahawk/2010/07/the-two-randy-vicars.html
I think Mann was playing with words. Clearly “robust” means you get a hockey stick no matter what data you put in.
Schroedinger says:
August 23, 2012 at 1:30 pm
I’m woundering–did these “other studies in scientific journals” also use equivalent Mannianesque procedures? (The NAS defends Mann and his tactics, don’t they?)
Multiple wrongs (regardless of how many) don’t make a single right.
Svend Ferdinandsen says:
August 23, 2012 at 1:07 pm
It is not a problem for the proxies to be correlated. Indeed, if they are all linear combinations of the inputs (to tree growth), they should be correlated. What can’t be correlated (for PCA to work properly) are the inputs, i.e., the source signals driving the proxies, e.g., ring width, that PCA is attempting to extract. The components that result from any method (principal components, eigenvectors, etc.) will be uncorrelated by definition. If the inputs are correlated, then clearly any uncorrelated components that were extracted are not representative of the inputs.
Schroedinger says:
August 23, 2012 at 1:30 pm
Many of us have. Besides the fact that they were hardly providing a glowing endorsement, none of those that “said” the NAS position really dug into any of the details.
Using the same data and same basic methods*, of course. That’s the problem. Take a subset of the data and the results vary widely. In general terms, this means either the data, the methods, or both, are ill-suited for the purpose for which they are provided.
Mark
* If you do not understand what I mean by the “same basic methods,” then you are not capable of understanding any of the criticisms anyway.