NOTE: This has been running two weeks at the top of WUWT, discussion has slowed, so I’m placing it back in regular que. – Anthony
UPDATES:
Statistician William Briggs weighs in here
Eduardo Zorita weighs in here
Anonymous blogger “Deep Climate” weighs in with what he/she calls a “deeply flawed study” here
After a week of being “preoccupied” Real Climate finally breaks radio silence here. It appears to be a prelude to a dismissal with a “wave of the hand”
Supplementary Info now available: All data and code used in this paper are available at the Annals of Applied Statistics supplementary materials website:
http://www.imstat.org/aoas/supplements/default.htm
=========================================
Sticky Wicket – phrase, meaning: “A difficult situation”.
Oh, my. There is a new and important study on temperature proxy reconstructions (McShane and Wyner 2010) submitted into the Annals of Applied Statistics and is listed to be published in the next issue. According to Steve McIntyre, this is one of the “top statistical journals”. This paper is a direct and serious rebuttal to the proxy reconstructions of Mann. It seems watertight on the surface, because instead of trying to attack the proxy data quality issues, they assumed the proxy data was accurate for their purpose, then created a bayesian backcast method. Then, using the proxy data, they demonstrate it fails to reproduce the sharp 20th century uptick.
Now, there’s a new look to the familiar “hockey stick”.
Before:
After:
Not only are the results stunning, but the paper is highly readable, written in a sensible style that most laymen can absorb, even if they don’t understand some of the finer points of bayesian and loess filters, or principal components. Not only that, this paper is a confirmation of McIntyre and McKitrick’s work, with a strong nod to Wegman. I highly recommend reading this and distributing this story widely.
Here’s the submitted paper:
(PDF, 2.5 MB. Backup download available here: McShane and Wyner 2010 )
It states in its abstract:
We find that the proxies do not predict temperature significantly better than random series generated independently of temperature. Furthermore, various model specifications that perform similarly at predicting temperature produce extremely different historical backcasts. Finally, the proxies seem unable to forecast the high levels of and sharp run-up in temperature in the 1990s either in-sample or from contiguous holdout blocks, thus casting doubt on their ability to predict such phenomena if in fact they occurred several hundred years ago.
Here are some excerpts from the paper (emphasis in paragraphs mine):
This one shows that M&M hit the mark, because it is independent validation:
In other words, our model performs better when using highly autocorrelated
noise rather than proxies to ”predict” temperature. The real proxies are less predictive than our ”fake” data. While the Lasso generated reconstructions using the proxies are highly statistically significant compared to simple null models, they do not achieve statistical significance against sophisticated null models.
We are not the first to observe this effect. It was shown, in McIntyre
and McKitrick (2005a,c), that random sequences with complex local dependence
structures can predict temperatures. Their approach has been
roundly dismissed in the climate science literature:
To generate ”random” noise series, MM05c apply the full autoregressive structure of the real world proxy series. In this way, they in fact train their stochastic engine with significant (if not dominant) low frequency climate signal rather than purely non-climatic noise and its persistence. [Emphasis in original]
Ammann and Wahl (2007)
…
On the power of the proxy data to actually detect climate change:
This is disturbing: if a model cannot predict the occurrence of a sharp run-up in an out-of-sample block which is contiguous with the insample training set, then it seems highly unlikely that it has power to detect such levels or run-ups in the more distant past. It is even more discouraging when one recalls Figure 15: the model cannot capture the sharp run-up even in-sample. In sum, these results suggest that the ninety-three sequences that comprise the 1,000 year old proxy record simply lack power to detect a sharp increase in temperature. See Footnote 12
Footnote 12:
On the other hand, perhaps our model is unable to detect the high level of and sharp run-up in recent temperatures because anthropogenic factors have, for example, caused a regime change in the relation between temperatures and proxies. While this is certainly a consistent line of reasoning, it is also fraught with peril for, once one admits the possibility of regime changes in the instrumental period, it raises the question of whether such changes exist elsewhere over the past 1,000 years. Furthermore, it implies that up to half of the already short instrumental record is corrupted by anthropogenic factors, thus undermining paleoclimatology as a statistical enterprise.
…
We plot the in-sample portion of this backcast (1850-1998 AD) in Figure 15. Not surprisingly, the model tracks CRU reasonably well because it is in-sample. However, despite the fact that the backcast is both in-sample and initialized with the high true temperatures from 1999 AD and 2000 AD, it still cannot capture either the high level of or the sharp run-up in temperatures of the 1990s. It is substantially biased low. That the model cannot capture run-up even in-sample does not portend well for its ability
to capture similar levels and run-ups if they exist out-of-sample.
…
Conclusion.
Research on multi-proxy temperature reconstructions of the earth’s temperature is now entering its second decade. While the literature is large, there has been very little collaboration with universitylevel, professional statisticians (Wegman et al., 2006; Wegman, 2006). Our paper is an effort to apply some modern statistical methods to these problems. While our results agree with the climate scientists findings in some
respects, our methods of estimating model uncertainty and accuracy are in sharp disagreement.
On the one hand, we conclude unequivocally that the evidence for a ”long-handled” hockey stick (where the shaft of the hockey stick extends to the year 1000 AD) is lacking in the data. The fundamental problem is that there is a limited amount of proxy data which dates back to 1000 AD; what is available is weakly predictive of global annual temperature. Our backcasting methods, which track quite closely the methods applied most recently in Mann (2008) to the same data, are unable to catch the sharp run up in temperatures recorded in the 1990s, even in-sample.
As can be seen in Figure 15, our estimate of the run up in temperature in the 1990s has
a much smaller slope than the actual temperature series. Furthermore, the lower frame of Figure 18 clearly reveals that the proxy model is not at all able to track the high gradient segment. Consequently, the long flat handle of the hockey stick is best understood to be a feature of regression and less a reflection of our knowledge of the truth. Nevertheless, the temperatures of the last few decades have been relatively warm compared to many of the thousand year temperature curves sampled from the posterior distribution of our model.
Our main contribution is our efforts to seriously grapple with the uncertainty involved in paleoclimatological reconstructions. Regression of high dimensional time series is always a complex problem with many traps. In our case, the particular challenges include (i) a short sequence of training data, (ii) more predictors than observations, (iii) a very weak signal, and (iv) response and predictor variables which are both strongly autocorrelated.
The final point is particularly troublesome: since the data is not easily modeled by a simple autoregressive process it follows that the number of truly independent observations (i.e., the effective sample size) may be just too small for accurate reconstruction.
Climate scientists have greatly underestimated the uncertainty of proxy based reconstructions and hence have been overconfident in their models. We have shown that time dependence in the temperature series is sufficiently strong to permit complex sequences of random numbers to forecast out-of-sample reasonably well fairly frequently (see, for example, Figure 9). Furthermore, even proxy based models with approximately the same amount of reconstructive skill (Figures 11,12, and 13), produce strikingly dissimilar historical backcasts: some of these look like hockey sticks but most do not (Figure 14).
Natural climate variability is not well understood and is probably quite large. It is not clear that the proxies currently used to predict temperature are even predictive of it at the scale of several decades let alone over many centuries. Nonetheless, paleoclimatoligical reconstructions constitute only one source of evidence in the AGW debate. Our work stands entirely on the shoulders of those environmental scientists who labored untold years to assemble the vast network of natural proxies. Although we assume the reliability of their data for our purposes here, there still remains a considerable number of outstanding questions that can only be answered with a free and open inquiry and a great deal of replication.
===============================================================
Commenters on WUWT report that Tamino and Romm are deleting comments even mentioning this paper on their blog comment forum. Their refusal to even acknowledge it tells you it has squarely hit the target, and the fat lady has sung – loudly.
(h/t to WUWT reader “thechuckr”)
@richard
The fact is that all those isolated places around the globe show periods of warming and cooling and they each show a peak of temperature (i.e. the MWP) in the period from 750 to 1250 AD. Together they indicate a peak near 1000 AD. This peak is the MWP.
How would you know when the peak was from historical and archeological proxies? How would you be able to compare France and Argentina, Australia and Nigeria? There simply is no system for establishing peak MWP without temperature proxies. And remember, you can’t use these any more (if you have accepted this paper), but you are willing to fall back on historical records and archeological evidence. I studied quite a lot of both, and I have to tell you that very few in either profession would go much beyond “some evidence that it was a bit warmer here.” Even the forensic archeologists I know would use lots of ifs, buts and maybes. So the evidence for the Incredible Moving Medieval Warming Period™ looks very slim indeed.
If a CAGW theorist made the kind of claims you are making, you would quite rightly destroy them. Historical and archeological records were only ever useful in establishing an alternative record for confirmation of temperature proxies, not in verifying those actual temperatures.
You are way out on a limb here. Come back in. 🙂
@Gail Combs August 18, 2010 at 5:24 am
The US ranks in or near the top 20 in scientific literacy. If you look at the questions in standardized scientific literacy tests you’ll find a couple that Americans don’t do well on that are related to evolution of life and age of the earth. Obviously the reason is that these clash with certain religious beliefs which are more widely held in the U.S. If you subtract the results of those two questions then the US rises well into the top ten and exceeds the average of western Europe.
There’s nothing really wrong with science education in the US nor the scientific literacy of the adult population in comparison with the rest of the world. There is very little practical impact in not believing mud-to-man evolution nor in the belief that God created the universe. This is why the U.S. has been and continues to be a leader in scientific discovery and engineering accomplishment – although no small part of that is the success of the constitutional republic form of government and capitalism which combine to encourage and provide great funding for scientific discovery and practical application thereof.
Duckster,
Go back and read my presentation, http://www.kidswincom.net/climate.pdf, with an open mind. In analyzing the the ice core isotope depletion data with several different statistical techniques, I was able to identify 13, statistically significant, naturally occuring cyles. The shortest significant cycle from the combined data was around twenty years and the longest was around 100,000 years. Plotting the best fit of all these natural cycles clearly shows both the MWP and LIA, as well as the Roman warm period.
If you question my ability to use statistics correctly, Google “Fred H. Haynie”+ statistics.
I’ve read this and it will be very educational for you.
Henry@Fred Haynie
Fred
“Go back and read my presentation, http://www.kidswincom.net/climate.pdf, with an open mind”
Henry
I liked this presentation . However, I could not figure out where we are now, within all these cycles. Do you expect cooling or warming?
CRS, Dr.P.H. says:
August 17, 2010 at 9:53 pm
“…As a public health scientist, I’m very comfortable with bringing in the high-level academic & practicing biostatisticians to help out in data analysis, study design etc. From the McShane/Wyner publication, we can see that the Hockey Team have taken the exact opposite tack….and the reasons why they would seek to do this (their entire argument falls apart upon statistical analysis)….”
I agree. Working in big Pharma, where our every scientific move is heavily scrutinized by various agencies around the world (such as the FDA) we work with statisticians as much as possible. In fact, when we want to put some real “teeth” behind our results, we most definitely bring in the statisticians. It’s a great relationship: they don’t necessarily know the science and we don’t necessarily know the math but by working together we achieve our greatest and most sound results. In other words, the statisticians are indispensible in our industry to ensure our results show that our drugs are both safe and efficacious. The fact that Mann et al. avoids statisticians makes me highly suspicious of their science.
So far, and I am reading this paper in digestible chunks, the model was able to reproduce the Mann blade, meaning that using the observations Mann used, the model these authors used was as good as the one Mann used in drawing the blade portion of the stick. And both models reproduced a rise that matches the data Mann chose as the observed temperature data. What the authors’ model could not do was backcast a trend that matched Mann’s backcast.
So we have a couple of questions.
1. What is the essential difference between the two models? Where is the code different from each other?
2. Do the models have a CO2 variable that reproduced the recent rise? Okay, fine. But I would have used a real control model. Is there another naturally occurring weather pattern variation that can send temperatures soaring (IE ENSO and Artic oscillations anyone)? These model runs, in my opinion, should also have a real (not made up random stuff) statistical model control: A model that is based on a training run of oceanic/atmospheric conditions tied to temperature data without adding a CO2 statistic. If both models (CO2 driven dynamic model and non-CO2 statistical only model) can predict a fast temperature rise, the hypothesis must continue to be null.
My criticism: The paper lacks a real control model and instead uses random data as the control. Why not get the two birds in the bush?
Fred H. Haynie says:
August 18, 2010 at 6:59 am
Duckster,
Go back and read my presentation, http://www.kidswincom.net/climate.pdf, with an open mind.
—-Reply:
Thanks, Mr. Haynie. That’s an excellent presentation!
This is an interesting paper. Watts has copied (but not highlighted) the following acknowledgment in the paper’s conclusion:
“Nonetheless, paleoclimatoligical reconstructions constitute only one source of evidence in the AGW debate. Our work stands entirely on the shoulders of those environmental scientists who labored untold years to assemble the vast network of natural proxies.”
In summary, there is still much work to be done in improving our measurements and our statistical analysis of natural temperature proxies. But the fundamental science of AGW by way of GHG emissions accelerating since the beginning of the industrial revolution is unchallenged. We have a problem on our hands, and it’s time we acknowledged this reality and moved on to addressing it somehow.
Duckster, temperature proxies are a dime a dozen. But to say you can’t use them anymore is going too far. The Pacific Decadel Oscillation was discovered when studying the ups and downs of salmon population and migration habits. Ship logs were, and still are, a treasure trove of temperature, winds, storm tracks, and currents. Salmon behavior turned out to be such a good proxie for the PDO that marine fisheries now use the PDO to plan out fishing trips.
Dave Springer, good post. Those science literacy quizzes should remove the bias from questions that have bearing on religious faith. The faithful will always answer these from a theological point of view, returning to science when questions go elsewhere. You can verify this easily by asking how a religious person would think about an “origins” question versus, for example, a chemical science question. They easily and readily report than such questions will always cause them to view possible answers from a religious point of view, thus destroying the question’s ability to measure scientific knowledge and understanding.
Besides, science fact/theory knowledge is different from understanding the scientific method. When I took the multiple subjects exam for teachers in Oregon, there were both kinds of questions: Multiple choice and scientific method essay. Interestingly, there were no questions related to how old the Earth/Universe is or how it was made. The essay questions were worth many more points than a single multiple choice question.
lattituide, you should go back and read the document. That figure shows that they stopped using the proxy at 1968 and then used the statistical method to try and predict the warming from 1969 to 2000. At the same time they put in the real recorded figures (the black line).
The point I was making is that the whole thing is rubbish. The scientists don’t use the proxies to predict the future. They use them as a single datum point for historical action in a physics model.
So it doesn’t matter what statistical model you use. If the proxies track the 130 year temprature record well then we can assume they also track the derived historical temperature record well.
What they cannot do is predict future warming using the historical temperature record by any statistical model known to man. Which is one of the reasons we know that WE are driving the current Global Warming.
However I’m sure it will be discussed to death here and trumpeted as some kind of justification for vilifying Mann.
Personally it’s not worth any more comments.
I think this snippet from the paper says it all.
“Finally, we construct and fit a full probability model for the relationship
between the thousand year old proxy database and Northern Hemisphere
average temperature, providing appropriate pathwise standard errors
which account for parameter uncertainty.”
They should have constructed their model using the local temperature series applicable to each proxy database, rather than to the hemispherical mean. After all a tree ring proxy predicts local temperature, not hemispherical mean.
I think it is an example of garbage in garbage out as I have seen posted numerous times on this site.
Stephan says:
August 18, 2010 at 6:33 am
It seems obvious that the team is really frightened about this one and are tryin to push the line that it confirms Mann et al what a joke! Deltoid represents what Australian education and science has become… 3rd world level…. period.
That site is almost unbelievable.
Some poor new poster asks a polite question and a pack of piranhas descend on him/her and hurl really unrestrained vile abuse.
I think that WUWT should provide a link to Deltoid and label it as ” a site where the science of AGW gets logically explained”
bob says:
“They should have constructed their model using the local temperature series applicable to each proxy database, rather than to the hemispherical mean. After all a tree ring proxy predicts local temperature, not hemispherical mean. I think it is an example of garbage in garbage out as I have seen posted numerous times on this site.”
Bob, the statistical study used Michael Mann’s own data. The difference is that they are statisticians and Mann is a rank amateur on statistics.
[I am giving Mann an easy excuse as an incompetent, rather than as a scientific charlatan who tortured his carefully selected data until it said what he wanted it to say.]
Did Ryan Z say this:
In summary, there is still much work to be done in improving our measurements and our statistical analysis of natural temperature proxies. But the fundamental science of AGW by way of GHG emissions accelerating since the beginning of the industrial revolution is unchallenged. We have a problem on our hands, and it’s time we acknowledged this reality and moved on to addressing it somehow.
I doubt if I would agree with this statement, and I am not sure if the problem is CO2.
Suggest you read my blog:
http://letterdash.com/HenryP/more-carbon-dioxide-is-ok-ok
NeilT says:
“So it doesn’t matter what statistical model you use. If the proxies track the 130 year temprature record well then we can assume they also track the derived historical temperature record well.
What they cannot do is predict future warming using the historical temperature record by any statistical model known to man. Which is one of the reasons we know that WE are driving the current Global Warming.”
Huh? Your last sentence is a complete non-sequitur, and makes the usual argumentum ad ignorantium that is constantly employed by the alarmist contingent.
The “current global warming” is no different whatever from past global warming cycles. Even arch-alarmist Phil Jones acknowledges that fact.
Making a baseless assumption that what we see today is from a demonstrably different cause, than the same exact cycles happening in the past, is what always trips up the Chicken Licken crowd.
Construct a testable, replicable experiment using raw data, showing that CO2 is the cause of the current [mild] warming cycle. If you do you will convince me. But so far, all the hand-waving over what is clearly a natural event has been completely unconvincing.
CO2 may cause a small amount of warming. If so, it has turned out to be much less than assumed. If the minor warming from CO2 is insignificant, then there is no cause to throw good taxpayer money after bad for any more expensive, wild-eyed ‘global warming studies.’
and Smokey says:
“Bob, the statistical study used Michael Mann’s own data. The difference is that they are statisticians and Mann is a rank amateur on statistics.
[I am giving Mann an easy excuse as an incompetent, rather than as a scientific charlatan who tortured his carefully selected data until it said what he wanted it to say.]”
And I’m saying that although they may be competent statisticians, they did not apply statistics properly to the problem.
They made a calibration error in calibrating the proxies to the hemispherical mean, rather than to the local mean.
I mean it is as simple as that, a proxy predicts local temperature, not the hemispherical mean.
If I want to choose a coat to wear, I take a tree ring sample from my backyard, not averages from 93 samples across the northern hemisphere.
Just because they are good statisticians doesn’t mean they are setting up the problem they are trying to solve correctly.
bob says:
August 18, 2010 at 9:00 am
“They should have constructed their model using the local temperature series applicable to each proxy database, rather than to the hemispherical mean. After all a tree ring proxy predicts local temperature, not hemispherical mean.”
Yeh bob, they were probably totally unaware of that particular option being available. If only they’d asked bob before hand.
From the paper,
“3.6. Proxies and Local Temperatures. We performed an additional test which
accounts for the fact that proxies are local in nature (e.g., tree rings in Montana)
and therefore might be better predictors of local temperatures than
global temperatures. Climate scientists generally accept the notion of ”teleconnection”
(i.e., that proxies local to one place can be predictive of climate
in another possibly distant place). Hence, we do not use a distance restriction
in this test. Rather, we perform the following procedure.”…it goes into great detail of the testing procedure. Later, “The results of this test are given by the second boxplot in Figure 9. As can be seen, this method seems to perform somewhat better than the pure global method. However, it does not beat the empirical AR1 process or Brownian Motion. That is, random series that are independent of global temperature are as effective or more effective than the proxies at predicting global annual temperatures in the instrumental period. Again, the proxies are not statistically significant when compared to sophisticated null models.“
bob, it seems they regarded the local vs. global considerations and actually did the testing. Apparently the results were just as dismal.
bob says:
“… a proxy predicts local temperature, not the hemispherical mean. If I want to choose a coat to wear, I take a tree ring sample from my backyard, not averages from 93 samples across the northern hemisphere.”
That is exactly what MBH critics have been saying: Mann carefully selected proxies that reflected very local conditions [if they reflect anything at all]. But he foisted off his erroneous results as proof of anthropogenic global warming.
Seriously, get a copy of A.W. Montford’s The Hockey Stick Illusion. You will see the endless shenanigans used by Mann, the IPCC and the climate journals to promote a scientifically untenable agenda in order to keep the taxpayer loot flowing into their pockets.
Why do you think it is that 12 years after MBH98, Mann still refuses to allow skeptical scientists to see his methodologies? He’s not protecting nuclear defense secrets here — this is only the weather and climate; what’s the big secret? But Mann is still hiding his work product from the public that paid for it. Can anyone justify that?
bob says: (August 18, 2010 at 9:45 am )
“And I’m saying that although they may be competent statisticians, they did not apply statistics properly to the problem.”
But Mann, not a statistician, applied the statistics properly to the problem?
Pamela writes:
So far, and I am reading this paper in digestible chunks, the model was able to reproduce the Mann blade, meaning that using the observations Mann used, the model these authors used was as good as the one Mann used in drawing the blade portion of the stick. And both models reproduced a rise that matches the data Mann chose as the observed temperature data. What the authors’ model could not do was backcast a trend that matched Mann’s backcast.
BINGO!
I have no idea why that is so hard for people to understand. I also see a lot of comment about this paper calling into question the use of proxies to find temperature signals. No. It is not questioning that, but simply using the specific ones Mann used for this particular study.
Henry Pool August 18, 2010 at 7:44 am
The proxie data used to identify the cycles is calibrated to measured Western Equatorial Pacific SSTs where a great bit of energy goes into the atmosphere. The timing of each cycle is an average of 10,000 years of cycles. There is variability in the wave length between individual cycles. I did not analyze that variability but estimated that it could range within about plus or minus 25%. That said, in the presentation it stated that the proxie data suggested that we were on the up slope of a 20 year cycle and expected to be on a down slope within a couple of years. Based on recent satalite data, we have been on that down slope for several years. That said, the longer wave length cycles suggest we are on an up slope that will have a maximum at around 2090. This is consistent with a 308 year cycle that is statistically significant for CO2 data.
Re: Tom Scharf says:
August 17, 2010 at 11:22 am
When faced with a problem like this where the authors have
shown legitimate problems with methods and have reached
reasonable conclusions, you don’t fight the methods or conclusions,
you will attack the very basis of their paper. When you are guilty,
argue the law.
The elegant feature of the McS & W2010 paper is the use of
the Mann et alia data as the basis for the various reconstructions,
backcasts, and comparative statistical processes. They
(the Team) picked the facts to use… and proffered conclusions
based on their pick of the litter.
Since Mike Mann and the Team selected the proxies and temp
reports in the first place, there’s really not much left to refute
in McS & W other than the application of the maths in the statistical
treatments.
Having the database nailed to the side of the shed for all to
ponder may induce Mann to give up his actual coding… or look
rather foolish.
Let this rather rusty stats man presume Brigg’s yellow line is the most devastating peer review line drawn through an iconic hockey stick the world has ever seen, based on McShane and Wyner’s peer review of same. What that means is the world can be as 95% confident in the straight yellow line of temperature through the IPCC’s world’s best historic climate record gatherings as any other line within that 95% confidence band. In which case it completely verifies what Congress, Mckintyre and Mckitrick with Wegman told us all some years ago and were reviled and pilloried for in the greatest scientific scandal of our age- Advocacy pseudo-science.
If all that is true there is some immediate Nobel stripping to be done and some great names to be added to the roll call of Science, to retrieve the Nobel from the gutter and place it back on its rightful pedestal.
I think the problem with climatology is the failure to separate the problem in smaller units in order to make it accessible to critique. In computer programming it is called modular programming. The goal is to separate the problem in small modules so each of them is easy to manage and any programmer can criticize(debug) them.
In this case, scientists should have decomposed the problem in several units(tree growth in relation to time, correlations between recent tree growth and recent temperatures, reconstruction of past temperature). Each of these units should have been so easy, the methodology would have been clearly expressed and a lot of people would have been able to criticize them.
In computer programming, I might fail to do a good modularity if I think too much about the final program instead of looking to the different part needed. I think science based on goals will always fail against science based on modularity and application of the scientific method because the former will often fail to be accessible to a larger pool of critics.