Guest Post by Willis Eschenbach
Anthony Watts, Lucia Liljegren , and Michael Tobis have all done a good job blogging about Jeff Masters’ egregious math error. His error was that he claimed that a run of high US temperatures had only a chance of 1 in 1.6 million of being a natural occurrence. Here’s his claim:
U.S. heat over the past 13 months: a one in 1.6 million event
Each of the 13 months from June 2011 through June 2012 ranked among the warmest third of their historical distribution for the first time in the 1895 – present record. According to NCDC, the odds of this occurring randomly during any particular month are 1 in 1,594,323. Thus, we should only see one more 13-month period so warm between now and 124,652 AD–assuming the climate is staying the same as it did during the past 118 years. These are ridiculously long odds, and it is highly unlikely that the extremity of the heat during the past 13 months could have occurred without a warming climate.
All of the other commenters pointed out reasons why he was wrong … but they didn’t get to what is right.
Let me propose a different way of analyzing the situation … the old-fashioned way, by actually looking at the observations themselves. There are a couple of oddities to be found there. To analyze this, I calculated, for each year of the record, how many of the months from June to June inclusive were in the top third of the historical record. Figure 1 shows the histogram of that data, that is to say, it shows how many June-to-June periods had one month in the top third, two months in the top third, and so on.
Figure 1. Histogram of the number of June-to-June months with temperatures in the top third (tercile) of the historical record, for each of the past 116 years. Red line shows the expected number if they have a Poisson distribution with lambda = 5.206, and N (number of 13-month intervals) = 116. The value of lambda has been fit to give the best results. Photo Source.
The first thing I noticed when I plotted the histogram is that it looked like a Poisson distribution. This is a very common distribution for data which represents discrete occurrences, as in this case. Poisson distributions cover things like how many people you’ll find in line in a bank at any given instant, for example. So I overlaid the data with a Poisson distribution, and I got a good match
Now, looking at that histogram, the finding of one period in which all thirteen were in the warmest third doesn’t seem so unusual. In fact, with the number of years that we are investigating, the Poisson distribution gives an expected value of 0.2 occurrences. In this case, we find one occurrence where all thirteen were in the warmest third, so that’s not unusual at all.
Once I did that analysis, though, I thought “Wait a minute. Why June to June? Why not August to August, or April to April?” I realized I wasn’t looking at the full universe from which we were selecting the 13-month periods. I needed to look at all of the 13 month periods, from January-to-January to December-to-December.
So I took a second look, and this time I looked at all of the possible contiguous 13-month periods in the historical data. Figure 2 shows a histogram of all of the results, along with the corresponding Poisson distribution.
Figure 2. Histogram of the number of months with temperatures in the top third (tercile) of the historical record for all possible contiguous 13-month periods. Red line shows the expected number if they have a Poisson distribution with lambda = 5.213, and N (number of 13-month intervals) = 1374. Once again, the value of lambda has been fit to give the best results. Photo Source
Note that the total number of periods is much larger (1374 instead of 116) because we are looking, not just at June-to-June, but at all possible 13-month periods. Note also that the fit to the theoretical Poisson distribution is better, with Figure 2 showing only about 2/3 of the RMS error of the first dataset.
The most interesting thing to me is that in both cases, I used an iterative fit (Excel solver) to calculate the value for lambda. And despite there being 12 times as much data in the second analysis, the values of the two lambdas agreed to two decimal places. I see this as strong confirmation that indeed we are looking at a Poisson distribution.
Finally, the sting in the end of the tale. With 1374 contiguous 13-month periods and a Poisson distribution, the number of periods with 13 winners that we would expect to find is 2.6 … so in fact, far from Jeff Masters claim that finding 13 in the top third is a one in a million chance, my results show finding only one case with all thirteen in the top third is actually below the number that we would expect given the size and the nature of the dataset …
w.
Data Source, NOAA US Temperatures, thanks to Lucia for the link.
Just pulling your leg, Willis.
Willis
Nice work. As Phil Jones is unable to use a spreadsheet I doubt if his high profile work is statistically sound. Don’t know about others like Mann as there is so much sound and light surrounding his work. I see him as the lynch pin so his expertise in statistics and analysis is obviously highly relevant
tonyb
Mr Masters is an intelligent weatherman, so he must already know that the US heatwave is primarily due to blocking, as was the Moscow heatwave a couple years ago.
Therefore the real question is “does CO2 cause an increase in blocking events?”.
Mr Masters may be able to tell me otherwise, but I’ve seen no hint that this is the case from the literature. But I’ve seen many times that low solar activity is linked with increased jet stream blocking.
I would therefore put the onus on Mr Masters to show that the null hypothesis is false: ie this event is related to solar activity, given the Ap index recently hit its lowest value for over 150 years.
It surely looks like a Poisson process and the 1 in 1.6 million figure is absolutely bollocks, but isn’t the interesting question how / if the lambda has changed over the years? Let’s say calculated from the data of a 30-year-or-so sliding period?
Unless you are certain of the underlying distribution, curve fitting and reading off the tails may lead to large errors. For example, you might try fitting a Burr distribution. It would be interesting to see if you get a similar result.
2.6 times in 1374 trials is a 1 in 528 chance.
Curiously, I happen to turn 44 years old this year which is also 528 months. (12 * 44)
cd_uk says:
July 11, 2012 at 1:45 am
Naw, you don’t have to be a pedant, cd_uk, it’s a choice, and one I’d advise you to give a miss.
But if you are going to be a pendant, you should at least be a good one. Mathworld says a histogram is:
Since that is exactly what I’ve done, it is indeed a histogram. Take a look at the Mathworld example, looks like mine.
w.
PS—By the way, what I believe you are talking about is generally called a “column chart”. In a “bar chart” the bars run horizontally, while in a column chart they run vertically.
The original mis-use of statistics is the sort that landed an innocent woman in prison for child abuse. Though eventually exonerated and released, she died by suicide. That’s how dangerous these people are!
It looks like a very nice analysis, thanks for it.
I’d just like to see a somewhat more solid proof that poisson distribution is the correct one to use in this case than “The first thing I noticed when I plotted the histogram is that it looked like a Poisson distribution.”. Both number of extreme records and number of their streaks is going to decline over time on normal data, but I suspect streaks are going to decline way faster because we’re working with fixed interval rather than portion of the record. Is the poisson distribution invariant to that?
Reading Willis’ response to rgb at 12:02 above puts me in mind of Richard Feynman talking about what his wife said with regard to Feynman joining the team to find out what happened to bring Challenger down….something along the lines of “You better do it…you won’t let go….you’ll keep circling around, looking at it from a different perspective than others…and it needs doing.” Ah, Willis, what a treasure you are….and in such good company.
I do agree with the general feeling that the “ridiculously long odds” are on very shaky ground, But I don’t agree with Willis’ statistical model. It may be that for n distinctly smaller than n the distribution of “n months in top third out of 13” is roughly approximated by a Poisson distribution, but it’s a leap of faith that the approximation is valid for n equal to 13. There is bound to be edge phenomenons.
For instance, the model predicts that “14 months out of 13 would be in the top third” happens about once in 1374 tries. On the other hand, we can be absolutely sure that this won’t happen until we get two Mondays in a week. This is an edge effect, The model breaks down for completely trivial reasons as soon as n is greater than 14, so we should not trust it too much for n equal to 13.
Great analysis. What really saddens me is that so many after years of study appear to have not done so well at basic statistics which is part of most courses where some analysis is likely to be required. May be it is just that many climatologists just can’t do statistics. I don’t know just looking for some rational explanation for the outrageous claims of so many over the last few years.
Willis – Beautiful work simply and elegantly explained. Entirely as we’ve come to expect from you.
Such calculations as p^N are based on independent random events which this is not.
With 20 and 60 year oscillations high event will occur roughly 60 years apart and the change of one being higher than the next is 1 / [the number of such events].
Over the last “118 years” there have been 3 (maybe even only 2) such events. Placing the odds of a 13 months of such significant highs occurring (without warming) at close to – 1 in 3 – sometime over this short time period at the height of the harmonic cycle.
With a small natural warming such has been going on since the LIA the chance is very close to 100%.
So if there is a claim that 13 months of co-joined, not independent, warm whether at the peak of this 60 year cycle is proof of a small natural warming trend (since 1895 or even the LIA) – the chances of that are approximately 66% likely.
To be fair though such limited datasets are usually rated against [N-1] events making even that an overstatement of the chance: more like 50% likely.
Statistics is for those who understand its intricacies, and I do not. It appears to me that extreme events may happen (but not necessarily) if certain principal conditions are satisfied. From my (non-climate related) experience this is most likely happen with cyclical events when one of two extreme plateaux (plural ?) is reached. If there is a 65 year cycle in the climate events (AMO at peak etc) , than it looks as the conditions are right for such events to be more frequent than usual.
Just a speculation of an idle mind.
So just to clarify – the actual odds of a running 13 month “month in top third” based on historical observational data is 2.6 out of 1374, or 1:528 (ish) ?
Maybe I should wait on Jeff opening a betting shop.
Willis, you say that picking the appropriate model for the situation is the central, crucial, indispensable, and often overlooked first step of any statistical analysis.
But the Poisson distribution is unbounded at the upper end. So the distribution that you fitted to your histogram also suggests that we should expect to find one instance (0.939) of a 13-month period in which 14 of the temperatures are in the top third. And it wouldn’t be that surprising to find a 13-month period with 15 (expected frequency 0.326) or 16 (expected frequency 0.106) of the individual months in the top third.
Does this really sound like the appropriate model?
Also, lambda in the Poisson distribution is the expected value of the mean of the data. So if you fit a Poisson distribution, you are determining that the mean number of months falling in their top third in a 13-month period is 5.213. (Note: the fact that you arrive at 5.213 seems odd to me, as I’d expect only 4.333 months out of every 13 on average to be in the top third. Am I missing something here?). However, it seems to me that your discovery that the mean of your distribution (5.213) remains the same when you oversample the same dataset is unsurprising. And it doesn’t really endorse the choice of Poisson as a distribution.
Nice work Willis. I like your approach.
So in over 1300 13 month periods, there has been just one where every month is in the top 3rd.
Yes, this was the right way to do the math (critic of earlier posts).
Have to look at the details more, like the reasoning for using 13 months, and using The warmest third instead of, say, warmest 10 percent, but at least the tools are good.
Willis, is your and his data Raw or after it has been mangled by Quality Control algorithms?
This sort of statistical mistake is reminiscent of certain other problems that do not necessarily follow simple intuition about stochastic events. The famous ‘birthday’ problem comes to mind– in a group of 50 people, there is about a 95% probability of two having the same birthday. In the case of the birthday problem, it is the difference between the probability of two particular people having the same birthday compared with any two people in a larger group. It raises the interesting question: If you included all possible combinations of six months out of a sequence of thirteen (rather than sequential months), would you arrive at an even higher probability? I suspect so, despite the fact that you would be reducing the correlations by spreading out the sample.
Major cities might almost have been designed to set higher max and higher min temperatures. Paint them black and carbon-dioxide could be awarded anothers recordbreaker’s medal – by dumb judges.
Willis Eschenbach says:
“I look because at the core, I’m trying to understand the data”
And that makes all the difference! It seems to me that the climate change action advocates are not trying to understand the data but are trying to understand how to use the data to promote the cause.
Nice work debunking this “wolf cry”.
Also, even if he were right about the stats, evidence for a warming climate is not proof it’s man made nor quantify how dangerous it might be.
Good job Willis.
The same weather pattern has been in place in the south for at least 30 years.
I recall my wife getting her citizenship on the steps of Tom Jefferson’s house in 1977 in 105 degree heat. Happens every year just a little more intense this year. Shade is strategic terrain.
Willis
I can see you’re conversational skills are as about as an inept as your stats.
Histograms deal in bins not categories: i.e. ranges 0-1, 1-2, 2-3 hence the x-axis labelling should be at each tick not between ticks you are plotting categories (0, 1, 2, 3, 4) where’s the bin range. As for Mathsworld it should know better. If you don’t believe me you can look at even elementary statistical packages such Excel: histogram vs bar chart functionality – note they are not the same.