Guest Post by Willis Eschenbach
I read a curious statement on the web yesterday, and I don’t remember where. If the author wishes to claim priority, here’s your chance. The author said (paraphrasing):
If you’re looking at any given time window on an autocorrelated time series, the extreme values are more likely to be at the beginning and the end of the time window.
“Autocorrelation” is a way of measuring how likely it is that tomorrow will be like today. For example, daily mean temperatures are highly auto-correlated. If it’s below freezing today, it’s much more likely to be below freezing tomorrow than it is to be sweltering hot tomorrow, and vice-versa.
Anyhow, being a suspicious fellow, I thought “I wonder if that’s true …”. But I filed it away, thinking, I know that’s an important insight if it’s true … I just don’t know why …
Last night, I burst out laughing when I realized why it would be important if it were true … but I still didn’t know if that was the case. So today, I did the math.
The easiest way to test such a statement is to do what’s called a “Monte Carlo” analysis. You make up a large number of pseudo-random datasets which have an autocorrelation structure similar to some natural autocorrelated dataset. This highly autocorrelated pseudo-random data is often called “red noise”. Because it was handy, I used the HadCRUT global surface air temperature dataset as my autocorrelation template. Figure 1 shows a few “red noise” autocorrelated datasets in color, along with the HadCRUT data in black for comparison.
Figure 1. HadCRUT3 monthly global mean surface air temperature anomalies (black), after removal of seasonal (annual) swings. Cyan and red show two “red noise” (autocorrelated) random datasets.
The HadCRUT3 dataset is about 2,000 months long. So I generated a very long string (two million data points) as a single continuous long red noise “pseudo-temperature” dataset. Of course, this two million point dataset is stationary, meaning that it has no trend over time, and that the standard deviation is stable over time.
Then I chopped that dataset into sequential 2,000 data-point chunks, and I looked at each 2,000-point chunk to see where the maximum and the minimum data points occurred in that 2,000 data-point chunk itself. If the minimum value was the third data point, I put down the number as “3”, and correspondingly if the maximum was in the next-to-last datapoint it would be recorded as “1999”.
Then, I made a histogram showing in total out of all of those chunks, how many of the extreme values were in the first hundred data points, the second hundred points, and so on. Figure 2 shows that result. Individual runs of a thousand vary, but the general form is always the same.
Figure 2. Histogram of the location (from 1 to 2000) of the extreme values in the 2,000 datapoint chunks of “red noise” pseudodata.
So dang, the unknown author was perfectly correct. If you take a random window on a highly autocorrelated “red noise” dataset, the extreme values (minimums and maximums) are indeed more likely, in fact twice as likely, to be at the start and the end of your window rather than anywhere in the middle.
I’m sure you can see where this is going … you know all of those claims about how eight out of the last ten years have been extremely warm? And about how we’re having extreme numbers of storms and extreme weather of all kinds?
That’s why I busted out laughing. If you say “we are living today in extreme, unprecedented times”, mathematically you are likely to be right, even if there is no trend at all, purely because the data is autocorrelated and “today” is at one end of our time window!
How hilarious is that? We are indeed living in extreme times, and we have the data to prove it!
Of course, this feeds right into the AGW alarmism, particularly because any extreme event counts as evidence of how we are living in parlous, out-of-the-ordinary times, whether hot or cold, wet or dry, flood or drought …
On a more serious level, it seems to me that this is a very important observation. Typically, we consider the odds of being in extreme times to be equal across the time window. But as Fig. 2 shows, that’s not true. As a result, we incorrectly consider the occurrence of recent extremes as evidence that the bounds of natural variation have recently been overstepped (e.g. “eight of the ten hottest years”, etc.).
This finding shows that we need to raise the threshold for what we are considering to be “recent extreme weather” … because even if there are no trends at all we are living in extreme times, so we should expect extreme weather.
Of course, this applies to all kinds of datasets. For example, currently we are at a low extreme in hurricanes … but is that low number actually anomalous when the math says that we live in extreme times, so extremes shouldn’t be a surprise?
In any case, I propose that we call this the “End Times Effect”, the tendency of extremes to cluster in recent times simply because the data is autocorrelated and “today” is at one end of our time window … and the corresponding tendency for people to look at those recent extremes and incorrectly assume that we are living in the end times when we are all doomed.
All the best,
w.
Usual Request. If you disagree with what someone says, please have the courtesy to quote the exact words you disagree with. This avoids misunderstandings.
“How hilarious is that? We are indeed living in extreme times, and we have the data to prove it!
Of course, this feeds right into the AGW alarmism, particularly because any extreme event counts as evidence of how we are living in parlous, out-of-the-ordinary times, whether hot or cold, wet or dry, flood or drought …”
Well, the way I see it is that we have a recent pattern that is in the 1/40 part of the externe, if I understand your distribution.
So two points:
1. How often should we be in “extremes”. Three times in the past century?
2. If there is variability, why don’t you people accept the possibility that the “pause” is variation…
davidmhoffer says:
April 24, 2014 at 5:56 pm
In other words, assuming the data is an undulating wave, it doesn’t much matter how you cut it up into smaller segments, you’re pretty much guaranteed to have extrema at both ends of the segment.
Thanks, David … I think that was very well worded and well explained.
oops … sorry, above was me … WordPress has taken over here…
If you’re looking at any given time window on an autocorrelated time series, the extreme values are more likely to be at the beginning and the end of the time window.
=====================================
??
It appears to me that the “beginning” and “end” need to be defined. The last 5 years of the T record are less extreme then the preceding five years and they are not random. If the end and beginning are defined as the first third and the last third, then you are covering 2/3rds of the series, and so more likely to have extremes within those segments.
(Likely I do not follow this at all.)
David M…”In other words, assuming the data is an undulating wave, it doesn’t much matter how you cut it up into smaller segments, you’re pretty much guaranteed to have extrema at both ends of the segment.”
====================================
What if you stard and end the undulation on the mean?
What if you start and end the undulation on the mean?
Even if we assume Brownian motion, a general closed form solution cannot be produced. It depends on the shape of the distribution of the difference between neighboring points. Because of the central limit theorem the effect isn’t huge, but there’s still an effect. I’ll try to work this out for a normal distribution but the integrals as a pain in the … (and I don’t have time).
What I like is that you can scale the autocorrelation (change the sigma of f(N)-f(N-1)) and it doesn’t make any difference.
David A;
What if you start and end the undulation on the mean?
>>>>>>>>>>>>>>>>.
Hmmmm. Well in my thinking out loud thoughts above, I presumed that each segment was smaller than the over all cycle. So if I understand your question, you’d be using a segment size that equals the entire cycle rather than a segment size that is only part of the cycle. But you could only do that if the over all trend across an entire cycle was zero, and you’d be able to manipulate what extrema showed up at what end simply by choosing where the start point was (ie peak to peak or valley to valley) Or, assuming that there is an underlying trend that is positive (and in Hadcrut there is a positive trend) by choosing a segment that starts and ends on the mean would be tough to do. You’d essentially have to choose an artificial start and end point over a part of the data that where those end points are at the mean value, which would be less than a complete cycle. Could you find such a segment? Probably. But it would be vary rare in comparison to other segments of the same length with random start and end points.
gary bucher says:
April 24, 2014 at 4:15 pm
So…any idea why this happens? It seems counter intuitive to say the least
================================================
Let me fathom a guess..
I would dare say that this is a manufactured illusion because TODAY is always considered an extreme time as it is ALWAYS at the end of the last data set. SO in its easiest form, this is a self fulfilling prophecy for the CAGW fear mongers. No matter when the cut is made in the data, the first and last will always be extreme.
Talk about creating your own perception of reality… (and all by accident for most)
Well Done WIllis!
First, you haven’t shown the data set is stationary – it’s simple an assumption or wild eyed guess.
Second, it’s called a temperature anomaly because it’s neither the temperature nor the mean deviation – the mean was pulled from where the Sun doesn’t shine so it has a linear trend. In any case, for partial correlations you need to demean the data and throw out the end points.
Third, the auto-correlation function is an even function, i.e.,
int[ f(u)*g(u-x)*du] = int[ f(u)*g(u+x)*du]
and auto-correlation function should have a maximum at zero lag which should be in the center of the plot (not on the left hand side.)
Try using R to do the calculation.
MarkY says: “When I grow up, I wanna be a statistician. Then I won’t have to tell my Mom I’m a piano player in a whorehouse (kudos to HSTruman).”
I believe H. Allen Smith said it first. He told his newspaper friends not to let his parents, in town for the weekend, know he was a journalist, that he’d told them the above.
Jeff L says:
April 24, 2014 at 6:50 pm
It is the extreme values (max & min) of the time series data points within the time window.
I’m not calculating the ACF of anything but the HadCRUT3 data. (Actually, I calculate the AR and MA coefficients of an ARIMA model of the HadCRUT3 data.) Then I used those coefficients to generate the temperature pseudo-data, so that it would resemble the HadCRUT3 data (see Fig. 1).
Welcome,
w.
Another very interesting one, Willis.
Is it possible to tie this in with Dr Roy Spencer’s climate null hypothesis? Current parameters such as extreme weather events, global temperatures, etc., have all been exceeded in the past — and not just the deep geologic past, but within the current Holocene.
Anyway, a very interesting hypothesis. Thanks for sharing.
++++++++++++++++++++++++++
trafamadore says:
why don’t you people accept the possibility that the “pause” is variation…
Maybe that is because to be properly labeled a “pause”, global warming would have had to resume. It may resume, or it may not. We don’t know.
But unless global warming resumes, the proper way to express the past 17+ years is to say that global warming has stopped. Sorry about all those failed runaway global warming predictions.
Oops, I meant you need to remove the linear trend.
There is another reason for ” it was the n hottest of the instrumental record”. The instrumental record is an S form with the hottest years at the top. Any year in the last 17 is guaranteed to be one of the top 17.
Humans have a natural tendency to “autocorrelate”. It is a perennial search for portents.
Here is my logical explanation…
An auto correlated time series is similar to a continous function in mathematics, since neighbouring points are more likely to be near each other.
For a continuous function, all global maxima occur either at local maxima or at the endpoints.
All local maxima occur at critical points (places where a function is either non-differentiable or the derivative is zero).
If you consider the space of all continuous functions, all points in the domain are equally likely to be critical points..
So that means that endpoints are more likely to be global maxima. They are equally likely as all other points to be critical points, and in addition, there are classes of continous functions where they are the maxima even when they are not critical points.
Michael D says:
April 24, 2014 at 4:43 pm
“I disagree with the suggestion that this is related to Benford’s law.”
Willis Eschenbach says:
April 24, 2014 at 4:57 pm
“Thanks, Gary. I don’t think it’s related to Benford’s distribution, it’s another oddity entirely.”
This from Wolfram: http://mathworld.wolfram.com/BenfordsLaw.html
One striking example of Benford’s law is given by the 54 million real constants in Plouffe’s “Inverse Symbolic Calculator” database, 30% of which begin with the digit 1. Taking data from several disparate sources, the table below shows the distribution of first digits as compiled by Benford (1938) in his original paper.
Scrolling down to the large table we find a broad range of electic data that fits, including populations of countries, areas of rivers, engineering/physics data such as specific heats of materials, etc. etc. I believe your extreme “high”s are the “1s” and the “lows” are the “9s” of the Benford distribution.
A similar idea is to look at the frequency of records – floods, temperatures, rainfall, snow… and the like as a random distribution of a set of numbers. In an N=200 (years) for example, counting the first year’s data point as a record, their will be approximately Ln N records in the 200 yr stretch. Even though the distribution of such data is not in fact perfectly random, it is surprising that you get something close to the actual number of records for the data set (I’ve done this for Red River of the North floods). Maybe Briggs or McIntyre might weigh in on the topic.
tchannon says:
April 24, 2014 at 6:12 pm
Neither did I. However, it’s easy to produce in R. I used the function TK95 from the package RobPer. Turns out it’s the same story. Here’s pink noise:
w.
Cinaed Simson says:
April 24, 2014 at 8:06 pm
Dear heavens, my friend, such unwarranted certainty. Of course I measured the mean, the trend, and the heteroskedasticity of the random data. As expected, the random data generator generates stationary data, no surprise there. And I was going to assume that, when I thought no, someone might ask me, and I’ve never checked it … so I did. Stationary.
However, instead of asking, you’ve made an unpleasant accusation that I’m either assuming it is stationary (wrong), or just guessing (also wrong).
Cinaed Simson. I’ll remember your name. Next time, rather than assuming bad faith, foolishness, or bad motives on my part, just ASK!
I’ll give you an example. You say “Willis, how do you know that your pseudo-data is stationary?”
See how easy it is?
Lay off the accusations. Not appreciated, not polite.
w.
Cinaed Simson says:
April 24, 2014 at 8:06 pm
For a person who doesn’t understand what I said, you certainly are unpleasant. Re-read my explanation. I don’t show the autocorrelation in Figure 2. Read the explanation of Figure 2 again, and the caption.
The whole post was prepared in R. What are you talking about?
w.
Willis – Good thinking, nice work! Following on from your post, I thought I would investigate the notion that nine of the last 10 years being the warmest “ever” was unprecedented. Answer : NO. It also happened back in 1945 and 1946. [I used Hadcrut4 from http://www.metoffice.gov.uk/hadobs/hadcrut4/data/current/time_series/HadCRUT.4.2.0.0.annual_ns_avg.txt%5D
The whole mathematical basis behind stealth radar technology applied to warplanes was done in closed form equations by the Russians and published in open math journals. The US defense industry found it and used it to create the F-117 and the B-2. The first time they took a scale mock up of the F-117 out to a radar test range they (US scientists) thought their instrumentation was broken; “how come we cannot see that metal object over there with our radar?”. The secret was the shape of the plane, then they applied radar absorbing coating and the plane virtually “disappeared” from the radar screen.
#####################
Wrong. wrong wrong.
First the F-117 and B2 used entirely different codes for RCS prediction. The F117 was limited to flat plates because the radiative transfer codes where limited to flat objects.. The used a program called echo1 yes taked from an obscure soviet math paper. northrop at the time did not have access to echo1.
The b2 was designed using an entirely different set of code far superior to echo1. It could handle curved surfaces ( very specific surfaces ). The algorithms were not from a soviet paper. The chinese gentlemen who wrote them was in my group
Autocorrelations are related to power spectra by the Fourier transform. And power spectra are the square of the magnitude spectrum (typically a Fourier transform of the times signal). What happens, even with just white noise? The power spectrum is flat? Not with the FFT!
Of course, no single white noise is flat. But if one takes the AVERAGE magnitude spectrum of a large set of white noise signals (millions say), it is supposed to trend more and more flat – a “schoolboy” exercise. If we take the magnitude spectrum as the magnitude of the FFT (the fast Discrete Fourier transform), it gets remarkably flat, save at one or two frequencies, where it is down by a factor of about 90%. One of the frequencies is 0. The other is fs/2 (half the sampling frequency) if we have an even number (N) of samples. The exact ratio seems to be 2^(3/2)/pi = 0.9003163. Astounding. IT SHOULD BE FLAT!
Well I had seen this for years and never found, or worked hard on an explanation until two years
ago:
http://electronotes.netfirms.com/EN208.pdf
(For example, Fig. 2 there.)
In essence, (and I think I got it right) it is because the DFT X(k) of a time sequence x(n) is of course by definition:
X(k) = SUM { x(n) e^(-j*2*pi*n*k)/N) }
This is a REAL random walk if k=0 (the exponential becomes just 1), and if k=N/2 and N is even, and has the “drunkard’s walk” normal distribution. For all other values of k (most values) we have a sum of vectors with COMPLEX random magnitudes (two dimensional random walk), and that’s a Rayleigh distribution (hence the different mean when we average FFTs at each k).
Einstein I believe thought that Nature was subtle, but not malicious.
Walpurgis says: “Interesting. Also, how likely would it be that you would want to go into a career in climatology if you believed the climate isn’t changing much and won’t until a few thousand years after you retire. What would you write your thesis on? What would you do every day?”
Well you could do paleoclimatology. That’s a lot of fun. Or you could do prehistorical-historical climatology (Discontinuity in Greek civilization Paperback, Rhys Carpenter; also R.A. Bryson). Interesting summary here: http://www.varchive.org/dag/gapp.htm
Or you could do cosmoclimatology. Hendrik Svensmark, Nir Shaviv, Jan Veizer, Eugene Parker and Richard Turco manage to keep busy. http://www.thecloudmystery.com/The_Cloud_Mystery/The_Science.html
Or you could do anthropological climatology. Elizabeth Vbra found that interesting enough to edit Paleoclimate and Evolution, with Emphasis on Human Origins. http://yalepress.yale.edu/yupbooks/book.asp?isbn=9780300063486
During the last 20 years climatologists, geophysicists and other scientists have revealed a few pages of the book of Earth’s climate system.
Still, our ignorance is greater than our knowledge and will continue to languish until scientists free themselves from the view that the science is settled..
Good find Willis. This looks to be of fundamental importance. However, trying to explain this to some Joe down at the bar who is freaked out “weird climate” is going to take some work.
“Of course, this applies to all kinds of datasets. For example, currently we are at a low extreme in hurricanes … but is that low number actually anomalous when the math says that we live in extreme times, so extremes shouldn’t be a surprise?”
I don’t see how that can apply. Your graph shows large magnitudes at the ends, not unusually small values.
This is all about ‘random walks’ and averaging.
The data is based on continual summing of a random ( gaussian distributed ) series. At the beginning the data is very short and that ‘random’ distribution has not been sufficiently sampled for the subset to accurately represent the gaussian distribution. Thus the chance of having a run of numbers in one direction or the other is greater.
A similar argument applies at the end. Since the middle of a reasonable long window has been well enough sampled to average out , the nearer you get the to end the stronger the chance is of a temporary run off to one side.ie the last few points are not a sufficient sample and can provide a non average deviation.
I’m wondering what the profile of your graph is. My guess is 1/gaussian