Guest Post by Willis Eschenbach
I read a curious statement on the web yesterday, and I don’t remember where. If the author wishes to claim priority, here’s your chance. The author said (paraphrasing):
If you’re looking at any given time window on an autocorrelated time series, the extreme values are more likely to be at the beginning and the end of the time window.
“Autocorrelation” is a way of measuring how likely it is that tomorrow will be like today. For example, daily mean temperatures are highly auto-correlated. If it’s below freezing today, it’s much more likely to be below freezing tomorrow than it is to be sweltering hot tomorrow, and vice-versa.
Anyhow, being a suspicious fellow, I thought “I wonder if that’s true …”. But I filed it away, thinking, I know that’s an important insight if it’s true … I just don’t know why …
Last night, I burst out laughing when I realized why it would be important if it were true … but I still didn’t know if that was the case. So today, I did the math.
The easiest way to test such a statement is to do what’s called a “Monte Carlo” analysis. You make up a large number of pseudo-random datasets which have an autocorrelation structure similar to some natural autocorrelated dataset. This highly autocorrelated pseudo-random data is often called “red noise”. Because it was handy, I used the HadCRUT global surface air temperature dataset as my autocorrelation template. Figure 1 shows a few “red noise” autocorrelated datasets in color, along with the HadCRUT data in black for comparison.
Figure 1. HadCRUT3 monthly global mean surface air temperature anomalies (black), after removal of seasonal (annual) swings. Cyan and red show two “red noise” (autocorrelated) random datasets.
The HadCRUT3 dataset is about 2,000 months long. So I generated a very long string (two million data points) as a single continuous long red noise “pseudo-temperature” dataset. Of course, this two million point dataset is stationary, meaning that it has no trend over time, and that the standard deviation is stable over time.
Then I chopped that dataset into sequential 2,000 data-point chunks, and I looked at each 2,000-point chunk to see where the maximum and the minimum data points occurred in that 2,000 data-point chunk itself. If the minimum value was the third data point, I put down the number as “3”, and correspondingly if the maximum was in the next-to-last datapoint it would be recorded as “1999”.
Then, I made a histogram showing in total out of all of those chunks, how many of the extreme values were in the first hundred data points, the second hundred points, and so on. Figure 2 shows that result. Individual runs of a thousand vary, but the general form is always the same.
Figure 2. Histogram of the location (from 1 to 2000) of the extreme values in the 2,000 datapoint chunks of “red noise” pseudodata.
So dang, the unknown author was perfectly correct. If you take a random window on a highly autocorrelated “red noise” dataset, the extreme values (minimums and maximums) are indeed more likely, in fact twice as likely, to be at the start and the end of your window rather than anywhere in the middle.
I’m sure you can see where this is going … you know all of those claims about how eight out of the last ten years have been extremely warm? And about how we’re having extreme numbers of storms and extreme weather of all kinds?
That’s why I busted out laughing. If you say “we are living today in extreme, unprecedented times”, mathematically you are likely to be right, even if there is no trend at all, purely because the data is autocorrelated and “today” is at one end of our time window!
How hilarious is that? We are indeed living in extreme times, and we have the data to prove it!
Of course, this feeds right into the AGW alarmism, particularly because any extreme event counts as evidence of how we are living in parlous, out-of-the-ordinary times, whether hot or cold, wet or dry, flood or drought …
On a more serious level, it seems to me that this is a very important observation. Typically, we consider the odds of being in extreme times to be equal across the time window. But as Fig. 2 shows, that’s not true. As a result, we incorrectly consider the occurrence of recent extremes as evidence that the bounds of natural variation have recently been overstepped (e.g. “eight of the ten hottest years”, etc.).
This finding shows that we need to raise the threshold for what we are considering to be “recent extreme weather” … because even if there are no trends at all we are living in extreme times, so we should expect extreme weather.
Of course, this applies to all kinds of datasets. For example, currently we are at a low extreme in hurricanes … but is that low number actually anomalous when the math says that we live in extreme times, so extremes shouldn’t be a surprise?
In any case, I propose that we call this the “End Times Effect”, the tendency of extremes to cluster in recent times simply because the data is autocorrelated and “today” is at one end of our time window … and the corresponding tendency for people to look at those recent extremes and incorrectly assume that we are living in the end times when we are all doomed.
All the best,
w.
Usual Request. If you disagree with what someone says, please have the courtesy to quote the exact words you disagree with. This avoids misunderstandings.
Wonderful explanation of a wonderful insight, Willis. Just what we expect from you.
Michael D, agreed. This has nothing to do with Benford’s Law. But thanks for bringing my attention to it.
For example, currently we are at a low extreme in hurricanes
For additional things that are extreme now, see:
http://www.theweathernetwork.com/news/articles/extreme-weather-events-not-making-headlines/25948/
Extreme weather events NOT making headlines
Dr. Doug Gillham
Meteorologist, PhD
1. GREAT LAKES ICE COVERAGE
“Current ice coverage is over double the previous record for the date (April 23) which was set in 1996.”
2. SLOW START TO TORNADO SEASON
3. LACK OF VIOLENT TORNADOES
4. QUIET 2013 TROPICAL SEASON
My gut feeling is you have only proved your time series is band-limited both in low and high frequencies.
“Usual Request. If you disagree with what someone says, please have the courtesy to quote the exact words you disagree with. This avoids misunderstandings.”
=================
Please define a “misunderstanding” 🙂
u.k.(us) says:
April 24, 2014 at 5:12 pm
Me thinking you’re talking about one thing, while in fact you’re talking about something totally different.
Or me going postal because someone didn’t quote my words for the fiftieth time and is accusing me of something I never said …
Either one works for me,
w.
Steve from Rockwood says:
April 24, 2014 at 5:12 pm
Thanks, Steve, and you may be right about the cause. However, I wasn’t speculating on or trying to prove the underlying causes of the phenomenon.
Instead, I was commenting on the practical effects of the phenomenon, one of which is that we erroneously think we are living in extreme times.
w.
Maybe this can be understood in an inductive fashion. Suppose you have points 1 through N and N has, say, the highest value. Now add point N+1. If the series is autocorrelated, this new point has a 50% chance of being the new highest point.
So, compare the “chances” of point N staying the highest if we add another N points. If there’s no autocorrelation, it’s 50%. With autocorrelation it’s obviously lower.
I haven’t figured out a quantitative result (yet) but the result seems intuitive.
More to come (I hope).
Thanks Willis, I am continually amazed by the things that you unearth that the rest of us sail on past. What Lindzen said is obviously true, but this goes well beyond and into unexpected territory.
@Willis. I’m with you on the extreme times bit and always look forward to your thought provoking posts.
Is the effect stronger for shorter series? Eg what about a 160 point long series (to reflect the hottest year on record claims), or 16 point long series (to reflect hottest decade)
Well that was a head scratcher as it seems counter intuitive until I thought it through a bit more. Ignoring the red noise for a moment and just considering HadCrut alone, this makes a lot of sense. Hadcrut is sort of an undulating wave. Cut it into pieces smaller than the entire wave form and you get four possible scenarios:
1, your sample is over all a negative trend, resulting in high extrema at one end and low extrema at the other,
2. your sample is over all a positive trend, and the reverse of 1 applies.
3. your sample spans a peak in the undulating wave, in which case you have low extrema at both ends
4. your sample spans a bottom in the undulating wave, so you have high extrema at both ends
In other words, assuming the data is an undulating wave, it doesn’t much matter how you cut it up into smaller segments, you’re pretty much guaranteed to have extrema at both ends of the segment.
What an interesting analysis Willis. I’d never in a million years have thought this would be the case, but now that you’ve pointed it out, it makes sense!
Robert of Ottawa wrote; (CAPS added by myself)
“Any mathematical issue that depends upon an integral from minus to plus infinity (correlation, Fourier transform, etc.) is NOT ACCURATE WITH A FINITE SERIES. Hence the great interest in Window Functions:”
Exactly correct, this is one of the limits taught early in signal processing. Most signal processing (especially digital, versus analog computing) is an APPROXIMATION to a closed form equation (you know the ones with the integral sign). That is why there a dozens of windowing functions. These artifacts can easily be mistaken for real information, but they very rarely are.
As an interesting historical aside; the old Soviet Union was far behind the “West” in terms of digital computing power. But they had many quite good mathematicians. They solved many integrals with closed form equations (i.e. to get the accurate answer for the integral of function “abc”, plug the limits and the values into this closed form equation). The “West” just hammered it with digital signal processing. I have an old Russian reference text (translated version) from the 1970’s (long out of print) that has closed form equations for hundreds of integrals, 20 per page, 700 or so pages. And the closed form solutions are exact (up to the number of decimal places you use, of course). Finding the closed form solution for an integral is like a puzzle, there is no exact algorithm to follow, you just try hunches, I wonder if the derivative of function “qwy” is the answer?
The whole mathematical basis behind stealth radar technology applied to warplanes was done in closed form equations by the Russians and published in open math journals. The US defense industry found it and used it to create the F-117 and the B-2. The first time they took a scale mock up of the F-117 out to a radar test range they (US scientists) thought their instrumentation was broken; “how come we cannot see that metal object over there with our radar?”. The secret was the shape of the plane, then they applied radar absorbing coating and the plane virtually “disappeared” from the radar screen.
Another example of these artifacts is the use of digital random number generators (like the rand()) function in Excel ™. It does not produce a true random number sequence, it is good enough for most work, but you can see frequency components in the data that are artifacts from the random number generator algorithm. At one time there was a company that had a electronic circuit which digitized the noise across a resistor (designed to maximize the noise) and sold it as a “true random number generator. The digital versions have become better with time (more bits to work with) so I think that device is no longer on the market.
Cheers, Kevin.
Is this related to John Brignall’s “Law of League Tables”?
http://www.numberwatch.co.uk/2004%20february.htm
“All measures used as the basis of a league table always improve.
Corollary 1 All other measures get worse to compensate.
Corollary 2 What you measure is what you get.”
I believe what the NumberWatch master intends to convey is that the top (or bottom) record reported tends to be taken as a standard against which subsequent measures are evaluated. I’d thought he was making a point about psychology but the analysis here makes me wonder if I overlooked something…
The “red noise” or “Brownian motion” assumption is essential to finding a closed form solution. In my example of adding the N+1th point, knowing the value of the Nth point needs to be complete knowledge. (This is sometimes called “memoryless.”) If there are longer autocorrelations (trends, periodicity, etc.) the problem gets harder, and all bets are off on the endpoint effect — it could grow or disappear.
Well I have heard this before, I can’t remember when or where, I think I just thought the idea was a crank. I didn’t gel with it at all.
When you study ‘highly correlated data or red noise’, I’m am fairly certain you will find that it exhibits all the characteristics of ‘highly correlated data or red noise’.
I suggest Willis you use pink noise, not red noise.
This is a tricky and contentious subject. Pink noise is 1/f noise, is very common in natural processes, related to chaos. A lot of opinions go on about red noise, beware.
Unfortunately pink noise is not so simple to produce.
I do not know what would happen if you try.
That was extremely interesting. Thanks
BTW, we just had another extreme solar event. X 1.3 flare (R3). Perhaps it is just me, but there have been many more of those in the last few months than I have viewed over the last several years I have been monitoring such (Perhaps Leif can comment on such). Also, have been watching the USGS pages and the ping pong of quakes across the Pacific. Chile, Nicaragua, Mexico and yesterday British Columbia, in that order. All significant events with similar sized events in between them across the Pacific. Addtionally, a recent anomolous event, from what I can tell, between South Africa and Antarctica.
Just some novice observations.
Regards Ed
Actually, there is a direct relationship between Benford’s Law and convolution, and autocorrelation is just convolution of a sequence with itself. See a really good description of how and why here: http://www.dspguide.com/ch34/1.htm
“”””””……tchannon says:
April 24, 2014 at 6:12 pm
I suggest Willis you use pink noise, not red noise…….””””””
Well 1/f noise is pretty common in analog MOS or CMOS circuits; and it is inherent. PMOS transistors, tend to have lower 1/f noise than NMOS, so analog designers (good ones), tend to use PMOS in input stages, even though NMOS gives higher gm values for a given gate area.
It is common to use very large area PMOS devices, in analog CMOS to reduce the 1/f noise corner frequency.
I designed (and built) an extremely low noise, and low 1/f corner frequency CMOS IC, using truly enormous PMOS transistors. It was a very high current gain feedback amplifier for an extremely sensitive high speed photo-detector.
1/f noise seems to defy logic, since it tends to infinity as f tends to zero. Actually it is not a catastrophe, since you can prove that the noise power is constant in any frequency octave (or decade; so the noise power doesn’t go to infinity, since the lower the frequency, the less often it happens.
I have often claimed, that the “big bang” was nothing more than the bottom end of the 1/f noise spectrum. Get ready for the next one.
As to Willis’ new conundrum; is not a truncated data sequence akin to the transient portion of the startup of an inherently narrow band signal.
An ideal sine wave signal, only has zero bandwidth, if you disallow for turning it on, or switching it off. When you do either of those things, you get a transient, that dies out with time leaving the steady state signal.
So if your signal begins with an off to on step function, which it does in Willis’ chopped chunks, you are going to get the overshoot, of a brick wall filter response.
Is that not what is going on here ??
I suppose this explains how you can have a once-every-100-year storm, and then a second once-every-hundred-year-storm only a few weeks later. I recall this happening with a couple of snowstorms that hit Boston in February of 1978, and that I became rude and sarcastic towards the people who used the phrase “once-every-hundred-year-storm.”
Sorry about that, you people who used that phrase, and who are still alive 36 years later.
However I have to confess this doesn’t make a lick of sense to me. It seems to me that if you snipped a random 100 years from the history of weather, the once-every-100-year-storm might come on any year, and not be more likely to come in year-one or year-hundred.
Likely there is something I don’t understand. However, armed with my incomplete intellectual grasp, I am heading off to Las Vegas, convinced I can beat the odds.
Appologies, I neglected my citations.
http://earthquake.usgs.gov/
http://www.swpc.noaa.gov/ftpmenu/warehouse.html
In the original statement :
“If you’re looking at any given time window on an autocorrelated time series, the extreme values are more likely to be at the beginning and the end of the time window.”
Is the “extreme values ” referring to the “extreme values” of the input time series ? or the “extreme values” of the output ACF function? From your calculation, it would appear you are looking at the input time series, but in that case, there is no need to calculate the ACF … or am I mis-understanding your calculation (or perhaps what you mean by ” autocorrelated time series”) & you are looking at the extreme values of the ACF output with the x axis on figure 2 being the lag times.
Thanks for the clarification.
I think the term “red noise” is throwing folks off here. Willis is talking about pure Brownian motion. That is known as red noise but thinking about this in terms of spectrum is a rabbit trail. Willis is speaking of a series with no periodicity.