Guest Post by Basil Copeland
Figure 1
Each month, readers here at Watt’s Up With That, over at lucia’s The Blackboard, and elsewhere, anxiously await the latest global temperature estimates, as if just one more month of data will determine one way or the other the eternal destiny of AGW (the theory of “anthropogenic global warming”). For last month, July, the satellite estimates released by UAH and RSS were up sharply, with the UAH estimate up from essentially zero in June, to +0.41°C in July, while the RSS estimate was up from +0.081°C to +0.392°C. Does this sharp rise presage the resumption of global warming, after nearly a decade of relative cooling? Or is it just another in a series of meandering moves reminiscent of what statisticians know as a “random walk?”
I have not researched the literature exhaustively, but the possibility that global temperature follows a random walk was suggested at least as early as 1991 by A.H. Gordon in an article in The Journal of Climate entitled “Global Warming as a Manifestation of a Random Walk.” In 1995 Gordon’s work was extended by Olavi Kӓrner in a note in the Journal of Climate entitled “Global Temperature Deviations as a Random Walk.” Statistician William Briggs has written about climate behaving like a random walk on his blog.
Now even I will confess that the notion that global temperature, as a manifestation of climate processes, might be essentially random is difficult to accept. But I am coming around to that view, based on what I will present here, that monthly global temperature variations do, indeed, behave somewhat like a random walk. The qualifier is important, as I hope to show.
So, what is a “random walk” and why do some think that global temperature behaves, even if only somewhat, like a random walk? And what does it matter, anyhow?
While there are certainly more elegant definitions, a random walk in a time series posits that the direction of change at any point in time is essentially determined by a coin toss, i.e. by chance. As applied to global temperature, that is the same as saying that in any given month, it is just as likely to go up as it is to go down, and vice versa. Were global temperature a true random walk, there would be no underlying trend to the data, and any claimed evidence of a trend would be spurious. One of the best known “features” of a random walk is that in a time series it appears to “trend” up or down over extended periods of time, despite the underlying randomness of the direction of change at each point in time.
So why might we think global temperature follows a random walk? One reason is suggested by a close look at Figure 1. Figure 1 is the familiar HadCRUT3 time series of monthly global temperature anomalies since 1850, with a simple linear trend line fit through the data. When we look close, we see long periods, or “runs,” in which the data are above or below the trend line. If the data were truly generated by a linear process with random variations about the trend, we’d expect to see the deviations scattered approximately randomly above and below the trend line. We see nothing of the kind, suggesting that whatever is happening isn’t likely the result of a linear process.
On the other hand, when we perform what is a very simple transformation in time series analysis to the HadCRUT3 data, we get the result pictured in Figure 2.
Figure 2
A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, i.e., comparing one month’s observation to the observation from 12 months preceding. Since 12 months have intervened in computing this difference, it is equivalent to an annual rate of change, or a one month “spot” estimate of the annual “trend” in the undifferenced, or original, series. When we look at Figure 2, it has the characteristic appearance of a random walk.
But we can do more than just look at the series. We can put a number to it: the Hurst exponent. Here’s a very understandable presentation of the Hurst exponent:
“The values of the Hurst Exponent range between 0 and 1.
-
A Hurst Exponent value H close to 0.5 indicates a random walk (a Brownian time series). In a random walk there is no correlation between any element and a future element and there is a 50% probability that future return values will go either up or down. Series of this type are hard to predict.
-
A Hurst Exponent value H between 0 and 0.5 exists for time series with “anti-persistent behaviour”. This means that an increase will tend to be followed by a decrease (or a decrease will be followed by an increase). This behaviour is sometimes called “mean reversion” which means future values will have a tendency to return to a longer term mean value. The strength of this mean reversion increases as H approaches 0.
- A Hurst Exponent value H between 0.5 and 1 indicates “persistent behavior”, that is the time series is trending. If there is an increase from time step [t-1] to [t] there will probably be an increase from [t] to [t+1]. The same is true of decreases, where a decrease will tend to follow a decrease. The larger the H value is, the stronger the trend. Series of this type are easier to predict than series falling in the other two categories.”
So what is the Hurst exponent for the series depicted in Figure 2? It is 0.475, which is very near the value of 0.5 which indicates a pure random walk. And when we exclude the data before 1880, which may be suspect because of a dearth of surface locations in computing the HadCRUT3 series, the Hurst exponent is 0.493, even closer to 0.5. So by all appearances, the global temperature series has the mark of a random walk. But appearances can be deceiving. In Figure 3 I fit a Hodrick-Prescott smooth to the data:
Figure 3
In the upper pane, the undulating blue line depicts the smoothed value derived using Hodrick-Prescott smoothing (lambda is 129,000, for the curious). In the lower panel are detrended seasonal differences, i.e., what is left after removing the smoothed series. Conceptually, the smoothed series can be taken to represent the “true” underlying “trend” in the time series, while the remainder in the bottom pane represents random variations about the trend. In other words, at times, the annual rate of change in temperature is consistently (or persistently, as we shall see) rising, while at other times it is consistently falling. That is, there are trends in the trend, or cycles, if you will. And while it is not obvious, because of the scaling involved, these are essentially the same cycles that Anthony and I have attributed to a lunisolar influence on global temperature trends. That should not be so surprising. In our paper, we smoothed the data first with Hodrick-Prescott smoothing, and then differenced it. Here we’re differencing it first, to show the random walk nature of the series, and then smoothing the differences. But either approach reveals the same pattern of cycles in global temperature trends over time.
Looking more closely at the smoothed series, and the random component (labeled “Cyclical component” in Figure 3), we have an interesting result when we compute the Hurst exponents for the two series. The Hurst exponent for the smoothed (blue line) series is 0.835, while the Hurst exponent for the detrended random component (bottom pane) is 0.383. The first is in the range associated with “persistent” behavior, while the second is in the range associated with “anti-persistent” behavior. Let’s discuss the latter first.
Anti-persistence is evidence of mean reversion or what is also sometimes called “regression toward the mean.” Simply put, when temperatures spike in one direction, there is a strong probability that they will subsequently revert back toward a mean value. Ignoring all other factors, this property would suggest that the dramatic rise in the temperature anomaly for July should lead to subsequent declines back toward some underlying mean or stable value. I think this is probably more what Gordon or Kӓrner had in mind for the physical processes at work when they proposed treating global temperatures as a random walk. I.e.,shocks to the underlying central tendency of the climate system from processes such as volcanism, ENSO events, and similar climate variations create deviations from the central tendency which are followed by reversions back toward the mean or central tendency. Carvalho et. al (2007), using rather complicated procedures, recently laid claim to having first identified the existence of anti-persistence in global temperatures. We’ve identified it here in a much simpler, and more straightforward, fashion. (I’m not trying to take away from the usefulness or significance of their work. Their procedures demonstrate the spatial-temporal nature of anti-persistence in global temperatures, especially on decadal time periods. I think WUWT readers would find their Figure 10 especially interesting, for while they do not use the term, it demonstrates “the great climate shift” of 1976 rather dramatically.)
With respect to the smoothed series, the Hurst exponent of 0.835 indicates persistent behavior, i.e. if the series is trending upward, it will have a tendency continue trending upwards, and vice versa. But that is to be expected from the cyclical undulations we observe in the smoothed series. As to the possible physical processes involved in generating these cycles, after Anthony and I posted our paper, comments by Leif Svalgaard prompted me to perform a “superposed epoch analysis” (also known as a “Chree analysis”) on these cycles:
While Leif contends that the analysis should be performed on the raw data, in this case I would beg to differ. As shown in Figure 3, the raw data is dominated by the essentially random character of the monthly changes, completely obscuring the underlying cycles in the data that emerge when we filter out (detrend) the raw data. Arguably, what we have in the blue line in Figure 3 is a “signal” that has been extracted from the “noise” depicted in the bottom pane. Now as such, the “signal” may mean something, or it may not. That is where Figure 4 comes in to play. The peaks in the cycles depicted by the blue line in Figure 3 show a strong correspondence to maximums in the lunar nodal cycle (the “luni” part of our suggestion of a “lunisolar” influence on global temperature trends). They also show a strong correspondence in solar maxima associated with odd numbered solar cycles, especially beginning with solar cycle 17. Are these correspondences mere coincidence? Anthony and I think not. While each may play an independent role in modulating global temperatures, since the 1920’s the solar and lunar influences appear to have been roughly in phase to strongly influence temperature trends on a bidecadal time frame. In other words, Figure 4 may be revealing the physical processes at work in explaining the persistence revealed by the Hurst exponent for the blue line in Figure 3.
Taken together, the two Hurst exponents – one for the true “signal” in the series, and the other for the “noise” in the series – essentially offset each other, leaving us with a Hurst exponent for the unsmoothed, raw, seasonal difference of ~0.5, i.e., essentially a random walk. And so on a monthly basis, the global temperature anomalies we await anxiously are essentially unpredictable. However, if the cycles in the smoothed series can be plausibly related to physical processes, as Anthony and I believe, that gives us a clue as the “general direction” of the monthly anomalies over time.
In our paper together, Anthony and I presented the following projection using a sinusoidal model based on the same cycles shown in the blue smooth in Figure 3:
Figure 5
The light purple line in Figure 5 is, essentially, a continuation, or projection, of the blue smooth in Figure 3. From this, we derived a projection for the HadCRUT3 anomaly (light blue in Figure 5) which has it essentially meandering between 0.3 and 0.5 for the foreseeable future (here, roughly, the next two decades).
But the monthly values will vary substantially around this basically flat trend, with individual monthly values saying little, if anything, about the long term direction of global temperature. In that sense, global temperature will be very much like a random walk.
Paul Vaughan,
Two notes: firstly, whilst I agree that a blanket rejection of smoothing is wrong, smoothing can have negative impacts on the analysis being performed. In particular, any Hurst-like analysis which is influenced by spectral power at different scales will clearly be affected by smoothing, which implicitly modifies spectral power at different scales. Anyone who uses smoothing prior to Hurst type analyses immediately raises questions of why it is needed, and should provide a clear justification as to how they prevent said smoothing from interfering with results (or how they have calibrated out the interference).
Secondly, that paper (Carvalho et al) presents very strange results. One of their diagrams (the 6-7.5 year Hurst exponent field) looks nothing like that which would be expected, and indeed looks largely like floor-to-ceiling noise from their estimator. As noted in the Koutsoyiannis papers that I link above, the Hurst exponent of available temperature data is consistently high (near, but below, 1) on scales from months up to 10,000 years. Without digging deep into their analysis, I am only guessing as to why this difference exists, but the most common reason is a failure to adequately remove the strong annual cycles in the temperature data (this particularly applies to local temperature measures). Failure to fully remove this will introduce the exactly type of effect they are observing in their paper. I note they do attempt to remove the annual cycle, but I would like to see how effective it is. Even a small residual component would produce a random field very much like that seen in their figure 4(f). This is compounded by their similar decision to analyse the Hurst coefficient over small bandwidths (which makes little sense in the broader context of the Hurst phenomenon). Without seeing their data and code I cannot be sure this is a problem, but it certainly makes my sceptical side think I would not trust this study without some of these questions answered.
Paul Vaughan (19:53:15) :
Basil, thanks for drawing my attention to this:
Carvalho, L.M.V.; Tsonis, A.A.; Jones, C.; Rocha, H.R.; & Polito, P.S. (2007). Anti-persistence in the global temperature anomaly field. Nonlinear Processes in Geophysics 14, 723-733.
http://www.uwm.edu/~aatsonis/npg-14-723-2007.pdf
“[…] significant power exists in the 4-7 years band corresponding to ENSO. Such features, however, are broadband features and do not represent periodic signals; they are the result of nonlinear dynamics (e.g., Eccles and Tziperman, 2004). As such they should not be removed from the records.”
I didn’t catch that. And I’m not sure I agree. Without going back and looking, I’m quite sure the MTM spectrum analysis of the global temperature series has a harmonic at ~4.7 years. Let’s look:
http://wattsupwiththat.files.wordpress.com/2009/05/figure4.png?w=510&h=374
The paper is still good, and harmonics, or cycles, can exist along side mean-reversion, as two different mechanisms at work influencing global temperature. That’s what I’m trying to show in this post.
By the way, I still cannot help but observe that 4.7 years is close to a harmonic (1/4) of the lunar nodal cycle, and about half the length of recent solar cycles, and may be a harmonic of a bidecadal beat cycle combining the two. The cycles we are observing here (in the blue line in Figure 3) are of a magnitude (amplitude) that could be completely attributed to typical variations in TSI. Couple that with the variation of the lunar nodal cycle, and we have an adequate physical basis, it seems to me, for the cyclical part of the time series.
Then, on top of that, we have the anti-persistent response to random effects upon temperature described by Carvalho et al, and demonstrated in the bottom pane of Figure 3. I suspect you’ll find a lot of ENSO in the latter, but that still doesn’t rule out a harmonic in ENSO as well.
Works for me. 😉
Spence_UK (05:54:27) :
Paul Vaughan,
Two notes: firstly, whilst I agree that a blanket rejection of smoothing is wrong, smoothing can have negative impacts on the analysis being performed. In particular, any Hurst-like analysis which is influenced by spectral power at different scales will clearly be affected by smoothing, which implicitly modifies spectral power at different scales. Anyone who uses smoothing prior to Hurst type analyses immediately raises questions of why it is needed, and should provide a clear justification as to how they prevent said smoothing from interfering with results (or how they have calibrated out the interference)..
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations (and noise). We (Anthony and I) demonstrated this in our paper, most effectively (in my view) with the following wavelet image:
http://wattsupwiththat.files.wordpress.com/2009/05/figure3.png?w=402&h=536
Figure 3 is just a different representation of what you see in the image linked above. It speaks for itself, and and the image above shows that the smoothing in question is not “interfering with results.” The Hurst exponent, as applied then to the two components of the time series, simply helps further describe the difference between the two.
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations
Basil, low pass filters which filter out high frequency oscillations is the exact same thing as modifying spectral power at different scales.
And it is clear that the smoothing does affect the Hurst exponent just as I said it would (see point 4 from PaulM’s comment above).
PaulM (09:58:26) :
Basil,
This article starts well but then goes seriously downhill when you start smoothing the data before applying your tests. You must never do this or your results are meaningless. Here’s a quote from the blog of statistician William Briggs:
What Paul Vaughn said: Paul Vaughan (19:18:40) : . Smoothing has its uses (and its abuses). I agree with you in part, here:
4. The figure of 0.835 for the smoothed series is meaningless. It is purely a consequence of the smoothing. The more you smooth, the bigger this number will be. You can generate as many bogus ’signals’ as you like by smoothing in different ways. It is instructive to do this with data from a random number generator. Leif is right, the analysis should be performed on the raw data. Reading the comments more carefully I see that Spence_uk made this point too.
Not that it is meaningless (I do not agree there), but that it is a result or consequence of the smoothing. Of course it is. The Hurst exponent doesn’t “prove” anything, and is not cited for such. It is simply a descriptive statistic. The smoothing is justified, or not, on other grounds. Have you read the paper Anthony and I wrote? We deal there extensively with the smoothing we’re using, and show that it is no different, in its results, that can be shown otherwise with wavelet analysis and spectrum analysis. It is useful, however, as yet another way of looking at the data. We’ve justified the smoothing on other grounds. Here the Hurst exponent is simply calculated as a descriptive statistic to show that what appears as a random walk in the raw temperature data may be in fact the result of offsetting persistence (cycles, captured by the smoothing), and anti-persistence (in yearly variations in annual trend, or rate of growth).
BTW, thanks for your independent observations, and taking the time to investigate what happens if we take simple first differences, rather than the seasonal difference. You wrote:
3. If I do a month-to-month difference, as some people have suggested, instead of the seasonal difference, I get a much smaller number, around 0.1 – I don’t understand why this is!
I get 0.24. What software are you using? (I’m using gretl.) As for the number being noticeably smaller, I’m not sure that should be so surprising. That is simply saying that the degree of anti-persistence is greater in monthly fluctuations than it is in annual fluctuations. That makes sense to me, in that monthly fluctuations will tend to revert to the mean more quickly than shocks measured on an annual basis. Actually, given my numbers — 0.24 for monthly, and 0.38 for seasonal — the difference is about what I would expect.
Thanks for the dialog.
Phil M (05:15:36) :
Flanagan points out, by differentiating first, you are making any ‘linear trend’ into a constant offset
– and then analysing the result to show that we have a ‘random walk’ about a ‘constant offset’ (which is the differential off the linear trend)
– so in effect, you are saying we have a ‘random walk’ about a ‘linear trend’
– so, what is your point, exactly?
I think you left off your snark tags.
All,
There is a general point regarding the Hurst coefficinet/exponent I fail to get, and that is how 0.5 is associated with a random walk.
So if anyone who can point out the air gap in the statements below, I would be grateful.
Gausian (white noise) is not correlated, its Range (defined as the integral of the differences to the running mean) varies as (n)^0.5 and its SD varies as (n)^0.0 and hence (R/S)(n) varies as (n)^0.5 which as I understand it gives H=0.5. The same is true if the series is just a series of coin tosses (1,1,-1,1,-1,-1,-1,1, …).
A random walk can be formed by taking the integral (or sum) of Gaussian noise (or coin tosses). Or putting it the other way the first differential or difference series, of a random walk, will have the Hurst coefficient of the Gaussian or toin coss i.e. H=0.5
The random walk itself will have H=~1.
As I understand it H=0 implies strongly anti-persistence, (the differentials of Gaussian (white) Noise have this property),
H=0.5 implies neutrality (non-autocorrelated or white noise)
H=~1 implies strong persistence, (the integrals of white noise have this property).
Now either that is a load of tosh, or identifying H=0.5 with a random walk (as opposed to the first differential of a random walk) is misleading me.
Is there a different usage of the term “random walk” that I am not familiar with?
Anyone help me out?
Alexander
In the above I only meant “running mean” to imply that it varies with (n) not that it varies during the integration step.
Alexander
Stevo (12:57:38) :
Basil,
Are you aware of ARIMA processes?
Yes, of course I am. And were I trying to forecast temperature trends over the next few months I might well make use of that approach. But ARIMA models are largely black box models which do not require any understanding of underlying physical processes, and are not particularly useful, in my view, in understanding the long term dynamics of underlying physical processes. Have you ever looked at the confidence intervals of an ARIMA forecast? After a few months, you are in la la land.
Re: my own: Alex Harvey (08:20:10)
I have got the noise colours wrong I am sure (oops), but I hope that the rest of it stands, that my notion of a random walk would have H=~1 and a non-autocorrelated series H=~0.5.
Alexander
Leif Svalgaard (09:51:16) :
Basil (08:15:04) :
Any extrapolation out for two decades, let alone for the next century (a la IPCC) involves what we call in my field “heroic assumptions.”
Except that the lunar influence can be calculated accurately for thousands of years and the solar influence is well-known for centuries and reasonably well-known also for thousands of years, so your input to the ’signal’ is well-known, there your hindcast is not an extrapolation, but an application of known inputs.
Fair enough (that these inputs are known), except that the hindcast (or forecast) is not being driven by those inputs, per se. Maybe a better mathematician than I am can construct such a model some day. But, to address an earlier question (wanting to see Figure 5 back to 1850), the hindcast/forecast comes from the following:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png?w=510&h=300
It is a sinusoidal fit to what is in effect the same thing you see in the blue line in Figure 3 of this post. I.e., the red line in the above linked image, and the blue line in Figure 3, are for all intents and purposes the same. Now as a fit, it purports to explain 60% (from R^2) of the variation we see in the blue line in Figure 3. It does a reasonably good job (in my opinion) of modeling the frequency of the cycles, but less well capturing their amplitude (which is where most of the lack of fit occurs). Empirically, the dominant frequencies are 20.69 and 9.63 years.
I suppose what you are really getting at here is that if these really do come from the lunar nodal cycle and the solar cycle, we ought to be able to model the blue line from “first principles” rather than just fit something empirically to it. Keeling & Whorf tried that with the lunar nodal cycle, but it didn’t stimulate much interest. And I can imagine that it would be even more difficult to add in the effect of the solar cycle. I would be more than happy if someone took up the challenge of doing that.
FWIW (not much in your eyes, I imagine), but Figure 4 here (the Chree analysis) is derived from the red cycles in the image linked to above. As such, I think it is descriptively helpful, in isolating what we attempted to show the “S’s” and “L’s” in the image linked to above. Whether it “proves” anything is another matter altogether, I suppose. Being a Popperian, I don’t think we ever really “prove” anything. I would like to think that it is one of those things that are “suggestive” of something deserving further investigation. In a Popperian sense, I think Figure 3 (and everything that was done to get there) is a perfectly valid approach to developing a falsifiable hypothesis.
I know, of course, what comes next: “How do you propose to falsify it?” Well, we can wait and see if Figure 4 is replicated in any way during future solar and lunar nodal cycles.
You also asked:
And, perhaps, comment on the fact that the average distance after N steps is not zero but SQRT(N), so no strong probability of reverting to the mean.
That’s true of a pure random walk. But I haven’t suggested that is what we have here. Were we to look more closely at Figure 2, we’d find that the mean is not zero, so we might here have a “random walk with drift.” It (the mean) is 0.0053302, equivalent to a long term decadal trend of 0.053°C.
(FYI, I’m not attributing this trend to a solar influence, but rather the variations around any such trend, such as we see in the blue line in Figure 3. You’ve previously said you would expect to find variation of this magnitude from TSI variations, and have treated such as unremarkable. So I do not really understand the basis for all your opposition here.)
More substantively, though, I never claim a probability of reverting to the mean in the data represented by Figure 2. That claim is reserved for the data represented in the bottom panel of Figure 3, which I make clear evidences anti-persistence, i.e. is not a random walk.
Again, thanks to all who have commented. There are still some comments and observations worthy of reply, but I’ll have to come back to them later.
Basil
Spence_UK (08:10:36) :
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations
Basil, low pass filters which filter out high frequency oscillations is the exact same thing as modifying spectral power at different scales.
And it is clear that the smoothing does affect the Hurst exponent just as I said it would (see point 4 from PaulM’s comment above).
1) Okay, I understand your first point, which is true. Which is why Anthony and I performed the wavelet transforms on the raw data, as well as the smoothed data. What that shows is that the cycles seen in the smoothing are not artifacts of the smoothing, but are there in the raw, untransformed, and unsmoothed, data.
2) I didn’t deny that smoothing would affect the Hurst exponent. I said “of course” it will. That doesn’t change the basic conclusion here, which is that the seasonal variation, which appears to be a random walk, can be decomposed into two components, one which exhibits persistence (yes, because of the smoothing), and the other which exhibits anti-persistence.
Alex Harvey (08:20),
This is a good question and one that it took a while for me over at climateaudit to “get” (Prof. Koutsoyiannis had to rather spell it out for me!)
A random walk does not have H=0.5. In fact, H is undefined for a random walk because it is not a stationary process, and the Hurst exponent is defined for stationary processes only.
A key relationship (which some estimators use to derive H), is a=2H-1, where a is the relationship between the time series spectrum where spectral power is proportional to f^(-a).
So H=0.5 yields a f^(0) relationship with spectral power, that is pure white noise
H=1 yields a f^(-1) relationship with spectral power, that is 1/f noise, excess noise or long-term persistence.
On this basis, a pure random walk would have spectral power of f^(-2). According to the relationship above, this would give H=1.5, but this is incorrect, because random walks are not stationary. (1/f noise is stationary, it has a defined population mean, but the sample mean is not a good estimator of the population mean). But by the original definition of H (laid down by Mandelbrot in, I think, Mandelbrot and van Ness), H is undefined for the random walk.
In the event of a random walk, the time series represents a storage parameter (or integral), and the series must be differenced and the Hurst analysis applied to that series. So, in principle, if global temperature were a random walk, the raw data would have no defined Hurst exponent (and many estimators would incorrectly report H>1), and it would be correct to take first differences.
I would suggest this is inappropriate for the global temperature series, as there are good reasons to doubt that it is a random walk:
1. The power spectrum is not proportional to 1/(f^2)
2. A pure random walk would eventually wander off to infinity, linking with Leif’s observations above
3. Hurst estimators do not yield values of H>1 for temperature series
However, once first differences are taken, the resulting series may yield Hurst coefficients in the valid range 0 < H < 1.
Alex, I think what you say is exactly right, the difference of a random walk should have H=0.5 (not the random walk itself). But that’s exactly what Basil is doing, so what’s the problem?
Basil thanks for your comments.
I’m using a matlab code called hurst_exponent.m off the matlab file exchange.
As long as you are fully aware that you would get the same result (H incresing as you smooth more) with random data then I’m happy. What appears to be a random walk in the temperature data could in fact be … a random walk. And the frequency of the ‘signal’ you see depends your choice of smoothing, so you can choose your smoothing parameter to get a match with whatever cycle you want. I still think your post is a bit misleading around fig 3.
PaulM,
The problem is in the assumption of a random walk. If you’re running MATLAB, I can show you why more quantitatively. Download “hurst estimators.zip” by Chu Chen at the MATLAB file exchange. This gives you a number of different estimators. Grab yourself a temperature series (I had the HadCRU kicking around from Oct 08 on my hard disk so I used that) and we’ll create a random walk with the following command (I choose 1904 because that is how many points I had in my HadCRU series):
randwalk = cumsum(randn(1904, 1));
H = hurst_estimate(randwalk, ‘method’)
H = hurst_estimate(HadCRU, ‘method’)
(Where method is one from the list). Now try some estimators both on the random walk and the temperature series. The results that I tried came out with (methods on top row):
methods: aggvar / higuchi / boxper / peng / absval
randwalk: 0.994 / 1.011 / 1.697 / 1.534 / 0.990
HadCRU: 0.923 / 0.951 / 0.990 / 0.732 / 0.909
Note the variation of estimates for the random walk include some very high (actually invalid) values; some estimators are constrained to be more or less 0 to 1, and yield (incorrectly) seemingly valid figures. This results in inconsistent values across estimators; this is typical of a random walk.
On the other hand, the HadCRU does not yield high values for these estimators. Why? Because the HadCRU data does not behave like a random walk. So those estimators (such as peng and boxper) that are unbounded do not yield invalid values for H.
(A variety of values is still seen, over a much narrower range, highlighting some of the difficulties associated with estimating H)
IMHO, I suspect the reason Basil sees antipersistence is because he is differencing a stationary series; the reason he sees persistence is because of the smoothing function. Applying Hurst analysis to the raw data without differencing shows that global temperature is stationary, not a random walk, but exhibits very strong long term persistence across all scales.
Dr. Jose Sanchez writes in reply to Stef:
“Dr Jose Sanchez (15:50:07) :
Stef wrote: “Dr Sanchez, the articles you gave links for are all from different sources, and this latest article is a guest post by Basil Copeland. It might surprise you that different people actually have different opinions. WUWT publishes articles and information from many sources. You do realise that not every single article is penned by Mr. Watts don’t you?”
Yes, I do realize they all come from different sources. And I also realize that journalists around the world are using this website as a source to spread misinformation about the theory of global warming. And I also do realize that an editor of such a website is supposed to have basic knowledge of the theory, and not publish whatever information is convenient at the time because it agrees with the position they are trying to push forward, regardless of how much it contradicts previous information they gave. There is a concept called “responsible journalism” that Mr Watts ought to read up upon. Until then, all credibility from this website is thrown out the window.”
He he…. ,
You are welcome to point out what “misinformation” you claim to read here,with your own counterpoints.I am sure Anthony and everyone else would like to see what you come up with.I am always ready to read of credible counterpoints,as this would teach me greater understanding of the topic under consideration.Did you have a counterpoint to offer here?
You are knocking an award winning science blog,with absolutely no regard to the idea,that it is that way because of the quality open civil exchanges that goes on here.Your replies seems to be one of avoiding the actual topic and just tar us a little.
Please tell us why you think some one on a topic is wrong with some details and leave out the worthless put down of this blog?
Spence_UK,
Many thanks for your very detailed response, I think I will get it now.
PaulM,
Thanks to you as well,
As to: “what’s the problem “?
I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.
Alexander
Dr Jose Sanchez (15:50:07) :
This guy seems to be one of a type we get here occasionally who drops in for a quick AGW rant and then disappears again. We’ve got ‘collapsing wave’ going “la la la la la” on another thread at the moment. We had another one a week ago or so, who was actually very eloquent but seemed to be on speed for the night, and then disappeared again. There’ll be another one along again soon. It’s a bit like two different universes making a close approach for a brief interaction before both going off on their own ways…
Alex Harvey (14:28:47) :
I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.
Seasonal differencing isn’t the same as a seasonal adjustment. In the case of monthly temperature anomalies, each month is baselined against its average for the base period, so that relative to the middle of the base period, each month should have a value of zero. (Or so I understand.) As long as you do not believe that each month has a different trend, then a seasonal difference will simply measure the annual rate of change from one month to the same month of the next year. Again, as long as there is no reason to believe that the tread is different for different months of the year, I cannot imagine what your problem is with this. It is certainly not arbitrary. The rationale is clearly stated.
G. Karst (08:22:37) : E.M.Smith (02:46:34) : Your analysis should be another talking point… a thread on it’s own merit. I would be interested to read a full discussion including “AGW convinced” rebuttals, on your thesis. You have given us much food for thought. Thx.
Thanks! But I didn’t really have a “thesis” going into this. About as close as I got was “An average of things broadly going up ought to go up” but that’s more a statement of a mathematical property than a thesis.
As of now, I think I do have a thesis: GIStemp is a filter that attempts to remove the impact of adding and removing thermometers from the record, distributed in time and space; but it fails to fully insulate from an order of magnitude change in thermometers in a largely divergent geography, leading to a warming of the “anomaly maps” based on a change of number and location of thermometers. GIStemp is not a perfect filter.”
All I did was to respect the data and ask it what it had to say, then shut up and listen. Others, I fear, tortured the data until they say whatever is desired. It’s not about me, or my thesis, it’s all about the data and what they have to say. You just need to listen…
Jimmy Haigh (15:23:35) : This guy seems to be one of a type we get here occasionally who drops in for a quick AGW rant and then disappears again.
There are certain linguistic and stylistic clues that imply some of these folks are the same person. Unfortunately folks who hang out together a lot tend to soak up a common “accent” and style. So it’s also possible it’s just a group of folks who hang out together in the echo chamber a lot. An inspection of IP addresses would be helpful, and I presume that the folks running this site are well aware of that.
At any rate, I find them a “useful irritant”. They show where there is a weakness in an argument by being all over it. They show where there is a weakness in their argument by giving lame counters (and you get good practice at rebuttal). And most importantly, they serve a wonderful purpose by showing the truly stellar arguments. Those will be greeted with silence.
Learn to see that “negative space”. The lack of flack from The Team. It tells you where you have hit gold… It warms my soul to hear their silence and know that I’ve got it right.
Re: Spence_UK (05:54:27)
6.4a is the period of the terrestrial polar motion group-wave (aside from ~1920-1940 when it slowed down, shifting 180 degrees). Maybe people aren’t seeing this because related signals vary regionally. (Increased pressure in one area is balanced by decreased pressure in another area, etc. – i.e. pressure doesn’t redistribute the same way temp does – & people do tend to fixate on global averages.)
Thanks for the links.
–
Re: Basil (06:57:00)
2 cautionary notes:
1) ENSO’s period is certainly not stationary.
2) Solar variables & temperature variables are in (broad-sense) anti-phase prior to ~1931 (back to ~1765). This appears to be related to Jupiter-Neptune’s phase relationship with the LNC (which has a beat period of ~205 years) and terrestrial north-south asymmetry. [Note: Jupiter-Neptune is the highest-frequency heavyweight-beat in solar system dynamics.] (Also: Remember that NH temps dominate global averages.) It is proving to be a devil of a challenge to get people to clue in to the preceding even though it should be as plain is day to anyone handling the terrestrial polar motion time series with sufficient computational skill. For too long people were bent on exploring how solar system dynamics affect the sun; somehow (?) this prevented people from looking closer to home (i.e. Earth’s shells) (?). The Chandler wobble phase reversal (centred ~1931) shows up in all kinds of terrestrial time series (including SOI, regional precipitation, & aa (if one knows how to look)). So: Be careful if you only work with frequency info, because you’ll miss important stuff that is plain as day if you use *time-frequency info.
Here’s another reference:
MacMynowski, D.G.; & Tziperman, E. (2008). Factors affecting ENSO’s period. Journal of Atmospheric Science 65(5),1570-1586.
http://www.cds.caltech.edu/~macmardg/pubs/MacMynowski-Tziperman-2008.pdf
sunsettommy writes: “You are knocking an award winning science blog,with absolutely no regard to the idea,that it is that way because of the quality open civil exchanges that goes on here.”
Award winning science blog??? Did it get an award for not understanding the difference between climate and weather? Did it get an award for propagating a myth about “global cooling” based upon a single data point for the month of June, while ignoring the long term trend? Did it get an award for not understanding the basics of applied statistics?
I’m sorry, but no wonder why there is so much misinformation about global warming. If you can’t filter out the junk from the legitimate criticisms, and people use your blog as an authoritative source, then you have become the major source of the misinformation. Scientific discussion does not look for data to fit the argument they are trying to push forward. Instead, it looks at the data without bias, and then draws a conclusion about it, even if that conclusion may disagree with what we hope the truth is. For two months the articles here tried to suggest that we are now in global cooling (by looking at the June data and selective points in July that were not representative of the big picture), and then once the new data came out that was contrary to what blog editor wanted to believe, the argument was changed to “individual monthly values saying little, if anything, about the long term direction of global temperature.” There is absolutely nothing scientific about that.
You will never hear Lindzen saying that we are in a global cooling trend now based upon the data you were trying to put forward in June and July. Listen to the scientists rather than trying to pretend like you are one while misleading others into believing you have a clue about global warming.
Jose Sanchez,
Yes, this is an award winning site. Why does that bother you so much? Are you upset that realclimate failed? Or Tamino, or the Rabett, or climateprogress, or any of the other censorship prone purveyors of climate alarmism?
Speaking of alarmism, why should we listen to those Chicken Littles, who keep telling us the sky is falling? You may not be aware of it if you inhabit one of the echo chambers named above, but the planet’s current temperature is right at about the same level it was at thirty years ago: click.
Instead of going ballistic when someone mentions the “Best Science” site, maybe you should stick around and learn something.
Alex Harvey (14:28:47) “I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.”
Bear in mind that the seasonal “correction” is based on assumptions (usually defining a rigid annual structure based on a 30 year span (a climatology) …as opposed to a flexible one that varies locally (in time), for a contrasting example…)
Let me give an example that might help people get around this:
Should there be a very strong annual term (with harmonics at monthly-multiples) in the sun’s motion about the solar system center of mass? Well guess what: There IS …if you work with monthly summaries …..and as soon as you difference: THAT GETS AMPLIFIED, but it is a spurious effect that a sensible analyst would recognize (…& perhaps remove with annual-smoothing).
This is just one example. There is no substitute [such as rule’s of thumb] for careful, context-specific thinking. (Sometimes a person “knows” that smoothing makes sense in a given context, but hasn’t thought-through why. A good recent example: McLean et al’s (2009) use of the RATPAC series that got them into trouble (which they did not explain) with averages for the 4 seasons – (that series is not monthly-resolution).)
On a practical note: I have learned to expect bitter opposition to the use of smoothing whether it is warranted or not – i.e. what a headache — a more efficient education system might be a long-term solution – (smoothing isn’t even usually addressed until 4th year stats courses).
We should dump the convention of posting anomalies to webpages. If people want anomalies, empower them to choose what type of anomaly is most appropriate in a _specific_ analysis context. For example, if someone is differencing twice with a series that is based on a pre-defined climatology, they might have a problem that they should not have (depending on the nature of the series).