Guest Post by Basil Copeland
Figure 1
Each month, readers here at Watt’s Up With That, over at lucia’s The Blackboard, and elsewhere, anxiously await the latest global temperature estimates, as if just one more month of data will determine one way or the other the eternal destiny of AGW (the theory of “anthropogenic global warming”). For last month, July, the satellite estimates released by UAH and RSS were up sharply, with the UAH estimate up from essentially zero in June, to +0.41°C in July, while the RSS estimate was up from +0.081°C to +0.392°C. Does this sharp rise presage the resumption of global warming, after nearly a decade of relative cooling? Or is it just another in a series of meandering moves reminiscent of what statisticians know as a “random walk?”
I have not researched the literature exhaustively, but the possibility that global temperature follows a random walk was suggested at least as early as 1991 by A.H. Gordon in an article in The Journal of Climate entitled “Global Warming as a Manifestation of a Random Walk.” In 1995 Gordon’s work was extended by Olavi Kӓrner in a note in the Journal of Climate entitled “Global Temperature Deviations as a Random Walk.” Statistician William Briggs has written about climate behaving like a random walk on his blog.
Now even I will confess that the notion that global temperature, as a manifestation of climate processes, might be essentially random is difficult to accept. But I am coming around to that view, based on what I will present here, that monthly global temperature variations do, indeed, behave somewhat like a random walk. The qualifier is important, as I hope to show.
So, what is a “random walk” and why do some think that global temperature behaves, even if only somewhat, like a random walk? And what does it matter, anyhow?
While there are certainly more elegant definitions, a random walk in a time series posits that the direction of change at any point in time is essentially determined by a coin toss, i.e. by chance. As applied to global temperature, that is the same as saying that in any given month, it is just as likely to go up as it is to go down, and vice versa. Were global temperature a true random walk, there would be no underlying trend to the data, and any claimed evidence of a trend would be spurious. One of the best known “features” of a random walk is that in a time series it appears to “trend” up or down over extended periods of time, despite the underlying randomness of the direction of change at each point in time.
So why might we think global temperature follows a random walk? One reason is suggested by a close look at Figure 1. Figure 1 is the familiar HadCRUT3 time series of monthly global temperature anomalies since 1850, with a simple linear trend line fit through the data. When we look close, we see long periods, or “runs,” in which the data are above or below the trend line. If the data were truly generated by a linear process with random variations about the trend, we’d expect to see the deviations scattered approximately randomly above and below the trend line. We see nothing of the kind, suggesting that whatever is happening isn’t likely the result of a linear process.
On the other hand, when we perform what is a very simple transformation in time series analysis to the HadCRUT3 data, we get the result pictured in Figure 2.
Figure 2
A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, i.e., comparing one month’s observation to the observation from 12 months preceding. Since 12 months have intervened in computing this difference, it is equivalent to an annual rate of change, or a one month “spot” estimate of the annual “trend” in the undifferenced, or original, series. When we look at Figure 2, it has the characteristic appearance of a random walk.
But we can do more than just look at the series. We can put a number to it: the Hurst exponent. Here’s a very understandable presentation of the Hurst exponent:
“The values of the Hurst Exponent range between 0 and 1.
-
A Hurst Exponent value H close to 0.5 indicates a random walk (a Brownian time series). In a random walk there is no correlation between any element and a future element and there is a 50% probability that future return values will go either up or down. Series of this type are hard to predict.
-
A Hurst Exponent value H between 0 and 0.5 exists for time series with “anti-persistent behaviour”. This means that an increase will tend to be followed by a decrease (or a decrease will be followed by an increase). This behaviour is sometimes called “mean reversion” which means future values will have a tendency to return to a longer term mean value. The strength of this mean reversion increases as H approaches 0.
- A Hurst Exponent value H between 0.5 and 1 indicates “persistent behavior”, that is the time series is trending. If there is an increase from time step [t-1] to [t] there will probably be an increase from [t] to [t+1]. The same is true of decreases, where a decrease will tend to follow a decrease. The larger the H value is, the stronger the trend. Series of this type are easier to predict than series falling in the other two categories.”
So what is the Hurst exponent for the series depicted in Figure 2? It is 0.475, which is very near the value of 0.5 which indicates a pure random walk. And when we exclude the data before 1880, which may be suspect because of a dearth of surface locations in computing the HadCRUT3 series, the Hurst exponent is 0.493, even closer to 0.5. So by all appearances, the global temperature series has the mark of a random walk. But appearances can be deceiving. In Figure 3 I fit a Hodrick-Prescott smooth to the data:
Figure 3
In the upper pane, the undulating blue line depicts the smoothed value derived using Hodrick-Prescott smoothing (lambda is 129,000, for the curious). In the lower panel are detrended seasonal differences, i.e., what is left after removing the smoothed series. Conceptually, the smoothed series can be taken to represent the “true” underlying “trend” in the time series, while the remainder in the bottom pane represents random variations about the trend. In other words, at times, the annual rate of change in temperature is consistently (or persistently, as we shall see) rising, while at other times it is consistently falling. That is, there are trends in the trend, or cycles, if you will. And while it is not obvious, because of the scaling involved, these are essentially the same cycles that Anthony and I have attributed to a lunisolar influence on global temperature trends. That should not be so surprising. In our paper, we smoothed the data first with Hodrick-Prescott smoothing, and then differenced it. Here we’re differencing it first, to show the random walk nature of the series, and then smoothing the differences. But either approach reveals the same pattern of cycles in global temperature trends over time.
Looking more closely at the smoothed series, and the random component (labeled “Cyclical component” in Figure 3), we have an interesting result when we compute the Hurst exponents for the two series. The Hurst exponent for the smoothed (blue line) series is 0.835, while the Hurst exponent for the detrended random component (bottom pane) is 0.383. The first is in the range associated with “persistent” behavior, while the second is in the range associated with “anti-persistent” behavior. Let’s discuss the latter first.
Anti-persistence is evidence of mean reversion or what is also sometimes called “regression toward the mean.” Simply put, when temperatures spike in one direction, there is a strong probability that they will subsequently revert back toward a mean value. Ignoring all other factors, this property would suggest that the dramatic rise in the temperature anomaly for July should lead to subsequent declines back toward some underlying mean or stable value. I think this is probably more what Gordon or Kӓrner had in mind for the physical processes at work when they proposed treating global temperatures as a random walk. I.e.,shocks to the underlying central tendency of the climate system from processes such as volcanism, ENSO events, and similar climate variations create deviations from the central tendency which are followed by reversions back toward the mean or central tendency. Carvalho et. al (2007), using rather complicated procedures, recently laid claim to having first identified the existence of anti-persistence in global temperatures. We’ve identified it here in a much simpler, and more straightforward, fashion. (I’m not trying to take away from the usefulness or significance of their work. Their procedures demonstrate the spatial-temporal nature of anti-persistence in global temperatures, especially on decadal time periods. I think WUWT readers would find their Figure 10 especially interesting, for while they do not use the term, it demonstrates “the great climate shift” of 1976 rather dramatically.)
With respect to the smoothed series, the Hurst exponent of 0.835 indicates persistent behavior, i.e. if the series is trending upward, it will have a tendency continue trending upwards, and vice versa. But that is to be expected from the cyclical undulations we observe in the smoothed series. As to the possible physical processes involved in generating these cycles, after Anthony and I posted our paper, comments by Leif Svalgaard prompted me to perform a “superposed epoch analysis” (also known as a “Chree analysis”) on these cycles:
While Leif contends that the analysis should be performed on the raw data, in this case I would beg to differ. As shown in Figure 3, the raw data is dominated by the essentially random character of the monthly changes, completely obscuring the underlying cycles in the data that emerge when we filter out (detrend) the raw data. Arguably, what we have in the blue line in Figure 3 is a “signal” that has been extracted from the “noise” depicted in the bottom pane. Now as such, the “signal” may mean something, or it may not. That is where Figure 4 comes in to play. The peaks in the cycles depicted by the blue line in Figure 3 show a strong correspondence to maximums in the lunar nodal cycle (the “luni” part of our suggestion of a “lunisolar” influence on global temperature trends). They also show a strong correspondence in solar maxima associated with odd numbered solar cycles, especially beginning with solar cycle 17. Are these correspondences mere coincidence? Anthony and I think not. While each may play an independent role in modulating global temperatures, since the 1920’s the solar and lunar influences appear to have been roughly in phase to strongly influence temperature trends on a bidecadal time frame. In other words, Figure 4 may be revealing the physical processes at work in explaining the persistence revealed by the Hurst exponent for the blue line in Figure 3.
Taken together, the two Hurst exponents – one for the true “signal” in the series, and the other for the “noise” in the series – essentially offset each other, leaving us with a Hurst exponent for the unsmoothed, raw, seasonal difference of ~0.5, i.e., essentially a random walk. And so on a monthly basis, the global temperature anomalies we await anxiously are essentially unpredictable. However, if the cycles in the smoothed series can be plausibly related to physical processes, as Anthony and I believe, that gives us a clue as the “general direction” of the monthly anomalies over time.
In our paper together, Anthony and I presented the following projection using a sinusoidal model based on the same cycles shown in the blue smooth in Figure 3:
Figure 5
The light purple line in Figure 5 is, essentially, a continuation, or projection, of the blue smooth in Figure 3. From this, we derived a projection for the HadCRUT3 anomaly (light blue in Figure 5) which has it essentially meandering between 0.3 and 0.5 for the foreseeable future (here, roughly, the next two decades).
But the monthly values will vary substantially around this basically flat trend, with individual monthly values saying little, if anything, about the long term direction of global temperature. In that sense, global temperature will be very much like a random walk.
Steve Fitzpatrick (07:59:16) :
Basil,
Thanks for an interesting post; I had not seen this type of analysis before.
1) It seems to me that by using the 1-year differences to transform the raw data (Figure 2) that you are taking something of a “first derivative” of the data, which ought to magnify the importance of short term variations, while reducing any longer term variation in the original data to small differences in the trend above or below the baseline. Since the overall trend for the raw data set is clearly positive, the average value for all the transformed data should be slightly above zero, with a value that depends on the slope of the least squares fit trend in the original data divided by the number of data points in the raw data set less 1. Do I understand the transform you have done correctly?
You understand it perfectly, Steve.
Would a series of similar transforms with different time steps (2 yrs, 3 yrs, 4 yrs, etc.) not increasingly show the longer term variations in the raw data, while “filtering out” the short term variation? What is the reason for choosing 1 year changes as opposed to some other period?
It is conventional to look at time series on monthly, quarterly, or annual bases. I’ve never heard of anyone using as a whole unit, a period of 2 yrs, 4yrs, etc.
2. If I understand correctly, you have assigned cause for the recent temperature history (the 20th century to now) to the “lunisolar” influence you described, and based on this, you project variation in average temperature around a flat trend for the next 20+ years (as shown in Figure 5). If this is true, then it seems to me that you are implicitly assigning a value of near zero for climate sensitivity to radiative forcing..
No, because solar isn’t the only factor at work here. The other is the lunar nodal cycle (and actually, maybe some longer term oceanic or atmospheric influences, since the sinusoidal model has frequencies of ~15 and ~54 years as well). We’re not sure about the exact physical mechanism(s) here, but I think these forces play out through long term trends or shifts in atmospheric processes, with meridional flows dominating at times, and zonal flows dominating at other times. Through all of this, the solar “forcing” can remain relatively constant (though not entirely, as TSI does fluctuate some over the solar cycle).
Fair enough, it could be very low. But in this case, how can the climate have any measurable sensitivity to variation in TSI over the solar cycles? I think that Leif suggests an average solar cycle signature of about 0.075C in the historical data due to variation in TSI of about 1.4 watt/M^2 at the top of the atmosphere. This is consistent with a relatively low climate sensitivity, but not a near-zero sensitivity. If the sensitivity to radiative forcing is in fact near zero, then by what mechanism do you think the solar cycle shows up in the temperature data?
Leif calculates the impact of variations in TSI on temperature to be, I think, about 0.07 K. So look carefully at the variation indicated on the y-axis of the following graph:
http://i25.tinypic.com/25f2b00.jpg
This is an amplified version of the wavy blue line in Figure 3. Just eyeballing it, the average peak to trough change in range of change might be something on the order of 0.05°C, at least of an order of magnitude comparable to the 0.07 K Leif computes. In other words, over the course of a solar cycle, the annual rate of change increases ~0.05°C, and then declines by that amount. Now of course, that’s a Mark I eyeball estimate, and some of the cycles are less, a some a bit more. The largest peak to trough value is ~0.10°C around 1940, and during the anomalous period of the 1920’s, the change was less than 0.02°C. But accuracy of my Mark I eyeball estimates aside, these variations, which (again) you see in the wavy blue line in Figure 3, are ballpark order of magnitude what they should be according to Leif.
By the way, for all of those who are skeptical of the use of seasonal differencing here, and wonder what would be the case if we used monthly data, I refer you to the comparable figure from the paper Anthony and I wrote:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png
Note well the difference in order of magnitude on the y-axis. That is because in that research, we smoothed the monthly observations, and then differenced the monthly observations. There was never any seasonal differencing to transform the observations into an annual rate of return. The data in the above figure are represented on the y-axis as monthly rates of change. E.g., just multiply the values in the last graph by 12 and you will have what you see in the preceding one.
But the resulting pattern of variation over time is the same!
This, I think, ought to settle the issue as far as the question of seasonal differencing is concerned. It is, as the lawyers like to say, and distinction without a difference.
Paul Vaughan (17:43:57) :
Solar variables & temperature variables are in (broad-sense) anti-phase prior to ~1931 (back to ~1765). This appears to be related to Jupiter-Neptune’s phase relationship with the LNC (which has a beat period of ~205 years) and terrestrial north-south asymmetry.
Fascinating stuff. I’m certainly aware of the anti-phase correlation in solar and temperature variables prior to the 1930’s. But this is the first time I’ve seen this explanation for it.
The Chandler wobble phase reversal (centred ~1931) shows up in all kinds of terrestrial time series (including SOI, regional precipitation, & aa (if one knows how to look)). So: Be careful if you only work with frequency info, because you’ll miss important stuff that is plain as day if you use *time-frequency info.
Well, I like to think that’s exactly what we’re looking at here — a time-frequency representation of the data:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png
So is that the Chandler wobble we see in the phase shift in this diagram circa 1930?
Re: Basil (19:40:16)
There’s a loose 1:2:3 resonance that breaks ~1931:
http://www.sfu.ca/~plv/1930sHarmonicPhaseDifference.PNG
http://www.sfu.ca/~plv/1931UniquePhaseHarmonics.png
Wavelet methods are one way to go for nonstationary series — you need a method that gives a view of how period varies – like this:
http://www.sfu.ca/~plv/ChandlerPeriod.PNG
http://www.sfu.ca/~plv/ChandlerPeriodAgassizBC,CanadaPrecipitationTimePlot.PNG
Basil & Peter Vaughan,
This annual differencing would do two thing, remove a seasonal cycle, if present, and progressively filter out some of the frequency components.
Basil Wrote: “A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, …”
Now as the series is already seasonally adjusted, the choice of 12 months is not necessary to remove the seasonal component of the earths temperature record. It will remove variance as in the 12,6,4,3, and 2 month periods but is this a necessity given what else this diverencing does to the data.
Significantly it alters the frequency specturm of the series in a different way to what the choice of 1 month, 3 months or 6 months, etc., (each with a different Hurst coefficient), would do.
As you choose to estimate a Hurst coefficient on the filtered data (and to compare it with 0.5), the choice of the time interval materially affects the result. If you get H=~0.5 using 12 months, someone can say that it is due to a subjective choice of the interval, with no other need to pick 12 months.
Now there is a small residual seasonal component in the HadCRUT3 data partly due to the length of climatology only spanning about 1/4 of the total period, if you wish to remove this seasonal component you can construct a new climatology spanning the entire period of the data and subtract it. I expect that would not have a significant effect on the estimated Hurst coefficient.
Obviously you can filter the data in anyway that suits your purposes but to do so, and then to attach any particular significance to a resulting Hurst coefficient seems a little strange.
There is one obvious filters to use and that is monthly differencing. Why not do that?
Given that there is no signifiacnt seasonal signal and that it is possible to minimise any residual seasonal signal by subtracting a new climatology, I can not see that a 12 month interval as being anything but a subjective choice.
Besides progressively filtering out signals with long periods what else does it achieve? Why not some other time constant?
Peter your:
“Should there be a very strong annual term (with harmonics at monthly-multiples) in the sun’s motion about the solar system center of mass? Well guess what: There IS …if you work with monthly summaries …..and as soon as you difference: THAT GETS AMPLIFIED, but it is a spurious effect that a sensible analyst would recognize (…& perhaps remove with annual-smoothing).”
Sorry I could not follow that at all. Anyway the HadCRUT3 data has very little variance in the seasonal frequency bands (12months, 6 months, 4 months, etc.).
Now if you had the raw (non adjusted by subtracting a climatology) HadCRUT3 data. You would find that it was dominated by the seasonal variation and you would look to do something to remove it. The choice would be to subtract a climatology, or perform 12 month differencing, I can see no reason fro doing both.
I need is a convincing reason to do both.
Alternatively you could also subject it to a low pass filter.
Now I have previously described 12 month differencing as a low pass filter, now that is not strictly the case. For periods longer than 6 years it is a type of low pass filter, but from 6 years down to 1.2 years it amplifies the signal (the strong El Nino Signal is in this band). Basically it stops frequencies at (0,1,2,3,4,…) Cycles/Yr and amplifies frequencies around (0.5,1.5,2.5,…) Cycles/Yr.
At the stop frequencies it removes all variance, at peak frequencies it increases the variance by a factor of 4.
I would think twice before I would rely on data transformed in this way to tell me much about cyclic trends. Choosing monthly differencing would not have produced the various pass and stop bands but it would have produced a very different Hurst coefficient.
Alexander
Patrick Davis (23:04:07) :
“VG (22:03:06) :
“Its always been a ramdom walk, and will continue to be, unless Krakatoa erupts again in our lifetime. That’s our only chance of experiencing anything remotely close to “climate change”
It has…
http://www.volcanodiscovery.com/volcano-tours/photos/krakatau/june09.html
Probably why we’re getting some truely awesome sunsets recently”.
No, the bulk of the SO2 that colors our sskies today was produced by Sarychev Volcano, plume altitude 21 km and 3 other medium volcanic eruptions in the NH this year, including Mt. Redoubt.
http://volcanoes.suite101.com/article.cfm/sarychev_peak_volcano_june_12_2009
Re MY Above:
It should have read: “Now I have previously described 12 month differencing as a HIGH pass filter, …”
Basil,
“Yes, of course I am.”
Good. So why on Earth did you use a random walk model, when it quite obviously doesn’t fit? Random walks do have the spurious trend property you discuss, but their wandering is unbounded, and climate considered over many centuries very obviously isn’t. There can be no ‘reversion to mean’ with a random walk, because random walks have no mean. Random walks don’t revert.
ARIMA processes are certainly the wrong model too (long term climate has periodic components, and some external forcing), but they offer a closer match to the particular features of climate data you seem to be discussing here. Was it because of a desire for simplicity in explanation?
“And were I trying to forecast temperature trends over the next few months I might well make use of that approach.”
I’d never even considered the possibility you might be trying to forecast anything.
“But ARIMA models are largely black box models which do not require any understanding of underlying physical processes, and are not particularly useful, in my view, in understanding the long term dynamics of underlying physical processes.”
ARIMA models can be constructed based on physical processes, and random walks can be considered to be black box models, too. It’s like arguing that straight line models don’t require an understanding of physics. It’s true that you can fit a straight line to anything, whether you understand it or not, but a lot of straight lines are the result of a physical understanding. The same goes for ARIMA, or any other model.
In any case, a random walk is an ARIMA model. ARIMA(0,1,0) to be precise. Like I said above, it’s a generalisation.
And I don’t see where you derive the random walk characteristics from any underlying physical processes, either.
“Have you ever looked at the confidence intervals of an ARIMA forecast? After a few months, you are in la la land.”
Have you ever looked at the confidence intervals for a random walk forecast? They increase in proportion to the square root of time, (centred on the last observation,) and therefore increase without bound. Random walk forecasts explore la la land far more deeply than many ARIMA processes.
Alex Harvey (05:33:00) :
Basil & Peter Vaughan,
This annual differencing would do two thing, remove a seasonal cycle, if present, and progressively filter out some of the frequency components.
Basil Wrote: “A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, …”
Now as the series is already seasonally adjusted, the choice of 12 months is not necessary to remove the seasonal component of the earths temperature record. It will remove variance as in the 12,6,4,3, and 2 month periods but is this a necessity given what else this diverencing does to the data.
You are beating a dead horse here, Alex. I’ve shown that the same pattern of smoothed oscillations can be extracted from the monthly data, without any seasonal differencing. You are barking up the wrong tree, this dog won’t hunt, and whatever other mixed metaphor you choose, you are raising a non-issue here.
To wit, the Hurst exponent of the smoothed series derived using monthly data with no seasonal differencing is 0.792, versus the 0.835 computed using the model derived with seasonal differencing. For the point being made, that degree of difference is insignificant.
Obviously you can filter the data in anyway that suits your purposes but to do so, and then to attach any particular significance to a resulting Hurst coefficient seems a little strange.
The Hurst exponent is just a convenient descriptive statistic that describes the degree of persistence (or lack of persistence) in a time series. It is what it is, and nothing more. Instead of harping on use of the Hurst exponent, you could better contribute to the discussion of whether or not the cycles revealed by the smoothing mean anything. Obviously, because smoothing is involved, and particularly because of the resulting cycles in the data, the Hurst exponent increases.
In truth, while you’ve been all over me for seasonally differencing the data, and for applying the Hurst exponent to the resulting smooth, the really more significant conclusion to be drawn from the Hurst exponent is the anti-persistence demonstrated in the residuals once the “trend” is removed. This wouldn’t happen with a strictly linear process with iid residuals. So here the Hurst exponent may be telling us (descriptively) something significant about the data.
And lest you think that is an artifact of the seasonal differencing, when I use only the monthly observations (no seasonal differencing), fit the HP smooth to the data, and subtracting out the residual variation around the smooth, the Hurst exponent of the residual (visualize here the bottom pane of Figure 3, but now just from month to month variation) is 0.395, compared to the 0.383 I get using the seasonally differenced data . So what is the big deal? Do you have any comments on that?
Choosing monthly differencing would not have produced the various pass and stop bands but it would have produced a very different Hurst coefficient.
But it didn’t. Unless you think 0.395 is “very different” from 0.383 (or 0.792 is “very different” from the 0.835. No slight intended, but are you sure you know what you are talking about? Maybe you do, in an abstract context, but not here, that is you really do not seem to fully comprehend what has been done, so your knowledge about the subject is being misapplied. More over, as I’ve shown, as far as the HP smoothing working like a low pass filter, I get to the same place whether I use monthly data, or seasonally difference the data first. So you are incorrect there, also, that “monthly differencing would not have produced the various pass and stop bands.”
Look, rather than just criticize, if you think something was done wrong, and that the result stems from doing it wrong, and that if you do it “right” you get a different result, why not demonstrate it?
Truth is, I’ve met your every challenge. I’ve shown that your complaints are much ado about nothing, that I can get there without seasonal differencing.
Having done that, do you have anything else to bring to the discussion?
Stevo (08:05:08) :
Have you ever looked at the confidence intervals for a random walk forecast? They increase in proportion to the square root of time, (centred on the last observation,) and therefore increase without bound. Random walk forecasts explore la la land far more deeply than many ARIMA processes.
Fair rejoinder. But in the end, your focus on the best way to model a random walk seems to miss the point. The real point is not the possibility that Figure 2 is a random walk, or whether there are better ways to model it. The real point is what the disaggregation of the series into the two components presented in Figure 3 says about possible physical processes influencing global temperature trends. My frustration with much of the discussion, such as with you and Alex, is that it focuses only on questions of methodology, without actually showing that the issues you are trying to raise have anything to do with the implications this analysis might have on the science (or physical processes) under consideration.
Do not get me wrong. Methodology is important, especially where it can be shown that the methodology is leading to wrong or unjustified conclusions. But neither you nor Alex have done that. You are arguing methodology on abstract grounds completely devoid of any connection to the main point of the post. In your case, you want to argue about the best way to model a random walk. I.e., you are hung up on Figure 2, when my point isn’t about modeling a random walk at all, but about what is implied by Figure 3. And I don’t need a random walk to get to Figure 3, and I fail to see how arguing about to best model a random walk advances the discussion I’d prefer to have over Figure 3.
I don’t know if this will be the end of the discussion or not. I imagine it is winding down. What I am about to say does not apply to you specifically, but is a general observation about internet discussions such as this one. A lot of people jump in and begin to critique a novel presentation before they fully understand what the presenter is saying, or doing. I live by a long standing rule that before I criticize someone, or what someone is doing, I try to understand their point of view, whether I agree with it or not. I’m not sure I’m getting the same consideration here.
But that is okay. I’ve taken the time to respond, respectfully, to a lot of criticism here. And that is fine. I can take the heat, because I know that only by exposing our beliefs or reasoned considerations to the possibility of refutation can we truly be assured that we are not fooling ourselves (by confirmation bias). But frankly, the most vocal criticism has only served to reinforce my conviction that there is something worthwhile in what I’m doing, because with the possible exception of Leif’s issue with the Chree analysis, none of the criticism has amounted to much.
And the jury is still out on the Chree analysis. When I get the time, I’m going to do it Leif’s way, and see what happens.
Basil (12:04:49) “A lot of people jump in and begin to critique a novel presentation before they fully understand what the presenter is saying, or doing.”
There will always be those who will opportunistically engage in obfuscation about differencing & smoothing (regardless of whether applied sensibly or not).
One suggestion I can share from years of experience teaching online stats: Keep posts brief & to-the-point if your aim is to avoid a wasteland of comments & inquiries from online skiers (those who skim & skip).
At least we know from the comments that people are enthusiastic to participate in discussions about this type of analysis. I look forward to more (in chewable chunks).
Thank you for presenting.
Smokey wrote: “Speaking of alarmism, why should we listen to those Chicken Littles, who keep telling us the sky is falling? You may not be aware of it if you inhabit one of the echo chambers named above, but the planet’s current temperature is right at about the same level it was at thirty years ago: click.
“Instead of going ballistic when someone mentions the “Best Science” site, maybe you should stick around and learn something.”
Smokey, has it ever occurred to you to actually think about what I have said above? For example, about the problem of using short term data, and about the problem of picking and choosing data points to make it appear as if something is happening which is not really reflective of reality? In fact, may I ask you if you have any understanding of the random walk topic posted here? Because if you did, then you wouldn’t be showing graphs that show that if you start from a place of your choosing which is not too far back, and go to a point where it is particularly convenient for your argument, and then concluding that this is a sign of a cooling trend. Or at least, you wouldn’t be showing that to a person who has a background and statistics, who clearly can see the misuse of such statistics.
In fact, Smokey, you are just another example of the point I have made in several posts here. People like you, who love to throw around your charts but do not understand basic statistics and concepts behind the theory, are misleading people into believe things that are not true. And as I have said above, rather than listening to what the legitimate skeptics are saying, you try to invent the argument yourself even though you have a complete misunderstanding of the basics.
Obviously, Smokey, this is not sinking in. So let me try to rephrase it as a challenge to you. I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend. To make this tractable for you, here is where you can get a list of scientists in the top 500:
http://www.eecg.utoronto.ca/~prall/climate/climate_authors_table.html
By the way, Smokey, I am not surprised that your “award winning” refers to website popularity, and not scientific popularity. As I have said, this website is the reason why there is so much misinformation about the science. It is the source!
Well, gosh, Dr. Sanchez, let’s look at the longer term, then.
We’ve warmed ~0.7C or so this century, globally. If the GHCN adjusted data is to be trusted. We can see the imprint of PDO cycles in the record. We also know that 1976 through 2001, the six major cycles went from cold to warm, one by one. We’ve seen a relatively flat trend since 2001, with a downturn starting in 2007.
The IPCC premise seems to be that the entire 21st century will warm at the rate of 1979 – 1998, or even faster. At this early date, that seems not to be the case. If the PDO follows its past patterns, we’ll be in a negative phase for another two to three decades. For the IPCC to be correct, there would have to be a roaring warming after that point (or even from this point).
I agree that CO2 will put an upward pressure on temperatures. But the IPCC scenario is dependent on strong positive feedbacks, and so far the feedbacks appear, if anything, to be negative. This completely ignores the solar issue, which influence (if any) is as yet unknown.
The hockey stick has been shown to be in error (to be charitable). We do not seem to be in an imminent crisis situation, and we are in an age of hugely advancing wealth-fed technology.
If we continue to warm at a rate of 0.7C or even somewhat more over the 21st century, there is no crisis. So it’s not even a matter of reversing past trends.
That is the “shorter” longterm picture. The next decade’s worth of data will show us a lot more. I do not see that this places us in a position where we must react drastically with a guesswork-based solution. In my opinion, we need to monitor the situation carefully and, above all, openly. If there is a crisis, we will be far, far better equipped to deal with it in a decade, unless, of course, we do not eliminate a large percentage of world growth in the meantime.
Dr Jose Sanchez,
I’ll do better than find one scientist out of 500. The OISM Petition was signed by over thirty thousand people with advanced degrees in the hard sciences. The petition reads:
Savor that last paragraph. Re-read it. Again. It will do you good. They are saying very clearly that there is nothing to be alarmed about.
I am not misrepresenting anything by linking to a graph of four government/university agencies, plus the ARGO data, all of which show cooling for most of the past decade. When every metric shows cooling, and when tens of thousands of scientists state that there is nothing to worry about, I wonder why you would say the opposite?
Spin your statistics any way you want, but the real world is telling us that it has been cooling for most of the past decade.
I’ll give him 3: Richard Lindzen, Roy Spencer, and Roger Pielke Sr. Quite frankly, I don’t think there are any scientists, top 500 or otherwise, that don’t agree temperatures have been either cooling or leveling off for the last decade You’d be stupid not to. Even Gavin admits as much every time he says it’s not inconsistent with the models.
So, why do so many Drs. make the authority argument? Are they afraid of the technical argument?
Mark
They can make any appeal they like. But appeal to authority without releasing data is rather telling. No skeptic would be able to get away with that. Which is fine, because no one at all should be able to get away with that.
Smokey,
what part of “I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend.” did you fail to understand?
Oh, besides, I see your OSIM assorted MDs and engineers and raise you 84% of the AAAS members saying “Warming is due to human activities” (just 4% claiming there is no evidence of warming) and 70% considering global warming a very serious, 22% considering it a somewhat serious problem and just 2% contending it to be no problem at all. The numbers are from a recent poll of 2500 members of the AAAS by the Pew Research Center. They also document the warped public perception of the view of scientists and the strong divide along political affiliation in the US.
Basil,
Thankyou for your respectful responses.
Part of the reason I don’t criticise your main point is that I don’t see any reason to disagree with it. That’s partly because I’m already of the view that a lot of the short term ‘trends’ are spurious randomness anyway, the result of stochastic processes with memory, but partly because the issues with methodology obscure precisely what you’re doing and what you mean by it for me.
But the main reason I raise such objections is in the hopes of improving the arguments. When I start reading through an argument and suddenly run into something I know is wrong, it jerks me up short. I find it distracting and off-putting. Even when it is irrelevant to the rest of the argument, it is still an error. And if the rest of the argument continues to refer to and rely upon it, it makes things very difficult to follow. And worse, even irrelevant points get picked up and remembered as background knowledge by other readers, and repeated elsewhere.
It’s one reason why I keep ranting about it every time I see someone repeat the line that greenhouse gases “trap” radiation, even if it’s just by way of a scene-setting intro and not relevant to the main argument, because other people then go on to waste huge amounts of effort to debating and debunking a mechanism that isn’t even the ‘official’ understanding of the greenhouse effect anyway. Letting the small errors pass perpetuates the myths.
Criticism isn’t always opposition. I do sometimes find that when I criticise a sceptic’s argument, people assume I’m therefore an AGW-believer. I had hoped that citing Climate Audit might allay that, but it doesn’t always work. Many of the arguments in this area, on both sides, are in some small detail wrong. That includes many of my own. Nobody likes that; but it’s unfortunately inevitable. On the whole, I’ve generally found sceptics to be more open-minded with regard to improving them.
You have paid more attention than many would, for which I thank you. I apologise for the misunderstanding. Your conviction that there is something worthwhile in what you are doing is not something I have any wish to argue against. My aim is to help, because I think the details do matter.
Basil,
Perhaps we have a terminology problem in case so, I will clarify.
I am referring to 12 month differencing X(13)-X(1) etc., of the HadCRUT3 time series for which I get H=0.415 (You gave us H=0.475)
and to 1 month differencing X(2) – X(1) for which I get H=0.228.
I do not think that you have given us a H value for this filter of the HadCRUT3. It you have I apologise but could you give it again.
So for me one filter gives a series with H close to 0.5 and the other definitely does not.
I felt that you were drawing some significance to its closeness to zero and I am saying that it could be seen as happenstance as H values differs if you use different filters.
I have tried examples of other filters quite arbitrarily just because I or others have your choice might be seen as arbitrary and suggested such filters as alternatives.
For X(25)-X(1) I get H=0.533.
This filter has 11 stop bands (inside the pass band) with amplification in its pass bands.
Also for simple high pass filter (time constant = 1/(2pi)yrs) I get H=0.238.
This filter definitely does not have any stop bands (inside the pass band), only a minima at f=0, also and at no frequency does it amplify the variance.
Another method of minimising the seasonal component that, for me, leaves the H value largely unchanged and close to 1.
Here I am referring to subtracting a climatology based on the means (over the whole interval) for each month,
I get H=0.980 after removing a climatology. This is the same value (3dp) I get for the unfiltered series.
This type of filtering has the same stop bands as 12 month differencing but they are much finer. It does remove the seasonal cycle without doing a lot else. In this case it removed about 27% of total variance whereas 12 month differencing removed > 60%. It is also not a high pass filter. It is flat across the spectrum except for narrow stop bands at 12,6,4, … month periods. This is almost certainly why it leaves the H value almost unchanged.
As I mentioned above 12 month differencing also amplifies the variance across the majority of the width of the pass bands so the 60% does not give such a good guide to how much more targeted subtracting a climatology is. After adjusting for this amplification by dividing the series by 2 to reduce the maximum gain in the pass band to unity, >90% of the variance has been removed by 12 month differencing).
So the filtering effect of the 12 month interval does produce stop bands at 1,2,3,4,5 Cycles/Yr whereas I find a 1 month interval does not.
If you are saying that one month differencing produces stop bands can you tell me where they are?
For clarification I have not been included the behaviour for f->0 as this minimum is common to high pass filters I am referring to stop bands inside the pass band.
For reference: The number of stop bands (inside the pass band) is given by N/2 – 1 (N=interval in months) for even months, and (N-1)/2 for odd months. I doubt that I am wrong here. If you wish to count minima at the top of the pass band (6 cycles/yr in this case) you add one for even months.
So I definitely get different Hurst coefficients depending on my choice of filter.
Now I make no bones about what I do and do not know. I do know the results I get and if I discover I am wrong I will take it on the chin.
You say that you can get there without seasonal differencing yet you chose to do it, and I thought that you were drawing some significance from a H value ~0.5. If you no longer think that was a point worth making just snip it out.
I would do so because I think it is not sound. I find it to be highly dependent on your choice of a filter that for me seems to serve no purpose other than to provide this value. I am not sure why you added this stage which you say you do not need, and except for saying that people commonly do this I am not sure you have done much more to justify doing it.
Now I think you were asking for some specifics so I have provided some, hopefully this will clarify the discrepancy between our points of view on the choice of filters.
BTW at no point have I referred to your processing beyond this point, as I have no idea what Hodrick-Prescott smoothing would do to Hurst coefficients, so I could not comment. But I shall look to it.
At this stage I am concentrating on figure 2 not figure 3. I have not commented on the H values for figure 3, so need to restate them.
Alexander
bluegrue (05:46:28),
With all the false-alarmist huffing and puffing, they still can’t come up with more than a small fraction of 31,000 scientists who say:
What part of ‘no convincing scientific evidence’ do you people fail to understand?
Regarding Jose’s silly challenge, let me put that out of its misery. He linked to a list comprised of less than one-tenth the number of signers of the OISM Petition, which is limited to those with degrees in the hard sciences.
But Jose’s list? His list includes poseurs with degrees in things like Community Ecology, Advertising, Prediction [heh], Politics, Downscaling, Market Research, Economics, etc.
Furthermore, about 40% of Jose’s list is comprised of UN/IPCC political appointees, who have their marching orders, no matter what they privately think. Those IPCC individuals have traded their credibility for job security.
And Pew? Give me a break. They were easily the most error prone, inaccurate polling in the last election; Pew polls are used to skew results. In other words, for propaganda. How else can we take an all-or-nothing question like “Warming is due to human activities”? Does that mean 100% of warming? Hang your hat on that push poll if you want, but it is not credible when it frames questions like that.
I won’t embarrass the inept Jose Sanchez for his foolish challenge: “I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend.”
Jose’s own list includes internationally esteemed names like Ross McKitrick, Roy Spencer, Lubos Motl, Fred Seitz, William Gray, Freeman Dyson, Nils-Axel Morner, Pielke, Sr. & son, Edward Wegman, Roger Revelle, Chris Landsea, Syun-Ichi Akasofu, Nir Shaviv, Bob Carter, Craig Idso, Tim Ball, Willie Soon, Henrik Svensmark, Piers Corbyn, Benny Peiser, Hans Erren, Joe D’Aleo, Vincent Gray, and plenty of other AGW skeptics. The list also has the names of disreputable individuals like Caspar Amman, Rajendra Pachauri, William Connolley and Michael Mann, who wouldn’t recognize integrity if it bit ’em on the ankle.
Jose was simply winging it with his fingers crossed, hoping that no one would check the list he posted. I’d be willing to wager $10,000 that I can find someone on his list who thinks we’re in a cooling trend, if you or Jose are game. Otherwise, why bother? We all know the answer to Jose’s foolish challenge.
So you and Jose lose both the “consensus” claim and the challenge, hands down. You couldn’t even come up with one-tenth the number of scientists that the skeptic side has. Which is the way it should be: skepticism is an absolute requirement of the scientific method — a requirement that the false-alarmist side has given up and surrendered to greed. Now, they’re just grant hogs with both front feet in the public trough.
Alex Harvey (08:57:49) :
and to 1 month differencing X(2) – X(1) for which I get H=0.228.
I do not think that you have given us a H value for this filter of the HadCRUT3. It you have I apologise but could you give it again.
Yes I did post this above, but in a reply to PaulM, who wrote:
3. If I do a month-to-month difference, as some people have suggested, instead of the seasonal difference, I get a much smaller number, around 0.1 – I don’t understand why this is!
To which I responded:
“I get 0.24. What software are you using? (I’m using gretl.) As for the number being noticeably smaller, I’m not sure that should be so surprising. That is simply saying that the degree of anti-persistence is greater in monthly fluctuations than it is in annual fluctuations. That makes sense to me, in that monthly fluctuations will tend to revert to the mean more quickly than shocks measured on an annual basis. Actually, given my numbers — 0.24 for monthly, and 0.38 for seasonal — the difference is about what I would expect.”
I think the explanation holds up as well for any point you are trying to make of this.
At this stage I am concentrating on figure 2 not figure 3. I have not commented on the H values for figure 3, so need to restate them.
No, I do not need to restate them. You need to do me the courtesy of reading the whole post, and get off of your fixation over Figure 2. I’m not discussing this any more with you, if after all this time you haven’t even bothered to read the entire original post (or read it so cursorily that you do not recall that what you now ask me to restate in in the post, as was a major focus of the post). Frankly, you have just vindicated my previous post (in reply to Stevo). You are wasting my time. I’ve got more to do on this, so I shall consider our discussion at an end, and move on to other matters.
Stevo (06:02:58) :
Basil,
Thankyou for your respectful responses.
And thanks to you, too. I sometimes get impatient, as in my previous reply to Alex, but I do appreciate vigorous discussion and challenge, so long as I think something constructive can come of it. So again, thanks.
AAAS has about 125,000 members, 84% of which is 105,000, if you really want to play the numbers game.
Community Ecology has nothing to do with your local town hall, but is relevant for impact assessment; do you have anything specific to say against GC Hurtt? Or FW Zwiers or C Rosenzweig? You mocked them for being listed under “prediction”. And why would you, Smokey, of all people, mock JS Armstrong, the only one listing advertising? He signed the 2009 newspaper ad by the Cato Institute. His most cited paper on climate is published in E&E and his paper on Polar bear population is published in Interfaces, a curious choice, given that the journal’s scope is “Learn how to overcome the difficulties and issues encountered in applying operations research and management science to real-life situations.”. Too bad that most of his citations are not for his work on climate, otherwise you would have had a winner. I have not checked further.
Given how shallow your reading seems to be, I’m stepping out of this discussion.
Basil,
could you please reply to my questions, too? You seem to have missed them. I’d just like to see, whether your analysis is sensitive to long-term trends.
Re: Alex Harvey (08:57:49)
Alex,
I would like to discuss this with you, but Hurst coefficients are new to me and my plate is already full for the foreseeable weeks. I’ve downloaded the papers others have cited above. I thank you & others for sharing your enthusiasm for the methods under discussion – I hope to look into this in the future.
Also: I hope we (WUWT participants) might start having some in-depth discussions about wavelet methods moving forward. It’s tricky when not everyone in the audience knows a method. For example, I suspect many readers might better-understand the info in my wavelet plots (upthread) if I simply present the following plain time-plots instead:
http://www.sfu.ca/~plv/(J,N)o2&Pr.png
http://www.sfu.ca/~plv/PhaseConcordancePxySI.png
http://www.sfu.ca/~plv/(J,N),r..png
http://www.sfu.ca/~plv/Pr,JN4,r..,m4..png
Perhaps these will lead readers unfamiliar with wavelet methods to a better appreciation for the post-1940 stability of the Chandler Wobble period.
We are certainly well-into an era when specialized statistical-computing is leading to communication break-downs.
Regards,
Paul.
I have been looking at Hodrick-Prescott filters, and the following seem to be true, but if anyone knows better please tell.
Directly from the definition it appears to me to be a low pass filter with the folowing gain.
G(w) = 1/(1+L*(w/A)^4)
where L is the lambda value and “A” is the number of elements per unit time.
Hence its slope is to 1/f^4, with a cut-off frequency given by:
fc =A/(2*pi*L^(.25))
In this case a=12 (months in a year), and L=12900 giving fc=0.1 cycles/yr.
It does appear to have one very useful property in the the phase lag is zero.
It appears to be have the Buterwortth property of maximal flatness but differs in that G(w) is of the form 1/(1+f(w)) as opposed to the 1/Sqrt(1+f(w)) for the Butterworth filters I know. Also the zero phase lag is not typical.
At first sight it does not seem to be a real time realisable filter in that the minimisation process that describes the HP filter is over the entire series and would require foresight to impliment.
I also suspect that it generalises into a range of filters in a fashion similar to the Butterworth family.
That said it basically is a moderately steep low phase filter. In this case it attenuates signals with a periods less than 10 years.
By the time it gets to down to 2 years its gain is down to ~1/600. I expect that this may be why it does not care if it is coupled to a 12 monthly differenced filter or a 1 monthly differenced filter. Either combination will have a pass band somewhere around 10 years and similar performance both above and below that period.
Alexander Harvey