Guest Post by Basil Copeland
Figure 1
Each month, readers here at Watt’s Up With That, over at lucia’s The Blackboard, and elsewhere, anxiously await the latest global temperature estimates, as if just one more month of data will determine one way or the other the eternal destiny of AGW (the theory of “anthropogenic global warming”). For last month, July, the satellite estimates released by UAH and RSS were up sharply, with the UAH estimate up from essentially zero in June, to +0.41°C in July, while the RSS estimate was up from +0.081°C to +0.392°C. Does this sharp rise presage the resumption of global warming, after nearly a decade of relative cooling? Or is it just another in a series of meandering moves reminiscent of what statisticians know as a “random walk?”
I have not researched the literature exhaustively, but the possibility that global temperature follows a random walk was suggested at least as early as 1991 by A.H. Gordon in an article in The Journal of Climate entitled “Global Warming as a Manifestation of a Random Walk.” In 1995 Gordon’s work was extended by Olavi Kӓrner in a note in the Journal of Climate entitled “Global Temperature Deviations as a Random Walk.” Statistician William Briggs has written about climate behaving like a random walk on his blog.
Now even I will confess that the notion that global temperature, as a manifestation of climate processes, might be essentially random is difficult to accept. But I am coming around to that view, based on what I will present here, that monthly global temperature variations do, indeed, behave somewhat like a random walk. The qualifier is important, as I hope to show.
So, what is a “random walk” and why do some think that global temperature behaves, even if only somewhat, like a random walk? And what does it matter, anyhow?
While there are certainly more elegant definitions, a random walk in a time series posits that the direction of change at any point in time is essentially determined by a coin toss, i.e. by chance. As applied to global temperature, that is the same as saying that in any given month, it is just as likely to go up as it is to go down, and vice versa. Were global temperature a true random walk, there would be no underlying trend to the data, and any claimed evidence of a trend would be spurious. One of the best known “features” of a random walk is that in a time series it appears to “trend” up or down over extended periods of time, despite the underlying randomness of the direction of change at each point in time.
So why might we think global temperature follows a random walk? One reason is suggested by a close look at Figure 1. Figure 1 is the familiar HadCRUT3 time series of monthly global temperature anomalies since 1850, with a simple linear trend line fit through the data. When we look close, we see long periods, or “runs,” in which the data are above or below the trend line. If the data were truly generated by a linear process with random variations about the trend, we’d expect to see the deviations scattered approximately randomly above and below the trend line. We see nothing of the kind, suggesting that whatever is happening isn’t likely the result of a linear process.
On the other hand, when we perform what is a very simple transformation in time series analysis to the HadCRUT3 data, we get the result pictured in Figure 2.
Figure 2
A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, i.e., comparing one month’s observation to the observation from 12 months preceding. Since 12 months have intervened in computing this difference, it is equivalent to an annual rate of change, or a one month “spot” estimate of the annual “trend” in the undifferenced, or original, series. When we look at Figure 2, it has the characteristic appearance of a random walk.
But we can do more than just look at the series. We can put a number to it: the Hurst exponent. Here’s a very understandable presentation of the Hurst exponent:
“The values of the Hurst Exponent range between 0 and 1.
-
A Hurst Exponent value H close to 0.5 indicates a random walk (a Brownian time series). In a random walk there is no correlation between any element and a future element and there is a 50% probability that future return values will go either up or down. Series of this type are hard to predict.
-
A Hurst Exponent value H between 0 and 0.5 exists for time series with “anti-persistent behaviour”. This means that an increase will tend to be followed by a decrease (or a decrease will be followed by an increase). This behaviour is sometimes called “mean reversion” which means future values will have a tendency to return to a longer term mean value. The strength of this mean reversion increases as H approaches 0.
- A Hurst Exponent value H between 0.5 and 1 indicates “persistent behavior”, that is the time series is trending. If there is an increase from time step [t-1] to [t] there will probably be an increase from [t] to [t+1]. The same is true of decreases, where a decrease will tend to follow a decrease. The larger the H value is, the stronger the trend. Series of this type are easier to predict than series falling in the other two categories.”
So what is the Hurst exponent for the series depicted in Figure 2? It is 0.475, which is very near the value of 0.5 which indicates a pure random walk. And when we exclude the data before 1880, which may be suspect because of a dearth of surface locations in computing the HadCRUT3 series, the Hurst exponent is 0.493, even closer to 0.5. So by all appearances, the global temperature series has the mark of a random walk. But appearances can be deceiving. In Figure 3 I fit a Hodrick-Prescott smooth to the data:
Figure 3
In the upper pane, the undulating blue line depicts the smoothed value derived using Hodrick-Prescott smoothing (lambda is 129,000, for the curious). In the lower panel are detrended seasonal differences, i.e., what is left after removing the smoothed series. Conceptually, the smoothed series can be taken to represent the “true” underlying “trend” in the time series, while the remainder in the bottom pane represents random variations about the trend. In other words, at times, the annual rate of change in temperature is consistently (or persistently, as we shall see) rising, while at other times it is consistently falling. That is, there are trends in the trend, or cycles, if you will. And while it is not obvious, because of the scaling involved, these are essentially the same cycles that Anthony and I have attributed to a lunisolar influence on global temperature trends. That should not be so surprising. In our paper, we smoothed the data first with Hodrick-Prescott smoothing, and then differenced it. Here we’re differencing it first, to show the random walk nature of the series, and then smoothing the differences. But either approach reveals the same pattern of cycles in global temperature trends over time.
Looking more closely at the smoothed series, and the random component (labeled “Cyclical component” in Figure 3), we have an interesting result when we compute the Hurst exponents for the two series. The Hurst exponent for the smoothed (blue line) series is 0.835, while the Hurst exponent for the detrended random component (bottom pane) is 0.383. The first is in the range associated with “persistent” behavior, while the second is in the range associated with “anti-persistent” behavior. Let’s discuss the latter first.
Anti-persistence is evidence of mean reversion or what is also sometimes called “regression toward the mean.” Simply put, when temperatures spike in one direction, there is a strong probability that they will subsequently revert back toward a mean value. Ignoring all other factors, this property would suggest that the dramatic rise in the temperature anomaly for July should lead to subsequent declines back toward some underlying mean or stable value. I think this is probably more what Gordon or Kӓrner had in mind for the physical processes at work when they proposed treating global temperatures as a random walk. I.e.,shocks to the underlying central tendency of the climate system from processes such as volcanism, ENSO events, and similar climate variations create deviations from the central tendency which are followed by reversions back toward the mean or central tendency. Carvalho et. al (2007), using rather complicated procedures, recently laid claim to having first identified the existence of anti-persistence in global temperatures. We’ve identified it here in a much simpler, and more straightforward, fashion. (I’m not trying to take away from the usefulness or significance of their work. Their procedures demonstrate the spatial-temporal nature of anti-persistence in global temperatures, especially on decadal time periods. I think WUWT readers would find their Figure 10 especially interesting, for while they do not use the term, it demonstrates “the great climate shift” of 1976 rather dramatically.)
With respect to the smoothed series, the Hurst exponent of 0.835 indicates persistent behavior, i.e. if the series is trending upward, it will have a tendency continue trending upwards, and vice versa. But that is to be expected from the cyclical undulations we observe in the smoothed series. As to the possible physical processes involved in generating these cycles, after Anthony and I posted our paper, comments by Leif Svalgaard prompted me to perform a “superposed epoch analysis” (also known as a “Chree analysis”) on these cycles:
While Leif contends that the analysis should be performed on the raw data, in this case I would beg to differ. As shown in Figure 3, the raw data is dominated by the essentially random character of the monthly changes, completely obscuring the underlying cycles in the data that emerge when we filter out (detrend) the raw data. Arguably, what we have in the blue line in Figure 3 is a “signal” that has been extracted from the “noise” depicted in the bottom pane. Now as such, the “signal” may mean something, or it may not. That is where Figure 4 comes in to play. The peaks in the cycles depicted by the blue line in Figure 3 show a strong correspondence to maximums in the lunar nodal cycle (the “luni” part of our suggestion of a “lunisolar” influence on global temperature trends). They also show a strong correspondence in solar maxima associated with odd numbered solar cycles, especially beginning with solar cycle 17. Are these correspondences mere coincidence? Anthony and I think not. While each may play an independent role in modulating global temperatures, since the 1920’s the solar and lunar influences appear to have been roughly in phase to strongly influence temperature trends on a bidecadal time frame. In other words, Figure 4 may be revealing the physical processes at work in explaining the persistence revealed by the Hurst exponent for the blue line in Figure 3.
Taken together, the two Hurst exponents – one for the true “signal” in the series, and the other for the “noise” in the series – essentially offset each other, leaving us with a Hurst exponent for the unsmoothed, raw, seasonal difference of ~0.5, i.e., essentially a random walk. And so on a monthly basis, the global temperature anomalies we await anxiously are essentially unpredictable. However, if the cycles in the smoothed series can be plausibly related to physical processes, as Anthony and I believe, that gives us a clue as the “general direction” of the monthly anomalies over time.
In our paper together, Anthony and I presented the following projection using a sinusoidal model based on the same cycles shown in the blue smooth in Figure 3:
Figure 5
The light purple line in Figure 5 is, essentially, a continuation, or projection, of the blue smooth in Figure 3. From this, we derived a projection for the HadCRUT3 anomaly (light blue in Figure 5) which has it essentially meandering between 0.3 and 0.5 for the foreseeable future (here, roughly, the next two decades).
But the monthly values will vary substantially around this basically flat trend, with individual monthly values saying little, if anything, about the long term direction of global temperature. In that sense, global temperature will be very much like a random walk.
Conceptually, I hate the idea of global temperature movements being a random walk. But I must admit that the math behind this model appears to be much more solid and much more supported by the record than does that of any other model. Dammit.
Random walk is a bit tricky. After N steps of length L, the walker will be a distance L*SQRT(N) away from his starting point, which increases with N. And if he has strayed away to some distance, there will not be a strong probability that his next step will take him closer to the mean. That probability is still 50%, like if I had tossed Heads three times in a row, there is no higher probability that the next toss yields Tails; that probability is still 1/2.
good good very good
Its always been a ramdom walk, and will continue to be, unless Krakatoa erupts again in our lifetime. That’s our only chance of experiencing anything remotely close to “climate change”
OT but the Australian Senate has rejected the ETS. The current government may have a surprise if they dissolve the parliament and call an election of the issue next year. Looks like the arctic pivotal AGW argument may be disappearing.
http://arctic-roos.org/observations/satellite-data/sea-ice/observation_images/ssmi_ice_area.png
It will be indeed very interesting to have a look at this by January 2010.
When I was young it was called the drunkard’s walk, the allusion being to the drunkard clinging to the lampost and trying to walk away from it: and always ending up back in the same place: the lampost.
Yes you are entirely right, to assume that you can get any meaningful data from these short term variations is utter folly.
People try to do this and use what they suppose to be some statistical test that supposedly proves or otherwise the validity of their analysis only shows that they do not understand what they are doing.
No matter, it amuses them.
And me too, although I get weary of their idiocy from time to time.
Kindest Regards
Well I sometimes try and find some correlation of temperature with something but with no luck so far. We seek it here, we seek it there, those warmies seek it everywhere. Is it in carbon? or is it in helium? That damned, elusive Pimpernel.
Hang on minute! I regularly read here the assertion that due to negative feedbacks in the climate system, increasing greenhouse gasses could not possibly have a significant affect on global temperature. Now you’re asserting that it is a random walk. Seems to me that it’s your sequence of climate theories that most resembles a random walk.
But isn’t that exactly what has happened since the middle of the Little Age?
A slow overall sinusoid trend rising ever since 1750, with a shorter 60 year cycle superimposed on that primary signal?
“But the monthly values will vary substantially around this basically flat trend, with individual monthly values saying little, if anything, about the long term direction of global temperature. In that sense, global temperature will be very much like a random walk.”
Hence the need to look at other meteorological data than temperature such as barometric pressure evolution.
While Leif contends that the analysis should be performed on the raw data, in this case I would beg to differ. As shown in Figure 3, the raw data is dominated by the essentially random character of the monthly changes, completely obscuring the underlying cycles in the data that emerge when we filter out (detrend) the raw data.
It works the other way around: the find the cycles, you detrend the raw data and do the Chree analysis on the residuals [i.e. raw – trend]. The beauty of the Chree analysis is that it automatically dampens the noise and shows you any signal that is keyed to your chosen epochs [points in time or events].
I don’t know about a random walk – I’ve had a few of those back home from the pub.
But what the HADCRUT temperature plot always reminds me of is a digitised version of the opening clarinet glissando from George Gerschwin’s ‘Rhapsody in Blue’. Mann’s hockey stick even more so!
So after removing the trend (in figure 2) it behaves randomly. What am I missing here?
“VG (22:03:06) :
Its always been a ramdom walk, and will continue to be, unless Krakatoa erupts again in our lifetime. That’s our only chance of experiencing anything remotely close to “climate change”
It has…
http://www.volcanodiscovery.com/volcano-tours/photos/krakatau/june09.html
Probably why we’re getting some truely awesome sunsets recently.
The science has been settled for years, according to the politicians. And yet according to scientists there are still so many interesting questions to be argued over and ultimately answered. Life would be so boring if the science were truly settled. This is all fascinating, especially taken with the Scafetti presentation and paper discussed at theAirVent.
Leif Svalgaard (21:42:58) :
Random walk is a bit tricky. After N steps of length L, the walker will be a distance L*SQRT(N) away from his starting point, which increases with N. And if he has strayed away to some distance, there will not be a strong probability that his next step will take him closer to the mean. That probability is still 50%, like if I had tossed Heads three times in a row, there is no higher probability that the next toss yields Tails; that probability is still 1/2.
Her is a visualization of William Fellers classic coin toss distribution (scrool down to Coin Flips and Brownian Motion) interesting behavior whilst probability remains unchanged.
http://orlingrabbe.com/Chaos3.htm
Hey, thanks for your work!
I’m trying to explain this to my son – can you help me? I’m wondering why you use the transformed HadCRUT3 data – why don’t you just apply the Hurst exponent to the normal HadCRUT3 time series? Does that make a random walk too? I feel like that would be more convincing, because it looks like the transformed graph isn’t much like the time series. Or is the transformed graph a better one to use, as it seems to show no temperature rising at all!
Keep it up!
There are many weather stations with daily data, which should also show random walks when detrended. The detrending causes me worries. The ability to predict the temperature on Day 1 from that on Day 0 is greater than the ability to predict the temeperature on a day in Year 1 from the same date in Year 0. This is because there are degrees of change in the dependence of one figure on the next. In some forms of statistical analysis, it is desirable to show independence is complete.
Given that the random walk approach should work on a daily data set, and that it does not require a global database, would there be value in repeating the exercise on a daily basis? There are many datasets that go back to 1860 or so.
The disadvantage in using HadCrut3 data is that they have been adjusted in ways that are not disclosed. A prudent scientist might be bothered about the quality of the data.
Well it is a highly chaotic system……..
I like drunkards walk ( a jones (22:20:21) 🙂
The intro needs work really I don’t think you really mean this:
“……..a random walk in a time series posits that the direction of change at any point in time is essentially determined by a coin toss, ..”
That cannot be correct.
My calculation of average combined temperatures at 32 locations spread across the 2.5 million square kilometres of Western Australia from the early 20th century compared to June 2009 (based on official BoM figures) was minima down .12 degrees C and maxima up .75 degrees C. For July 2009, the comparison showed minima up .31 degrees and maxima up .64 degrees. That’s more like a drunken walk than a random walk.
For the state’s five major cities, the average minima (early 20th C compared to July 09) didn’t change at all but the average max was up 1.39 degrees C. For the other 27 smaller locations, the average min was up .37 degrees C and the average max was up .5 degrees C.
Comparing a recent one month average with a 30 year average from a hundred years ago is a bit irrelevant but I still do it for fun every month at http://www.waclimate.net
Basil,
I don’t understand the reason for this analysis. One can easily have a random walk component with a drift component that would be non-stationary in the untransformed data and stationary in the differenced data. Whether or not the global temperature anomaly time series has a random walk component is not particularly interesting.
What is interesting is whether the underlying stochastic (random) process that produces a given time series (such as the global temperature anomaly) is invariant with respect to time. To understand whether this is the case, one differences the time series, that is, one subtracts the value in a given period from that of a chosen earlier period. If the resulting transformed time series is white noise (that is, random) then we may say that the original time series is stationary.
That being the case, you did not explain well why differencing a given month’s anomaly with a 12-month moving average of the anomaly makes sense for the global temperature anomaly. The only reason to do so is if the correlation between a given monthly anomaly and the average of the previous 12 months is higher than for any other combination of months or any given month. However, anyone who has worked with this data knows that the differencing offering the strongest correlation is the one period difference. It is also a fact that the one period difference transformation shows that the original global temperature anomaly time series is not stationary. In other words, there are underlying dynamic in the global temperature anomaly time series.
I suspect that your Hurst Exponent test result is simply due to your decision to smear out differences between the anomaly of a given month with the moving average of the preceeding 12 months. That should create a lower correlation and thus a differencing that appears to be stationary.
Finally, be careful – one can have a random walk with an upward drift with the upward drift being the result of an increase in the global temperature anomaly and yet have a underlying random walk. It doesn’t prove anything.
Cheers
Surely there is no such thing as “random” or “luck”.
The use of the coin toss analogy is not a good one. There is nothing random about a coin toss. If you toss a coin in an identical manner 10 times in a row, you will get the same result 10 times in a row. The average human cannot be that precise, but it is just physics.
Surely it is the same with climate? There is no “random”, the climate just reacts to the initial conditions. Just because Man might not understand them all (as is evident by the ongoing debate about CO2 in AGW), this does not mean they are random.
Isn’t “random” in this case a matter of imprecise measurement etc?
Sorry, I meant to say that one can have a random walk with an upward drift with the upward drift being the result of an increase in the global temperature anomaly due to an increase in atmospheric carbon dioxide.
Cheers
“Each month, readers here at Watt’s Up With That, over at lucia’s The Blackboard, and elsewhere, anxiously await the latest global temperature estimates, as if just one more month of data will determine one way or the other the eternal destiny of AGW (the theory of “anthropogenic global warming”). For last month, July, the satellite estimates released by UAH and RSS were up sharply, with the UAH estimate up from essentially zero in June, to +0.41°C in July,”
HAHAHAHHAHAHAHHA! At the end of the month of June, this website and heaps of right-wing journalists like (such as Andrew Bolt and Mudoch) were claiming that the 30 years of warming had been erased and now the world is cooling. But as soon as the data makes a sudden change, the story changes to “random walk”. Honestly, the only random walk is by those who are changing their theories on a monthly basis on why man is not responsible for climate change!
All credibility is thrown out the window!
This is a very interesting and thought provoking article and Basil deserves our thanks for putting it together. So is global temperature rising sharply due to mans activities? It depends greatly on start point and methodolgy. The following comes from a longer article I wrote recently;
Do we accept that Hadleys start point of 1850 (the LIA) in which 20 poorly distributed stations worldwide is meaningful (or as meaningful as a ‘global’ temperature gets.)
Or do we take Hansens 1880 figures based on slightly better station distribution (but still very poor) that relies on 1200km gridded cells?
The following links may go some way to explaining how we got to where we are .
So first up, what is a global temperature and how was it created?
Link 1 Wikipedia’s explanation of global temperature with a colour globe showing location of weather stations world wide.
http://en.wikipedia.org/wiki/File:GHCN_Temperature_Stations.png
Link 2 Even better explanation with graphs
http://www.appinsys.com/GlobalWarming/GW_Part2_GlobalTempMeasure.htm
Link 3 This piece is taken from link 2 and is a blink chart illustrating station change
http://climate.geog.udel.edu/~climate/html_pages/air_ts2.html
(go to first item- ‘stations locations’ and click) You will get a media player animation illustrating the ever changing location and number of weather stations from 1950. Look for the startling changes since 1990.
Link 4 Over the years four major temperature data sets have evolved, this link shows how each are compiled
http://www.yaleclimatemediaforum.org/2008/04/common-climate-misconceptions-global-temperature-records/
Link 5 James Hansen was foremost in developing a co2 hypotheses which he combined with his work on calculating global temperatures. He believed this definitively supported his view of a proven link between rising (man made) co2 and rising temperatures over the past 130 years or more. Due to this and various other papers (also cited here) he has become a pivotal figure and is responsible for a temperature data set called Giss. This paper is from Hansen in 2009 which shows how Giss is compiled
http://data.giss.nasa.gov/gistemp/
Link 6 the link below is Hansen’s original 1987 paper which is still much used by the climate industry as proof of temperature change since 1880.
http://pubs.giss.nasa.gov/docs/1987/1987_Hansen_Lebedeff.pdf
It was a good piece of detective work from a highly competent and motivated scientist and the following year he used this document as the basis for his talk to Congress on catastrophic warming linked to rising man made co2 emissions- allegedly after ensuring the air conditioning was turned off to ensure his message had a greater impact.
If you look at figure 4 of this paper (after first reading how many times the word ‘estimates’ is used to excuse the interpolation of data to compensate for the lack of numerical or spatial coverage) you will see that it shows the tiny numbers of stations in the Northern and Southern Hemispheres from which the data was initially derived.
In 1850 in the whole of the NH there were 60 weather stations and in the SH there were 10. Hansen chose to use data from 1880 believing the 1850 compilations were too sparse (although they are still frequently cited)
By about 1900 we theoretically had 50% coverage in the NH (if you accept very large gridded squares of 1200km as ample coverage with which to record inconsistent data) and it took until 1940 for the same coverage in the SH.
The Sea surface temperatures (SST) also cited here has been hotly contested due to the nature of the ships data being used-you might have followed the long debate on CA about Buckets and water intakes. (As an aside, quite by chance I met someone who served on a ship and took these water temperatures, and the word haphazard is far too kind a word to use)
Following James Hansen’s 1987 paper many people have attempted to deconstruct his global temperatures, describing either the concept of a single global temperature as flawed, or querying the quality of the data- particularly the further back in history the data refers to.
(G S Callendar wrote his influential co2 thesis in 1938 and even then used only a total of 200 stations worldwide, many of which he was not impressed with-the numbers he believed could be relied on for the period in question here -pre 1900- numbered in the few dozens.)
Link 7 This from IPCC reviewer Vincent Gray querying the meaning of global temperatures
http://nzclimatescience.net/index.php?Itemid=32&id=26&option=com_content&task=view
Link 8 this rebuttal from Vincent Gray of the nature of Hansen’s data in general
http://www.fcpp.org/pdf/The_Cause_of_Global_Warming_Policy_Series_7.pdf
Link 9 this rebuttal from Ross McKitrick of Hansen’s data
http://www.uoguelph.ca/~rmckitri/research/jgr07/M&M.JGRDec07.pdf
Link 10 this refers to the fuss about McKitrick’s paper which was hotly refuted by various people as it queried the very core of AGW data.
http://www.sourcewatch.org/index.php?title=Talk:Ross_McKitrick
Link 11 this technical interrogation of the calculations from Climate Audit
http://www.climateaudit.org/?p=2015
So we have several sets of parallel discussions whereby the meaning or worth of a single global temperature is queried in the first place, and the reliability of the information gathered is contested. This revolves mainly around changes in weather station locations, numbers, methodology and general consistency, plus UHI and therefore the value of the overall reliability of the information derived.
I do not want to detract from Basil’s paper so will curtail the article here. Suffice to say deriving anomalies from fairly meaningless information will come up with a fairly meaningless and suspect ‘global’ temperature which in itself is a strange concept.
Tonyb
We all know that the strong La Nina of a while back has been fading away and we are now on the verge of weak El Nino conditions.
The observed movement of the air circulation systems slightly more towards the poles (but with strong resistance from certain cold air masses) and the increased rate of energy transfer from ocean to air can be expected to result in exactly that which we now observe.
The sole question is whether any of it is caused by human influence. I see no evidence of that unless one can demonstrate that the changed rate of energy release from the oceans is affected by human activity.
We can only affect the air. The air cannot affect the oceans significantly.
Humanity is acquitted.
Not being any kind of statistician, nor any kind of climatologist, but my reaction is to be sceptical of this theory.
A random walk has the feature that future course changes are independent of its history. Like a roulette table.
Physical systems just don’t work that way. They have a state, and they respond to physical laws. The current state of any system that includes any kind of feedback (+ve or -ve) will affect the future course of the system. Lorentz attractors and butterfly wings taught me that at high school.
Craig Allen (22:23:39) :
Hang on minute! I regularly read here the assertion that due to negative feedbacks in the climate system, increasing greenhouse gasses could not possibly have a significant affect on global temperature. Now you’re asserting that it is a random walk. Seems to me that it’s your sequence of climate theories that most resembles a random walk.
Do you mean to say the AGW thermogeddonists didn’t test various null hypotheses before they led us down the road of dismantling modern civilisation?
Caramba!
Basil Copeland
could you please add a constant 0.008°C/year to the seasonally differentiated temperature data and report the Hurst exponent of that new series? I’d be interested to see the changes in the Hurst exponent induced by doubling the warming during the 20th century, i.e. whether your analysis is sensitive to this order of magnitude of linear trend at all.
Another question, why do you differentiate instead of simply detrending the data with a linear fit or other suitably smoothed version of the data? Seasonal differencing seriously changes the spectrum of the noise, adding multiple dips in the FFT spectrum. Furthermore, why use seasonal differences at all instead of simple monthly differences, after all we are dealing with temperature anomalies that have the annual cycle removed already.
1. The Hurst exponent is used to analyse the behaviour of self-similar or scaling systems, NOT random walks. The two are quite different and unrelated. A random walk is a storage measure and should be differenced prior to Hurst-type analysis. As noted by Leif, evidence suggests global temperature does not follow the characteristics of a random walk (although I note in your presentation you do acknowledge this)
2. The Hurst exponent is a fixed value applied to a series, not a variable parameterised across scale. If the Hurst exponent did vary with scale, it would suggest non-Hurst like behaviour of the system.
3. The Hurst exponent is VERY difficult to estimate. Furthermore, because it assesses the behaviour of a time series across scales, it is quite wrong to smooth a series and then estimate the Hurst coefficient. The estimator will not work correctly in these conditions.
4. The Hurst exponent for global mean temperature has been estimated to be in the region 0.9 to 0.95, and this number is consistent across all temperature series (HadCRU, Satellite) as well as historical proxies (e.g. Greenland and Antarctic ice core measures of temperature).
For more info, please see some of the following presentations by Demetris Koutsoyiannis. For an estimate of H over different scales, see pages 28-32 of the following presentation:
From climate certainties to climate stochastics (Opening Lecture)
And for a nice general overview on the Hurst phenomenon and climate:
The Hurst phenomenon and climate
Each month, readers here at Watt’s Up With That, over at lucia’s The Blackboard, and elsewhere, anxiously await the latest global temperature estimates, as if just one more month of data will determine one way or the other the eternal destiny of AGW (the theory of “anthropogenic global warming”).
No – not quite. Looking at the monthly figures can give an indication whether what we think might be happening in the oceans and atmosphere is actually happening. For example – do the satellite readings ‘lag’ the surface readings in response to ENSO fluctuations.
Also the GISS July anomaly of +0.6 implies that the surface is (or was), relatively speaking, cooler than the lower troposphere – yet GISS are continually being accused of ‘fraud’.
I’m confused. A few weeks ago, you were telling us that temperatures in the USA were below average:
http://wattsupwiththat.com/2009/08/10/noaa-july-temperature-below-average-for-the-u-s/
and then you told me of these record cold temperatures for July:
http://wattsupwiththat.com/2009/08/02/roundup-of-some-interesting-july-weather-records/
And at the end of June, you told me the the June global temperature anomaly was important:
http://wattsupwiththat.com/2009/07/03/uah-global-temperature-anomaly-for-june-09-zero/
“Given the U.S. Senate is about to vote upon the most complex and costly plan to regulate greenhouse gases, while the EPA suppresses earlier versions of the chart shown below from a senior analyst, this should give some pause to those who are rational thinkers.”
and now, all of a sudden, the argument has changed to “But the monthly values will vary substantially around this basically flat trend, with individual monthly values saying little, if anything, about the long term direction of global temperature.”
So please, please explain to the rational thinkers on why the data in June matters but the data in July does not, and why the articles about record cool temperatures in certain cities in the USA in July was important but the global average temperature for July is not?
Please, please, explain that.
Sorry, on re-reading my previous post (01:46:17) I noticed my point (1) from my previous post is not well written, and does not read as I intended it. What I was attempting (poorly) to say, is that I recognise you acknowledge the relationship between the random walk and the storage element, but what you are missing is that the behaviour of the raw data is not consistent with that of a random walk, which is the point Leif makes.
That said, a conventional Hurst analysis of the raw global temperature data IS interesting, and I would strongly recommend people read the presentations that I linked above.
Doesn’t this discussion lead back to the understanding that our climate/weather is a chaotic & non-linear system. The idea that predictability is possible rather than just pattern recognition does suggest that people are still struggling with the concept of chaos.
Even if we could control every variable which affects the Earth’s climate/weather system, variation would still occur as a chaotic, non-linear system can shift without external changes to forcings.
That global temperature is a random walk is no surprise.
Relax Dr Jose Sanchez, it’s what we call ‘discussion’. Someone presents an idea and we talk about its merits or otherwise. I know that may be a difficult concept, especially if you are a regular at Real Climate, but that’s how we do things here.
Dr Jose Sanchez (00:50:04) : At the end of the month of June, this website and heaps of right-wing journalists like (such as Andrew Bolt and Mudoch) were claiming that the 30 years of warming had been erased and now the world is cooling. But as soon as the data makes a sudden change, the story changes to “random walk”. Honestly, the only random walk is by those who are changing their theories on a monthly basis on why man is not responsible for climate change! All credibility is thrown out the window!
Here, maybe this will help you:
http://chiefio.wordpress.com/2009/08/13/gistemp-quartiles-of-age-bolus-of-heat/
At the bottom I added a bonus section. When you look at 150 years of the best 10% of the thermometers in the GHCN record (27% of the data) you get no warming.
Not in summer. Not in winter. Not over time. Nada. Nothing. Zip.
Temperatures are stable to within a few 1/10C over that 150 years.
All your “30 years of warming” are concentrated in a few short lived thermometers, flitting into existence, then evaporating.
The temperature rise only happens in winter, not in summer.
The temperature rise only happens in a subset of the the stations.
The temperature rise does not happen in the stable stations.
The spatial domain is not uniformly heated.
The time domain is not uniformly heated.
The heating has a strong SEASONAL pattern.
All of these things say it can not be CO2.
Further: the pattern of thermometer variation strongly implies that all of the “global warming” is simply an artifact of changing what thermometers are read and in the record at any given time. A semi-random event.
I can think of little that would confirm the thesis of this posting more than the observation that on any given day we take a random sample of the air temperatures, but that the central tendency is very stable since (as the top 10% of stations show) the mean just has not changed significantly in 150 years.
Or to put it more bluntly, the climate is astoundingly stable, but the weather tomorrow is nearly random. Central tendency is solid, any given day and place, not so much.
Dermot O’Logical (01:35:19) : A random walk has the feature that future course changes are independent of its history. Like a roulette table.
Physical systems just don’t work that way. They have a state, and they respond to physical laws.
You may well be right on the physical level, but this article talks about the reported temperature. We have a strongly non-Nyquist compliant set of thermometers in the spacial and in the time domain. At any given time you may have dramatically more, or less, of them under a hot or cold mass of air on any one day.
That daily “average temperature” can vary dramatically just from how many thermometers are on the cold vs warm side of the Canada Express…
But over a long period of time, the average tends to a constant.
Thus we have the random daily component, but the strong central tendency in the longer term.
It’s not about the physical system, it’s all about a very poorly distributed set of thermometers, not read frequently enough.
Dr Jose Sanchez (02:13:35) :
“I’m confused… Please, please, explain that. ”
Dr Sanchez, the articles you gave links for are all from different sources, and this latest article is a guest post by Basil Copeland. It might surprise you that different people actually have different opinions. WUWT publishes articles and information from many sources. You do realise that not every single article is penned by Mr. Watts don’t you?
Presumably you write to New Scientist or Scientific American on a regular basis asking them to “Please, please, explain that.” whenever they publish a new article with a different viewpoint?
TonyB (01:22:51) :
This is a very interesting and thought provoking article and Basil deserves our thanks for putting it together. So is global temperature rising sharply due to mans activities? It depends greatly on start point and methodolgy. The following comes from a longer article I wrote recently;
Do we accept that Hadleys start point of 1850 (the LIA) in which 20 poorly distributed stations worldwide is meaningful (or as meaningful as a ‘global’ temperature gets.)
Or do we take Hansens 1880 figures based on slightly better station distribution (but still very poor) that relies on 1200km gridded cells?
====
Er, uhm, no. We DON’T accept Hansen’s tortured and corrupted data as indication of anything but a lesson on “how to manipulate data and hide a methodology of corruption”.
A few points: 1850 did NOT mark the lowest point of the LIA, temperatures have been rising naturally and with no assistance from man since about the 1750- 1780 point, with another dip in 1816 from volcanic activity. Prior to that, they had been declining (again, with no assistance from man!) from the much earlier Medieval Warm Period. Temperatures have been rising sinicie the LIA, oscillating with a 70 year period about a sinusoid baseline that id itself now rising.
Hansen’s methods of smearing data, of recalculating and erasing old data, and of multiply re-calculating and re-averaging vaguely sourced and well-hidden data are NOT clear, nor are they explained, nor are they repeatable. They are NOT based on logic nor on valid and validated computerprogramming techniques, nor have they EVER been audited or passed through ANY review.
Further ..”Link 6 the link below is Hansen’s original 1987 paper which is still much used by the climate industry as proof of temperature change since 1880.”
Hansen’s computer-gaming scenarios (his “climate models”) do NOT provide
ANY proof of anything except “garbage in = garbage out” – though I do agree with your term using “the climate industry” – today’s 60 billion dollars spent on climat epolitics – and the 1.6 trillion demanded by that industry, are far, far larger than all but a few industries worldwide. Hansen’s models approximate output values based on assumed inputs, assumed constants, and assumed changes. NONE of which assumptions can be “proved” correct now, in the model, or in the future.
SOME of his assumptions can be ASSUMED correct under some circumstances as coarse approximations of ideal simplifications of a few carefully selected physical processes. Most important, NONE of Hansen’s predictions have proven true in even the short 10-20 real-world cases, and he can offer NO reason why his predictions are such failures – but he demands we extrapolate using HIS approximations not in 5-20 years, but over 200, 400, and 800 years.
A very reliable rule, when reading scientific papers, is that if the title is a question, the answer is no. When reading ~snip~ science blogs, that rule holds even more strongly.
Dr Sanchez
Dr of pyscology ? peut etre. Surely not physics!! or economics
So monthly temperature data is reproducible by taking a normal distribution round 0 sigma .5 say and picking a random figure from it and calling it a monthly anomaly.
Do this for a big enough dataset and stretches would compare (within alarmist error bars) favourably with the existing records?
This article and Tonyb’s response are very helpful to me as I was pursuing similar lines — but felt someone must have already done it. Now, I can head in a more productive direction based on this information.
Tonyb, at this moment I tend to lean in the direction you indicate:
“…deriving anomalies from fairly meaningless information will come up with a fairly meaningless and suspect ‘global’ temperature which in itself is a strange concept.”
Nonetheless, I would like to prove (or disprove) this to myself in a defensible way. I am in the “baby step” mode right now. Again, thanks to the authors for saving me some time.
I’ve always had a problem with the anomaly approach to global temperature since it is a second order differencing scheme that introduces noise due to measurement error and lack of resolution. What if you applied this analysis to a true global average temperature, i.e., for each day add up all the temperatures and simply divide by the number of stations?
I am not sure that I follow your argument.
In deciding if the temperature record represents a random walk would it not be better to give the Hurst Exponent of the raw data to begin with. I expect it will show long term persistence. Can you give us this value?
Your seasonal differencing must act as a high pass filter removing the long term trend and its associated persistence. I can not see any need for this step, the HadCRUT3 data has had its seasonal variations removed when the climatology was subtracted from the actual temperatures. Given that there is no need to remove monthly details, choosing to make the subtraction X(i)-X(i-12) seems arbitary and just defines the pass band of the filter.
I think that one of us must have got something wrong.
Alexander
I remember the January 2009 temp anomalies showing a rather large positive spike. Everyone then spoke about the return of AGW.
No doubt there is a major random component and each month’s data is rather irrelevant – however much partisan observers emphasise deviations which favour their point of view and ignore others! But it passes the time while the serious trends build up 🙂
However Basil’s posting seems to be shading dangerously close to trying to suggest that there is no underlying linear trend. Maybe that wasn’t the intention but some people seem to be taking it that way. The choice of the differential series to analyse certainly masks the underlying trend, since the eye (and possibly the random walk analysis?) can’t easily see whether the sum of all positives is greater than the sum of all negatives. I don’t know the details but I suspect the Hurst analysis only looks at short-term sample-to-sample differences rather than long term integrations.
To me the most likely analysis of temperature data is the sum of:
a) A more-or-less linear trend (C02, LIA rebound?)
b) Several long-term cycles (PDO, solar)
c) A whole pile of random noise
For me, the only real question is whether the “more-or-less” is indeed more than linear (positive feedback) or less than linear (negative feedback). I think we’re just going to have to wait to find out…
I have been interested in the fact that whatever happens to the worlds ‘temperatures’ there seems to be almost no good correlation with any theory…after the theory is gone over with a fine tooth comb, anyway.
Despite vociferous arguements on either side strongly trying to prove that their sides evidence is overwhelming, the resulting cause and effect of such influences is very weak regarding the actual climate response. Solar cycles seem to have a weak influence, as does CO2. All related ocean oscillations seem to have modest, but temporary effects both cooling and warming. Particulate pollution seems to both warm and cool the atmosphere depending on the type and action. There are conflicting actions of cloud type, cloud cover and relative humidity. Large dust storms and volcanic eruptions are ‘random’ and apply temporary forcings. Ice cover/extent is variable and apparently unpredictable. Cosmic rays have weak and variable influences, Etc.
In the end, from my ‘stand back from the trees and look at the forest’ point of view, it seems as though the strong arguments are actually weak and the strong forcing theories are actually weak as well. It seems as though the random walk theory is about as good as any existing theory at this point in time.
Looking at past temperature cycles (as good as any proxy can be anyway) long term and short term temperature fluctuations are as normal as can be and that todays temperature cycles can easily be fit into ancient cycles absolutes and periods. It makes little sense to me that anyone can say with a straight face that they understand climate/climate change or that they can predict changes or modify the climate natural progress.
There probably will be some climate model that eventually makes some reasonably accurate predictions, but that day is obviously a long ways off. I was attending many climate modelling presentations back in the mid-70’s where attempts were being made to predict future weather/climate. Clearly, after 30+ years the difficulty in accurate modelling is as onerous as ever despite the incredible advances made to date.
This is one field where natural effects are slow and minimal, man’s effects are slow and minimal and predictions are consequently heavily inaccurate over such small time frames.
Patience here is an obvious virtue, here.
Jim, too.
On reading a bit more about Hurst exponents (but still not really understanding them!) it seems they can detect long-term drift. But like “bluegrue” above I’m worried that the differentiating step turns the linear trend into a fixed offset which the Hurst process will probably then ignore (what’s the betting the first thing it does is subtract out the series mean?).
Lots of interesting comments to respond to.
First, a thanks to
Stef (03:13:17) :
for pointing out that one cannot casually respond “you said this here, and you said this there” to posts up here on WUWT. Not everything posted here comes from Anthony directly. And he frequently puts up posts from people with differing perspectives.
That out of the way.
Craig Allen (22:23:39) :
Hang on minute! I regularly read here the assertion that due to negative feedbacks in the climate system, increasing greenhouse gasses could not possibly have a significant affect on global temperature. Now you’re asserting that it is a random walk. Seems to me that it’s your sequence of climate theories that most resembles a random walk.
Did you read it all? I suggested that while the aggregate, or raw, data, looks like a random walk, when we decompose it into a “signal” and residual “noise,” the “signal” exhibits persistence, and the “noise” exhibits anti-persistence. If you want to find your GHG effect, look for it in the “signal” (not that it is necessarily there, just that that is where you’d find it in the framework of this analysis). And what Stef wrote applies too.
Leif Svalgaard (22:27:11) :
It works the other way around: the find the cycles, you detrend the raw data and do the Chree analysis on the residuals [i.e. raw – trend].
Except that we do not need Chree analysis to identify the cycles. That’s already been done by differencing and smoothing the differences. Why should it matter how we’ve identified the cycles (and just to be clear, we — Anthony and I in our paper together — demonstrated that the same cycles can be derived using wavelet transforms, and MTM spectrum analysis)? Having identified some cycles, Figure 4 is what it is. If you want to chalk the apparent correspondence of the peaks in the cycles to solar cycle and lunar nodal cycle peaks as just fortuitous circumstance, what more can I say?
Arden Halliday (23:12:30) :
Hey, thanks for your work!
I’m trying to explain this to my son – can you help me? I’m wondering why you use the transformed HadCRUT3 data – why don’t you just apply the Hurst exponent to the normal HadCRUT3 time series? Does that make a random walk too?
A Hurst exponent applied to the untransformed data will undoubtedly be high. But that does not rule out a random walk. The high Hurst exponent in the untransformed series is an artifact of serial correlation. The conventional approach to determining whether or not a random walk exists is to difference the data, as we’ve done here.
John Tofflemire (00:18:00) :
Basil,
I don’t understand the reason for this analysis. One can easily have a random walk component with a drift component that would be non-stationary in the untransformed data and stationary in the differenced data. Whether or not the global temperature anomaly time series has a random walk component is not particularly interesting.
What is interesting is whether the underlying stochastic (random) process that produces a given time series (such as the global temperature anomaly) is invariant with respect to time. To understand whether this is the case, one differences the time series, that is, one subtracts the value in a given period from that of a chosen earlier period. If the resulting transformed time series is white noise (that is, random) then we may say that the original time series is stationary.
Well, in the end, I did what you are saying, but didn’t come up with the result you might have been looking for.
Yes, one can have a random walk with drift. But that (as normally done) implies an underlying linear process. I do not think that applies here. We do not have (linear) drift, we have a stochastic process varying around a series of oscillations (cycles) in the underlying system. It is of interest whether or not those oscillations have a physical basis or explanation. We (Anthony and I) do.
As to the character of the stochastic process, a Hurst exponent of 0.383 suggests pink noise, if not white.
You also wrote:
That being the case, you did not explain well why differencing a given month’s anomaly with a 12-month moving average of the anomaly makes sense for the global temperature anomaly.
I didn’t difference a 12 month moving average. No moving average was applied before differencing.
TonyB (01:22:51) :
This is a very interesting and thought provoking article and Basil deserves our thanks for putting it together. So is global temperature rising sharply due to mans activities? It depends greatly on start point and methodolgy. The following comes from a longer article I wrote recently;
Do we accept that Hadleys start point of 1850 (the LIA) in which 20 poorly distributed stations worldwide is meaningful (or as meaningful as a ‘global’ temperature gets.)
I, too, question inclusion of data prior to 1880. You can see some obvious difference in Figure 3. That said, it would not change anything in this analysis to have restricted the data to 1880 and on (except to give some who are looking for anything to criticize something to harp about. Note, that doesn’t impugn all critics. Some critics, perhaps most here, if not all, are honorable in their intentions).
Spence_UK (01:46:17) :
4. The Hurst exponent for global mean temperature has been estimated to be in the region 0.9 to 0.95, and this number is consistent across all temperature series (HadCRU, Satellite) as well as historical proxies (e.g. Greenland and Antarctic ice core measures of temperature).
But that number isn’t all that meaningful because of the autoregressive nature of the time series. Yes, there is persistence on that order of magnitude in the raw data, but as a correlation against time, some of it is spurious. By differencing the data, we control for serial correlation. Now we still see persistence, but we also see now where it comes from — the oscillating blue “trend” line in Figure 3. Note, besides the persistence that comes from the cycles, there may be a small underlying trend. I’ll deal with that in a follow up response specifically on that.
Dermot O’Logical (01:35:19) :
Not being any kind of statistician, nor any kind of climatologist, but my reaction is to be sceptical of this theory.
A random walk has the feature that future course changes are independent of its history. Like a roulette table.
Physical systems just don’t work that way. They have a state, and they respond to physical laws.
Please read the whole thing, or if you have, read more carefully. Your “state” is in the bottom pane of Figure 3. It is not random. It is mean reverting.
Thanks to all who are commenting, whether you agree or not.
RA COOK replied to my comment
“TonyB (01:22:51) :
This is a very interesting and thought provoking article and Basil deserves our thanks for putting it together. So is global temperature rising sharply due to mans activities? It depends greatly on start point and methodolgy. The following comes from a longer article I wrote recently;”
Yes, I am absolutely aware of climate history since the lasrt ice age and that temperature has recopvereed naturally from the depths of the LIA. This is all in another part of the article but I did not want to divert peopple away from Basil’s article so curtyailed my own at the poiunt you see.
Personally I think we place far to much credence on many of the foundations of Agw w
Spence-UK .
.
Yes .
Nobody should write about the Hurst coefficient without reading D.Koutsoyianis papers first .
The purpose of the Hurst analysis is to identify (or better said to TRY to identify) power laws in probability density distributions .
If H = 0.5 then there is no power law and the process is iid (Gaussian , normal , white noise) .
Random walk is gaussian but the climatic parameters are clearly not gaussian therefore they can’t be random walk .
.
If H is not 0.5 then the process may be represented by a fractional gaussian noise (or a power law) .
May .
Or must not .
In any case it is not gaussian .
From the methodological point of view you are of course right to say that it is a heresy to apply the Hurst analysis to “smoothed” data or to moving averages because the Hurst analysis PRECISELY looks at autocorrelations at DIFFERENT scales from the smallest to the biggest .
By smoothing first , the small scales are destroyed and the computed “Hurst coefficient” looses any significance .
.
There is a third possibility too .
Dan Hughes applied the Hurst analysis to a known low dimensional chaotic process (Lorenz system) .
The Hurst analysis failed because the computed “Hurst coefficient” depended on scale what it should not .
However it stayed constant over large intervals of scales .
.
That allowed 2 conclusions :
– Deterministic chaos is not random (neither gaussian nor not gaussian) but that is something that people familiar with chaos theory already know .
– If you have not a huge amount of data (e.g a very long period of time) you might fall on a scale interval where there SEEMS to be a constant Hurst coefficient and therefore you’d think that you see a random process (gaussian or not gaussian) while in reality the system is chaotic . You only had bad luck that the Hurst coefficient didn’t vary at scales at which you looked .
Typically this applies to climate . Once we have some 10 000 years of daily temperature data , then we may be able to draw some conclusions from a Hurst analysis of this time series .
So only some 9900 years to wait 🙂
Sanchez, Politics aside, WUWT is website where open minds come to play… As a “Dr” of something or other, how is it that you don’t get that? Oh, wait…
In other news Athropogenic Continental Drift (h/t to Kate at SDA LOL!
Sorry my above comment ‘escaped’ before I finished and edited it!
As well as the nonsense of parsing historic Global temperatures to fractions of a degree as if they had been compiled in laboratory conditions, the manner in which ‘sea level rise since 1700’ has been put together also warrants an article in itself.
Tonyb
All I ever needed to learn about the climate I learned from my uncle Albert, the farmer. His words to describe the excursions of the temperature were: “one extreme always follows another”. So, when the summers were really hot he cut a lot of firewood.
For anyone interested in understanding the math & principles involved, an excellent related reference is: “The Misbehavior of Markets,” by Benoit Mandelbrot.
While the subject matter, stock market trends & crowd behavior, is very different than the focus of the analysis here, the underlying principles are basically the same. Such parallels between disciplines are very common…and reviewing related material in a diffent contexts is usually very helpful in increasing one’s understanding & appreciation of the basic analytical subject matter. But [more importantly] such breadth in exposure also helps one in both applying such tools, and [even more importantly still] is spotting the underlying patterns & opportunities for extracting new insights.
Besides, anyone enjoying the above “random walk” analysis will likely enjoy Mandelbrot’s book regardless.
The “random walk” jargon, by the way, hails to a classic book by Burton Malkiel, “A Random Walk Down Wall Street” 1st published in 1973.
I notice that the Danish Center for Snow and Ice shows the arctic temps dropping below freezing about two weeks before normal, and remarkably Arctic ROOS shows the ice area currently increasing (about two weeks before normal.) All the while UAH shows atmospheric temps substantially up. This just gets curiouser and curiouser.
It appears that our understanding of the earth’s climate and what drives it is still pretty primitive. One thing is obvious, though, the inexorable linear rise in Co2 that we are seeing doesn’t seem to correlate with ANYTHING (except perhaps increased agricultural yields.)
Sorry, Basil, but this must be grumpy day for me. Global temperatures — as we know them today — are not random, but crooked, fixed, cooked. Mosly lies. Until we locate those “thermometers” sited appropriately and read over a long period of time, we will never know. (See E.M. Smith and part of what I imagine Anthony’ssurface station projec will address.) The climate system may be chaotic, but the climate has changed over geological time — which involves the temperaure of land and ocean — and this truth is not random. I’m interested in the truths, not the lies of statistics from altered raw data.
More grumpy — the link to Orlin Grabbe on “Chaos and Fractals in Financial Markets”, you know, the one who is “an internationally recognized derivatives expert”. These are the guys who, in the guise of helping us to minimize risk, hide the fact that they have been dumping empy-of-any-value “equities” on unsuspecting (trusting) investors worldwide leading us to the disaster we are facing today. How about some substance, some truth in financial instruments, just like thermal energy instruments.
Until truth is determined in the HadCrut3 data ( global temperature anomalies), anything you do with it must fall under the GIGO principle. Good grief and bad grief! Physicists on an earlier thread cannot even agree on the definition of heat — a noun or a verb? thermal energy of a “substance” or transfer of energy? Help
Talking of obsessively following every new monthly value… About a month ago, when we’re were talking about surface-satellite ‘lag’ in
http://wattsupwiththat.com/2009/07/14/giss-for-june-way-out-there/
I said, “…I’m tempted to make a SWAG of UAH=+0.3 by September, and a crossover with GISS (adjusted for baselines) by the end of the year.”.
GISS for July is out at 0.6. Baseline-adjusted comparisons here:
http://www.woodfortrees.org/plot/wti/last:12/plot/hadcrut3vgl/last:12/offset:-0.15/plot/gistemp/last:12/offset:-0.24/plot/uah/last:12/plot/rss/last:12
OK, it’s a bit earlier than expected, but mine’s a pint of Doom Bar 🙂
This is Hansen’s very first climate projection from 1981.
Overall, includes some description of the random walks of the climate, temperature impact per doubling lines lower than later versions (basic math must have changed in the interim).
http://img23.imageshack.us/img23/7720/image002x.png
Quite often people confuse “random” with “arbitrary” , which results in unfortunate conclusions. When prior information is neglected ( Bayes theorem ) the result may bear little resemblance to reality.
But in this drunkard’s walk the drunkard gets further away from the lamp-post as the square-root of time as Leif has pointed out. Take a gaussian process (brownian motion, random walk, drunkard’s walk, whatever you wish to call it) and add an integrator to it. The integrator is important because other wise you’ll just bob around a long-term mean. The integrator produces something like 1/f noise. If the integrator as an infinite time constant your process will possess such a long memory about past events that you will never even be able to calculate a mean value–the central limit theorem does not apply to such a process and a mean value becomes meaningless.
The Earth probably does not have an integrator with infinite time constant but, there may be very long time constants, which means that no matter how long the time series one examines, there will always appear to be a linear trend.
By the way, the person who first pointed this out for geophysical processes was Benoit Mandelbrot, of fractals fame, back in 1968. He called it the Joseph Effect (seven fat years, etc). Mandelbrot looked at river discharge data, but the lesson has been there all along for people to apply to weather/climate. I have tried to point out on many occasions that negative feedback with sufficiently long characteristic time will make time series indistinguishable from the temprature series for any time duration one cares to examine, but the concept just doesn’t seem to click with anyone except my engineering students.
I’ve been interested in this notion for some time now. I’m no statistician, but I diddled around with Excel’s random number function to produce graphics that look amazingly like the average global temperature charts we’re all used to seeing.
Here’s a link to a web discussion group that I’ve posted some of this stuff on:
http://wc5.worldcrossing.com/webx?14@@.1de4fb6e/313
Some of the results I got with Excel:
http://i40.tinypic.com/wwb2hj.jpg
http://i41.tinypic.com/qq92ti.gif
http://i44.tinypic.com/11gj8t1.gif
Does it prove anything? I’m smart enough to know that trying to predict which way it’s all going to go is a fool’s errand. 100 years out? Ha ha ha ha ha ha ha ha ha!
But even a drunk wandering around a lamp post is moving around the sun, in a solar system circling the galaxy, in a galaxy moving across the cosmos, just as as the random walk of a snow flake takes place in the larger context of a moving storm system. What is the larger system that contains the temperature random walk? Just asking.
I’ve been interested in this notion for some time now. I’m no statistician, but I diddled around with Excel to create with random numbers graphics that look amazingly like the average global temperature charts we’re all used to seeing.
Here’s a link to a web discussion group that I’ve posted some of this stuff on:
http://wc5.worldcrossing.com/webx?14@@.1de4fb6e/313
Some of the results I got with Excel:
http://i40.tinypic.com/wwb2hj.jpg
http://i41.tinypic.com/qq92ti.gif
http://i44.tinypic.com/11gj8t1.gif
Does it prove anything? I smart enough to know that trying to predict which way it’s all going to go is a fool’s errand.
WoodforTrees
Looking at your chart I would say that someone has incorrectly calculated an algorithm, or adjusted in one go figures that were too ‘low’ in previous months.
Do you know of anyone that looks at the individual monthly figures used for global stations? If not separately perhaps as the total for the NHemisphere and the SH. THis is most useful as an actual temperature-not an anomaly. It would be interesting to see where this additional warmth is coming from. The fact that the US went trhe other way (I know it is just 2% of Earths land mass) suggests this is not global but regional.
Tonyb
Basil (05:44:03) :
If you want to chalk the apparent correspondence of the peaks in the cycles to solar cycle and lunar nodal cycle peaks as just fortuitous circumstance, what more can I say?
You can show a version of Figure 5 extended backwards to 1850 and to 1600 [the latter clearly without the observed data, but with the calculated values].
And, perhaps, comment on the fact that the average distance after N steps is not zero but SQRT(N), so no strong probability of reverting to the mean.
Spence_UK (01:46:17) :
I want to follow up here on the issue of computing a Hurst exponent from the raw monthly temperature analyses. The calculation is simple enough. For the HadCRUT data I am using, it is 0.967, close to the numbers you reference. Now this would ordinarily imply a very “strong trend” in the data. But before we can conclude that, we have to consider the impact of serial correlation. Here are two simple and straight forward estimates of the order of linear trend in the data:
(I’ll try to get this for format as “code” but without a preview function, I have no way of knowing it it will work.)
OLS estimates using the 1914 observations 1850:01-2009:06
Dependent variable: hc_g
HAC standard errors, bandwidth 9 (Bartlett kernel)
coefficient std. error t-ratio p-value
-----------------------------------------------------------
const -0.522158 0.0230692 -22.63 1.08E-100 ***
time 0.000367397 2.24001E-05 16.40 1.15E-056 ***
OLS estimates using the 1913 observations 1850:02-2009:06
Dependent variable: hc_g
HAC standard errors, bandwidth 9 (Bartlett kernel)
coefficient std. error t-ratio p-value
-----------------------------------------------------------
const -0.125384 0.0122823 -10.21 7.35E-024 ***
time 8.86101E-05 9.16300E-06 9.670 1.25E-021 ***
hc_g_1 0.759139 0.0235876 32.18 6.12E-182 ***
We’re interested here in the coefficients for the “time” variable. Since these are monthly data, we can multiply by 120 to derive an equivalent “decadal” trend rate. In the first case, which is the regression for the trend line represented in my original Figure 1, the decadal trend rate is 0.044°C. But there is a high degree of serial correlation (Durbin-Watson statistics = 0.48). In the second case, we control for serial correlation by adding a one month lag of the temperature as an explanatory variable (Durbin-Watson is now 2.40). The resulting trend estimate is now slashed by about 75%, to a decadal trend rate of 0.011°C.
So a high Hurst exponent, by itself, isn’t enough to indicate that we’ve accurately captured the real trend in the data.
And if anyone is curious, a similar trend analysis of the wavy blue line in Figure 3 yields this:
OLS estimates using the 1890 observations 1852:01-2009:06
Dependent variable: hpt_sd_hc_g
HAC standard errors, bandwidth 9 (Bartlett kernel)
coefficient std. error t-ratio p-value
----------------------------------------------------------------
const -0.00228909 0.00187566 -1.220 0.2225
time 2.83920E-06 1.68064E-06 1.689 0.0913 *
hpt_sd_hc__12 0.808243 0.0391915 20.62 2.21E-085 ***
Now, be careful, because here the annual trend is reflected in the constant term. It starts off negative, but has been becoming less negative over time, and at the end of the data the decadal equivalent is 0.031°C.
For those of you looking for a “drift” in the data over time that might be capturing AGW, it will be in the upward trend in the wavy blue line. Were this order of “drift” to continue, the decadal rate will have increased to about 0.065°C in another hundred years. That works out to about 0.43°C (interpolating) increase over the next century. If that’s all there is to the AGW impact, I think we can hold off on cap and trade for a while.
While each may play an independent role in modulating global temperatures, since the 1920’s the solar and lunar influences appear to have been roughly in phase to strongly influence temperature trends on a bidecadal time frame. In other words, Figure 4 may be revealing the physical processes at work in explaining the persistence revealed by the Hurst exponent for the blue line in Figure 3.
Interesting statistical analysis!
But if you want to pursue publication you need to add [……..] “but these physical processes cannot explain global temperature increase in the industrial era.
The July MSU temp jump of .42c seems to mostly result from the readings in Antarctica, where they show a really big jump of 3.11c
The historical series for the South Pole land is volatile and it has spikes often, the 3.11 though is high and while there was a 3.3c reading in May, 2002, the 3.11 appears to be the second highest in the 1979 to present series
I would bet that August will show a much lower temp
woodfortrees (Paul Clark) (05:16:39) :
No doubt there is a major random component and each month’s data is rather irrelevant – however much partisan observers emphasise deviations which favour their point of view and ignore others! But it passes the time while the serious trends build up 🙂
However Basil’s posting seems to be shading dangerously close to trying to suggest that there is no underlying linear trend. Maybe that wasn’t the intention but some people seem to be taking it that way.
Paul,
While you were posting this, I was writing up my preceding reply in which I tried, somewhat unsuccessfully, to post up some formatted statistics relevant to your concern. You are right in thinking that it is not my intention of saying there is NO trend. But where I would look for it is in the blue wavy line of Figure 3. And it is there, though the significance level is marginal (90%), and is smaller than we get with a linear fit through the raw data.
Again, everyone, please, please, understand that what I’m seeing, and saying, is not that there is no trend, or that temperature is entirely a random walk. What I’m am saying is that I think the monthly anomalies “look” like a random walk, in part because of the high volatility of the stochastic component, and because the two components of the series have characteristics that seem to be offsetting, i.e. the true underlying “trend” (the wavy blue line in Figure 3) shows persistence, while the “random” or “stochastic” part (bottom pane of Figure 3) shows anti-persistence (mean reversion). With mean reversion working against the trend, we get something that looks like a random walk. But that is not to say that there are no physical processes at play. In the trending part, Anthony and I see a lunisolar influence in the oscillations, and there is an underlying “drift” that may, or may not, be due to AGW (and I say AGW, and not GHG, to allow for such impacts as UHI). In the random or stochastic part, we’re seeing the characteristic of a stable physical system (which climate has to be for life as we know it to exist) which responds to shocks by reverting back toward the mean.
Can I be any clearer?
Basil,
Thanks for an interesting post; I had not seen this type of analysis before.
1) It seems to me that by using the 1-year differences to transform the raw data (Figure 2) that you are taking something of a “first derivative” of the data, which ought to magnify the importance of short term variations, while reducing any longer term variation in the original data to small differences in the trend above or below the baseline. Since the overall trend for the raw data set is clearly positive, the average value for all the transformed data should be slightly above zero, with a value that depends on the slope of the least squares fit trend in the original data divided by the number of data points in the raw data set less 1. Do I understand the transform you have done correctly? Would a series of similar transforms with different time steps (2 yrs, 3 yrs, 4 yrs, etc.) not increasingly show the longer term variations in the raw data, while “filtering out” the short term variation? What is the reason for choosing 1 year changes as opposed to some other period?
2. If I understand correctly, you have assigned cause for the recent temperature history (the 20th century to now) to the “lunisolar” influence you described, and based on this, you project variation in average temperature around a flat trend for the next 20+ years (as shown in Figure 5). If this is true, then it seems to me that you are implicitly assigning a value of near zero for climate sensitivity to radiative forcing. Fair enough, it could be very low. But in this case, how can the climate have any measurable sensitivity to variation in TSI over the solar cycles? I think that Leif suggests an average solar cycle signature of about 0.075C in the historical data due to variation in TSI of about 1.4 watt/M^2 at the top of the atmosphere. This is consistent with a relatively low climate sensitivity, but not a near-zero sensitivity. If the sensitivity to radiative forcing is in fact near zero, then by what mechanism do you think the solar cycle shows up in the temperature data?
Well, what you would have here is a random walk for the “velocity” (i.e. the trend) around its average value. The fact that the Hurt coefficient is so close to 0.5 indicates a Gaussian distribution probability for it which, for large systems, is quite expected because of the central limit theorem. What do you think?
And I might add, with reference to all the concern about whether there is a trend, that there would be a trend component to the sinusoidal model that underlies Figure 5. But it may, for the next two decades, be “masked” by the behavior of the sinusoidal components of the model, leaving the anomaly series like we see it in Figure 5.
But do not read too much into this. Any extrapolation out for two decades, let alone for the next century (a la IPCC) involves what we call in my field “heroic assumptions.” And I’m quite well aware of that. What is useful in such analyses are exposing the assumptions that underlie them, so we can evaluate their plausibility.
E.M.Smith (02:46:34) :
Your analysis should be another talking point… a thread on it’s own merit. I would be interested to read a full discussion including “AGW convinced” rebuttals, on your thesis. You have given us much food for thought. Thx.
I’ve promoted the random walk model of the temperature for quite some time, e.g.
http://motls.blogspot.com/2009/01/record-breaking-years-in-autocorrelated.html
http://motls.blogspot.com/2009/01/weather-and-climate-noise-and.html
so you shouldn’t expect any devastating criticism from me.
Of course, there’s a lot of influences that are (at least qualitatively) well-understood, like the influence of the orbital variations, solar activity, greenhouse gas increases, or the fluctuations of the cosmic rays.
There are also many approximate predictions that are possible because of various lags – such as the delayed influence of El Ninos.
Still, the bulk of the global mean temperature change may be a completely Brownian motion, up to variations of order +-10 deg C where the regulating mechanisms have to become important, like in an AR(1) process, because the temperatures have been stabilized in a 20-deg-C window for milions of years.
But below this +-10 deg C, i.e. below tens of thousands of years, the motion can be completely random. The scaling laws seem to work well. The typical temperature jump after Y centuries is sqrt(Y) degrees Celsius, or one half of it.
For 1 century, the temperature jumps by 1 deg C (or 0.5). After 4 centuries, it’s 2 deg C (or 1). After 9 centuries, it’s 3 deg C (or 1.5). After 100 centuries i.e. 10,000 years, it’s 10 deg C (or 5). This sqrt-scaling law follows from random walk and works remarkably well. These critical exponents contain a lot of actual knowledge about the climate, and one simply shouldn’t deny them just because he would like the world to be predictable. It doesn’t seem to be!
Since you mention the Hurst Exponent:
http://www.itia.ntua.gr/en/docinfo/849/
However I would also note that the supposition that one can use HadCrut for analysis is dangerous, given all the evidence that the trends have biases.
Basil,
Thanks for the explanation – I had thought people were reading more into what you were saying than you intended. It feels to me that what you say in your latest comment (08:15) about the sinusoidal components masking the overall trend is the closest we’ve come to a sane model here for a while. Your search for the causes of the cyclical components is fascinating and valuable – just as long as everyone realises that when they turn positive we will return to – and for a while exceed – the long term trend.
That is, of course, unless there is a very large, very long (100+ years) cycle we’re not aware of yet – but as Steve Fitzpatrick says above, if you attribute *everything* to cycles you have rather thrown the baby out with the bathwater – that is, unless you tear up the basic CO2/radiation physics.
TonyB: The only adjustment done on that graph is to bring everything to the same baseline – see http://www.woodfortrees.org/notes#baselines for details. To believe that GISS, UAH and RSS had all suddenly felt the need to adjust their data upwards in June/July is stretching the conspiracy theory a bit, I think 🙂
Actually, that graph rather illustrates Basil’s discussion here – it makes it very obvious that the random component of the temperature series is a lot bigger than any trends or cycles, at least in the short term. That’s why it’s rather pointless (but fun) to speculate about each new month’s value – although, as John Finn points out above, it does give some interesting pointers to some short-term heat distribution mechanisms.
Darn. The phrase I used earlier
“negative feedback with sufficiently long characteristic time will make time series indistinguishable from the temprature series for any time duration one cares to examine”
should read
“integrated negative feedback with sufficiently long characteristic time will make time series indistinguishable from the temperature series for any time duration one cares to examine”
The integrator is absolutely vital. What provides the integrator? The ocean, soil, albedo as determined with ice, snow, and plants, and so forth all provide some memory of distant past climate and are therefore “integrators”. The characteristic times are not known to me, but some processes involving albedo might be thousands of years.
By the way, I am new to this site, and I should thank everyone here, Basil in this case, who takes time to post these interesting guest papers, those who add erudite commentary, and the moderator who keeps it pretty civil. I really enjoy this site.
All,
More good questions and comments continue to come in. I will not be able to respond in any detail until later this evening (Central Daylight Time in the US).
Basil
Basil (08:15:04) :
Any extrapolation out for two decades, let alone for the next century (a la IPCC) involves what we call in my field “heroic assumptions.”
Except that the lunar influence can be calculated accurately for thousands of years and the solar influence is well-known for centuries and reasonably well-known also for thousands of years, so your input to the ‘signal’ is well-known, there your hindcast is not an extrapolation, but an application of known inputs.
Basil (07:39:33),
Many thanks for your courteous reply.
The Hurst phenomenon is a form of serial correlation – so when applying Hurst type analysis, there is no need to account for the serial correlation separately. It is a separate and distinct form of serial correlation to Markovian models. They are commonly referenced as “Short Term Persistence”, or STP for short, for Markovian dependency, or “Long Term Persistence”, or LTP, for Hurst dependency.
As the two are simply different forms of serial correlation, either method can be used to account for the serial correlation. The method chosen is the method applicable to the time series of interest.
Deciding which type of serial correlation should be used is a difficult question. Since STP with very long time constants look very much like LTP series, statistics alone cannot determine which model is more suitable, either requiring enormous lengths of data or a full understanding of the physics (as noted in Tom Vonk’s very good comment), neither of which are readily available. Prof. Koutsoyiannis provides example of simple chaotic systems which exhibit LTP; this does not prove climate behaves in this way, but it does show that LTP can emerge from simple chaotic systems. (Of course, not all chaotic systems exhibit LTP, so we must keep an open mind).
Tools and estimators designed for STP time series do not always work when applied to LTP time series. Tools for LTP analysis are separate and commonly misunderstood. As suggested, I would strongly recommend reading some of Demetris Koutsoyiannis’ body of work, which does a very good job of explaining the concepts (and common misunderstandings) about LTP series.
You say that a Hurst exponent of 0.967 is a strong indicator of a trend. This is incorrect. A high Hurst exponent shows strong scaling behaviour, i.e. large low-frequency oscillations strongly visible in the time series. However, in the presence of LTP this is a stationary phenomenon and should not be confused as a deterministic trend that needs to be accounted for.
To me, the LTP/STP issue for climate helps to underline how little we know about climate, and how a simple, justifiable change of assumptions can render the 20th century warming statistically insignificant (e.g. Cohn and Lins “Naturally Trendy”)
Basil,
This article starts well but then goes seriously downhill when you start smoothing the data before applying your tests. You must never do this or your results are meaningless. Here’s a quote from the blog of statistician William Briggs:
“Now I’m going to tell you the great truth of time series analysis. Ready? Unless the data is measured with error, you never, ever, for no reason, under no threat, SMOOTH the series! And if for some bizarre reason you do smooth it, you absolutely on pain of death do NOT use the smoothed series as input for other analyses! “
I would say, instead, a levogyre walk.
Basil,
Are you aware of ARIMA processes? They’re a generalisation of random walks that have the same spurious trend properties but avoid the difficulties of unbounded wandering away from the mean. (Strictly speaking, a random walk has no mean.) The similarity has been noticed before.
Climate Audit discussed them at length, a long way back.
(http://www.climateaudit.org/?p=300 and others.)
Sorry, here’s a better one.
http://www.climateaudit.org/?p=332
I prefer to think of it as more like 1/f noise; the longer you wait, the bigger the transient events will be.
In any case, if one of my students fitted that straight line to that data he would fail my course. There’s no a priori reason that the function should be a straight line.
My eye sees some 30 year ramps; up and down, with some longer term up; but wait long enough and it will coem down again.
Stef wrote: “Dr Sanchez, the articles you gave links for are all from different sources, and this latest article is a guest post by Basil Copeland. It might surprise you that different people actually have different opinions. WUWT publishes articles and information from many sources. You do realise that not every single article is penned by Mr. Watts don’t you?”
Yes, I do realize they all come from different sources. And I also realize that journalists around the world are using this website as a source to spread misinformation about the theory of global warming. And I also do realize that an editor of such a website is supposed to have basic knowledge of the theory, and not publish whatever information is convenient at the time because it agrees with the position they are trying to push forward, regardless of how much it contradicts previous information they gave. There is a concept called “responsible journalism” that Mr Watts ought to read up upon. Until then, all credibility from this website is thrown out the window.
As George E. Smith notes, you could still be looking at some form of colored noise in climate. It is a common misconception to equate “randomness” with gaussian, or white, noise (i.e. the system is completely unpredictable and each time step is governed by nothing more than a coin-toss).
Instead of constructing a difference time series of January to January, February to February and so on (I’m not sure exactly what that tells you), why not look at the month-to-month difference time series which would be a more standard analysis?
If the difference (or “jump”) time series is really gaussian, then a frequency spectrum of that difference time series should exhibit zero slope in log-log space (that is, the “magnitude” of the jump at any frequency should be independent of the frequency).
What you tend to find is that most natural phenomena exhibit some form of fractal or power-law scaling (at least over a certain range of scales) and it can be easily identified in their frequency spectrum. For example, take Earthquakes- you tend to get a lot of small events and very few large events – the resulting power-spectrum exhibits a positive straight-line slope in a log-log plot of Magnitude v.s. Frequency – this is the famous “Gutenberg Richter” relation where the slope indicates something about the scaling and recurrence of Earthquakes in that part of the Earth. Take something closer to home – rainfall – you tend to get a lot of small events, and few large events (floods) – that is, power-law scaling. What this type of scaling implies is the system as a whole is far from being Gaussian (and therefore unpredictable), but exhibits structure and is instead, COMPLEX (in the chaos and complexity sense – see http://en.wikipedia.org/wiki/Complex_system), where it may offer some predictability over certain time-scales (e.g. a weather forecast).
A while back (here http://wattsupwiththat.com/2009/03/16/synchronized-chaos-and-climate-change/) Anthony posted a very interesting article which proposed that the climate system might act like synchronized chaos. If that is true (and I think it is), you should see power-law scaling in the frequency spectrum and not pure gaussian (white) noise.
I’d therefore be interested to see what a power spectrum of the straight month-to-month difference time series looks like.
Fig. 2 looks like tree ring data to me. Only tree ring data has long meaas and grabens in it’s semi-random walk.
Those hockey sticks to the moon are unreal. Rural data sets don’t show hockey sticks.
1/f (flicker noise) is very common in real world “dirty” environments.
No-one knows why it happens: the slower (or lower the frequency) the greater the noise, with no lower frequency limit. (counter intuitive)
This item might interest some.
http://www.dsprelated.com/showarticle/40.php
Wait until it gets to “1/f noise has been observed in the strangest places- electronics, traffic density on freeways, the loudness of classical music, DNA coding, and many others.”
Try some searches.
How about “1/f model for long-time memory of the ocean surface temperature”
http://www.mi.uni-hamburg.de/fileadmin/files/forschung/theomet/docs/pdf/fraelukble04.pdf
Or “1/f (One-Over-F) Noise in Meteorology and Oceanography”
http://www.nslij-genetics.org/wli/1fnoise/1fnoise_meteo.html
As a student of economics who studied complex modeling and its utter futility as a predictive mechanism, I applaud this post!
I’ve always felt that modeling for the climate was always about on par with modeling in financial systems – and if we think Climate Modeling is overfunded just trust me when I say that it’s a drop in the bucket compared to what the Financial World throws into their analysis and a monkey throwing darts at the WSJ still picks a better portfolio than the average mutual fund.
It’s like staring into the sun, our primate brains can’t handle the truth 😉
“Yes, I do realize they all come from different sources. And I also realize that journalists around the world are using this website as a source to spread misinformation about the theory of global warming. And I also do realize that an editor of such a website is supposed to have basic knowledge of the theory, and not publish whatever information is convenient at the time because it agrees with the position they are trying to push forward, regardless of how much it contradicts previous information they gave. There is a concept called “responsible journalism” that Mr Watts ought to read up upon. Until then, all credibility from this website is thrown out the window.”
===
Er. no.
See, Dr Sanchez, it is Mann, Steig, Hansen, and their very, very-well-paid international socialist comrades who are ignoring and riduculing irresponsible results and who are NOT being open about the sources, methods, and even raw data. It is those to who we’ve paid 70 billions in PUBLIC money who are irresponsible and who are displaying an amazing ignorance of basic science. (ie, “Don’t cook the books and manipulate the data and conceal your equations behind lies about security.”
Presenting opposing and exploratory ideas – even those which do NOT agree with the 32,000 real scientists who publically hold skeptical views about global warming.
A privately-supported web blogger is not supposed to be able to find out that your vauted Mann graphs are totally false -> thereby showing your much-publicized IPCC reports for the past 15 years are false. But they (M&M) did that. So who are the “responsible” ones? If a private blogger is actually allowing and encouraging open discussions, who is the “responsible” one – he who promotes discussion and finds real data, or those in academica who conceal it and present lies?
(By the way, when Mann, Hansen, Gore, and their European counterparts present and publicize THEIR lies to manipulate public opinion and public policies threatening the lives of billions, threatening the world with extremist policies that will kill hundreds of millions, what members of the “press” have actually come “here” for their data? How many tens of thousands of lies are being promoted by the AGW ecotheists in their “religion” that are sourced by your missing “press” who NEVER read or quote ANY opposing views?
You are, politely put, also promoting that same propaganda. You are, therefore, also responsible for those deaths. Sleep well with those thoughts.
Re: PaulM (09:58:26)
I can’t let that comment pass. You are propagating the myths that exist about smoothing, which has LEGITIMATE purposes. Some people (including top statisticians I know) have simply not thought about it carefully. I encourage those ‘opposed to smoothing’ to pause for a second to ponder why climatologists have adopted the convention of using anomalies. Spatiotemporal heterogeneity exists. Time-integration is not ‘bad’, but one should assess how parameter estimates vary with scale (as opposed to cherry-picking select scales without providing context for an audience).
George E. Smith (13:53:23) :
“In any case, if one of my students fitted that straight line to that data he would fail my course. There’s no a priori reason that the function should be a straight line.”
===
Yes, there IS a reason for that curve to be straight: Hansen has declared ALL global warming is directly proportional to CO2 levels and CO2 levels have been going up.
Therefore, the “best fit curve” through ANY set of data points IS Hansen’s straight line.
End of discussion. As Dr Sanchez has declared, “The science is settled.”
Basil, thanks for drawing my attention to this:
Carvalho, L.M.V.; Tsonis, A.A.; Jones, C.; Rocha, H.R.; & Polito, P.S. (2007). Anti-persistence in the global temperature anomaly field. Nonlinear Processes in Geophysics 14, 723-733.
http://www.uwm.edu/~aatsonis/npg-14-723-2007.pdf
“[…] significant power exists in the 4-7 years band corresponding to ENSO. Such features, however, are broadband features and do not represent periodic signals; they are the result of nonlinear dynamics (e.g., Eccles and Tziperman, 2004). As such they should not be removed from the records.”
This seems consistent with W.W. Hsieh’s observation that post-1950 NH El Nino response has been nonlinear. This gives cause to review carefully the appropriateness of COWL signal removal (cold oceans – warm land).
I tracked this down:
Eccles, F.; & Tziperman, E. (2004). Nonlinear effects on ENSO’s period. Journal of Atmospheric Science 61, 474-482.
http://www.seas.harvard.edu/climate/eli/reprints/Eccles-Tziperman-2004.pdf
Dr. Sanchez, please note that your views are as welcome here as any. We run a fairly open discussion, here. Compared with pro-CO2/AGW websites, we run a very open shop. I have approved your posts as have other moderators.
Views here are very, very varied. Personally, I am a “lukewarmer”. I think CO2 has had an effect, but seriously doubt the positive feedback assertions, and I question the adjustment procedures of GISS and NOAA. And I deplore the reluctance of many on the severe warming side to disclose data, methods, code, etc. At this point, I tend to be more of a “sea witch” than a “sun worshiper”.
In short, I think there has been some warming, but it has been overestimated and I doubt that 21st century warming will be several times that of the 20th century, as the IPCC estimates.
Many here disagree with this pov (from either direction), but at least we can discuss our views: it takes considerable talent to be repeatedly deleted and even more to be banned entirely.
So feel free to hang out and take part in the debate. Try to be (reasonably) polite and respectful and you are welcome here.
re Ken (06:17:38) :
The “random walk” jargon, by the way, hails to a classic book by Burton Malkiel, “A Random Walk Down Wall Street” 1st published in 1973.
It is an excellent book, but not the originator of the term. I don’t claim to
know the first use, but William Feller’s excellent text of 1950 has a chapter
on the topic.
A few more thoughts:
1. Basil is doing the right thing in differencing the series before applying the test.
2. The Hurst exponent can only be estimated. When I repeat his calculation (using different software) I get slightly lower numbers, about 0.41 for fig 2, rising to about 0.44 if I throw out the earlier data.
3. If I do a month-to-month difference, as some people have suggested, instead of the seasonal difference, I get a much smaller number, around 0.1 – I don’t understand why this is!
4. The figure of 0.835 for the smoothed series is meaningless. It is purely a consequence of the smoothing. The more you smooth, the bigger this number will be. You can generate as many bogus ‘signals’ as you like by smoothing in different ways. It is instructive to do this with data from a random number generator. Leif is right, the analysis should be performed on the raw data. Reading the comments more carefully I see that Spence_uk made this point too.
5. Essential reading about smoothing: http://wmbriggs.com/blog/?p=195
6. realitycheck – the spectrum of the difference is fairly flat. In fact it increases slightly with frequency. What would that indicate?
7. Dr Sanchez, you are so amazingly wrong I hardly know where to start. The point is that there are many possible ‘explanations’ of the temperature record that are discussed, explored and criticised here. We don’t know which is correct. The misinformation spread by most journalists is that there is only one. The duty of a responsible journalist is to question and challenge things.
Flanagan points out, by differentiating first, you are making any ‘linear trend’ into a constant offset
– and then analysing the result to show that we have a ‘random walk’ about a ‘constant offset’ (which is the differential off the linear trend)
– so in effect, you are saying we have a ‘random walk’ about a ‘linear trend’
– so, what is your point, exactly?
Paul Vaughan,
Two notes: firstly, whilst I agree that a blanket rejection of smoothing is wrong, smoothing can have negative impacts on the analysis being performed. In particular, any Hurst-like analysis which is influenced by spectral power at different scales will clearly be affected by smoothing, which implicitly modifies spectral power at different scales. Anyone who uses smoothing prior to Hurst type analyses immediately raises questions of why it is needed, and should provide a clear justification as to how they prevent said smoothing from interfering with results (or how they have calibrated out the interference).
Secondly, that paper (Carvalho et al) presents very strange results. One of their diagrams (the 6-7.5 year Hurst exponent field) looks nothing like that which would be expected, and indeed looks largely like floor-to-ceiling noise from their estimator. As noted in the Koutsoyiannis papers that I link above, the Hurst exponent of available temperature data is consistently high (near, but below, 1) on scales from months up to 10,000 years. Without digging deep into their analysis, I am only guessing as to why this difference exists, but the most common reason is a failure to adequately remove the strong annual cycles in the temperature data (this particularly applies to local temperature measures). Failure to fully remove this will introduce the exactly type of effect they are observing in their paper. I note they do attempt to remove the annual cycle, but I would like to see how effective it is. Even a small residual component would produce a random field very much like that seen in their figure 4(f). This is compounded by their similar decision to analyse the Hurst coefficient over small bandwidths (which makes little sense in the broader context of the Hurst phenomenon). Without seeing their data and code I cannot be sure this is a problem, but it certainly makes my sceptical side think I would not trust this study without some of these questions answered.
Paul Vaughan (19:53:15) :
Basil, thanks for drawing my attention to this:
Carvalho, L.M.V.; Tsonis, A.A.; Jones, C.; Rocha, H.R.; & Polito, P.S. (2007). Anti-persistence in the global temperature anomaly field. Nonlinear Processes in Geophysics 14, 723-733.
http://www.uwm.edu/~aatsonis/npg-14-723-2007.pdf
“[…] significant power exists in the 4-7 years band corresponding to ENSO. Such features, however, are broadband features and do not represent periodic signals; they are the result of nonlinear dynamics (e.g., Eccles and Tziperman, 2004). As such they should not be removed from the records.”
I didn’t catch that. And I’m not sure I agree. Without going back and looking, I’m quite sure the MTM spectrum analysis of the global temperature series has a harmonic at ~4.7 years. Let’s look:
http://wattsupwiththat.files.wordpress.com/2009/05/figure4.png?w=510&h=374
The paper is still good, and harmonics, or cycles, can exist along side mean-reversion, as two different mechanisms at work influencing global temperature. That’s what I’m trying to show in this post.
By the way, I still cannot help but observe that 4.7 years is close to a harmonic (1/4) of the lunar nodal cycle, and about half the length of recent solar cycles, and may be a harmonic of a bidecadal beat cycle combining the two. The cycles we are observing here (in the blue line in Figure 3) are of a magnitude (amplitude) that could be completely attributed to typical variations in TSI. Couple that with the variation of the lunar nodal cycle, and we have an adequate physical basis, it seems to me, for the cyclical part of the time series.
Then, on top of that, we have the anti-persistent response to random effects upon temperature described by Carvalho et al, and demonstrated in the bottom pane of Figure 3. I suspect you’ll find a lot of ENSO in the latter, but that still doesn’t rule out a harmonic in ENSO as well.
Works for me. 😉
Spence_UK (05:54:27) :
Paul Vaughan,
Two notes: firstly, whilst I agree that a blanket rejection of smoothing is wrong, smoothing can have negative impacts on the analysis being performed. In particular, any Hurst-like analysis which is influenced by spectral power at different scales will clearly be affected by smoothing, which implicitly modifies spectral power at different scales. Anyone who uses smoothing prior to Hurst type analyses immediately raises questions of why it is needed, and should provide a clear justification as to how they prevent said smoothing from interfering with results (or how they have calibrated out the interference)..
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations (and noise). We (Anthony and I) demonstrated this in our paper, most effectively (in my view) with the following wavelet image:
http://wattsupwiththat.files.wordpress.com/2009/05/figure3.png?w=402&h=536
Figure 3 is just a different representation of what you see in the image linked above. It speaks for itself, and and the image above shows that the smoothing in question is not “interfering with results.” The Hurst exponent, as applied then to the two components of the time series, simply helps further describe the difference between the two.
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations
Basil, low pass filters which filter out high frequency oscillations is the exact same thing as modifying spectral power at different scales.
And it is clear that the smoothing does affect the Hurst exponent just as I said it would (see point 4 from PaulM’s comment above).
PaulM (09:58:26) :
Basil,
This article starts well but then goes seriously downhill when you start smoothing the data before applying your tests. You must never do this or your results are meaningless. Here’s a quote from the blog of statistician William Briggs:
What Paul Vaughn said: Paul Vaughan (19:18:40) : . Smoothing has its uses (and its abuses). I agree with you in part, here:
4. The figure of 0.835 for the smoothed series is meaningless. It is purely a consequence of the smoothing. The more you smooth, the bigger this number will be. You can generate as many bogus ’signals’ as you like by smoothing in different ways. It is instructive to do this with data from a random number generator. Leif is right, the analysis should be performed on the raw data. Reading the comments more carefully I see that Spence_uk made this point too.
Not that it is meaningless (I do not agree there), but that it is a result or consequence of the smoothing. Of course it is. The Hurst exponent doesn’t “prove” anything, and is not cited for such. It is simply a descriptive statistic. The smoothing is justified, or not, on other grounds. Have you read the paper Anthony and I wrote? We deal there extensively with the smoothing we’re using, and show that it is no different, in its results, that can be shown otherwise with wavelet analysis and spectrum analysis. It is useful, however, as yet another way of looking at the data. We’ve justified the smoothing on other grounds. Here the Hurst exponent is simply calculated as a descriptive statistic to show that what appears as a random walk in the raw temperature data may be in fact the result of offsetting persistence (cycles, captured by the smoothing), and anti-persistence (in yearly variations in annual trend, or rate of growth).
BTW, thanks for your independent observations, and taking the time to investigate what happens if we take simple first differences, rather than the seasonal difference. You wrote:
3. If I do a month-to-month difference, as some people have suggested, instead of the seasonal difference, I get a much smaller number, around 0.1 – I don’t understand why this is!
I get 0.24. What software are you using? (I’m using gretl.) As for the number being noticeably smaller, I’m not sure that should be so surprising. That is simply saying that the degree of anti-persistence is greater in monthly fluctuations than it is in annual fluctuations. That makes sense to me, in that monthly fluctuations will tend to revert to the mean more quickly than shocks measured on an annual basis. Actually, given my numbers — 0.24 for monthly, and 0.38 for seasonal — the difference is about what I would expect.
Thanks for the dialog.
Phil M (05:15:36) :
Flanagan points out, by differentiating first, you are making any ‘linear trend’ into a constant offset
– and then analysing the result to show that we have a ‘random walk’ about a ‘constant offset’ (which is the differential off the linear trend)
– so in effect, you are saying we have a ‘random walk’ about a ‘linear trend’
– so, what is your point, exactly?
I think you left off your snark tags.
All,
There is a general point regarding the Hurst coefficinet/exponent I fail to get, and that is how 0.5 is associated with a random walk.
So if anyone who can point out the air gap in the statements below, I would be grateful.
Gausian (white noise) is not correlated, its Range (defined as the integral of the differences to the running mean) varies as (n)^0.5 and its SD varies as (n)^0.0 and hence (R/S)(n) varies as (n)^0.5 which as I understand it gives H=0.5. The same is true if the series is just a series of coin tosses (1,1,-1,1,-1,-1,-1,1, …).
A random walk can be formed by taking the integral (or sum) of Gaussian noise (or coin tosses). Or putting it the other way the first differential or difference series, of a random walk, will have the Hurst coefficient of the Gaussian or toin coss i.e. H=0.5
The random walk itself will have H=~1.
As I understand it H=0 implies strongly anti-persistence, (the differentials of Gaussian (white) Noise have this property),
H=0.5 implies neutrality (non-autocorrelated or white noise)
H=~1 implies strong persistence, (the integrals of white noise have this property).
Now either that is a load of tosh, or identifying H=0.5 with a random walk (as opposed to the first differential of a random walk) is misleading me.
Is there a different usage of the term “random walk” that I am not familiar with?
Anyone help me out?
Alexander
In the above I only meant “running mean” to imply that it varies with (n) not that it varies during the integration step.
Alexander
Stevo (12:57:38) :
Basil,
Are you aware of ARIMA processes?
Yes, of course I am. And were I trying to forecast temperature trends over the next few months I might well make use of that approach. But ARIMA models are largely black box models which do not require any understanding of underlying physical processes, and are not particularly useful, in my view, in understanding the long term dynamics of underlying physical processes. Have you ever looked at the confidence intervals of an ARIMA forecast? After a few months, you are in la la land.
Re: my own: Alex Harvey (08:20:10)
I have got the noise colours wrong I am sure (oops), but I hope that the rest of it stands, that my notion of a random walk would have H=~1 and a non-autocorrelated series H=~0.5.
Alexander
Leif Svalgaard (09:51:16) :
Basil (08:15:04) :
Any extrapolation out for two decades, let alone for the next century (a la IPCC) involves what we call in my field “heroic assumptions.”
Except that the lunar influence can be calculated accurately for thousands of years and the solar influence is well-known for centuries and reasonably well-known also for thousands of years, so your input to the ’signal’ is well-known, there your hindcast is not an extrapolation, but an application of known inputs.
Fair enough (that these inputs are known), except that the hindcast (or forecast) is not being driven by those inputs, per se. Maybe a better mathematician than I am can construct such a model some day. But, to address an earlier question (wanting to see Figure 5 back to 1850), the hindcast/forecast comes from the following:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png?w=510&h=300
It is a sinusoidal fit to what is in effect the same thing you see in the blue line in Figure 3 of this post. I.e., the red line in the above linked image, and the blue line in Figure 3, are for all intents and purposes the same. Now as a fit, it purports to explain 60% (from R^2) of the variation we see in the blue line in Figure 3. It does a reasonably good job (in my opinion) of modeling the frequency of the cycles, but less well capturing their amplitude (which is where most of the lack of fit occurs). Empirically, the dominant frequencies are 20.69 and 9.63 years.
I suppose what you are really getting at here is that if these really do come from the lunar nodal cycle and the solar cycle, we ought to be able to model the blue line from “first principles” rather than just fit something empirically to it. Keeling & Whorf tried that with the lunar nodal cycle, but it didn’t stimulate much interest. And I can imagine that it would be even more difficult to add in the effect of the solar cycle. I would be more than happy if someone took up the challenge of doing that.
FWIW (not much in your eyes, I imagine), but Figure 4 here (the Chree analysis) is derived from the red cycles in the image linked to above. As such, I think it is descriptively helpful, in isolating what we attempted to show the “S’s” and “L’s” in the image linked to above. Whether it “proves” anything is another matter altogether, I suppose. Being a Popperian, I don’t think we ever really “prove” anything. I would like to think that it is one of those things that are “suggestive” of something deserving further investigation. In a Popperian sense, I think Figure 3 (and everything that was done to get there) is a perfectly valid approach to developing a falsifiable hypothesis.
I know, of course, what comes next: “How do you propose to falsify it?” Well, we can wait and see if Figure 4 is replicated in any way during future solar and lunar nodal cycles.
You also asked:
And, perhaps, comment on the fact that the average distance after N steps is not zero but SQRT(N), so no strong probability of reverting to the mean.
That’s true of a pure random walk. But I haven’t suggested that is what we have here. Were we to look more closely at Figure 2, we’d find that the mean is not zero, so we might here have a “random walk with drift.” It (the mean) is 0.0053302, equivalent to a long term decadal trend of 0.053°C.
(FYI, I’m not attributing this trend to a solar influence, but rather the variations around any such trend, such as we see in the blue line in Figure 3. You’ve previously said you would expect to find variation of this magnitude from TSI variations, and have treated such as unremarkable. So I do not really understand the basis for all your opposition here.)
More substantively, though, I never claim a probability of reverting to the mean in the data represented by Figure 2. That claim is reserved for the data represented in the bottom panel of Figure 3, which I make clear evidences anti-persistence, i.e. is not a random walk.
Again, thanks to all who have commented. There are still some comments and observations worthy of reply, but I’ll have to come back to them later.
Basil
Spence_UK (08:10:36) :
The smoothing here doesn’t do that. It simply acts as a low pass filter, filtering out high frequency oscillations
Basil, low pass filters which filter out high frequency oscillations is the exact same thing as modifying spectral power at different scales.
And it is clear that the smoothing does affect the Hurst exponent just as I said it would (see point 4 from PaulM’s comment above).
1) Okay, I understand your first point, which is true. Which is why Anthony and I performed the wavelet transforms on the raw data, as well as the smoothed data. What that shows is that the cycles seen in the smoothing are not artifacts of the smoothing, but are there in the raw, untransformed, and unsmoothed, data.
2) I didn’t deny that smoothing would affect the Hurst exponent. I said “of course” it will. That doesn’t change the basic conclusion here, which is that the seasonal variation, which appears to be a random walk, can be decomposed into two components, one which exhibits persistence (yes, because of the smoothing), and the other which exhibits anti-persistence.
Alex Harvey (08:20),
This is a good question and one that it took a while for me over at climateaudit to “get” (Prof. Koutsoyiannis had to rather spell it out for me!)
A random walk does not have H=0.5. In fact, H is undefined for a random walk because it is not a stationary process, and the Hurst exponent is defined for stationary processes only.
A key relationship (which some estimators use to derive H), is a=2H-1, where a is the relationship between the time series spectrum where spectral power is proportional to f^(-a).
So H=0.5 yields a f^(0) relationship with spectral power, that is pure white noise
H=1 yields a f^(-1) relationship with spectral power, that is 1/f noise, excess noise or long-term persistence.
On this basis, a pure random walk would have spectral power of f^(-2). According to the relationship above, this would give H=1.5, but this is incorrect, because random walks are not stationary. (1/f noise is stationary, it has a defined population mean, but the sample mean is not a good estimator of the population mean). But by the original definition of H (laid down by Mandelbrot in, I think, Mandelbrot and van Ness), H is undefined for the random walk.
In the event of a random walk, the time series represents a storage parameter (or integral), and the series must be differenced and the Hurst analysis applied to that series. So, in principle, if global temperature were a random walk, the raw data would have no defined Hurst exponent (and many estimators would incorrectly report H>1), and it would be correct to take first differences.
I would suggest this is inappropriate for the global temperature series, as there are good reasons to doubt that it is a random walk:
1. The power spectrum is not proportional to 1/(f^2)
2. A pure random walk would eventually wander off to infinity, linking with Leif’s observations above
3. Hurst estimators do not yield values of H>1 for temperature series
However, once first differences are taken, the resulting series may yield Hurst coefficients in the valid range 0 < H < 1.
Alex, I think what you say is exactly right, the difference of a random walk should have H=0.5 (not the random walk itself). But that’s exactly what Basil is doing, so what’s the problem?
Basil thanks for your comments.
I’m using a matlab code called hurst_exponent.m off the matlab file exchange.
As long as you are fully aware that you would get the same result (H incresing as you smooth more) with random data then I’m happy. What appears to be a random walk in the temperature data could in fact be … a random walk. And the frequency of the ‘signal’ you see depends your choice of smoothing, so you can choose your smoothing parameter to get a match with whatever cycle you want. I still think your post is a bit misleading around fig 3.
PaulM,
The problem is in the assumption of a random walk. If you’re running MATLAB, I can show you why more quantitatively. Download “hurst estimators.zip” by Chu Chen at the MATLAB file exchange. This gives you a number of different estimators. Grab yourself a temperature series (I had the HadCRU kicking around from Oct 08 on my hard disk so I used that) and we’ll create a random walk with the following command (I choose 1904 because that is how many points I had in my HadCRU series):
randwalk = cumsum(randn(1904, 1));
H = hurst_estimate(randwalk, ‘method’)
H = hurst_estimate(HadCRU, ‘method’)
(Where method is one from the list). Now try some estimators both on the random walk and the temperature series. The results that I tried came out with (methods on top row):
methods: aggvar / higuchi / boxper / peng / absval
randwalk: 0.994 / 1.011 / 1.697 / 1.534 / 0.990
HadCRU: 0.923 / 0.951 / 0.990 / 0.732 / 0.909
Note the variation of estimates for the random walk include some very high (actually invalid) values; some estimators are constrained to be more or less 0 to 1, and yield (incorrectly) seemingly valid figures. This results in inconsistent values across estimators; this is typical of a random walk.
On the other hand, the HadCRU does not yield high values for these estimators. Why? Because the HadCRU data does not behave like a random walk. So those estimators (such as peng and boxper) that are unbounded do not yield invalid values for H.
(A variety of values is still seen, over a much narrower range, highlighting some of the difficulties associated with estimating H)
IMHO, I suspect the reason Basil sees antipersistence is because he is differencing a stationary series; the reason he sees persistence is because of the smoothing function. Applying Hurst analysis to the raw data without differencing shows that global temperature is stationary, not a random walk, but exhibits very strong long term persistence across all scales.
Dr. Jose Sanchez writes in reply to Stef:
“Dr Jose Sanchez (15:50:07) :
Stef wrote: “Dr Sanchez, the articles you gave links for are all from different sources, and this latest article is a guest post by Basil Copeland. It might surprise you that different people actually have different opinions. WUWT publishes articles and information from many sources. You do realise that not every single article is penned by Mr. Watts don’t you?”
Yes, I do realize they all come from different sources. And I also realize that journalists around the world are using this website as a source to spread misinformation about the theory of global warming. And I also do realize that an editor of such a website is supposed to have basic knowledge of the theory, and not publish whatever information is convenient at the time because it agrees with the position they are trying to push forward, regardless of how much it contradicts previous information they gave. There is a concept called “responsible journalism” that Mr Watts ought to read up upon. Until then, all credibility from this website is thrown out the window.”
He he…. ,
You are welcome to point out what “misinformation” you claim to read here,with your own counterpoints.I am sure Anthony and everyone else would like to see what you come up with.I am always ready to read of credible counterpoints,as this would teach me greater understanding of the topic under consideration.Did you have a counterpoint to offer here?
You are knocking an award winning science blog,with absolutely no regard to the idea,that it is that way because of the quality open civil exchanges that goes on here.Your replies seems to be one of avoiding the actual topic and just tar us a little.
Please tell us why you think some one on a topic is wrong with some details and leave out the worthless put down of this blog?
Spence_UK,
Many thanks for your very detailed response, I think I will get it now.
PaulM,
Thanks to you as well,
As to: “what’s the problem “?
I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.
Alexander
Dr Jose Sanchez (15:50:07) :
This guy seems to be one of a type we get here occasionally who drops in for a quick AGW rant and then disappears again. We’ve got ‘collapsing wave’ going “la la la la la” on another thread at the moment. We had another one a week ago or so, who was actually very eloquent but seemed to be on speed for the night, and then disappeared again. There’ll be another one along again soon. It’s a bit like two different universes making a close approach for a brief interaction before both going off on their own ways…
Alex Harvey (14:28:47) :
I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.
Seasonal differencing isn’t the same as a seasonal adjustment. In the case of monthly temperature anomalies, each month is baselined against its average for the base period, so that relative to the middle of the base period, each month should have a value of zero. (Or so I understand.) As long as you do not believe that each month has a different trend, then a seasonal difference will simply measure the annual rate of change from one month to the same month of the next year. Again, as long as there is no reason to believe that the tread is different for different months of the year, I cannot imagine what your problem is with this. It is certainly not arbitrary. The rationale is clearly stated.
G. Karst (08:22:37) : E.M.Smith (02:46:34) : Your analysis should be another talking point… a thread on it’s own merit. I would be interested to read a full discussion including “AGW convinced” rebuttals, on your thesis. You have given us much food for thought. Thx.
Thanks! But I didn’t really have a “thesis” going into this. About as close as I got was “An average of things broadly going up ought to go up” but that’s more a statement of a mathematical property than a thesis.
As of now, I think I do have a thesis: GIStemp is a filter that attempts to remove the impact of adding and removing thermometers from the record, distributed in time and space; but it fails to fully insulate from an order of magnitude change in thermometers in a largely divergent geography, leading to a warming of the “anomaly maps” based on a change of number and location of thermometers. GIStemp is not a perfect filter.”
All I did was to respect the data and ask it what it had to say, then shut up and listen. Others, I fear, tortured the data until they say whatever is desired. It’s not about me, or my thesis, it’s all about the data and what they have to say. You just need to listen…
Jimmy Haigh (15:23:35) : This guy seems to be one of a type we get here occasionally who drops in for a quick AGW rant and then disappears again.
There are certain linguistic and stylistic clues that imply some of these folks are the same person. Unfortunately folks who hang out together a lot tend to soak up a common “accent” and style. So it’s also possible it’s just a group of folks who hang out together in the echo chamber a lot. An inspection of IP addresses would be helpful, and I presume that the folks running this site are well aware of that.
At any rate, I find them a “useful irritant”. They show where there is a weakness in an argument by being all over it. They show where there is a weakness in their argument by giving lame counters (and you get good practice at rebuttal). And most importantly, they serve a wonderful purpose by showing the truly stellar arguments. Those will be greeted with silence.
Learn to see that “negative space”. The lack of flack from The Team. It tells you where you have hit gold… It warms my soul to hear their silence and know that I’ve got it right.
Re: Spence_UK (05:54:27)
6.4a is the period of the terrestrial polar motion group-wave (aside from ~1920-1940 when it slowed down, shifting 180 degrees). Maybe people aren’t seeing this because related signals vary regionally. (Increased pressure in one area is balanced by decreased pressure in another area, etc. – i.e. pressure doesn’t redistribute the same way temp does – & people do tend to fixate on global averages.)
Thanks for the links.
–
Re: Basil (06:57:00)
2 cautionary notes:
1) ENSO’s period is certainly not stationary.
2) Solar variables & temperature variables are in (broad-sense) anti-phase prior to ~1931 (back to ~1765). This appears to be related to Jupiter-Neptune’s phase relationship with the LNC (which has a beat period of ~205 years) and terrestrial north-south asymmetry. [Note: Jupiter-Neptune is the highest-frequency heavyweight-beat in solar system dynamics.] (Also: Remember that NH temps dominate global averages.) It is proving to be a devil of a challenge to get people to clue in to the preceding even though it should be as plain is day to anyone handling the terrestrial polar motion time series with sufficient computational skill. For too long people were bent on exploring how solar system dynamics affect the sun; somehow (?) this prevented people from looking closer to home (i.e. Earth’s shells) (?). The Chandler wobble phase reversal (centred ~1931) shows up in all kinds of terrestrial time series (including SOI, regional precipitation, & aa (if one knows how to look)). So: Be careful if you only work with frequency info, because you’ll miss important stuff that is plain as day if you use *time-frequency info.
Here’s another reference:
MacMynowski, D.G.; & Tziperman, E. (2008). Factors affecting ENSO’s period. Journal of Atmospheric Science 65(5),1570-1586.
http://www.cds.caltech.edu/~macmardg/pubs/MacMynowski-Tziperman-2008.pdf
sunsettommy writes: “You are knocking an award winning science blog,with absolutely no regard to the idea,that it is that way because of the quality open civil exchanges that goes on here.”
Award winning science blog??? Did it get an award for not understanding the difference between climate and weather? Did it get an award for propagating a myth about “global cooling” based upon a single data point for the month of June, while ignoring the long term trend? Did it get an award for not understanding the basics of applied statistics?
I’m sorry, but no wonder why there is so much misinformation about global warming. If you can’t filter out the junk from the legitimate criticisms, and people use your blog as an authoritative source, then you have become the major source of the misinformation. Scientific discussion does not look for data to fit the argument they are trying to push forward. Instead, it looks at the data without bias, and then draws a conclusion about it, even if that conclusion may disagree with what we hope the truth is. For two months the articles here tried to suggest that we are now in global cooling (by looking at the June data and selective points in July that were not representative of the big picture), and then once the new data came out that was contrary to what blog editor wanted to believe, the argument was changed to “individual monthly values saying little, if anything, about the long term direction of global temperature.” There is absolutely nothing scientific about that.
You will never hear Lindzen saying that we are in a global cooling trend now based upon the data you were trying to put forward in June and July. Listen to the scientists rather than trying to pretend like you are one while misleading others into believing you have a clue about global warming.
Jose Sanchez,
Yes, this is an award winning site. Why does that bother you so much? Are you upset that realclimate failed? Or Tamino, or the Rabett, or climateprogress, or any of the other censorship prone purveyors of climate alarmism?
Speaking of alarmism, why should we listen to those Chicken Littles, who keep telling us the sky is falling? You may not be aware of it if you inhabit one of the echo chambers named above, but the planet’s current temperature is right at about the same level it was at thirty years ago: click.
Instead of going ballistic when someone mentions the “Best Science” site, maybe you should stick around and learn something.
Alex Harvey (14:28:47) “I do have a little problem in Basil’s approach in that I find the choice of “seasonal differencing” of seasonally corrected data a bit arbitary.”
Bear in mind that the seasonal “correction” is based on assumptions (usually defining a rigid annual structure based on a 30 year span (a climatology) …as opposed to a flexible one that varies locally (in time), for a contrasting example…)
Let me give an example that might help people get around this:
Should there be a very strong annual term (with harmonics at monthly-multiples) in the sun’s motion about the solar system center of mass? Well guess what: There IS …if you work with monthly summaries …..and as soon as you difference: THAT GETS AMPLIFIED, but it is a spurious effect that a sensible analyst would recognize (…& perhaps remove with annual-smoothing).
This is just one example. There is no substitute [such as rule’s of thumb] for careful, context-specific thinking. (Sometimes a person “knows” that smoothing makes sense in a given context, but hasn’t thought-through why. A good recent example: McLean et al’s (2009) use of the RATPAC series that got them into trouble (which they did not explain) with averages for the 4 seasons – (that series is not monthly-resolution).)
On a practical note: I have learned to expect bitter opposition to the use of smoothing whether it is warranted or not – i.e. what a headache — a more efficient education system might be a long-term solution – (smoothing isn’t even usually addressed until 4th year stats courses).
We should dump the convention of posting anomalies to webpages. If people want anomalies, empower them to choose what type of anomaly is most appropriate in a _specific_ analysis context. For example, if someone is differencing twice with a series that is based on a pre-defined climatology, they might have a problem that they should not have (depending on the nature of the series).
Steve Fitzpatrick (07:59:16) :
Basil,
Thanks for an interesting post; I had not seen this type of analysis before.
1) It seems to me that by using the 1-year differences to transform the raw data (Figure 2) that you are taking something of a “first derivative” of the data, which ought to magnify the importance of short term variations, while reducing any longer term variation in the original data to small differences in the trend above or below the baseline. Since the overall trend for the raw data set is clearly positive, the average value for all the transformed data should be slightly above zero, with a value that depends on the slope of the least squares fit trend in the original data divided by the number of data points in the raw data set less 1. Do I understand the transform you have done correctly?
You understand it perfectly, Steve.
Would a series of similar transforms with different time steps (2 yrs, 3 yrs, 4 yrs, etc.) not increasingly show the longer term variations in the raw data, while “filtering out” the short term variation? What is the reason for choosing 1 year changes as opposed to some other period?
It is conventional to look at time series on monthly, quarterly, or annual bases. I’ve never heard of anyone using as a whole unit, a period of 2 yrs, 4yrs, etc.
2. If I understand correctly, you have assigned cause for the recent temperature history (the 20th century to now) to the “lunisolar” influence you described, and based on this, you project variation in average temperature around a flat trend for the next 20+ years (as shown in Figure 5). If this is true, then it seems to me that you are implicitly assigning a value of near zero for climate sensitivity to radiative forcing..
No, because solar isn’t the only factor at work here. The other is the lunar nodal cycle (and actually, maybe some longer term oceanic or atmospheric influences, since the sinusoidal model has frequencies of ~15 and ~54 years as well). We’re not sure about the exact physical mechanism(s) here, but I think these forces play out through long term trends or shifts in atmospheric processes, with meridional flows dominating at times, and zonal flows dominating at other times. Through all of this, the solar “forcing” can remain relatively constant (though not entirely, as TSI does fluctuate some over the solar cycle).
Fair enough, it could be very low. But in this case, how can the climate have any measurable sensitivity to variation in TSI over the solar cycles? I think that Leif suggests an average solar cycle signature of about 0.075C in the historical data due to variation in TSI of about 1.4 watt/M^2 at the top of the atmosphere. This is consistent with a relatively low climate sensitivity, but not a near-zero sensitivity. If the sensitivity to radiative forcing is in fact near zero, then by what mechanism do you think the solar cycle shows up in the temperature data?
Leif calculates the impact of variations in TSI on temperature to be, I think, about 0.07 K. So look carefully at the variation indicated on the y-axis of the following graph:
http://i25.tinypic.com/25f2b00.jpg
This is an amplified version of the wavy blue line in Figure 3. Just eyeballing it, the average peak to trough change in range of change might be something on the order of 0.05°C, at least of an order of magnitude comparable to the 0.07 K Leif computes. In other words, over the course of a solar cycle, the annual rate of change increases ~0.05°C, and then declines by that amount. Now of course, that’s a Mark I eyeball estimate, and some of the cycles are less, a some a bit more. The largest peak to trough value is ~0.10°C around 1940, and during the anomalous period of the 1920’s, the change was less than 0.02°C. But accuracy of my Mark I eyeball estimates aside, these variations, which (again) you see in the wavy blue line in Figure 3, are ballpark order of magnitude what they should be according to Leif.
By the way, for all of those who are skeptical of the use of seasonal differencing here, and wonder what would be the case if we used monthly data, I refer you to the comparable figure from the paper Anthony and I wrote:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png
Note well the difference in order of magnitude on the y-axis. That is because in that research, we smoothed the monthly observations, and then differenced the monthly observations. There was never any seasonal differencing to transform the observations into an annual rate of return. The data in the above figure are represented on the y-axis as monthly rates of change. E.g., just multiply the values in the last graph by 12 and you will have what you see in the preceding one.
But the resulting pattern of variation over time is the same!
This, I think, ought to settle the issue as far as the question of seasonal differencing is concerned. It is, as the lawyers like to say, and distinction without a difference.
Paul Vaughan (17:43:57) :
Solar variables & temperature variables are in (broad-sense) anti-phase prior to ~1931 (back to ~1765). This appears to be related to Jupiter-Neptune’s phase relationship with the LNC (which has a beat period of ~205 years) and terrestrial north-south asymmetry.
Fascinating stuff. I’m certainly aware of the anti-phase correlation in solar and temperature variables prior to the 1930’s. But this is the first time I’ve seen this explanation for it.
The Chandler wobble phase reversal (centred ~1931) shows up in all kinds of terrestrial time series (including SOI, regional precipitation, & aa (if one knows how to look)). So: Be careful if you only work with frequency info, because you’ll miss important stuff that is plain as day if you use *time-frequency info.
Well, I like to think that’s exactly what we’re looking at here — a time-frequency representation of the data:
http://wattsupwiththat.files.wordpress.com/2009/05/figure6.png
So is that the Chandler wobble we see in the phase shift in this diagram circa 1930?
Re: Basil (19:40:16)
There’s a loose 1:2:3 resonance that breaks ~1931:
http://www.sfu.ca/~plv/1930sHarmonicPhaseDifference.PNG
http://www.sfu.ca/~plv/1931UniquePhaseHarmonics.png
Wavelet methods are one way to go for nonstationary series — you need a method that gives a view of how period varies – like this:
http://www.sfu.ca/~plv/ChandlerPeriod.PNG
http://www.sfu.ca/~plv/ChandlerPeriodAgassizBC,CanadaPrecipitationTimePlot.PNG
Basil & Peter Vaughan,
This annual differencing would do two thing, remove a seasonal cycle, if present, and progressively filter out some of the frequency components.
Basil Wrote: “A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, …”
Now as the series is already seasonally adjusted, the choice of 12 months is not necessary to remove the seasonal component of the earths temperature record. It will remove variance as in the 12,6,4,3, and 2 month periods but is this a necessity given what else this diverencing does to the data.
Significantly it alters the frequency specturm of the series in a different way to what the choice of 1 month, 3 months or 6 months, etc., (each with a different Hurst coefficient), would do.
As you choose to estimate a Hurst coefficient on the filtered data (and to compare it with 0.5), the choice of the time interval materially affects the result. If you get H=~0.5 using 12 months, someone can say that it is due to a subjective choice of the interval, with no other need to pick 12 months.
Now there is a small residual seasonal component in the HadCRUT3 data partly due to the length of climatology only spanning about 1/4 of the total period, if you wish to remove this seasonal component you can construct a new climatology spanning the entire period of the data and subtract it. I expect that would not have a significant effect on the estimated Hurst coefficient.
Obviously you can filter the data in anyway that suits your purposes but to do so, and then to attach any particular significance to a resulting Hurst coefficient seems a little strange.
There is one obvious filters to use and that is monthly differencing. Why not do that?
Given that there is no signifiacnt seasonal signal and that it is possible to minimise any residual seasonal signal by subtracting a new climatology, I can not see that a 12 month interval as being anything but a subjective choice.
Besides progressively filtering out signals with long periods what else does it achieve? Why not some other time constant?
Peter your:
“Should there be a very strong annual term (with harmonics at monthly-multiples) in the sun’s motion about the solar system center of mass? Well guess what: There IS …if you work with monthly summaries …..and as soon as you difference: THAT GETS AMPLIFIED, but it is a spurious effect that a sensible analyst would recognize (…& perhaps remove with annual-smoothing).”
Sorry I could not follow that at all. Anyway the HadCRUT3 data has very little variance in the seasonal frequency bands (12months, 6 months, 4 months, etc.).
Now if you had the raw (non adjusted by subtracting a climatology) HadCRUT3 data. You would find that it was dominated by the seasonal variation and you would look to do something to remove it. The choice would be to subtract a climatology, or perform 12 month differencing, I can see no reason fro doing both.
I need is a convincing reason to do both.
Alternatively you could also subject it to a low pass filter.
Now I have previously described 12 month differencing as a low pass filter, now that is not strictly the case. For periods longer than 6 years it is a type of low pass filter, but from 6 years down to 1.2 years it amplifies the signal (the strong El Nino Signal is in this band). Basically it stops frequencies at (0,1,2,3,4,…) Cycles/Yr and amplifies frequencies around (0.5,1.5,2.5,…) Cycles/Yr.
At the stop frequencies it removes all variance, at peak frequencies it increases the variance by a factor of 4.
I would think twice before I would rely on data transformed in this way to tell me much about cyclic trends. Choosing monthly differencing would not have produced the various pass and stop bands but it would have produced a very different Hurst coefficient.
Alexander
Patrick Davis (23:04:07) :
“VG (22:03:06) :
“Its always been a ramdom walk, and will continue to be, unless Krakatoa erupts again in our lifetime. That’s our only chance of experiencing anything remotely close to “climate change”
It has…
http://www.volcanodiscovery.com/volcano-tours/photos/krakatau/june09.html
Probably why we’re getting some truely awesome sunsets recently”.
No, the bulk of the SO2 that colors our sskies today was produced by Sarychev Volcano, plume altitude 21 km and 3 other medium volcanic eruptions in the NH this year, including Mt. Redoubt.
http://volcanoes.suite101.com/article.cfm/sarychev_peak_volcano_june_12_2009
Re MY Above:
It should have read: “Now I have previously described 12 month differencing as a HIGH pass filter, …”
Basil,
“Yes, of course I am.”
Good. So why on Earth did you use a random walk model, when it quite obviously doesn’t fit? Random walks do have the spurious trend property you discuss, but their wandering is unbounded, and climate considered over many centuries very obviously isn’t. There can be no ‘reversion to mean’ with a random walk, because random walks have no mean. Random walks don’t revert.
ARIMA processes are certainly the wrong model too (long term climate has periodic components, and some external forcing), but they offer a closer match to the particular features of climate data you seem to be discussing here. Was it because of a desire for simplicity in explanation?
“And were I trying to forecast temperature trends over the next few months I might well make use of that approach.”
I’d never even considered the possibility you might be trying to forecast anything.
“But ARIMA models are largely black box models which do not require any understanding of underlying physical processes, and are not particularly useful, in my view, in understanding the long term dynamics of underlying physical processes.”
ARIMA models can be constructed based on physical processes, and random walks can be considered to be black box models, too. It’s like arguing that straight line models don’t require an understanding of physics. It’s true that you can fit a straight line to anything, whether you understand it or not, but a lot of straight lines are the result of a physical understanding. The same goes for ARIMA, or any other model.
In any case, a random walk is an ARIMA model. ARIMA(0,1,0) to be precise. Like I said above, it’s a generalisation.
And I don’t see where you derive the random walk characteristics from any underlying physical processes, either.
“Have you ever looked at the confidence intervals of an ARIMA forecast? After a few months, you are in la la land.”
Have you ever looked at the confidence intervals for a random walk forecast? They increase in proportion to the square root of time, (centred on the last observation,) and therefore increase without bound. Random walk forecasts explore la la land far more deeply than many ARIMA processes.
Alex Harvey (05:33:00) :
Basil & Peter Vaughan,
This annual differencing would do two thing, remove a seasonal cycle, if present, and progressively filter out some of the frequency components.
Basil Wrote: “A common transformation in time series to investigate the possibility of a random walk is to “difference” the data. Here, because we are using monthly data, a particularly useful type of differencing is seasonal differencing, …”
Now as the series is already seasonally adjusted, the choice of 12 months is not necessary to remove the seasonal component of the earths temperature record. It will remove variance as in the 12,6,4,3, and 2 month periods but is this a necessity given what else this diverencing does to the data.
You are beating a dead horse here, Alex. I’ve shown that the same pattern of smoothed oscillations can be extracted from the monthly data, without any seasonal differencing. You are barking up the wrong tree, this dog won’t hunt, and whatever other mixed metaphor you choose, you are raising a non-issue here.
To wit, the Hurst exponent of the smoothed series derived using monthly data with no seasonal differencing is 0.792, versus the 0.835 computed using the model derived with seasonal differencing. For the point being made, that degree of difference is insignificant.
Obviously you can filter the data in anyway that suits your purposes but to do so, and then to attach any particular significance to a resulting Hurst coefficient seems a little strange.
The Hurst exponent is just a convenient descriptive statistic that describes the degree of persistence (or lack of persistence) in a time series. It is what it is, and nothing more. Instead of harping on use of the Hurst exponent, you could better contribute to the discussion of whether or not the cycles revealed by the smoothing mean anything. Obviously, because smoothing is involved, and particularly because of the resulting cycles in the data, the Hurst exponent increases.
In truth, while you’ve been all over me for seasonally differencing the data, and for applying the Hurst exponent to the resulting smooth, the really more significant conclusion to be drawn from the Hurst exponent is the anti-persistence demonstrated in the residuals once the “trend” is removed. This wouldn’t happen with a strictly linear process with iid residuals. So here the Hurst exponent may be telling us (descriptively) something significant about the data.
And lest you think that is an artifact of the seasonal differencing, when I use only the monthly observations (no seasonal differencing), fit the HP smooth to the data, and subtracting out the residual variation around the smooth, the Hurst exponent of the residual (visualize here the bottom pane of Figure 3, but now just from month to month variation) is 0.395, compared to the 0.383 I get using the seasonally differenced data . So what is the big deal? Do you have any comments on that?
Choosing monthly differencing would not have produced the various pass and stop bands but it would have produced a very different Hurst coefficient.
But it didn’t. Unless you think 0.395 is “very different” from 0.383 (or 0.792 is “very different” from the 0.835. No slight intended, but are you sure you know what you are talking about? Maybe you do, in an abstract context, but not here, that is you really do not seem to fully comprehend what has been done, so your knowledge about the subject is being misapplied. More over, as I’ve shown, as far as the HP smoothing working like a low pass filter, I get to the same place whether I use monthly data, or seasonally difference the data first. So you are incorrect there, also, that “monthly differencing would not have produced the various pass and stop bands.”
Look, rather than just criticize, if you think something was done wrong, and that the result stems from doing it wrong, and that if you do it “right” you get a different result, why not demonstrate it?
Truth is, I’ve met your every challenge. I’ve shown that your complaints are much ado about nothing, that I can get there without seasonal differencing.
Having done that, do you have anything else to bring to the discussion?
Stevo (08:05:08) :
Have you ever looked at the confidence intervals for a random walk forecast? They increase in proportion to the square root of time, (centred on the last observation,) and therefore increase without bound. Random walk forecasts explore la la land far more deeply than many ARIMA processes.
Fair rejoinder. But in the end, your focus on the best way to model a random walk seems to miss the point. The real point is not the possibility that Figure 2 is a random walk, or whether there are better ways to model it. The real point is what the disaggregation of the series into the two components presented in Figure 3 says about possible physical processes influencing global temperature trends. My frustration with much of the discussion, such as with you and Alex, is that it focuses only on questions of methodology, without actually showing that the issues you are trying to raise have anything to do with the implications this analysis might have on the science (or physical processes) under consideration.
Do not get me wrong. Methodology is important, especially where it can be shown that the methodology is leading to wrong or unjustified conclusions. But neither you nor Alex have done that. You are arguing methodology on abstract grounds completely devoid of any connection to the main point of the post. In your case, you want to argue about the best way to model a random walk. I.e., you are hung up on Figure 2, when my point isn’t about modeling a random walk at all, but about what is implied by Figure 3. And I don’t need a random walk to get to Figure 3, and I fail to see how arguing about to best model a random walk advances the discussion I’d prefer to have over Figure 3.
I don’t know if this will be the end of the discussion or not. I imagine it is winding down. What I am about to say does not apply to you specifically, but is a general observation about internet discussions such as this one. A lot of people jump in and begin to critique a novel presentation before they fully understand what the presenter is saying, or doing. I live by a long standing rule that before I criticize someone, or what someone is doing, I try to understand their point of view, whether I agree with it or not. I’m not sure I’m getting the same consideration here.
But that is okay. I’ve taken the time to respond, respectfully, to a lot of criticism here. And that is fine. I can take the heat, because I know that only by exposing our beliefs or reasoned considerations to the possibility of refutation can we truly be assured that we are not fooling ourselves (by confirmation bias). But frankly, the most vocal criticism has only served to reinforce my conviction that there is something worthwhile in what I’m doing, because with the possible exception of Leif’s issue with the Chree analysis, none of the criticism has amounted to much.
And the jury is still out on the Chree analysis. When I get the time, I’m going to do it Leif’s way, and see what happens.
Basil (12:04:49) “A lot of people jump in and begin to critique a novel presentation before they fully understand what the presenter is saying, or doing.”
There will always be those who will opportunistically engage in obfuscation about differencing & smoothing (regardless of whether applied sensibly or not).
One suggestion I can share from years of experience teaching online stats: Keep posts brief & to-the-point if your aim is to avoid a wasteland of comments & inquiries from online skiers (those who skim & skip).
At least we know from the comments that people are enthusiastic to participate in discussions about this type of analysis. I look forward to more (in chewable chunks).
Thank you for presenting.
Smokey wrote: “Speaking of alarmism, why should we listen to those Chicken Littles, who keep telling us the sky is falling? You may not be aware of it if you inhabit one of the echo chambers named above, but the planet’s current temperature is right at about the same level it was at thirty years ago: click.
“Instead of going ballistic when someone mentions the “Best Science” site, maybe you should stick around and learn something.”
Smokey, has it ever occurred to you to actually think about what I have said above? For example, about the problem of using short term data, and about the problem of picking and choosing data points to make it appear as if something is happening which is not really reflective of reality? In fact, may I ask you if you have any understanding of the random walk topic posted here? Because if you did, then you wouldn’t be showing graphs that show that if you start from a place of your choosing which is not too far back, and go to a point where it is particularly convenient for your argument, and then concluding that this is a sign of a cooling trend. Or at least, you wouldn’t be showing that to a person who has a background and statistics, who clearly can see the misuse of such statistics.
In fact, Smokey, you are just another example of the point I have made in several posts here. People like you, who love to throw around your charts but do not understand basic statistics and concepts behind the theory, are misleading people into believe things that are not true. And as I have said above, rather than listening to what the legitimate skeptics are saying, you try to invent the argument yourself even though you have a complete misunderstanding of the basics.
Obviously, Smokey, this is not sinking in. So let me try to rephrase it as a challenge to you. I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend. To make this tractable for you, here is where you can get a list of scientists in the top 500:
http://www.eecg.utoronto.ca/~prall/climate/climate_authors_table.html
By the way, Smokey, I am not surprised that your “award winning” refers to website popularity, and not scientific popularity. As I have said, this website is the reason why there is so much misinformation about the science. It is the source!
Well, gosh, Dr. Sanchez, let’s look at the longer term, then.
We’ve warmed ~0.7C or so this century, globally. If the GHCN adjusted data is to be trusted. We can see the imprint of PDO cycles in the record. We also know that 1976 through 2001, the six major cycles went from cold to warm, one by one. We’ve seen a relatively flat trend since 2001, with a downturn starting in 2007.
The IPCC premise seems to be that the entire 21st century will warm at the rate of 1979 – 1998, or even faster. At this early date, that seems not to be the case. If the PDO follows its past patterns, we’ll be in a negative phase for another two to three decades. For the IPCC to be correct, there would have to be a roaring warming after that point (or even from this point).
I agree that CO2 will put an upward pressure on temperatures. But the IPCC scenario is dependent on strong positive feedbacks, and so far the feedbacks appear, if anything, to be negative. This completely ignores the solar issue, which influence (if any) is as yet unknown.
The hockey stick has been shown to be in error (to be charitable). We do not seem to be in an imminent crisis situation, and we are in an age of hugely advancing wealth-fed technology.
If we continue to warm at a rate of 0.7C or even somewhat more over the 21st century, there is no crisis. So it’s not even a matter of reversing past trends.
That is the “shorter” longterm picture. The next decade’s worth of data will show us a lot more. I do not see that this places us in a position where we must react drastically with a guesswork-based solution. In my opinion, we need to monitor the situation carefully and, above all, openly. If there is a crisis, we will be far, far better equipped to deal with it in a decade, unless, of course, we do not eliminate a large percentage of world growth in the meantime.
Dr Jose Sanchez,
I’ll do better than find one scientist out of 500. The OISM Petition was signed by over thirty thousand people with advanced degrees in the hard sciences. The petition reads:
Savor that last paragraph. Re-read it. Again. It will do you good. They are saying very clearly that there is nothing to be alarmed about.
I am not misrepresenting anything by linking to a graph of four government/university agencies, plus the ARGO data, all of which show cooling for most of the past decade. When every metric shows cooling, and when tens of thousands of scientists state that there is nothing to worry about, I wonder why you would say the opposite?
Spin your statistics any way you want, but the real world is telling us that it has been cooling for most of the past decade.
I’ll give him 3: Richard Lindzen, Roy Spencer, and Roger Pielke Sr. Quite frankly, I don’t think there are any scientists, top 500 or otherwise, that don’t agree temperatures have been either cooling or leveling off for the last decade You’d be stupid not to. Even Gavin admits as much every time he says it’s not inconsistent with the models.
So, why do so many Drs. make the authority argument? Are they afraid of the technical argument?
Mark
They can make any appeal they like. But appeal to authority without releasing data is rather telling. No skeptic would be able to get away with that. Which is fine, because no one at all should be able to get away with that.
Smokey,
what part of “I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend.” did you fail to understand?
Oh, besides, I see your OSIM assorted MDs and engineers and raise you 84% of the AAAS members saying “Warming is due to human activities” (just 4% claiming there is no evidence of warming) and 70% considering global warming a very serious, 22% considering it a somewhat serious problem and just 2% contending it to be no problem at all. The numbers are from a recent poll of 2500 members of the AAAS by the Pew Research Center. They also document the warped public perception of the view of scientists and the strong divide along political affiliation in the US.
Basil,
Thankyou for your respectful responses.
Part of the reason I don’t criticise your main point is that I don’t see any reason to disagree with it. That’s partly because I’m already of the view that a lot of the short term ‘trends’ are spurious randomness anyway, the result of stochastic processes with memory, but partly because the issues with methodology obscure precisely what you’re doing and what you mean by it for me.
But the main reason I raise such objections is in the hopes of improving the arguments. When I start reading through an argument and suddenly run into something I know is wrong, it jerks me up short. I find it distracting and off-putting. Even when it is irrelevant to the rest of the argument, it is still an error. And if the rest of the argument continues to refer to and rely upon it, it makes things very difficult to follow. And worse, even irrelevant points get picked up and remembered as background knowledge by other readers, and repeated elsewhere.
It’s one reason why I keep ranting about it every time I see someone repeat the line that greenhouse gases “trap” radiation, even if it’s just by way of a scene-setting intro and not relevant to the main argument, because other people then go on to waste huge amounts of effort to debating and debunking a mechanism that isn’t even the ‘official’ understanding of the greenhouse effect anyway. Letting the small errors pass perpetuates the myths.
Criticism isn’t always opposition. I do sometimes find that when I criticise a sceptic’s argument, people assume I’m therefore an AGW-believer. I had hoped that citing Climate Audit might allay that, but it doesn’t always work. Many of the arguments in this area, on both sides, are in some small detail wrong. That includes many of my own. Nobody likes that; but it’s unfortunately inevitable. On the whole, I’ve generally found sceptics to be more open-minded with regard to improving them.
You have paid more attention than many would, for which I thank you. I apologise for the misunderstanding. Your conviction that there is something worthwhile in what you are doing is not something I have any wish to argue against. My aim is to help, because I think the details do matter.
Basil,
Perhaps we have a terminology problem in case so, I will clarify.
I am referring to 12 month differencing X(13)-X(1) etc., of the HadCRUT3 time series for which I get H=0.415 (You gave us H=0.475)
and to 1 month differencing X(2) – X(1) for which I get H=0.228.
I do not think that you have given us a H value for this filter of the HadCRUT3. It you have I apologise but could you give it again.
So for me one filter gives a series with H close to 0.5 and the other definitely does not.
I felt that you were drawing some significance to its closeness to zero and I am saying that it could be seen as happenstance as H values differs if you use different filters.
I have tried examples of other filters quite arbitrarily just because I or others have your choice might be seen as arbitrary and suggested such filters as alternatives.
For X(25)-X(1) I get H=0.533.
This filter has 11 stop bands (inside the pass band) with amplification in its pass bands.
Also for simple high pass filter (time constant = 1/(2pi)yrs) I get H=0.238.
This filter definitely does not have any stop bands (inside the pass band), only a minima at f=0, also and at no frequency does it amplify the variance.
Another method of minimising the seasonal component that, for me, leaves the H value largely unchanged and close to 1.
Here I am referring to subtracting a climatology based on the means (over the whole interval) for each month,
I get H=0.980 after removing a climatology. This is the same value (3dp) I get for the unfiltered series.
This type of filtering has the same stop bands as 12 month differencing but they are much finer. It does remove the seasonal cycle without doing a lot else. In this case it removed about 27% of total variance whereas 12 month differencing removed > 60%. It is also not a high pass filter. It is flat across the spectrum except for narrow stop bands at 12,6,4, … month periods. This is almost certainly why it leaves the H value almost unchanged.
As I mentioned above 12 month differencing also amplifies the variance across the majority of the width of the pass bands so the 60% does not give such a good guide to how much more targeted subtracting a climatology is. After adjusting for this amplification by dividing the series by 2 to reduce the maximum gain in the pass band to unity, >90% of the variance has been removed by 12 month differencing).
So the filtering effect of the 12 month interval does produce stop bands at 1,2,3,4,5 Cycles/Yr whereas I find a 1 month interval does not.
If you are saying that one month differencing produces stop bands can you tell me where they are?
For clarification I have not been included the behaviour for f->0 as this minimum is common to high pass filters I am referring to stop bands inside the pass band.
For reference: The number of stop bands (inside the pass band) is given by N/2 – 1 (N=interval in months) for even months, and (N-1)/2 for odd months. I doubt that I am wrong here. If you wish to count minima at the top of the pass band (6 cycles/yr in this case) you add one for even months.
So I definitely get different Hurst coefficients depending on my choice of filter.
Now I make no bones about what I do and do not know. I do know the results I get and if I discover I am wrong I will take it on the chin.
You say that you can get there without seasonal differencing yet you chose to do it, and I thought that you were drawing some significance from a H value ~0.5. If you no longer think that was a point worth making just snip it out.
I would do so because I think it is not sound. I find it to be highly dependent on your choice of a filter that for me seems to serve no purpose other than to provide this value. I am not sure why you added this stage which you say you do not need, and except for saying that people commonly do this I am not sure you have done much more to justify doing it.
Now I think you were asking for some specifics so I have provided some, hopefully this will clarify the discrepancy between our points of view on the choice of filters.
BTW at no point have I referred to your processing beyond this point, as I have no idea what Hodrick-Prescott smoothing would do to Hurst coefficients, so I could not comment. But I shall look to it.
At this stage I am concentrating on figure 2 not figure 3. I have not commented on the H values for figure 3, so need to restate them.
Alexander
bluegrue (05:46:28),
With all the false-alarmist huffing and puffing, they still can’t come up with more than a small fraction of 31,000 scientists who say:
What part of ‘no convincing scientific evidence’ do you people fail to understand?
Regarding Jose’s silly challenge, let me put that out of its misery. He linked to a list comprised of less than one-tenth the number of signers of the OISM Petition, which is limited to those with degrees in the hard sciences.
But Jose’s list? His list includes poseurs with degrees in things like Community Ecology, Advertising, Prediction [heh], Politics, Downscaling, Market Research, Economics, etc.
Furthermore, about 40% of Jose’s list is comprised of UN/IPCC political appointees, who have their marching orders, no matter what they privately think. Those IPCC individuals have traded their credibility for job security.
And Pew? Give me a break. They were easily the most error prone, inaccurate polling in the last election; Pew polls are used to skew results. In other words, for propaganda. How else can we take an all-or-nothing question like “Warming is due to human activities”? Does that mean 100% of warming? Hang your hat on that push poll if you want, but it is not credible when it frames questions like that.
I won’t embarrass the inept Jose Sanchez for his foolish challenge: “I challenge you to find one scientists in the top 500 most published global warming researchers that claims we are now in a global cooling trend.”
Jose’s own list includes internationally esteemed names like Ross McKitrick, Roy Spencer, Lubos Motl, Fred Seitz, William Gray, Freeman Dyson, Nils-Axel Morner, Pielke, Sr. & son, Edward Wegman, Roger Revelle, Chris Landsea, Syun-Ichi Akasofu, Nir Shaviv, Bob Carter, Craig Idso, Tim Ball, Willie Soon, Henrik Svensmark, Piers Corbyn, Benny Peiser, Hans Erren, Joe D’Aleo, Vincent Gray, and plenty of other AGW skeptics. The list also has the names of disreputable individuals like Caspar Amman, Rajendra Pachauri, William Connolley and Michael Mann, who wouldn’t recognize integrity if it bit ’em on the ankle.
Jose was simply winging it with his fingers crossed, hoping that no one would check the list he posted. I’d be willing to wager $10,000 that I can find someone on his list who thinks we’re in a cooling trend, if you or Jose are game. Otherwise, why bother? We all know the answer to Jose’s foolish challenge.
So you and Jose lose both the “consensus” claim and the challenge, hands down. You couldn’t even come up with one-tenth the number of scientists that the skeptic side has. Which is the way it should be: skepticism is an absolute requirement of the scientific method — a requirement that the false-alarmist side has given up and surrendered to greed. Now, they’re just grant hogs with both front feet in the public trough.
Alex Harvey (08:57:49) :
and to 1 month differencing X(2) – X(1) for which I get H=0.228.
I do not think that you have given us a H value for this filter of the HadCRUT3. It you have I apologise but could you give it again.
Yes I did post this above, but in a reply to PaulM, who wrote:
3. If I do a month-to-month difference, as some people have suggested, instead of the seasonal difference, I get a much smaller number, around 0.1 – I don’t understand why this is!
To which I responded:
“I get 0.24. What software are you using? (I’m using gretl.) As for the number being noticeably smaller, I’m not sure that should be so surprising. That is simply saying that the degree of anti-persistence is greater in monthly fluctuations than it is in annual fluctuations. That makes sense to me, in that monthly fluctuations will tend to revert to the mean more quickly than shocks measured on an annual basis. Actually, given my numbers — 0.24 for monthly, and 0.38 for seasonal — the difference is about what I would expect.”
I think the explanation holds up as well for any point you are trying to make of this.
At this stage I am concentrating on figure 2 not figure 3. I have not commented on the H values for figure 3, so need to restate them.
No, I do not need to restate them. You need to do me the courtesy of reading the whole post, and get off of your fixation over Figure 2. I’m not discussing this any more with you, if after all this time you haven’t even bothered to read the entire original post (or read it so cursorily that you do not recall that what you now ask me to restate in in the post, as was a major focus of the post). Frankly, you have just vindicated my previous post (in reply to Stevo). You are wasting my time. I’ve got more to do on this, so I shall consider our discussion at an end, and move on to other matters.
Stevo (06:02:58) :
Basil,
Thankyou for your respectful responses.
And thanks to you, too. I sometimes get impatient, as in my previous reply to Alex, but I do appreciate vigorous discussion and challenge, so long as I think something constructive can come of it. So again, thanks.
AAAS has about 125,000 members, 84% of which is 105,000, if you really want to play the numbers game.
Community Ecology has nothing to do with your local town hall, but is relevant for impact assessment; do you have anything specific to say against GC Hurtt? Or FW Zwiers or C Rosenzweig? You mocked them for being listed under “prediction”. And why would you, Smokey, of all people, mock JS Armstrong, the only one listing advertising? He signed the 2009 newspaper ad by the Cato Institute. His most cited paper on climate is published in E&E and his paper on Polar bear population is published in Interfaces, a curious choice, given that the journal’s scope is “Learn how to overcome the difficulties and issues encountered in applying operations research and management science to real-life situations.”. Too bad that most of his citations are not for his work on climate, otherwise you would have had a winner. I have not checked further.
Given how shallow your reading seems to be, I’m stepping out of this discussion.
Basil,
could you please reply to my questions, too? You seem to have missed them. I’d just like to see, whether your analysis is sensitive to long-term trends.
Re: Alex Harvey (08:57:49)
Alex,
I would like to discuss this with you, but Hurst coefficients are new to me and my plate is already full for the foreseeable weeks. I’ve downloaded the papers others have cited above. I thank you & others for sharing your enthusiasm for the methods under discussion – I hope to look into this in the future.
Also: I hope we (WUWT participants) might start having some in-depth discussions about wavelet methods moving forward. It’s tricky when not everyone in the audience knows a method. For example, I suspect many readers might better-understand the info in my wavelet plots (upthread) if I simply present the following plain time-plots instead:
http://www.sfu.ca/~plv/(J,N)o2&Pr.png
http://www.sfu.ca/~plv/PhaseConcordancePxySI.png
http://www.sfu.ca/~plv/(J,N),r..png
http://www.sfu.ca/~plv/Pr,JN4,r..,m4..png
Perhaps these will lead readers unfamiliar with wavelet methods to a better appreciation for the post-1940 stability of the Chandler Wobble period.
We are certainly well-into an era when specialized statistical-computing is leading to communication break-downs.
Regards,
Paul.
I have been looking at Hodrick-Prescott filters, and the following seem to be true, but if anyone knows better please tell.
Directly from the definition it appears to me to be a low pass filter with the folowing gain.
G(w) = 1/(1+L*(w/A)^4)
where L is the lambda value and “A” is the number of elements per unit time.
Hence its slope is to 1/f^4, with a cut-off frequency given by:
fc =A/(2*pi*L^(.25))
In this case a=12 (months in a year), and L=12900 giving fc=0.1 cycles/yr.
It does appear to have one very useful property in the the phase lag is zero.
It appears to be have the Buterwortth property of maximal flatness but differs in that G(w) is of the form 1/(1+f(w)) as opposed to the 1/Sqrt(1+f(w)) for the Butterworth filters I know. Also the zero phase lag is not typical.
At first sight it does not seem to be a real time realisable filter in that the minimisation process that describes the HP filter is over the entire series and would require foresight to impliment.
I also suspect that it generalises into a range of filters in a fashion similar to the Butterworth family.
That said it basically is a moderately steep low phase filter. In this case it attenuates signals with a periods less than 10 years.
By the time it gets to down to 2 years its gain is down to ~1/600. I expect that this may be why it does not care if it is coupled to a 12 monthly differenced filter or a 1 monthly differenced filter. Either combination will have a pass band somewhere around 10 years and similar performance both above and below that period.
Alexander Harvey
Paul Vaughan,
Sorry to have seemed to ignore you but my last posting was started an age back.
I am no expert on either Hurst or wavelets but both are of interest to me. I do what I can, and it is always good to learn new things even if it does take me a lot longer than it once did.
Thanks for thanks, I am sure we all like to hear that it is not all a waste of time.
To Basil,
As you no longer wish to discuss it and have not given a justification for the particular choice of filter you used to get a H value close to 0.5 I will presume you do not have one.
And you are correct I do tend to stop at the point where an argument seems to be weak if things that follow reference it and do not seem to strengthen it.
I have moved on to HP filters which as I said I was not familiar with. Why should I get overly concerned with a section that I was not equiped to reason about.
I have posted a breif overview of how I see the performance of HP filters above which may or may not be correct, either way I expect, or at least hope, that it will interest others, maybe even your.
Hopefully that post and my long post on various filters will be leason to the circumspect.
Once I have done some more background, I will see if I can implement HP filtering and see If I get the same results.
Now you may or may not be interested in the nuts and bolts of what filtering does to signals; I certainly am.
Anyway I look forward to figure 3, hopefully I am saving the best to last.
Alexander Harvey
Basil,
I do like a bit of number-crunching, but I can’t help thinking that you are not using the best technique for determining whether or not global temperature follow a random walk. As I understand it, the Hurst exponent is most useful for detecting long-term predictability in a time series. While that means that there are implications for what you would calculate as the Hurst exponent for series that follows a random walk, it does not follow that it is a particularly powerful (in the statistical sense) for detecting random walks. Why use binoculars to look at bacteria when you have a microscope in the lab? To me, an obvious example of the microscope we have to hand is the Phillip-Perron (PP) unit root test. Since a random walk has a unit test, if this test allows you to reject the hypothesis of a unit root, you can dismiss the hypothesis of a random walk. According to my calculations, the seasonally differenced series has a PP test p-value of 0.01 (the p-value for the original series is also 0.01), so I would suggest that we can reject the random walk hypothesis with 99% confidence.
Jose Sanchez (15:50:07) :
It is not misinformation; CO2=AGW is misinformation. No one has falsified the theory of natural climate variability. The CO2=AGW hypothesis was an attempt to show that the natural warming of the planet is caused by increased carbon dioxide. That hypothesis has taken a couple more torpedoes with E.M. Smith’s demolishing of any unusual warming, and CO2 as a cause of anything but increased agricultural yields.
It’s best to be careful with words: CO2=AGW is not a theory. It is a hypothesis [actually more of a conjecture], and it is unable to predict the climate. The long accepted mainstream theory, which makes reliable predictions, is the theory of natural climate variability. Here and here are two good references.
Dr Jose Sanchez (15:50:07) “[…] journalists around the world are using this website as a source to spread misinformation about the theory of global warming.”
–
Dear, Dr. Sanchez,
Is that what you really think?
WUWT is a place where people come to discuss the complexities of natural climate variation.
Unlike other climate forums, participants here are not barred simply for being non-alarmist. If journalists are looking here it is probably in-part because the site welcomes a variety of comments about natural climate variation. On other climate sites you can’t reliably get honest, factual comments past moderation; if & when that changes, the journalists you are so worried about might have more potential sources of information.
If some of the comments here have been offensive, that is regrettable. Anthony & the moderators are being extraordinarily tolerant of nut-job alarmists posing as nut-job “deniers” to deliberately make WUWT look bad.
Do you have any interest in discussing natural climate variation? If so, please join us frequently. Feel welcome to ignore the partisan comments and focus on the complexity of nature. I think you will find there is lots of work to do – and that it is very interesting. Nature is far more beautiful & complex than any anthropogenic computer fantasy.
Regards,
Paul Vaughan
Ecologist, Parks & Wilderness Advocate
Re: Alex Harvey (17:29:13)
If you decide to learn wavelets, I recommend this site as an efficient starting-point:
http://www.ecs.syr.edu/faculty/lewalle/tutor/tutor.html
You have to be careful when calculating residuals from filtered data. I have not looked at Hodrick-Prescott filters specifically but looking at the math, it appears likely that it like most filters, induces a time lag in the data. This lag must be taken into account when calculating the residual i.e. you must back shift the filtered data by the lag so that the filtered output aligns in time with the input.
The easiest way to determine the lag is to drive the filter with an impulse and observe the delay (in time steps) before the output maximum appears. This may be a fractional time step in which case, interpolation is required to properly do the time alignment.