Uh oh, a significant error spotted in the just released IPCC AR5 SPM

Same same this time around: 95% probability does not equal confidence interval. Unscientific jiggery-pokery!
“Probabilistic estimates of quantified measures of uncertainty in a finding are based on statistical analysis of observations or model results, or both, and expert judgment2.”
2 In this Summary for Policymakers, the following terms have been used to indicate the assessed likelihood of an outcome or a result: virtually certain 99–100% probability, very likely 90–100%, likely 66–100%, about as likely as not 33–66%, unlikely 0–33%, very unlikely 0–10%, exceptionally unlikely 0–1%. Additional terms (extremely likely: 95–100%, more likely than not >50–100%, and extremely unlikely 0–5%) may also be used when appropriate. Assessed likelihood is typeset in italics, e.g., very likely (see Chapter 1 and Box TS.1 for more details).

When the IPCC pulls numbers out of their collective behinds this is what happens. I am praying their credibility will be shot sooner rather than later.

Forgot to post the URL for the shrinking atmosphere article…
http://artsandsciences.colorado.edu/magazine/2010/08/shrinking-atmosphere-linked-to-low-solar-radiation/

The Belgian news is still on the CAGW line. In the Netherlands there was at last more nuance. Marcel Crok pointed out in a very gentle way that the models don’t fit and the warming could be far less than assumed, whereupon an AGW supporter made the remark that the warming disappeared in the oceans. They don’t realise that models failing to predict the actual conditions, predicting for the year 2100 is scientific blasphemia.

My guess is Julia Slingo will not respond or acknowledge any problem, that is her role!

Over a year ago we had the SREX IPCC report that said.

March 2012
IPCC Special Report on Extreme Events and Disasters:
FAQ 3.1 Is the Climate Becoming More Extreme? […]None of the above instruments has yet been developed sufficiently as to allow us to confidently answer the question posed here. Thus we are restricted to questions about whether specific extremes are becoming more or less common, and our confidence in the answers to such questions, including the direction and magnitude of changes in specific extremes, depends on the type of extreme, as well as on the region and season, linked with the level of understanding of the underlying processes and the reliability of their simulation in models.
http://www.ipcc-wg2.gov/SREX/images/uploads/SREX-All_FINAL.pdf

Recently we had the draft Summary for Policymakers.

since the mid-20th century, and likely that human influence has more than doubled the probability of occurrence of heat waves in some locations (see Table SPM.1). {10.6}
http://www.climatechange2013.org/images/uploads/WGIAR5-SPM_Approved27Sep2013.pdf

Since a linear trend model plus error has least squares estimators that are asymptotically nomal, one standard error of the trend margin is about .2 /1.645 degrees C, or on order of magnitude 0.15. So 0.85, the quoted trend estimate, has a Z-score of about 5+, which is way more significant than 95, 99%, 99.99%, etc…. Seems to me the authors are downplaying the evidence. While I have not read the report, I wonder what is Doug arguing? That the AR(1) model is inappropriate? Sure. But statistics has advanced way beyond this the last few decades. Elaborating, put in a fractionally differenced long-memory component in the model if you want. You’re not going to knock that z-score below significant. Please reinforce your argument.

Weather forecasts use the same approach. If three weathermen say it will rain and one disagrees, there’s a 75% chance of showers.

IPCC news flash ,the global temperature trend going forward is going to be DOWN, not up.
Natural causes can explain the temperature rise from 1880-1998 from high solar activity, to a warm PDO post 1980, up through 1998, featuring more El Nino activity.

Summary for ScareMongers — 100 Ways to Spread Scare Stories and Make a Million!

Nick,
“Would you care to quote Julia Slingo saying AR(1) was rubbish?”
“However, considering the complex physical nature of the climate system, there is no
scientific reason to have expected that a linear model with first order autoregressive noise
would be a good emulator of recorded global temperatures, as the ‘residuals’ from a linear
trend have varying timescales ”
You could have found that yourself. Doug linked to it.
“But the IPCC is not claiming that linear+AR(1) is the best model of temperature. They are simply using it as the basis for calculating temperature change over the period.”
They’re claiming that these are 90% confidence intervals on the actual temperature rise. And implying, to a statistically non-literate audience who wouldn’t recognise the issues with the AR(1) choice, that they would be justified in thinking these have a 90% probability of covering the value that is being estimated. (Which from a Bayesian point of view is not true, either. That would be a ‘credible interval’, not a ‘confidence interval’. A common error, that.)
I know there’s this thing about “not giving ammunition to sceptics”, but wouldn’t it be simpler, more straightforward, and a lot less desperate, on hearing that the IPCC was basing its confidence intervals on an linear+AR(1) model, to simply say: “That’s wrong; they ought to have either picked a better model, explained the difficulty, or not given confidence intervals at all”?

Whatever happened to 1850 to the present??
Or is that Inconvenient??
regards.

“It’s used as a basis for computing the difference between temperatures at two times.”
And reporting confidence intervals. It’s the confidence intervals that are the issue.
And it isn’t a computation of the difference in temperatures at two times. To do that, you would subtract the temperature at one time from the temperature at the other. It’s a much simpler process. What you’re trying to do is something a lot more complicated – by “temperature” you don’t mean the temperature, but an underlying equilibrium temperature due to forcing that has short-term weather superimposed on top of it – a purely theoretical concept that assumes that’s how weather works. You’re trying to estimate the change in the underlying equilibrium, and using a low-pass filter to cut out the high-frequency ‘noise’ – a process that requires accurate statistical models of both signal and noise to do with any quantifiable validity.
The mainstream constantly conflate these two concepts – the observed temperature and the underlying equilibrium temperature – because it gives the impression that the statements are about direct empirical observation, while actually being about an unobservable parameter in a question-begging assumed model.
Had they simply given the OLS trend, you could have argued that it was merely informally descriptive, a rough and unscientific indication of how much temperatures generally had gone up, without making any comment on its significance. However, they stuck a confidence interval on it. Worse, they said there was a 90% likelihood of it covering the quantity being estimated. That gives the impression of a scientifically testable statistical statement. But the “confidence interval” here is a meaningless pair of numbers, because it relies for its validity on an assumption known not to be true.
“Regression fits are used for this in all kinds of fields, and they work well.”
Sadly so. That doesn’t make it right, though.
You might well know what they’re doing and that such estimates are to be treated cautiously, but the intended readers of this report don’t. They read it as authoritative science, and if they see confidence intervals being written down, by scientists, they’re going to assume they’re meaningful. In this case, as in so many others, they’d be wrong.

Not entirely meaningless, but surely deceptive. It is written in the Summary in such a way as to create a false impression. What it means is not stated clearly. SOP.

“I agree that AR(1) is not the only basis for calculating confidence intervals, and there is a case for others (discussed here). But it’s not meaningless.”
AR(1) is the wrong basis for calculating confidence intervals.
It’s meaningless if it’s based on an untrue assumption. Policy makers need to know how accurately you can state the amount of global warming observed. This does not answer that question.
And the IPCC didn’t fully explain the basis on which it is calculated – that’s something we had to deduce from what they did last time around (and buried in an appendix to the main report), and the fact that the interval they report this time matches that method. What the IPCC say is that we can be confident there’s a 90% likelihood that this interval covers the amount of global warming there has actually been. That’s not true.

The 0.65 and 1.06 °C figures are obviously P.O.O.M.A. numbers. [That stands for Preliminary Order of Magnitude Approximation. Really it does.]

To me, this is just more indication that the 95% number claimed by IPCC wasn’t derived mathematically …

Maybe, but if so, it was at least the result of impeccable risk assessment logic:
The clients must receive what they specified. Otherwise they will defund the project.

“I would say: An ARIMA(3,1,0)? Surely you jest in saying that is a thousand times more likely? I would sure like to see that likelihood comparison.”
Follow the earlier Met Office discussion at Bishop Hill. There are links back to Doug’s calculations, which Slingo confirms.
“Do prove me wrong, but the model you propose has a random walk component, meaning the variance increases linearly in time. That is clearly not the case with this data. What you propose isn’t even a stationary model, which should be the null hypothesis of any climate change argument.”
It’s an approximation for a subset of data, like a linear trend is.
It’s a standard procedure in time series analysis – if there are roots of the characteristic equation very close to the unit circle, it makes any short enough segment of the series look approximately as if it was on the unit circle (i.e. random-walk-like), and a lot of the standard tools don’t work or give invalid answers. So the standard approach on analysing a new time series is to first test for unit roots, and if “found”, take differences until the result is definitely stationary. It’s an approximate measure to handle situations when you don’t have a long enough sample to fully explore the data’s behaviour, and to avoid getting misleading results because of that.
Think of it as like the situation you get with the series x(t+1) = 0.999999999 x(t) + rand(t) where rand(t) is a zero-mean Gaussian random number series. Technically it’s AR(1) and stationary, but over any interval short of massive it’s going to look indistinguishable from x(t+1) = 1 x(t) + rand(t), which is a random walk. You don’t have enough data to resolve the difference.
Usually, after testing for unit roots and taking differences, the next step is to test to find what ARMA process best fits the result. This is where the ARIMA(3,1,0) model came from – it is the ARIMA process that best fits the short-term behaviour of the data. The process is analogous to fitting a polynomial to a short segment of a function to model its curves. It’s a local approximation that is not expected to apply indefinitely.

http://stevengoddard.files.wordpress.com/2013/09/screenhunter_1013-sep-28-00-13.jpg

Maybe this is too simplistic of a way to look at it, but say you have a system with several subsystems, each of which you are 99% confident that you have a sufficient understanding to model accurately. If there are 6 or more of these subsystems, is it possible for you to be 95% confident of the accuracy of your model of the entire system? 0.99^6 = 94.1%
Given that the climate has well over six subsystems, few of which if any, we are 99% confident that we can model accurately, let alone any interactive effects, it would seem nonsensical simply from a probability standpoint to claim 95% confidence.

Nullius in Verba:
You make some interesting comments about the suitability of various statistical methods to be used on different occasions.
Could you state what you currently feel would be the best sequence to use to analyse the various global temperature datasets?
I would really like to know what a person skilled in statistical analysis would say is the correct method or sequence to use.
Its a shame that most statistical methods give answers irrespective on whether the method should have been used in a particular case.

“Could you state what you currently feel would be the best sequence to use to analyse the various global temperature datasets?”
The primary problem, like I said, is the lack of a validated model for the background noise. This is where effort needs to be concentrated. Until we have one, none of the methods are going to give reliable answers.
More sophisticated studies in detection and attribution (‘detection’ is what we’re looking at here) use the big climate models to generate statistics on the background variation. There’s a bit more reason to pay attention to these, since they are at least partially based on physics. But there are lots of approximations and parameterisations and fudge-factors galore, they’re not validated in the sense required, they don’t match observations of climate in many different aspects and areas, and their failure to predict the pause (or rather, pauses of a similar length and depth) falsifies them even at the game they were primarily designed and built to play – global temperature.
Because they’re not validated, the fact that they can’t produce a rise like 1978-1998 doesn’t mean anything. But because they’re not validated, the fact they can’t produce a pause like 1998-2013 doesn’t mean anything either, except that one way or another the models are definitely invalidated. The pause doesn’t show that global warming theory is wrong, because it could just be that the models underestimated the natural variation.
If, one day, they can build a climate model that can be shown to predict climate accurately over the range of interest, then the correct approach would be to use this to generate statistics on trend sizes over various lengths, and use that to perform the trend analysis and confidence intervals and so on. That would be the right way of doing it. But we’re not there yet.
“I would really like to know what a person skilled in statistical analysis would say is the correct method or sequence to use.”
Thanks! But I’d only describe myself as ‘vaguely competent’ not ‘skilled’. There are a lot of people far better at this stuff than me!

@ dbstealey September 28, 2013 at 12:29 pm)
We’re dealing with kamikaze suicide-bomber types. Status quo daily wuwt operations alone can’t and won’t stop such a viciously aggressive force of nature.

Sorry for the double post. I felt for clarity (particularly wrt Brandon’s question) I should quote from Cohn and Lins’ conclusions:
[…] with respect to temperature data there is overwhelming evidence that the planet has warmed during the past century. But could this warming be due to natural dynamics? Given what we know about the complexity, long-term persistence, and non-linearity of the climate system, it seems the answer might be yes. Finally, that reported trends are real yet insignificant indicates a worrisome possibility: natural climatic excursions may be much larger than we imagine. So large, perhaps, that they render insignificant the changes, human-induced or otherwise, observed during the past century.with respect to temperature data there is overwhelming evidence that the planet has warmed during the past century. But could this warming be due to natural dynamics? Given what we know about the complexity, long-term persistence, and non-linearity of the climate system, it seems the answer might be yes. Finally, that reported trends are real yet insignificant indicates a worrisome possibility: natural climatic excursions may be much larger than we imagine. So large, perhaps, that they render insignificant the changes, human-induced or otherwise, observed during the past century.

Dudley, very good, yes it is repeated once 🙂
Paul, indeed, that is the null hypothesis (although some want to reverse it…)

Lund@clemson.edu says:
September 29, 2013 at 3:52 pm
This is total gibberish. Your words from previous post:
“Do prove me wrong, but the model you propose has a random walk component, meaning the variance increases linearly in time. That is clearly not the case with this data.”
There is no such “clearly”. A sample series of a random walk does not grow more noisy as you move away from the origin. It simply wanders from the origin, with 1-sigma bounds growing as the square root of time. There is no way to identify by inspection whether this data series has such a non-stationary component as it wanders quite a bit. So clearly, your assertion, that it was “clearly not the case”, is wrong.
“All I’m trying to tell you is that if one accounted for the longer memory issues in the series than AR(1) models (yeah, one can do better than AR(1)), you still wouldn’t get different conclusions.”
Incorrect. The PSD of the detrended temperature series looks like this. There are significant AR(2) processes with energy concentrated at frequencies associated with periods of about 65 and 21 years. You account for these, and I guarantee your conclusions will be wildly different.

“Were those numbers calculated, or just pulled out of some orifice?”
Not constructive.

Bart said “There is no such “clearly”. A sample series of a random walk does not grow more noisy as you move away from the origin. It simply wanders from the origin, with 1-sigma bounds growing as the square root of time.”
Bart, you are very wrong. Your first two sentences are complete contradictions.
If X_t is a random walk at time t, then
X_t = X_{t-1}+Z_t, t >= 1 , with X_0=0
and Z_t is some IID noise. So X_t = Z_1 + Z_2+ …. + Z_t and the variance of X_t = t times the variance of Z_1. I think you have been told this!!!! Standard deviations are the square root of the variance.

At the end of the day the “global’ climate is simply the sum of its factors – an equation. Some of the factors we know, or think that we know, and can attribute a value to them. There are many similar factors that we know, or think that we know, must feature, but to whose value we can only guess. We know that other factors exist, but we do not know how many and can attribute no weight to them. The equation is, for the moment unsolvable, It follows that any attempted solution, no matter how esoteric the statistical manipulation, can be allocated a very precise confidence level, and that level is Zero.

Now the discussion is getting somewhere.

(Note: This is the latest fake screen name for ‘David Socrates’, ‘Brian G Valentine’, ‘Buster Brown’, ‘Joel D. Jackson’, ‘beckleybud’, ‘Edward Richardson’, ‘H Grouse’, and about twenty others. The same person is also an identity thief who has stolen legitimate commenters’ names. All the time and effort he spent on writing 300 comments under the fake “BusterBrown” name, many of them quite long, are wasted because I am deleting them wholesale. ~mod.)

Nullius in Verba says:
September 30, 2013 at 2:17 pm
I am speaking of a finite random walk segment as well. Informally, in engineering applications, we usually model such a process as the sampled output of an integrator driven by wideband (effectively “white”, over the frequencies we are interested in), zero mean noise (Wiener Process). We generally assume the input noise is Gaussian which, due to the Central Limit Theorem, is generally a fairly safe assumption. If the input noise has an RMS measure of “sigma”, then the standard deviation of the random walk relative to the initial value grows as sigma * sqrt(time).
The point, though, is that the samples of the random walk are highly correlated in time. If x(t_k) is the process at kth sampled instant t_k, then the autocorrelation function is E{x(t_k)*x(t_n)} = sigma * min(t_k,t_n). This is not independent noise which flails wildly between the RMS bounds, but rather a slowly evolving trajectory which “walks” up and down over time. Such trajectories typically look like the sample series I plotted in the upper plot here. The standard deviation of sigma * sqrt(time) is a measure of the RMS spread over an ensemble of such processes at the given time, but not of a single sample of such a process.
If you filter out the processes in the temperature record associated with what I suspect is an oceanic-atmospheric resonance at ~65 years, and the ~21 year process which is probably associated with solar cycles, then you are left with something which looks very much like some of these random walk trajectories. There’s no way you can see this by inspection of the raw data. Hence, Lund’s claim that it was “clearly” not there was puzzling. I think it is possible that he momentarily forgot how random walks behave, expecting something more akin to the lower plot at my link above. At least, that was the impression under which I made my comments.
On the other hand, maybe he was saying that any such behavior was clearly not dominant. I can agree with that. Or, maybe he was saying that, as a random walk tends to infinity with time, there clearly cannot be an open ended accumulation over eons of time. I can agree with that, too. But, the answer to that, as you pointed out in a post above, is that over a finite interval of time, a true random walk can be indistinguishable from a stationary low frequency process which is ultimately bounded over time.

lund@clemson.edu
Yes, a fractionally integrated solution would be correct. I linked you to a peer-reviewed paper that showed when accounting for the fractionally integrated component, the 95% confidence intervals expand to include zero. I told you to read that paper and warned of the dangers (and difficulty) in estimating this value.
In summary: you claimed using such a model would have no effect, and I linked you to a peer-reviewed paper that does the calculation that shows it has a very large effect.
And without answering the scientific points I raise, you announce you are dropping out of the discussion at this point.
What a surprise.

Spence_UK says:
October 1, 2013 at 6:15 am
“Bart, the “quasi-periodic” cycles you describe are almost certainly a consequence of fractionally integrated random variations, and not deterministic cycles at all.”
They are definitely not deterministic cycles. I never suggested they were. But, they are very likely representative of ordinary resonant systems driven by random excitation.
The ~21 year cycles are likely due to quasi-periodic variations in solar activity. The Hale cycle has a nominal period of about this duration corresponding to oscillation of magnetic polarity (now that I think of it, this might be a manifestation of Svensmark-type cosmic ray modulation). A model for the sunspots, which displays characteristics very similar to observations, can be found here. See, for example, the simulations here and here, and compare to actual sunspot data here. Note, I am pointing out qualitative similarity, not quantitative replication. That analysis is TBD.
Partial differential equations on bounded domains typically produce solutions which can be expanded in a series of normal modes. Random excitation of these normal modes, with some inherent energy dissipation leading to damping, can be represented using a system model such as proffered above for the sun spots. This is standard operating procedure in, e.g., modeling structural vibrations, and is very widespread and well-established.
IMO, the ~65 year process is very likely such a normal mode excitation of the oceanic-atmospheric system. The random excitation doesn’t even actually have to be wideband – just nominally stationary with repeatable energy levels at the resonant frequency.
“Unfortunately you plot your PSD on a linear scale.”
It is better for seeing the spikes indicative of the resonances. A log-log scale is useful for observing power-law types of noise, but not very good for picking out resonances, as the varying scale distorts the Cauchy peaks.

Spence_UK: I must apologize. Your cited paper is certainly relevant. I didn’t read it the first two times you mentioned it. Perhaps I was misled by a lot of the misinformation here. I don’t normally look at this page. You have to yell pretty loud to be heard over all the noise. There is so much to correct.
But your reference contains exactly the type of analysis that I would think is hard to refute. The ARIMA(3,1,0) stuff is going the wrong way. While, I have not read your citation in detail, if the fractionally differenced p-value is .07, I would say okay, that increased the p-value way way more than I expected for the value of d quoted. This is not quite the case for the Continental US Series, which I have examined. I’m going to guess there are specifics (yearly instead of monthly series, interval of observation (does it contain the last few years, etc.) to dicker around, but I believe you.
BTW, I am still here. I just don’t have time to reply to everything.

Lund@clemson.edu
In which case I should apologise back for allowing my frustrations to cause me to question your motivations. Sorry for that. I’m glad you found the paper interesting.

Spence_UK says:
October 2, 2013 at 5:55 am
“The peaks you show are not special in this context.”
I always used to tell my students, you can’t rely on all this fancy math. It’s all based on models, and the models aren’t always applicable. You have to dig down into the actual data and do sanity checks.
If you can really look at this plot and not see the ~65 year component blazing in your eyes, then you need to take a break, and drop down to your local pub for a glass of perspective and soda.

http://stevengoddard.files.wordpress.com/2013/09/screenhunter_1013-sep-28-00-13.jpg
Add a bit of explanation and it makes a great tee-shirt sweatshirt.

Lund – charming. You need to learn that there are other conventions in other disciplines than the one in which you are engaged, and expand your mind.
“Learn what a random walk is.”
If you have an objection, state it clearly. My definition is standard, and consonant with descriptions widely available on the web.

Spence_UK says:
October 2, 2013 at 1:47 pm
” …all fractionally integrated time series will show strong “quasi-periodic” cycles near to the length of the time series.”
But, not all quasi-periodic cycles are phantoms of a power law process. Furthermore, you appear to be glossing over the fact that the ~65 year quasi-periodic cycle is half the length of the modern instrument record. There are very nearly two full cycles in evidence here.
This argument has been going on here longer than you may realize. Back in the early 2000’s, many people noticed that there appeared to be a ~65 year cyclicality to the temperature data. In particular, the run-up in temperatures from about 1910-1940 was almost precisely the same as that between about 1970-2000.
Even greater force was given to the argument when the turning point came roughly in 2005, right on schedule.
Fractionally integrated noise is a Martingale – the future does not depend on the past. These two events, the replication of the 1910-1940 and 1970-2000 increases and the turnabout in roughly 2005, would be astounding coincidences under the assumption that this is a figment of fractionally integrated noise.

Bart,
It is easy to see patterns in fractionally integrated noise, especially when you get to interpret those patterns post hoc.

Put Spence my team,dawg.. Bart, ???????, this is out of control.
[“on” my team? “in” ? Mod]
(Another fake identity -mod)

Bart says:
October 2, 2013 at 3:06 pm
(This sockpuppet can’t keep his screen names straight. -mod)
[1. The repeated STFU’s in repeated replies are not needed, nor useful, nor desired. Cut them out.<
2. What did you mean by “for fBm to exhibit repeating coh” ? Mod]

carol@stat.usc.edu says:
October 2, 2013 at 5:39 pm
If you are confused, perhaps you should ask actual questions. I am not equipped to interpret “?????”.

(Fake name. -mod)

I doubt this doofus calling himself “Lund” is actually Dr. Robert Lund of Clemson. Nobody granted such a position would make such a fool of himself, unless things at Clemson are even worse than most people assume.
So, for whatever person who has appropriated his title, before you say something even stupider like “what does the frequency response of an integral have to do with the frequency content of a random walk,” I will again point out that a Gaussian random walk is equivalent (and often the result of) sampling the output of an integrating process fed by wideband noise, in the limit as that input noise bandwidth approaches infinity, i.e., the standard Wiener Process.
Random walks look like the top plot here. Or, the plot here, for a non-Gaussian case. As is plainly evident, higher frequency motion is attenuated in each of these cases, in the latter because discrete accumulation, like continuous integration, attenuates higher frequencies.
This property is also immediately evident from the autocorrelation function, which I provided previously for the sampled data case: E{x(t_k)*x(t_n)} = sigma^2 * min(t_k,t_n). The cross correlation coefficient of nearby points is sqrt(min(t_k,t_n)/max(t_k,t_n)), which will be near unity when t_k is near t_n. That means the points tend to stay in the same neighborhood for an extended time, and fail to jump around significantly in narrow time intervals, i.e., their frequency content is weighted toward the low frequencies.
But then, this is obvious in the power law of -2 which such a process produces in a PSD estimate. I mean, this is really, really basic stuff.
So, guys… you’ve embarrassed your institution enough. How about you run along and play somewhere else now.

Robert Lund says:
October 3, 2013 at 8:55 am
“You at least have the covariance function of a random walk right:”
I had it right when you were still in diapers. I stated it well before this comment when you had first entered in. Apparently, you were in a blood frenzy, looking forward to tearing into someone who didn’t share your viewpoint, so you glossed over it.
“That is the whole premise on why a random walk, and hence why an ARIMA(3,1,0) model is inapprorpiate.”
It’s a flawed premise. Over a finite interval, AR processes with long time constants can behave essentially like a random walk. If the aim is prediction in the near term, there is nothing wrong with it.
“It torques me that you’ve insulted my math skills and my university. Really.”
It was intended to. It torques me that you paid so little attention to the things I stated, and made to ridicule me based on your incomplete knowledge and lack of experience outside your narrow field. Has the lesson been learned?

Robert Lund says:
October 3, 2013 at 10:27 am
[trimmed. Mod]
Do you even know how to derive the frequency response of a discrete AR system? Have you ever even heard of the Z-Transform? Do you have any idea how engineers design digital control systems? Do you know what a time constant is?
[trimmed. Mod]

Robert Lund says:
October 3, 2013 at 11:22 am
“Do I know what you are trying to say? Not in the slightest.”
Then, ask questions, instead of accusing me of not knowing what I am talking about. If you had asked nicely to begin with, we wouldn’t have had all this nastiness.
A simple example may help. Consider the difference equation
x(k+1) = 0.999*x(k) + w(k)
where w(k) is white noise. The time constant of this system is -T/log(0.999) = 995T, where T is the sample period. If you look at the output of this over a time less than a time constant, it will be very close to a random walk.

Bart,
Respectfully, this whole thread is about the nuances between a random walk, a fractionally differenced random walk, and a causal AR(1) model. Some of us here have decades of proficiency with the topic. I truly don’t know what you mean to say most of the time. Like in the example above, there is no period T. An AR(1) with autoregressive coefficient of .999 is stationary.
Here’s the deal: an autoregression with an autoregressive polynomial that has roots close to the unit circle will exhibit more persistence than one with roots far from it. Often, a sample path of such a series could be mistaken for a random walk. What we are trying to tell you is that there is something in between an AR(1) and a random walk, dubbed an ARFIMA model, that is the most appropriate error model here. And its stationary (a random walk is not).
Look, I’m sorry my time series class picked on you. It was a bad idea to tell them about this thread. Can we move on?

because this fBm obviously would have Hurst coefficient > 0.5, and so would tend to keep going in the same direction it was currently going for an extended time.
No, that is not how fractionally integrated noise behaves.
I’m all for making predictions from models, but you need to ensure that (1) you understand how the models actually behave and (2) your predictions need confidence intervals. Without these, predictions are worthless.

Bart, just look at the example plots (H=0.75, H=0.95) in the wiki article you just linked to. You will notice those plots have many local minima and maxima.
You know what local minima and maxima means? It means that if the current step goes up, the following step might just go down.
Long term persistence (H>0.5, H<=1.0) has a defined population mean, and although it can spend arbitrary periods of time to one side of that mean, it does *not* follow that the current step direction is dependent on the previous one
Also note that the change in step constitutes a change in the first derivative. Note that differentiation is the equivalent of dividing the power spectral density by a value linearly proportional to f. Since the power spectral density of LTP is 1/f, the first derivative of a long term persistent can be white, i.e. independent.
Demetris Koutsoyiannis has a nice term of phrase for this. He notes that the expression "long memory" often associated with long term persistence is a misnomer. In fact, the behaviour of long term persistent series is not one of memory, but much more one of amnesia. But the error you make is a common one.
If you want to see an example of a prediction based on fractionally integrated noise, I recommend you look at UC's blog here:
http://uc00.wordpress.com/2011/08/30/first-ever-successful-prediction-of-gmt-3-years-done/
UCs prediction is based on half integrated noise, back in 2008. Please note the central prediction dips slightly downward, conversely to your incorrect understanding. FWIW my instinct is that his confidence intervals are too narrow, but the presence of CIs allows his prediction to be tested. Please note UC’s recommendations on making predictions.
When your prediction rises to this standard, we can see if you can do as well as UC’s did.

Spence_UK says:
October 5, 2013 at 1:24 pm
“Note that differentiation is the equivalent of dividing the power spectral density by a value linearly proportional to f.”
I assume you were in a hurry, but I believe you meant to say “multiplying”, and differentiation multiplies the PSD by f squared.
“…the random walk does not have a defined population mean, whereas fractionally integrated noise does..”
I think we are possibly speaking of two different things, and need to more carefully delineate them.
First of all, it is not true that “the random walk does not have a defined population mean”. The expected value of a random walk, specifically the accumulation from zero of zero mean independent increments, is in fact zero. It is the excursion from zero which is expected to increase with time.
I suspect the property you were referring to was stationarity. But, fBm is not stationary. I am not sure about the process you and Dr. Lund refer to as “fractionally integrated noise”, but both you and he seem to maintain that it is stationary. Frankly, I do not see how this can be when the spectrum appears to still have a singularity at zero, but I have not looked closely at this yet.
In any case, if you have a difference of opinion with the Wikipedia page to which I linked, perhaps you should sign up as an editor and make your disagreement known.

Spence_UK says:
October 5, 2013 at 5:26 pm
“You will notice those plots have many local minima and maxima.”
Yes, but the point is what is likely. Sooner or later, they are likely to switch direction. The question is, how long do they tend, on average, to go largely in one direction or the other before switching to go the other way? I specifically do not mean every bump or bobble which switches directions, but the longer term, quasi-trends. This is largely determined by how strongly succeeding points are positively correlated on a particular timescale.
“But the key point here is that persistence does not result in the property that you claim exists (that the direction of the current step is tied to the previous step).”
I don’t think I see that, at least not yet from the point of view of this particular argument. As I mentioned, the differentiated PSD is weighted by the frequency squared, so you cannot get a flat spectrum except in the particular case of random walk. It is indeed true that a random walk is expected neither to go up nor down based on where it was previously heading – it is expected to stay the same because it is a Martingale – but that is the special case of H = 1/2. If you differentiate a process with H > 1/2, you still end up with a downward slope in the PSD, which suggests continued long range positive correlation.
IIRC, one of the links you presented previously estimated H = 0.92, so it appears to me there is still, from this point of view, long range correlation which is significantly positive for an extended interval within the temperature series. This interpretation appears to jibe with what the Wikipedia excerpt to which I linked was stating.
Getting back to the PSD question, one of the reasons I have always been bothered by descriptions of fBm is that it is generally rooted in these power law PSD descriptions. But, the PSD is not really even defined for non-stationary processes, so all we are seeing in PSD estimates is basically a 1-d projection of a 2-d entity. It’s a little like creatures of Flatland trying to work out what 3-d creatures look like based on the various cross-sections they observe. Now we, as 3-d creatures, could do that with enough cross sections, but the Flatlanders have no conception of the 3rd dimension, and so can never visualize it within their sphere (or, circle) of comprehension. Similarly, we have no widely utilized 2-d analysis tool of which I am aware which would allow us to fully comprehend what is going on in every case.
I see, e.g., no reason that the processes which produce 1/f signatures in PSD estimates should have a unique, all-encompassing description. Many non-stationary processes can produce approximately 1/f behavior when processed through a PSD estimation routine. I think better tools which observe the full dimensionality are needed. I could say a bit more on this topic, based on some of my memories of trying to hash out such an approach back when I was studying these topics, but the memories are a bit faded, and it would probably be difficult to get across the concepts in this venue. It had something to do with 2-d Laplace transforms, but that is as far as I can go into it at this moment, for whatever it is worth.
“Secondly, fractionally integrated time series … can be stationary processes.”
Hmm… that’s a little less general than your previous statement. I will have to read your link, and reaquaint myself with these things to either find out precisely what you mean, or ask questions which would help elucidate it. I doubt that will happen before this thread gets closed, so maybe we will take it up again at a later time.
But, again, I still believe that the two coincidences in the temperature data set to which I have referred previously indicate that there is a resonance involved, rather than just a random fractal drift, and this indicates that it is likely that temperatures in the next few decades will behave similarly to the era between roughly 1940-1970. I was hoping to make the question moot from your point of view, but it appears that sort of weak agreement will not happen on this thread.
But, I appreciate our civil discussion, as much as I regret the ugliness which passed between myself and Dr. Lund. If the clock runs out on us before there is time to say so, thanks for your time and insights.

Spence_UK says:
October 6, 2013 at 11:10 am
“Such is the nature of this debate.”
And, the nature of the internet, which suspends the normal bounds of propriety we observe when dealing with one another directly.
“When you put white noise through this, you get greater amplitudes at high frequencies than low. This means that we should expect differences to reverse… Flicker noise is more likely to reverse direction than continue its current direction, although the probability is close to 50% so it takes a reasonable number of samples to confirm this.”
But, doesn’t flicker noise have H < 0.5, which is the boundary noted in the Wikipedia article? So, isn't your finding actually consistent with this?

OK, apparently my interpretation of H is not quite right. This particular branch of stochastic processes has not been my bag for… decades. But…
“So I ran a short procedure in MATLAB to artificially generate flicker noise, and tested the probability that a positive step would be followed by a positive step.”
What we really want is not the conditional expectation of one sample at a time, but of many. What is the likelihood of an overall trend slope being, say, positive in the future, given that it has been positive in the past? And, what timelines are associated with the past trend, and the projected one?
I do not really have the time to formulate a precise statement of the question I am trying to ask which can be tested. But, given that the correlations are all positive, I suspect that some measure capturing this inchoate thought might well prove the Wikipedia statement correct in some sense. I am, at least, predisposed to believe that the author of the statement had something upon which to base it. Even if he was wrong, I don’t think we can make the determination until we know precisely what he meant.

Another note – the correlations are all positive, but that is with respect to the absolute values, not the relative change in value (step). That is, if one value is above the population mean, then the next is likely to be also above the population mean. That does not mean the rate of change will be.

Bart, sorry but your algebra is doing something odd. The first expectation operator is the normalised autocorrelation coefficient. But the second expectation operator you have used as if determining the expectation of the series itself – these are two completely different operators, and you’ve used them interchangeably.
As a result, your second expression is just a summation of autocorrelation coefficients, which tell us very little of interest. I agree the autocorrelation coefficients of fractionally integrated time series are all positive, but your conclusion about step of the data itself does not follow.
I don’t have much time right now but it is trivial to give an example to show your expression is wrong. Consider white noise, H=0.5. Your expression says E[(x(2)-x(1))*(x(1)-x(0)) should be zero. I can test this easily in MATLAB (or excel, or any other tool which can draw independent normally distributed random numbers):
>> x = randn(10000,1);
>> x2 = x(3:end);
>> x1 = x(2:end-1);
>> x0 = x(1:end-2);
>> mean((x2-x1).*(x1-x0))
ans = -0.9610
I did several more draws, and they are all around -1, within about 0.1, which is what I would expect (white noise necessarily tends to reverse the sign for the reasons I gave above). But your expression claims the expected value is zero. Why? Well the autocorrelation coefficients of white noise are zero, since the samples are independent. Your strange sum of autocorrelation coefficients will also be zero. But the expected value from the series is highly negative.
Sorry Bart, but your expression is simply not correct, and the wikipedia comment is still wrong.

Or, rather
x = [zeros(1,100000) ; cumsum(randn(2,100000))];
mean((x(3,:)-x(2,:)).*(x(2,:)-x(1,:)))
ans =
5.1115e-05

Or, you can do it your way, with a slight change:
x = cumsum(randn(1000000,1));
x2 = x(3:end);
x1 = x(2:end-1);
x0 = x(1:end-2);
mean((x2-x1).*(x1-x0))
ans =
5.0511e-04
though it takes more trials to get the number to generally come out insignificant. Over many trials, it tends to zero.
There appears, then, to be a disconnect between your definition of H and that used on the Wiki page, which may explain why we had different ideas about flicker noise. On the Wiki page, H = 0.5 is the designator for random walk, as is seen in my post previous relating their autocorrelation function for H = 0.5 to the standard one for random walk.

“H = 0.5 is the designator for random walk”
I am being informal. H = 0.5 is actually the designation for Wiener noise on the Wiki page, the sampled-in-time-data version of which is a random walk.

Bart,
H=0.5 is not the designator for a random walk. You are wrong.
The Hurst exponent H is undefined for a random walk.
H=0.5 is for white noise, i.i.d. gaussian.
Your equations are wrong. Full stop.

The same will hold mutatis mutandis when t1 > t2.

I know the equations are right the way I am using them. And, they agree with the Wiki site. That’s 2:1. If you are sure you are right, then I think the only conclusion can be that there is a schism in the conventions being used, and this is leading to confusion and frustration.
Sometimes… strike that… Often, the biggest part of the problem is defining the conventions and making sure everyone is on the same page.

Well, these guys, these, and these, contradict you. Honestly, I don’t care. But, clearly, if you take the autocorrelation function (not coefficient, function) provided on this page and provided in many other references as well, then for H > 0.5, succeeding delta’s are positively correlated, and this should lead to precisely what the article states, that this “dependence means that if there is an increasing pattern in the previous steps, then it is likely that the current step will be increasing as well.”
You haven’t explained why you think they are wrong, you have only asserted it. I do not know why you think there is a principle contained within your assertion which I should be understanding. You have provided no maths. You have provided no alternative autocorrelation function. You have provided no definitions. What I do not understand is why you think I should merely take your word for it that all these sources are wrong, and only you hold the truth. And then, I do not know what you expect me to do with that information once I have accepted it.

Is this autocorrelation function valid or not?
E{x(t2)x(t1)} = 0.5 * ( abs(t2)^2H + abs(t1)^2H – abs(t2-t1)^2H )
If it is, then with H = 0.5, it is equal to the minimum of t1 or t2, which is the autocorrelation function of a random walk.