From the “(pick one: 90% 95% 97%) certainty department, comes this oopsie:
Via Bishop Hill:
=============================================================
Doug Keenan has just written to Julia Slingo about a problem with the Fifth Assessment Report (see here for context).
Dear Julia,
The IPCC’s AR5 WGI Summary for Policymakers includes the following statement.
The globally averaged combined land and ocean surface temperature data as calculated by a linear trend, show a warming of 0.85 [0.65 to 1.06] °C, over the period 1880–2012….
(The numbers in brackets indicate 90%-confidence intervals.) The statement is near the beginning of the first section after the Introduction; as such, it is especially prominent.
The confidence intervals are derived from a statistical model that comprises a straight line with AR(1) noise. As per your paper “Statistical models and the global temperature record” (May 2013), that statistical model is insupportable, and the confidence intervals should be much wider—perhaps even wide enough to include 0°C.
It would seem to be an important part of the duty of the Chief Scientist of the Met Office to publicly inform UK policymakers that the statement is untenable and the truth is less alarming. I ask if you will be fulfilling that duty, and if not, why not.
Sincerely, Doug
============================================================
To me, this is just more indication that the 95% number claimed by IPCC wasn’t derived mathematically, but was a consensus of opinion like was done last time.
Your article asks “Were those numbers calculated, or just pulled out of some orifice?” They were not calculated, at least if the same procedure from the fourth assessment report was used. In that prior climate assessment, buried in a footnote in the Summary for Policymakers, the IPCC admitted that the reported 90% confidence interval was simply based on “expert judgment” i.e. conjecture. This, of course begs the question as to how any human being can have “expertise” in attributing temperature trends to human causes when there is no scientific instrument or procedure capable of verifying the expert attributions.
So it was either that, or it is a product of sleep deprivation, as the IPCC vice chair illustrated today:
There’s nothing like sleep deprived group think under deadline pressure to instill confidence, right?
Bart, sorry but your algebra is doing something odd. The first expectation operator is the normalised autocorrelation coefficient. But the second expectation operator you have used as if determining the expectation of the series itself – these are two completely different operators, and you’ve used them interchangeably.
As a result, your second expression is just a summation of autocorrelation coefficients, which tell us very little of interest. I agree the autocorrelation coefficients of fractionally integrated time series are all positive, but your conclusion about step of the data itself does not follow.
I don’t have much time right now but it is trivial to give an example to show your expression is wrong. Consider white noise, H=0.5. Your expression says E[(x(2)-x(1))*(x(1)-x(0)) should be zero. I can test this easily in MATLAB (or excel, or any other tool which can draw independent normally distributed random numbers):
>> x = randn(10000,1);
>> x2 = x(3:end);
>> x1 = x(2:end-1);
>> x0 = x(1:end-2);
>> mean((x2-x1).*(x1-x0))
ans = -0.9610
I did several more draws, and they are all around -1, within about 0.1, which is what I would expect (white noise necessarily tends to reverse the sign for the reasons I gave above). But your expression claims the expected value is zero. Why? Well the autocorrelation coefficients of white noise are zero, since the samples are independent. Your strange sum of autocorrelation coefficients will also be zero. But the expected value from the series is highly negative.
Sorry Bart, but your expression is simply not correct, and the wikipedia comment is still wrong.
Spence, it’s a linear operator.
E{(x(2) – x(1)) * (x(1) – x(0))} = E{x(2) * x(1)} – E{x(2) * x(0)} – E{x(1) * x(1)} + E{x(1) * x(0)}
You just plug in the formula. H = 0.5 is random walk with an initial starting point of zero, so yes, the expectation should be zero. You would test it like this:
>> x = cumsum(randn(3,100000));
>> mean((x(3,:)-x(2,:)).*(x(2,:)-x(1,:)))
ans =
2.3749e-04
Or, rather
x = [zeros(1,100000) ; cumsum(randn(2,100000))];
mean((x(3,:)-x(2,:)).*(x(2,:)-x(1,:)))
ans =
5.1115e-05
Note that with H = 0.5,
E{x(t2)x(t1)} = 0.5 * ( abs(t2)^2H + abs(t1)^2H – abs(t2-t1)^2H ) = min(t2,t1)
which is the normalized autocorrelation of a random walk.
Or, you can do it your way, with a slight change:
x = cumsum(randn(1000000,1));
x2 = x(3:end);
x1 = x(2:end-1);
x0 = x(1:end-2);
mean((x2-x1).*(x1-x0))
ans =
5.0511e-04
though it takes more trials to get the number to generally come out insignificant. Over many trials, it tends to zero.
There appears, then, to be a disconnect between your definition of H and that used on the Wiki page, which may explain why we had different ideas about flicker noise. On the Wiki page, H = 0.5 is the designator for random walk, as is seen in my post previous relating their autocorrelation function for H = 0.5 to the standard one for random walk.
Note that for normalized white noise
E{(x(2) – x(1)) * (x(1) – x(0))} = E{x(2) * x(1)} – E{x(2) * x(0)} – E{x(1) * x(1)} + E{x(1) * x(0)} = -1
which is essentially your monte carlo result.
“H = 0.5 is the designator for random walk”
I am being informal. H = 0.5 is actually the designation for Wiener noise on the Wiki page, the sampled-in-time-data version of which is a random walk.
“The first expectation operator is the normalised autocorrelation coefficient.”
This may be a source of confusion. It is normalized autocorrelation, period. The normalization is to a constant noise level. E.g., for a random walk modeled informally as the sampled-in-time integration of white noise, the white noise input is normalized to have standard deviation sigma = unity. The formula gives not a coefficient of correlation, but the actual correlation.
Bart,
H=0.5 is not the designator for a random walk. You are wrong.
The Hurst exponent H is undefined for a random walk.
H=0.5 is for white noise, i.i.d. gaussian.
Your equations are wrong. Full stop.
Spence_UK says:
October 8, 2013 at 11:03 am
Well, I’m sorry you are getting heated, Spence. Clearly, the Wiki site defines H = 0.5 as being a random walk. That may not be the convention with which you are familiar, but it is what the Wiki site is using.
Look at the autocorrelation function they give. For H = 0.5, it is the autocorrelation function for a random walk.
E{x(t2)x(t1)} = 0.5 * ( abs(t2)^2H + abs(t1)^2H – abs(t2-t1)^2H )
Plug in H = 0.5
E{x(t2)x(t1)} = 0.5 * ( abs(t2) + abs(t1) – abs(t2-t1))
If t2 > t1 and both are positive, then
E{x(t2)x(t1)} = 0.5 * ( abs(t2) + abs(t1) – abs(t2-t1)) = 0.5 * (t2 + t1 – (t2-t1)) = t1
The same will hold when t1 > t2. Hence
E{x(t2)x(t1)} = min(t1,t2)
It appears to me there is likely a clash of conventions here. But, under the convention they are using, they are right.
The same will hold mutatis mutandis when t1 > t2.
“H=0.5 is for white noise, i.i.d. gaussian.”
Perhaps you mean the increments are white?
I know the equations are right the way I am using them. And, they agree with the Wiki site. That’s 2:1. If you are sure you are right, then I think the only conclusion can be that there is a schism in the conventions being used, and this is leading to confusion and frustration.
Sometimes… strike that… Often, the biggest part of the problem is defining the conventions and making sure everyone is on the same page.
I’m not getting heated – I’m just telling you that you are wrong.
If the increments are white, you have a random walk, H is undefined for a random walk.
x = cumsum(randn(…));
Gives a random walk. Since H is undefined for this expression, you cannot use your equations at all for your analysis. H *is* defined for white noise, H=0.5. This is calculated in MATLAB as:
x = randn(…);
As we saw, the expected product between the current step and the next step for a time series of H=0.5 with zero mean and unity variance is approximately -1. This is simple and expected as I described above. But your equation does not give -1; it gives 0, because it is wrong.
The equation you give, less a constant applied to each component, is the equation describing the autocorrelation coefficient. You then interchangeably use this with the expected value of the series, which is an entirely different operator.
I’m explaining it in as simple terms as I can, but if you don’t understand the principles under which the maths is defined, I cannot help you.
Well, these guys, these, and these, contradict you. Honestly, I don’t care. But, clearly, if you take the autocorrelation function (not coefficient, function) provided on this page and provided in many other references as well, then for H > 0.5, succeeding delta’s are positively correlated, and this should lead to precisely what the article states, that this “dependence means that if there is an increasing pattern in the previous steps, then it is likely that the current step will be increasing as well.”
You haven’t explained why you think they are wrong, you have only asserted it. I do not know why you think there is a principle contained within your assertion which I should be understanding. You have provided no maths. You have provided no alternative autocorrelation function. You have provided no definitions. What I do not understand is why you think I should merely take your word for it that all these sources are wrong, and only you hold the truth. And then, I do not know what you expect me to do with that information once I have accepted it.
Ah, okay, I’ve realised why you’re confused. I’m referring to the Hurst exponent, H, in all my discussion here (and the equations you give above are the equations for the normalised autocorrelation *coefficient* of the Hurst exponent).
You have linked there to a different (but unfortunately similarly named) measure, called the generalised Hurst exponent, Hq.
These are different measures and *cannot* be used interchangeably as you have done here.
The coefficient used to measure the properties of fractionally integrated time series is the original Hurst exponent coined by Mandelbrot, H, not Hq. Dr Lund and I have argued that a climate follows a pattern associated with H>0.5, *not* Hq>0.5. These are quite different things and should not be confused in the way you have here.
As I explained, the original Hurst exponent is undefined for a random walk. And, as shown by my analysis above and by UC’s prediction of the behaviour of half integrated noise, the future expectation is in either direction, but the central prediction is slightly down (reversing the recent trend).
Is this autocorrelation function valid or not?
E{x(t2)x(t1)} = 0.5 * ( abs(t2)^2H + abs(t1)^2H – abs(t2-t1)^2H )
If it is, then with H = 0.5, it is equal to the minimum of t1 or t2, which is the autocorrelation function of a random walk.