Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead

After figure 3:
To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.
Note the word [that] — it appears to be missing…

Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.

Try same with CET(reliable rural station) you will probably get a much more pronounced cooling in the future

http://wattsupwiththat.files.wordpress.com/2013/09/clip_image034.png
Can you see? The warming will come back and will haunt us again!! 😀

As an electrical engineer, I’ve been wondering about using this approach for sometime. Mr. Patterson has done all the right things, so I don’t doubt the numbers per se. We don’t know is if the forecast portion is valid, but now that we have a model we can at least determine that going forward. What is interesting is the relative simplicity of the final result when compared with the climate models.

The maths is way over my head but if I understand the gist correctly you are saying that using standard DSP analysis techniques you can state with greater than 90% confidence that global temperature is cyclical and is heading down.
Unlike the IPCC you at least provide the maths!

If the same method was applied to data just through 1980, would it accurately predict the warming in the 80’s and 90’s?

A lot of work and for that many thanks. BUT, there is always a but with temperature, the data set used is altered to reflect current theories that temperatures must increase with rising CO2 levels. Raw temperature data is notoriously variable even if all collections are from correctly sited and using the same calibrated instruments there will be temperature differences even a few yards apart. So what is the real temperature of the earth’s surface? we really do not know because temperature can only be accurate if the item being measured is at temperature equilibrium which the earth never is.

‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.

Hadcrut 3 is not a balanced sample of weather stations. It may give you some indication of periodicity but with bias.
Just take a random sample, of weather stations, balanced by latitude and 70/30 @sea/inland,
http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/
pay particular attention to maxima, and plot:
http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/
I don’t understand why people want to do so difficult.

@Bruckner8
>Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?
Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem
The sequence of odd harmonics in the processed waves suggests that windowing artifacts remain, as the convolution of sine waves (which are always infinite in extend from an abstract mathematical viewpoint) multiplied by a sampling window (“square wave”) would suggest.http://en.wikipedia.org/wiki/Convolution_theorem
In AR modeling, the predictive modeling results depend on the time series being stationary, i.e. constant mean and variance. But historical temperature records are not stationary, and in fact it is the non-stationary part of the record that you are trying to model. So results of predicting values in the future based on values in the past will generally not be credible.
http://en.wikipedia.org/wiki/Wold's_theorem
😐

IanE says:
September 11, 2013 at 5:52 am
‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.
As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.

@Me

Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

As soon as I hit the “Post Comment” button I realized that my reponse was wrong because the 170-year wave in question is indeed “over-sampled”, not under-sampled.
But the reason I was so sure this couldn’t be done is that I’ve tried to reconstruct these kinds of “over-sampled partial waves” myself and have never been successful at it. But perhaps I going at it in the wrong way.
😐

This appears to be an example of the well known fallacy of Fourier Extrapolation.

If the above statistical analysis is valid ( as it appears)… why not submit it for publication and peer review. Surely the gatekeeping practices, of climate research, have somewhat abated since the climategate revelations. Or is this, just a pre-review before submission. GK

Hmmm, based on the data, we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome. Oh boy, who would have guessed?

Greg Goodman says:
September 11, 2013 at 6:31 am
… Running mean filter _introduce_ spurious signals.
Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

The 170 year cycle length is suspiciously close to the approximate cycle length of the solar system barycenter. the 170 year cycle continues to show up in the works of many different authors. as does the 55-60 year harmonic of 170, which closely approximates the observed length of the PDO.
Based on what we know about tidal prediction, DSP is likely to be at least as accurate as the global climate models in predicting future climate, and quite likely a whole lot more accurate.
if one tries to model the ocean tides using first principles, as is done with the climate models, the results are garbage. earth’s tides would be miniscule based on gravitational attraction alone.
the earth’s tides are predicted to a high degree of accuracy by the motion of the celestial bodies. this is because the earth’s oceans have over thousands and millions of years, come to oscillate in phase with the motion of the celestial bodies.
it is completely likely that s that climate oscillates similar to the tides, only on a longer timescale, as a result of a small but regular harmonic forcings based on the motion of the celestial bodies over thousands and millions of years.
this suggests that we can predict climate with some hope of accuracy using methods similar to what the author has proposed here today.

The relevant line from a very long article: “Even if we accept the premise of statistical significance, without knowledge of the underlying mechanism producing the periodicity, forecasting becomes a suspect endeavor. ”
Let’s not duplicate the errors of our adversaries and assume we know more than we know.

JP “Exactly why I used box care (i.e. non overlapping) filtering, not running mean.”
Could you explain what you mean by “box car” in more engineering/mathematical language , maybe I misunderstood what you mean by railway analogy. 😉

pochas says:
September 11, 2013 at 6:53 am
That fundamental of 170.7 years speaks strongly of the Jose cycle.If Patterson knew about it …
I do not. I did a cursory google search for “170 year climate cycle” but couldn’t find anything relevant, I would appreciate a reference if you have one.

@Greg Goodman – Box car decimation is simply dividing the recorded into N equal length segments and taking the average of each segment as the new sample (thus decimating by N). Unlike MA averaging, it introduces no correlation between samples.

@Jeff Patterson

September 11, 2013 at 6:49 am
Greg Goodman says:
September 11, 2013 at 6:31 am
… Running mean filter _introduce_ spurious signals.
Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

So the artifacts will not overlap, but they’re still there. The only window filter that I know of that has no artifacts is a Gaussian window. It produces no sidebands.
Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.
😐

@Greg Goodman – I’ve addressed your 1st concern elsewhere. The others are concerns about aliasing which is of course always a concern. I disagree with your statement regarding box care filtering – the zeros of the filters transfer function lie exactly on the aliasing spectral lines. It is a very effective re-sampling filter. Of course it can’t do anything for the aliases already present in the data, but as you can see from the PSD plots (e.g. Figure 9), the high frequencies are rolled of by some 40dB so any aliasing will be insignificant.

the sun’s motion around the solar system barycenter roughly repeats every 171.4 years as measured by its angular momentum. this is remarkably close to the author’s 170.7 year cycle length in the Hadcrut3 temperature data.
if one sets aside the argument over the mechanism by which climate oscillates with a period of 170 years, and simply accept that there is evidence for such an oscillation in the observed data, then one can go forward and accept that prediction is possible.
humans learned to predict the seasons long before we understood the mechanism. why then should we assume that we cannot predict climate without understanding the mechanism?
history shows that almost always the prediction comes BEFORE the understanding. when the prediction is shown to be correct by observation, this suggest the mechanism. and from there we investigate to discover the mechanism that governs the prediction.
for example, the motion of the planets in the heavens. we learned to predict this with amazing accuracy long before we discovered that gravity was the mechanism. modern science would tell us that such a thing was impossible, you cannot predict without a mechanism.
the facts are that humans invent mechanisms to explain predictions, and more often than not these mechanism are later shown to be false. this in no way affects the accuracy of the prediction. right answer, wrong reason is still a right answer.

John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”
The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.
No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.
Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. 🙁 )

RC Saumarez says:
The first seems to be a fundamental misubderstatnding of the relationship between the Fourier Transform and the Discrete (complex exponential) discrete Fourier series i.e. the FFT. In the former case there is an infinite resolution in frequency but requires the signal to be observed from – to + infinity. In the case of the FFT, if the fundamental frequency is determined by the length of the record. One might try to infer the presence of lower frequencies, but this lacks any formal justification.
I don’t think I misubderstatnd 🙂 anything. I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).
I addressed your record-length concern in the paper (see the section on record periodization).

RC Saumarez says:
September 11, 2013 at 7:28 am
You miss the point about Fourier Extrapolation.
>>Figure 13 is simply a re-run of the 20th century – i.e.: a periodicity.
Figure 13 contains 500 paths. They can’t all match the single path of the 20th century,
>>There is no theoretical justification for this approach – it simply says that the future will be a repeat of the past.
Which will be partially true in a the mean sense if the periodicity is present. As I took pains to point out though, that’s a big if and any projection depends on the major climatic parameters remaining constant over the projection period. I thought I was pretty clear on that.
What you have done here is to apply some extremely artificial constraints on the system by assuming that:
a) You have characterised the system correctly
b) The inputs to the system are stationary.
I am confused by (b). The projection assumes a periodic input process (and stationary climate process). Periodic signal + noise is decidedly a non-stationary process. Interesting though, the residual of the fit passes the Unit Root Test with a p vaule of ~ 10^-12. And the decimated record is at least weakly stationary (p = 10^-6).

JP: “I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).”
But in fitting your five harmonic model to the time series with no linear component you are just modeling the general slope with a truncated FS based on 170. RC point about repetition still stands. Plot your model out over 1000 years , it will be 170 year saw-tooth , more or less.

@Jeff Patterson
To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique is called record periodization. The data is regressed about a line connecting the record endpoints, dropping the last point in the resulting residual. This process eliminates the endpoint discontinuity while preserving the position of the spectral peaks (although it does extenuate the amplitudes at higher frequencies and modifies the phase of the spectral components).
There is a lot going on behind the scenes in this paragraph that I thinks needs more explanation.
“The data is” What data? The time series? [I think] Or the resulting PSD?
“regressed” how?
“about a line connecting the record endpoints”.
Did you draw a line from Point 1 to point 113? Did you do a least squares regression line across the [time-based?] data?
Figure 3, to 4
“Dropping the last point”. Fourier analysis has to assume that the time based data record repeats. You must think of the time series as a chart recorder plot where you wrap the end to the beginning as a cylindrical plot. There will be a splice where Point 113 meets Point 1. Dropping the last point doesn’t remove the splice. I don’t see what good that does.
The peaks are harmonically related, with odd harmonics dominating until the eighth. Since spectral resolution increases with frequency, we use the eighth harmonic of the periodized PSD to estimate the fundamental
Huh? Why should we conclude the peaks [in the Fig. 4 PSD] are harmonically related? Why use the eighth to estimate the fundamental harmonic? What if we used the 5th, 3rd, of 7th? This is not obvious.
The x-axis of Fig 3, Fig 4, Fig 7, Fig 9a, Fig 9b need some labels and units. They leave too much work for the reader.
From inspection of the PSD we form the harmonic model
I need more.
fit the model to the original (unperiodized) data
I give you credit for putting the 6 line parameter function of this step in the paper, but an illustration would have been nice.
Fig. 10 spaghetti graph.
What was allowed to vary between trials? What stayed the same?

Mr. Courtney, you don’t see a span that includes a cooling anomaly in the projection? I do. Watch and see, this is a theme you’ll see pop up here more and more: the recognition that that data does not exclude the fact that the overall effect of adding CO2 to the atmosphere could be cooling, not heating.

Greg Goodman says:
September 11, 2013 at 7:26 am
John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”
The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.
No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.
Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. 🙁 )

Greg,
Of course moving-average (“boxcar”) filters are very intuitive and simple to implement. But that doesn’t change the fact that they also produce spurious artifacts in their output.
Gaussian filters are “ideal” in the sense that they do not produce these artifacts. But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.
http://en.wikipedia.org/wiki/Gaussian_filter
“Gaussian filters have the properties of having no overshoot to a step function input while minimizing the rise and fall time. This behavior is closely connected to the fact that the Gaussian filter has the minimum possible group delay. It is considered the ideal time domain filter,…”

I am very surprised to see this.
You are imposing highly artificial constraints on the system. Your projected results (fig 13) are simply a periodic rerun of the 20th century. What you have done, in essense it to assume an impulse response based on auto-correlation, and perturbed this with a periodic input to get a result.
The point is that you cannot gain a statistical characterisation of your model because the persistance of the climate system is too long, compared to the record length, to allow segmentation of the record and improve estimated accuracy. It is inevitable that random inputs into a system with persistence will produce trends. The estimation of that persistence is very difficult and distinction between competing physical models almost impossible with current data. Nevertheless, trends that are comparable to that seen in the modern record are just possible with a randomly excited Hurst system. See a tutorial post:
http://judithcurry.com/2012/02/19/autocorrelation-and-trends/
The model assumes a periodic defined input and noise with some statistical properties. This is a stationary input, in the commonly accepted sense, because it has constant parameters with time. This is a huge assumption.
I would comment that this is the blind application of signal processing without giving sufficient thought to underlying physical assumptions: I wrote here, in a tutorial post
http://wattsupwiththat.com/2013/04/09/correlation-filtering-systems-and-degrees-of-freedom/
“Modern scripting programs such as “R” allow one to perform many signal processing calculations very easily. The use of these programs lies in not applying them blindly to data but in deciding how to use them appropriately. Speaking from bitter experience, it is very easy to make mistakes in signal processing and it is difficult to recognise them. These mistakes fall into three categories, programming errors, procedural errors in handling the signals and not understanding the theory as well as one should. While modern scripting languages are robust, and may largely eliminate straight programming errors, they most certainly do not protect one from making the others!”

joshv says:
September 11, 2013 at 5:12 am
“Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.”
Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out. As I have said before, one can obtain any result one desires with enough data manipulation and statistical analysis.

JD: “But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.”
If you add the proviso “for the same length filter” you are totally correct. I work from the spec I need so I compare two filters with similar FWHM and conclude I need a longer window to do the same job without unacceptable defects ( be it gaussian or triple-RM).
I actually favour 3RM for this kind of stuff since it has a zero in frequency. It’s marginally better than gauss for removing annual, for example.
http://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/

Jeff Patterson on September 11, 2013 at 6:46 am
Here’s a hopefully approachable summary:

– – – – – – – –
Jeff Patterson,
Thanks for your effort.
I particularly appreciate your care in describing the limitations and shortcomings of your methods.
John

Is this anything to do with Mr.T.Pratchett or the unseen university? There is certainly a whiff of Hex in the methodology and written conclusions.

JP: “The main point (beside the surprisingly good fit that can be achieved with such few harmonically related elements) was not the projection but rather that there exists plausible explanations for the observed data which do not require significant forcing from anthropogenic sources.”
Sorry, maybe I missed something in the article. What plausible explanation did this present?

sorry, “hypothesis”. My right ring fingertip is numb from a cut, hard to feel where the upper rlow lf keys is.

Pamela Gray says:
September 11, 2013 at 8:33 am
“Crosspatch, actually that is likely. There is a musicality test, standardized no less, that in its simplest form, has the examinee predict what note comes next after a short series of notes. Because the muical scale is mathematical and orderly one can if one has an ear for the “language” predict what comes next.”
This is very true in 1,4,5 chords in simple tunes such as blue grass and old time rock and roll, not so much in classical, at least not that I can tell. Too complex and many times have less repetitive sequences. But then I do not play classical. But torture the “data” enough and you will lose the tune.

Yay Agilent! (E44xx) I was wondering when someone would apply modern Information Theory or Signal Processing techniques to this problem, let alone modern statistical methods. I have yet to see any study doing bootstrap 🙁

Jim G says:
September 11, 2013 at 8:13 am
“Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out.”
That’s completely unfair. Jeff has argued convincingly why there is reason to believe that the periodicity is real.

Get off that tangent.

8/

the problem is you can use this method with so many curves…
so take any kind of data. price of wheat oranges or .., well anything..cut the last part of the data do the same analysis and..see if it can “foresee ” anything…i am pretty sure no…
i don’t buy that..because you don’t have any idea if the harmonics are related to something ,corespondind to a oscillating phenomena ,or just …a way to fit the curve.
May be this method can be interesting to find out if some frequancy you find smell “physical”

@Jeff Patterson

The residual _is_ stationary. That’s the point. We can certainly drive an AR process with stationary (and in this case also nearly Gaussian) noise + sine waves and get a valid projection.

Well, ideally, the residuals should be pure white noise, with mean=0
Overall I really like your experiment. Well done. A good teaching moment for us all!
But I have used similar ARIMA-based estimators myself to predict these kinds of time-series (stock-market) and managed to find a lot of “useless” predictability (i.e. in the linear sense). It always turned out that the event sequences that I couldn’t predict (the “innovations”) were mostly the same events that would have made me a rich man if my predictor worked the way I hoped it would.
To make these models perform better you need to know a lot more about the hidden processes which generate the observable outputs.
😐

It has been said that if it is good science, it does not require a statistical massage. I heartily endorse that sentiment.

@ Jim G and others.
I can take broadband noise and then pass it through a low-pass system. Provided the system has persistance, I will get a signal containing trends and this may appear quasi-periodic. I can fit a Fourier series to this signal and calculate an “input” that will create the output for a short length of recorded signal.
However, the input is noise and therefore the small segment of “input” is simply one snapshot of its underlying process. It is inevitable that if you do this you will find want appear to be cycles. If you repeat the “experiment”, you will find different periodicities in each repeat.
What one cannot do is assume that this snapshot will repeat – it is simply one sample of a noise record.
If you look at figure 13, it is the 21st century prediction and is simply a recapitulation of the 20th century with a basic triangular waveform so that the 20th and 21st centuries connect.
The logic underlying this post is, to my mind, hopelessly flawed.

I don’t have time this week. If I remember this discussion next week I will run it through a somewhat different set of processes and see If I get the same periodic repeats.

I appreciate the response posted here. rather than address each in turn, let me make some general comments on the method.
First, the concerns about aliasing are IMHO unwarranted. The monthly data record is highly low-passed, being averaged in both space and time. Also, presumably the climate process, containing as it does many integrative elements, imposes its own filtering of high frequency input variances. The power spectral density of the unaltered Hadcrut data shows the high frequency floor to be down some 60dB (1/1000) from the spectral peaks we are trying to extract. If one understands the aliasing process in the time domain, a moment’s reflection should reveal the mathematically provable theorem that the amplitude of an alias cannot exceed the amplitude of it source. Hence aliasing effects here are insignificant.
On filtering: In harmonic decomposition, filtering must be avoided like the plague because it alters the amplitudes and phases of the periodicity we’re attempting to extract. Loss of spectral resolution is also an issue, especially when trying to extract cycles whose periods exceed the record length. However, the monthly data has a strong annual component that must be removed prior to decomposition because the underlying assumption of the method is that all periodicities are harmonically related. Fortunely, if you know the period of the offending cycle there is a solution, Box care filtering as described above. This filter is ideal in that it places its transfer function zeros precisely on top of the spectral peak being removed. It is in essence a high-Q notch filter which has the least impact on the remaining spectrum. It is true that this filter has high sidelobes but in this case that’s fine because the power spectral density at the lobe peaks is extremely low as evidenced above.
On record periodization: First note this is only a test to verify that observed spectral peaking is not due to the discontinuity at the record end points- the harmonic decomposition is performed on the original, unperiodized data).
Fourier analysis (which is used here only to perform the PSD estimate) gets around the required infinite length record by simple assuming the record repeats ad infinitum. If the endpoints don’t line up (withing the average step-size of the sample), the discontinuity created by stitching the records together (that’s not how an FFT works but in principle the effect is the same) creates a periodic error that shows up as spectral peaks. We can test for this effect by subtracting a line that connects the data record endpoints and dropping the last point from the result to avoid repeating the zero-error point at the ends. The result is a record which has no discontinuity distinguishable from the noise present in the data. If the PSD peaks remain after this process, it indicates the spectral peaks are not record-length artifacts.
One reader asked “why use the eight harmonic?”. Because there are more cycles present of the eight harmonic and so it provides a better estimate of the fundamental.
Fig. 10 spaghetti graph.
What was allowed to vary between trials? What stayed the same?
Figure 10 is output of the AR process used to create “climate-like”. Each path is a 113 point sample of the process for a new randomized state and random, white, input function of the variance derived by the AR process estimation function.

If we started to fairly and objectively consider ringing, overshoot, beat frequency and other of these sorts of behaviors with respect to the global climate system, some interesting and perhaps frightening possibilities might start to become more apparent and would open the door for a whole new tone in the conversation.

I think DayHay makes the most important point.

Climate scientists got off on the wrong foot three decades ago, by prematurely assuming that there were just two important drivers of climate – namely the sun (insolation) and a CO2-driven greenhouse effect. These, most reckoned, were the “first-order” influences and all of the other factors were second-order or third-order and therefore could be largely ignored. CO2, they said, was the “control knob” of climate variability.
Mother Nature has unkindly demonstrated their basic error, by providing fifteen years of almost unchanged world average temperatures while the atmospheric CO2 level, for whatever reason, continued to increase significantly. The control knob was turned, and almost nothing happened! The most bizarre and ingenious ecclesiastical apologia forthcoming from the Jesuit intelligentsia would pale by comparison with some of the climate establishment’s amazing “explanations” for this hiatus. We wait with bated breath for the IPCC’s upcoming summary to see if it “explains” or simply ignores what nature has done to them.
This is why Jeff Patterson’s post is so exciting. It assumes absolutely nothing about the theories or explanations of climate scientists – or anything else – and simply mines the available data for plausible patterns which could provide some insight into future trends or actual mechanisms. And, unlike most other such attempts at naive analysis, it is performed by somebody who clearly understands the false moire and other mirages which can emerge from this type of statistical treatment. Such trend analysis is surely the best starting point for the badly-needed “reboot” of climate science.

Henry@all
I really feel sorry for you all because you are all missing an important point:
where would be without the sun? We’d be dead, would we not?
We know that earth has a number of factors (built-in) that confuse matters,
on mean average temp.
like
its own volcanic action,
turning of the iron core,
magnetic fields/ clouds
lunar interaction
to name but a few.
I am sure there a lot more.
So why do you all keep staring at those stupid figures for mean average temps. & their anomalies?
Surely anyone with brains must have figured out by now that maximum temperatures are a much better proxy to look at, as I found. But don’t look at only one station, because if you were to look at CET maxima alone you might get confused: they have a lot of weather…..there in London.
http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/
http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/

So, richardscourtney, how does this jive with Akasofu’s linear “recovery” from the LIA?

@Sam The First says: September 11, 2013 at 5:57 am
Menahwile in the broadsheet papers here in the UK, notably The Independent and The Guardian which are read by liberal / left opinion formers (inc teachers at all levels), the comments to this article below demonstrate that none of their readers is taking any notice of the long pause in warming and the overall cyclical record.
Will someone with the time and expertise please add a comment or two to explain what is really going on?
========================================================================
I’ve long been banned from CiF for dissent. The Indy, well, it’s best for me not to go there as it enrages me in no time at all. Occasionally, I will employ my rapier like wit to puncture some balloons, but ultimately, it’s a waste of time talking to such as comment there.

Reading 170 year cycles… I knew I read about this before!
People, try a quick blast on Google and you will be decimated by the amount of similar findings from differing sources. From war to famine. Even the 10Be records have something to say

This paper by Coughlin and Tung 2004 employs the nonlinear EMD (empirical mode decomposition) analysis to stratospheric temperatures, and finds a strong 11 year solar cycle.
This is the right kind of analysis to look for nonlinear forcing of the climate system by solar and other astrophysical cycles. Check out figure 3.

Mr. Patterson,
The quality of a five-harmonically-related-frequency model fit to the data surprises me. I’d be interested in seeing the quality of a similar fit using a five frequency model whose frequencies are not required to be harmonically related. That is, if the software routine you used to determined the values of harmonic sinewave amplitudes and phases that best fit the data can accommodate sinewave frequency as well, I’d be interested in seeing (a) how the best-fit estimated frequencies differ from their harmonic counterparts, and (b) the change, if any, of the residuals and residual spectral content. If the software you employed uses a weighted-least-squares cost function, my guess would be that giving that software an initial guess corresponding to your harmonic analysis results, the software would converge to a solution with frequencies, amplitudes, and phases differing only slightly from their “harmonic analysis” equivalents.

@Jeff Patterson at 11:31 am
Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the [known] line (ramp),
You don’t know the line. 113 points in the dataset is just an artifact of how long someone was recording data. When the data starts and ends is similarly an artifact. Different starts and different ends give different lines and therefore different transforms. I expect differences in the high frequencies and power at the low frequencies.
Also, the choice of a base “line” is arbitrary. Simplistic, but not necessarily the best choice. That base line might be a linear function of CO2 concentration, logarithmic function of CO2, or a non-linear function of sunspot number.

“””””…….Ric Werme says:
September 11, 2013 at 5:43 am
A few comments:
1) decimated by 12
I know that “decimate” came from the Roman practice of inspiring soldiers after battles where the commanders felt were uninspired. They lined up the soldiers and killed every tenth man, leaving the remaining 90% inspired to win the next battle. At first blush, this sound like torturing the data until it confesses or perhaps this is “dodecimated” – killing every 12th measurement. :-)…….””””””
Ric, In this instance “decimated is a highly specific technical term from digital signal processing; so don’t go looking for a highway lines with occupied crosses.
And no adherence to the ten implication is required.
And I know about enough about it to be a hazard to myself; but if an Agilent Cowboy, doesn’t know Digital Signal Processing, then nobody does.
My concern with the exercise, is that I wonder what result one could pull out of the phone numbers in the Manhattan telephone directory.
It’s not clear to me that the Hadcrud numbers actually comprise data that relates to planet earth.
And having seen in the past, a mathematical exercise, that recovered the value of one of the fundamental physical constants of science (the fine structure constant), to about 8 significant digits of precision (about 1/3 of the standard deviation of the best experimental value, at that time); yet the model had precisely zero input from the physical universe; it was literally just fetzing around with numbers; so I don’t go for all new things, as an early adopter. I’m often the last to see the light.
I have quite recently seen a movie called “Gone with the Wind”, for the very first time.

I should say that I enjoyed the post. Why start the series at year = 1900?
The following is well-said: If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.
The author estimates 10 parameters, after estimating the fundamental period. Why not just fit a 10th order polynomial or a Bayesian Adaptive Regression Spline model, or some well-selected set of 10 wavelets — as long as you are acknowledging that this is merely another curve-fitting exercise with no claim to predictive power, any old and new functions are just as good. If the fundamental frequency is evolving with time, in what sense is it a “fundamental” frequency.

The harmonic technique used is useful in freeing the process of reproducing a ramp determined by the length of the dataset.
The first second and third harmonics are just recreating the longer ramp. The most notable is perhaps the strong 8th.
170/8= 21.25 years. Also one of the three periods my 3 cosine analysis latched on to.
The suggestion of Schwabe solar cycle is obvious.

Pamela Gray: Comment 1: While data massaging here is extreme, it does demonstrate the futility in ordinary least squares statistics so adored by AGWing scientists.
This is non-linear least squares, after finding the fundamental period.

Jeff Patterson: As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.
That’s smart.

Greg Goodman says:
September 11, 2013 at 12:01 pm
JP: ” forget to add a comment on forecasting: In retrospect I wish I had left this section out”
Yep, forgetting the title of the post may be the best approach.
I didn’t choose the post title which I agree was unfortunate.

Nick Stokes says: How can you establish a 170 year periodicity from 113 years of data?
Basically, all he did was estimate a nonlinear function from the set of parameterized (phase and amplitude) family of sine curves. Using that estimated period, he then chose harmonics of it as the additional terms in a multiple non-linear regression. In harmonic regression, which is what this is, there is no non-arbitrary way to select the fundamental frequency: the standard in Fourier analysis is to let the full data series represent 1 full period, which may not have any relevance if you have managed to sample, say, over 1.25 times the period.

@Jeff Patterson at 12:21
You know the line exactly because you create it!
YES!
YOU create it. Not Nature. It is an artifact of the measurement time window and processing.
That is why you must try different subsets and get different lines to explore the changes in the resulting transform. Then look for similarities that are more likely in the signal and not an artifact of the processing.
It’s like trying different stacking velocities in seismic processing to improve the signal-noise ratio.

JP: “I looked over the referenced paper and while I certainly agree that aliased data is irretrievably corrupted, the question is by how much? The fundamental of the extracted model has a peak-to-peak amplitude of .6 degrees.”
No , I was not suggesting the whole long term trend is an alias. Neither did RC’s article on Climate etc suggest that degree of error. But if there is a lack of filtering it will not just be one frequency effected. RC demonstrated that this can lead to spurious decadal length trends (did you read the link?).
If there is error in the long term it is just as likely ‘bias correction’. I had the fortunate opportunity to discuss this at some length with John Kennedy in the comments. He was very forthcoming and helpful, though obviously defending their work.
What he told me of the validation process showed it was based on circular logic and geographically anecdotal “validation” from japanese data also using _buckets_. (Apparently japanese buckets don’t ‘bias’ like british and american ones!).
For the title , that’s editors for you . They’re always out for a catchy headline. 😉

Jeff, is there any sort of robustness tests for this method? Can the time series be analyzed on a shorter interval, and see if the predicted results for the out of sample period match?

Correct me if I’m wrong, but any non-random data points will show harmonics. I’m not sure what AGW would show but I would think it would simply be a S/N ratio of an AGW harmonic on top of natural harmonics. In fact, mixing the two (natural and AGW) would result in seeing the AGW mix in even pre-AGW data as it would be unable to blend out. I think the assumption using DSP is that temperature is the convolution of linear functions and that may not be the case.
My gut tells me that DSP methods will treat random steps as non-harmonic noise and will not detect random and non-linear changes in DC offset that is characteristic of AGW. My gut is that a rise in temperature that is a sequence of step changes that are random in time and magnitude will be washed out in the deconvolution and the individual residual components that make up long term AGW would be below the noise level. Put another way, AGW may manifest itself as a sequence of below Nyquist rate events that all lie below the noise level. If the residual magnitudes look like “black body radiation chart” with a peak that is below the noise level, and a frequency longer than Nyquist, I would suspect you could have AGW in that signal and not be accounting for it as non-random and non-periodic and quite possibly attributing it to natural variation. The fact that you can model the past as a harmonic is from the math that will always solve, less a residual. It has no predictive value, though, because a driving component is non-linear addition to data. The math is good at figuring out how many “mixers and sources” are present in the existing data (or rather how many it needs to reach a certain S/N ratio) but not so good at predicting how many mixers and sources will be present in the future. I believe if it’s random in time and magnitude, and small at each step, no amount of oversampling and decimation will detect a fingerprint to predict a future value. For example, ENSO might be a large harmonic source that describes a long variation it temperature. If reality is that AGW is tiny, independent, sources and mixer to inject AGW at random points (i.e. say in 0.01C in June of 2007 and then 0.012C for January 2008, and then another 0.005C in September 2009), the analysis of those “below the nyquist and noise level will either leave them out or model them as a harmonic that has a lower residual error even if it’s really an accumulated error that will continue. The method will always add a harmonic term to reduce the error if it can whether it’s physical or not. I think the only way to overcome it is massive oversampling and decimation but you may still run into the noise floor of the data before you can pick out small, random, non-periodic residual signals. It’s like describing something as a finite series and every time your error gets too large, you add another term to the series that reduces the error. That’s what DSP algos will do and it’s not discriminate as to whether the term is physically descriptive of the underlying phenomena. It’s a fit and the number of terms is a function of the tolerable error and the technique. It works well to compare two different datasets but it’s not predictive and I’m not sure you can lump all the “fit” into natural variation and claim only the residual is a man-made artifact.
An interesting exercise would be to break the data up into ranges always starting at 0. 0-11 years, 0-23 years, 0-37 years, 0-61 years (maybe just prime intervals) see what the solution was at the end of each of those periods and see how those functions compare to the full dataset. The PSD harmonics should not change as long as the dataset is longer than the fundamental mode nyquist rate. If the PSD harmonics are different depending on dataset choices, I think there is a problem with the method and it’s assumptions.

Matthew R Marler says:
September 11, 2013 at 1:05 pm
After choosing the base frequency, why did you limit yourself to harmonics of the base frequency? Is there relevant physical theory that the important frequencies are multiples of the base frequency?
Because the PSD indicated a harmonic relation between the spectral peaks and because harmonics are generated anytime sinusoids drive a non-linear response (thermodynamic equilibrium involves a quartic relationship to temperature if I remember correctly ) .

I do not see any spectral peaks here of any distinctive character. Which surprises me, because i would expect to see something distinctive in the neighborhood of 0.016 years^-1 (60-65 year cycle evident in the data). Rounded peaks are generally artifacts of finite record length, and not representative of actual underlying processes.

We know that the hadcru, giss and noaa data is severely adjusted to reduce the late 30’s early 40’s warming, to reduce the cooling from 1945 to 1975 and to enhance the warming from 1980.
The data sets also fail to account properly for urban heat island effect, for station deterioration and for the removal of many cooler stations from the dataset.
The question is; do these massive questionable adjustments coupled with the failure to account for the factors mentioned above, affect the sort of analysis you have performed here.

See – owe to Rich says: September 11, 2013 at 12:06 pm
Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?
Using the five sinusoids (amplitude, frequency, phase) given in JP’s paper, the match (sinusoids versus measured data) going back to 1850 is not very good. You ask a good question, and I’d like to hear JP’s answer. I’d put my plot of that data in this comment, but I don’t know how to include figures in a comment.

I think that this is a most useful, reasoned and practical contribution to the overall debate. Sure, it hasn’t ‘proved’ anything one way or the other (my math is far too weak and my neurons retiring too rapidly to follow the fine grain of it, so use the salt shaker here…).
But the attempt is praiseworthy in itself: sticking one’s head over the parapet is always a risky move, and I congratulate Jeff for having done so.

Willis Eschenbach says:
September 11, 2013 at 2:59 pm
The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data.
========
I’d also like to see this and encourage the author to post the results.

Nick Stokes says:
September 11, 2013 at 5:11 am
How can you establish a 170 year periodicity from 113 years of data?

Dunno. Maybe it’s like saying todays temperatures are unprecedented for 1000 years while hiding the decline.

Willis Eschenbach says:
September 11, 2013 at 2:59 pm
Hmm, I’ve seem to have struck some nerve with my hobby horse, that was not my intent. I long ago lost interest in the climate debate, knowing from the outset that model-based science is an oxymoron. My interest was re-sparked recently when I stumbled here while looking for some work related info on Wiener processes. There was an interesting post speculating on whether the climate was a random walk process. There was a link to the dataset so I pulled it in and ran a unit root test which failed with p=.7. The plot by eye looked to have some periodic patterns and knowing that periodic forcing functions can make a stationary process appear non-stationary, I looked at the PSD. I noticed the harmonic relationship between the spectral peaks so it seemed like a good candidate for harmonic decomposition, a technique we use widely for analyzing non-linear processes. After HD, the residual passed the URT at p=10^-12. I was also surprised by how few terms it took to get a good fit and that set me wondering if the periodicity was real or simply a curve fitting artifact.
Although I recognized the short record problem, it’s not one I normally encounter (if we need longer records we simply acquire more data – at 10 GSa/sec it doesn’t take long :>) I am an engineer and not a statistician so I used the tools I am familiar with to devise a significance test. The reasoning was simply. Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results. I do not see the flaw in the logic, nor in the method for testing the assertion. A s”simplistic” test is not necessarily a flawed test.
Perhaps the problem is revealed in your mischaracterization of my use of the AR model here:
“For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.”
Nowhere do I assume that it is AR data that we are seeing, nor do I anywhere attempt to prove that it is AR data we are seeing. The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record. If the HD fit were insignificant, it seems to me that one should easily achieve similar GOF results on the AR process data. As was shown in my post, this is not the case. It seemed an interesting result so I shared it.
As for halving the data, extracting a 170+ year of data from a 55 point record looks pretty hopeless. The HD algorithm fails to converge, as does the AR generated sequences 95% of the time. I may fool around with it some more if I get some time. As for hindcasting, I posted a link to a fifty year hindcast on the monthly data prior to your post.
Anyway, take it for what its worth (perhaps nothing). I need to get back to my day job.
Cheers.

@ Jeffery S. Patterson
If you happen to be reading this could you reprocess it after first passing it thru a high pass filter set at zero to remove the offset .. I would love to see what it shows?
It’s the same trick you do in your field to remove a DC voltage offset if my language is confusing.

In the end, Digital signal processing, is just following a mathematical algorithm; applying it to a set of numbers.
So it differs from computing an average, only in that the mathematical algorithm is different.
You can take the very same set of numbers, and calculate the average, using that particular algorithm, or you can do the more complex DSP algorithms, and get something else from exactly the same set of numbers.
Well of course, you can also apply the same mathematics to the set of numbers you can find in your local telephone directory and get an average, or whatever the DSP algorithms produce.
The mathematics is quite rigorous; but that doesn’t mean the result has any meaning.
In the case of DSP, there usually is a starting assumption that there IS a “signal” there to be found in the set of numbers. Some procedures make use of prior knowledge of what the signal is supposed to be, and when it is supposed to be present, so you aren’t looking for something at times when it is not there, so you just gather noise.; such as locking onto a Loran C signal or these days, a set of GPS signals.
But if you apply these methods to sets of numbers which are acquired without regard to the rules of sampled data systems (which all our measurements are), then there may be no signal there to acquire.
That’s the problem with “climate data”. It doesn’t even come close to satisfying the Nyquist criterion, particular for spatial sampling of Temperatures, and the common method of Temporal sampling, also fails on that score. A daily min max Temperature sampling, only suffices, if the diurnal Temperature variation is strictly sinusoidal with a 24 hour period. Now in climatism, the aim is not to recover the continuous Temperature function for processing, but just to get the average, which is the zero frequency component of the spectrum. You can’t even recover the average, if the daily Temperature cycle has even a second harmonic component, let alone any higher frequency components, such as the results of cloud cover variations during the day.
So the mathematics may be quite elegant; but the input “data” is just garbage.

Jeffrey – when I look at the FFT result, one thing strikes me – there is obviously periodicity, but the picture looks reminiscent of a response containng reverberation or echoes. Unfortunately, I don’t have easy access to cepstral processing, but I would be fascinated to see whether there were peaks in the cepstrum. There may be “impulse responses” present that are triggered by discrete “events”, and there may be many overlaid so that they convolve in the spectrum. Just my feeling, as there is no real trace of actual peaks in the spectrum. There are maxima, but relatively broad. It’s the minima that interest me…

@george e. smith says:
September 12, 2013 at 1:49 am

So the mathematics may be quite elegant; but the input “data” is just garbage.

You are right but you cheated and gave him the answer I wanted him to work it out.
I will give him credit he is at least thinking outside the usual box that climate science seems to be stuck in.
JP there is a range of techniques for non linear time invariant systems which would are far more suitable for doing the sort of analysis you are trying 🙂

I talked to a thermo engineer from Intel yesterday and he surprised me when he told me that 40-50% of the cooling of a small phone in his pocket was via radiation (hence a surface optimized for emissivity). That’s a lot, generally we don’t see more than about 10-15% of the cooling effect via radiation. Of course, outside the package, we don’t get much benefit from conduction and the smaller the enclosure…well, of course, the less area there is for heat spreading. When it’s your job to cool something (or, conversely, to heat something) and radiation becomes an important part of toolbox, that’s truly a game-changer. That’s why guys like me get the big bucks, I guess.
It’s unfair to expect thoughtful colleagues like Mr. Courtney to consider the balance between CO2 backradiation heating and CO2 cooling via convection…or to realize that if a projected temperature anomaly includes negative numbers, that indicates the theoretical span includes cooling correlated with increasing CO2 concentration. It’s all good, carry on, my friends.

@Rabbit @Matthew R Marler: Done:

Jeff Patterson: I’ve seem to have struck some nerve with my hobby horse,
Not so. Willis Eschenbach is a vigorous critic of most modeling that gets presented here. Like me, he mocked the pretended precision of the estimated period of “170.7”. I don’t think data halving is as useful a technique as waiting for the next 20 years of out-of-sample data. I think he missed your qualification that if the system is chaotic your model was useless for prediction. I think your defense of your choice of null distribution for null hypothesis testing is adequate for now.
I think it will be a miracle if your model proves to be reasonably accurate over the next 20 years, and I think on the whole it is less credible than Vaughan Pratt’s model (which also will, in my guess, require a “miracle” to be accurate over the next 20 years), but I put in in what I call “the pile of active models”. Because you provided the code, anyone who is interested can test.

Wayne: @Rabbit @Matthew R Marler:
Cute. What was the reference to “Science”? Did that article describe the method?