Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead

This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data. Digital Signal Processing analysis suggests cooling ahead in the immediate future with no significant probability of a positive anomaly exceeding .5°C between 2023 and 2113. See figures 13 and 14. Code and data is made available for replication. – Anthony

Guest essay by Jeffery S. Patterson, DSP Design Architect, Agilent Technologies

Harmonic Decomposition of the Modern Temperature Anomaly Record

Abstract: The observed temperature anomaly since 1900 can be well modeled with a simple harmonic decomposition of the temperature record based on a fundamental period of 170.7 years. The goodness-of-fit of the resulting model significantly exceeds the expected fit to a stochastic AR sequence matching the general characteristic of the modern temperature record.

Data

I’ve used the monthly Hadcrut3 temperature anomaly data available from http://woodfortrees.org/data/hadcrut3vgl/every as plotted in Figure 1.

clip_image002

Figure 1 – Hadcrut3 Temperature Record 1850-Present

To remove seasonal variations while avoiding spectral smearing and aliasing effects, the data was box-car averaged over a 12-month period and decimated by 12 to obtain the average annual temperature plotted in Figure 2.

clip_image004

Figure 2 – Monthly data decimated to yearly average

A Power Spectral Density (PSD) plot of the decimated data reveals harmonically related spectral peaks.

clip_image006

Figure 3 – PSD of annual temperature anomaly in dB

To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique is called record periodization. The data is regressed about a line connecting the record endpoints, dropping the last point in the resulting residual. This process eliminates the endpoint discontinuity while preserving the position of the spectral peaks (although it does extenuate the amplitudes at higher frequencies and modifies the phase of the spectral components). The PSD of the residual is plotted in Figure 4.

clip_image008

Figure 4 – PSD of the periodized record

Since the spectral peaking is still present we conclude these are not record-length artifacts. The peaks are harmonically related, with odd harmonics dominating until the eighth. Since spectral resolution increases with frequency, we use the eighth harmonic of the periodized PSD to estimate the fundamental. The following Mathematica (Mma) code finds the 5th peak (8th harmonic) and estimates the fundamental.

wpkY1=Abs[ArgMax[{psdY,w>.25},w]]/8

0.036811

The units are radian frequency across the Nyquist band, mapped to ±p (the plots are zoomed to 0 < w < 1 to show the area of interest). To convert to years, invert wpkY1 and multiply by 2p, which yields a fundamental period of 170.7 years.

From inspection of the PSD we form the harmonic model (note all of the radian frequencies are harmonically related to the fundamental):

(*Define the 5th order harmonic model used in curve fit*)

model=AY1*Sin[wpkY1 t+phiY1]+AY2*Sin[2*wpkY1* t+phiY2]+AY3*

Sin[3*wpkY1* t+phiY3]+AY4*Sin[4*wpkY1* t+phiY4]+AY5*

Sin[5*wpkY1* t+phiY5]];

vars= {AY1,phiY1,AY2,phiY2,AY3,phiY3,AY4,phiY4,AY5,phiY5 }

and fit the model to the original (unperiodized) data to find the unknown amplitudes, AYx, and phases, phiYx.

fitParms1=FindFit[yearly,model,vars,t]

fit1=Table[model/.fitParms1,{t,0,112}];

residualY1= yearly- fit1;{AY1→-0.328464,phiY1→1.44861,AY2→-

0.194251,phiY2→3.03246,AY3→0.132514,phiY3→2.26587,AY4→0.0624932,

phiY4→-3.42662,AY5→-0.0116186,phiY5→-1.36245,AY8→0.0563983,phiY8→

1.97142,wpkY1→0.036811}

The fit is shown in Figure 5 and the residual error in Figure 6.

clip_image010

Figure 5 – Harmonic model fit to annual data

clip_image012 clip_image014

Figure 6 – Residual Error Figure 7 – PSD of the residual error

The residual is nearly white, as evidenced by Figure 7, justifying use of the Hodric-Prescott filter on the decimated data. This filter is designed to separate cyclical, non-stationary components from data. Figure 8 shows an excellent fit with a smoothing factor of 15.

clip_image016

Figure 8 – Model vs. HP Filtered data (smoothing factor=3)

Stochastic Analysis

The objection that this is simple curve fitting can be rightly raised. After all, harmonic decomposition is a highly constrained form of Fourier analysis, which is itself a curve fitting exercise that yields the harmonic coefficients (where the fundamental is the sample rate) which recreate the sequence exactly in the sample domain. That does not mean however, that any periodicity found by Fourier analysis (or by implication, harmonic decomposition) are not present in the record. Nor, as will be shown below, is it true that harmonic decomposition on an arbitrary sequence would be expected to yield the goodness-of-fit achieved here.

The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period, although others have found significance in the 57-year (here 56.9 year) third harmonic. We can however, estimate the probability that the results are a statistical fluke.

To do so, we use the data record to estimate an AR process.

procY=ARProcess[{a1,a2,a3,a4,a5},v];

procParamsY = FindProcessParameters[yearlyTD["States"],procY]

estProcY= procY /. procParamsY

WeakStationarity[estProcY]

{a1→0.713,a2→0.0647,a3→0.0629,a4→0.181,a5→0.0845,v→0.0124391}

As can be seen in Figure 9 below, the process estimate yields a reasonable match to observed power spectral density and covariance function.

clip_image018 clip_image020

Figure 9 – PSD of estimated AR process (red) vs. data Figure 9b – Correlation function (model in blue)

clip_image022 clip_image024

Figure 10 – 500 trial spaghetti plot Figure 10b – Three paths chosen at random

As shown in 10b, the AR process produces sequences which in general character match the temperature record. Next we perform a fifth-order harmonic decomposition on all 500 paths, taking the variance of the residual as a goodness-of-fit metric. Of the 500 trials, harmonic decomposition failed to converge 74 times, meaning that no periodicity could be found which reduced the variance of the residual (this alone disproves the hypothesis that any arbitrary AR sequences can be decomposed). To these failed trials we assigned the variance of the original sequence. The scattergram of results are plotted in Figure 11 along with a dashed line representing the variance of the model residual found above.

clip_image026

Figure 11 – Variance of residual; fifth order HC (Harmonic Coefficients), residual 5HC on climate record shown in red

We see that the fifth-order fit to the actual climate record produces an unusually good result. Of the 500 trials, 99.4% resulted in residual variance exceeding that achieved on the actual temperature data. Only 1.8% of the trials came within 10% and 5.2% within 20%. We can estimate the probability of achieving this result by chance by examining the cumulative distribution of the results plotted in Figure 12.

clip_image028

Figure 12 – CDF (Cumulative Distribution Function) of trial variances

The CDF estimates the probability of achieving these results by chance at ~8.1%.

Forecast

Even if we accept the premise of statistical significance, without knowledge of the underlying mechanism producing the periodicity, forecasting becomes a suspect endeavor. If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.

Nevertheless, having come thus far, who could resist a peek into the future?

We assume the periodicity is externally forced and the climate response remains constant. We are interested in modeling the remaining variance so we fit a stochastic model to the residual. Empirically, we found that again, a 5th order AR (autoregressive) process matches the residual well.

tDataY=TemporalData[residualY1-Mean[residualY1],Automatic];

yearTD=TemporalData[residualY1,{ DateRange[{1900},{2012},"Year"]}]

procY=ARProcess[{a1,a2,a3,a4,a5},v];

procParamsY = FindProcessParameters[yearTD["States"],procY]

estProcY= procY /. procParamsY

WeakStationarity[estProcY]

clip_image030 clip_image032

A 100-path, 100-year run combining the paths of the AR model with the harmonic model derived above is shown in Figure 13.

clip_image034

Figure 13 – Projected global mean temperature anomaly (centered 1950-1965 mean)

clip_image036

Figure 14 – Survivability at 10 (Purple), 25 (Orange), 50 (Red), 75 (Blue) and 100 (Green) years

The survivability plots predict no significant probability of a positive anomaly exceeding .5°C between 2023 and 2113.

Discussion

With a roughly one-in-twelve chance that the model obtained above is the manifestation of a statistical fluke, these results are not definitive. They do however show that a reasonable hypothesis for the observed record can be established independent of any significant contribution from greenhouse gases or other anthropogenic effects.

0 0 votes
Article Rating
313 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
September 11, 2013 5:11 am

After figure 3:
To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.
Note the word [that] — it appears to be missing…

September 11, 2013 5:11 am

How can you establish a 170 year periodicity from 113 years of data?

joshv
September 11, 2013 5:12 am

Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.

Eliza
September 11, 2013 5:13 am

Unfortunately Hadcrut etc not reliable and even this would be showing cooling haha. BTW BBC is shouting NH ice lowest volume this PAST winter on its running red tape news please someone advise them.

Eliza
September 11, 2013 5:15 am

Try same with CET(reliable rural station) you will probably get a much more pronounced cooling in the future

Jonathan Abbott
September 11, 2013 5:15 am

Not being a statistician I understand about one term in ten but I think I get the gist. It all looks very impressive but appears to boil down to more curve fitting. At least the author admits to fundamental reasons why the forecast made could be entirely wrong.
Could anyone supply a summary of the the process used in plain English?

September 11, 2013 5:17 am

http://wattsupwiththat.files.wordpress.com/2013/09/clip_image034.png
Can you see? The warming will come back and will haunt us again!! 😀

Gary
September 11, 2013 5:18 am

Evaluating the math and the assumptions are way beyond me, but I would be more likely to accept the likelihood of this exercise being reasonable if a proven example of the method was demonstrated — something in signal processing which is well-understood both theoretically and experimentally.

September 11, 2013 5:22 am

As an electrical engineer, I’ve been wondering about using this approach for sometime. Mr. Patterson has done all the right things, so I don’t doubt the numbers per se. We don’t know is if the forecast portion is valid, but now that we have a model we can at least determine that going forward. What is interesting is the relative simplicity of the final result when compared with the climate models.

September 11, 2013 5:25 am

Jonathan Abbott says: Could anyone supply a summary of the the process used in plain English?
……………………
It is.

September 11, 2013 5:28 am

Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?
Because they lie in the same harmonic series (in a 3/2 ratio). The other harmonic periodicity noted (56.9) was in 3:1 ratio. (That’s what I gathered anyway.)

September 11, 2013 5:30 am

As Geofff says — it is in plain English. 😉
Plausible? Why not? I am doing similar work with other climate data using a similar technique. You can indeed spot these periodic form with less than the (normally required) full data stream.

Swiss Bob
September 11, 2013 5:36 am

The maths is way over my head but if I understand the gist correctly you are saying that using standard DSP analysis techniques you can state with greater than 90% confidence that global temperature is cyclical and is heading down.
Unlike the IPCC you at least provide the maths!

September 11, 2013 5:38 am

Compliments Jeffrey on a thorough naive evaluation of the observed temperature data updating us on the latest techniques, with estimates of the probabilities involved.
For comparison:
Armstrong & Kesten Green evaluate a naive no-change model and several simple models. See Scientific Forecasting eg for Public Policy
The near term projections are similar to a linear rise from the Little Ice Age plus a 50-60 year PDO model. See Syun Akasofu’s work provokes journal resignation
Nicola Scafetta developed an astronomical climate model using harmonic analysis to identify natural cycles plus an anthropogenic contribution. Scafetta recently published the invited review:
Solar and Planetary Oscillation Control on Climate Change: Hind-cast, Forecast and a Comparison with the CMIP5 GCMs
All of these appear to give global temperature predictions with higher near term probability than the current IPCC global climate models.
Jeffrey, It would be interesting to see who well your analysis techniques perform to hindcast/forecast from half the data to the other half.

Mike Lewis
September 11, 2013 5:41 am

If the same method was applied to data just through 1980, would it accurately predict the warming in the 80’s and 90’s?

Editor
September 11, 2013 5:43 am

A few comments:
1) decimated by 12
I know that “decimate” came from the Roman practice of inspiring soldiers after battles where the commanders felt were uninspired. They lined up the soldiers and killed every tenth man, leaving the remaining 90% inspired to win the next battle. At first blush, this sound like torturing the data until it confesses or perhaps this is “dodecimated” – killing every 12th measurement. 🙂
2) The obvious counter is that the relatively recent surge in CO2 levels and (possible) concomitant heating may be messing up any nice periodicities.
3) Would these techniques return similar results to the 2,450 year record described in http://wattsupwiththat.com/2011/12/07/in-china-there-are-no-hockey-sticks/ ?
And a free, bonus rant:
Ever since hurricane Katerina (and weak defenses) devastated New Orleans, new reporters have been saying that and other disasters have “decimated” the area. Bugs the heck out of me. I keep retorting that the reporter is a stupid idiot, but they keep on talking as though I’m not in the same room as said idiot.

johnmarshall
September 11, 2013 5:44 am

A lot of work and for that many thanks. BUT, there is always a but with temperature, the data set used is altered to reflect current theories that temperatures must increase with rising CO2 levels. Raw temperature data is notoriously variable even if all collections are from correctly sited and using the same calibrated instruments there will be temperature differences even a few yards apart. So what is the real temperature of the earth’s surface? we really do not know because temperature can only be accurate if the item being measured is at temperature equilibrium which the earth never is.

Chuck Nolan
September 11, 2013 5:46 am

WillR says:
September 11, 2013 at 5:11 am
After figure 3:
To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.
Note the word [that] — it appears to be missing…
———————————————-
Don’t add words, delete the word “is”.
cn

IanE
September 11, 2013 5:52 am

‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.

September 11, 2013 5:55 am

Eliza, what the BBC are doing (http://www.bbc.co.uk/news/science-environment-23964372) is to shift to ‘winter’ ice VOLUME. They don’t mention that it has recovered (http://psc.apl.washington.edu/wordpress/wp-content/uploads/schweiger/ice_volume/BPIOMASIceVolumeAnomalyCurrentV2_CY.png). But then this is the BBC now. As Jeremy Paxman said, they gave up the pretence of being impartial some time ago. This is what warmist organisations and individuals are doing now. As we find things not happening as planned, focus will be shifted to areas that somehow impart the feeling that we’re still doomed. It’s only when the global temperature actually starts to properly go into freefall that we will be able to prosecute warmists for their absurdity and ignorance. Just be patient (although none of us welcome cooler world) the Earth will show up the nonsense of AGW without sceptics having to do or say anything.

Editor
September 11, 2013 5:56 am

Nick Stokes says:
September 11, 2013 at 5:11 am
> How can you establish a 170 year periodicity from 113 years of data?
From the post:

The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period, although others have found significance in the 57-year (here 56.9 year) third harmonic.

So he found a 56.9 year cycle and extended it looking for the fundamental.
Here’s a mental exercise:
Draw a one cycle sine wave, marvel that it’s continuously differentiable and the results are more sinusoids. Get off that tangent.
Trim the ends so that only 113/170 of the cycle remains. Observe that “Hey, it looks like a clipped sine wave.” Apply appropriate techniques to determine the phase, amplitude, and period.
Back in my EE days, we barely had decent enough computers to do DSP, except in the CompSci dept, and I was focused on operating system and ARPAnet code. So I don’t know beans about DSP. Barring bothering to learn anything about it, it would be easy enough to take the digital data for that 113/170 th of a sine wave and convolve with with various test sines looking for the best match. So I’m sure there are mathematical techniques beyong Fourier transforms that can handle those fractional sines.

Sam The First
September 11, 2013 5:57 am

Menahwile in the broadsheet papers here in the UK, notably The Independent and The Guardian which are read by liberal / left opinion formers (inc teachers at all levels), the comments to this article below demonstrate that none of their readers is taking any notice of the long pause in warming and the overall cyclical record.
Will someone with the time and expertise please add a comment or two to explain what is really going on?
http://www.independent.co.uk/news/science/woolly-mammoths-became-extinct-because-of-climate-change–not-hunting-8808684.html

September 11, 2013 6:00 am

Hadcrut 3 is not a balanced sample of weather stations. It may give you some indication of periodicity but with bias.
Just take a random sample, of weather stations, balanced by latitude and 70/30 @sea/inland,
http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/
pay particular attention to maxima, and plot:
http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/
I don’t understand why people want to do so difficult.

September 11, 2013 6:04 am

Comment 1: While data massaging here is extreme, it does demonstrate the futility in ordinary least squares statistics so adored by AGWing scientists.
Comment 2: We know from the outset that the signal is representative of a complicated internal system (regardless of external possibilities) in its own right, with teleconnected yet varying unbalanced short and long term internal oscillations (ie decidedly non sine-waveish and very discordant). Therefore the assumption should always be that the variations are entirely internal, quite resistant to demonstrating affects of small external stimulus, and nearly impossible to predict going forward.

September 11, 2013 6:05 am

HenryP, quit calling your sample random. It is not.

Jonathan Abbott
September 11, 2013 6:07 am

To Geoff Sherrington:
It’s in plain English if you already understand the technical terminology.
*resigns self to spending a few hours reading up on signal processing techniques*
To Ian E:
You have it backwards. This is observing the wriggling trunk and trying to work out what the elephant looks like.
To all WUWTers:
I have seen doubts expressed in some places on the applicability of techniques developed for electronic signal processing when applied to the climate. Straw man? Valid?

John Day
September 11, 2013 6:08 am

@Bruckner8
>Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?
Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem
The sequence of odd harmonics in the processed waves suggests that windowing artifacts remain, as the convolution of sine waves (which are always infinite in extend from an abstract mathematical viewpoint) multiplied by a sampling window (“square wave”) would suggest.http://en.wikipedia.org/wiki/Convolution_theorem
In AR modeling, the predictive modeling results depend on the time series being stationary, i.e. constant mean and variance. But historical temperature records are not stationary, and in fact it is the non-stationary part of the record that you are trying to model. So results of predicting values in the future based on values in the past will generally not be credible.
http://en.wikipedia.org/wiki/Wold's_theorem
😐

Jeff Patterson
September 11, 2013 6:11 am

Nick Stokes says: How can you establish a 170 year periodicity from 113 years of data?
In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data. if the condition is not met, the high frequency components masquerade as low frequency components in a process referred to as aliasing. That is why we remove the high frequency components by filtering before decimating (re-sampling) the data.

September 11, 2013 6:12 am

Ric, don’t ya love “back in my day” times? Want to do an Anova, or Coanova? Take an MSDOS computer with a blank screen and program it (I programmed mine to say good morning to me every time I turned it on). Or grab a calculator, the clunky dark cream colored kind with sharp square corners, and do it that way. Hell, just use a pencil and paper. I remember getting all twitterpated when I got a little Mac SE with Statview installed on it. I thought it was Christmas every day.

richardscourtney
September 11, 2013 6:14 am

Friends:
I second the request of Mike Lewis at September 11, 2013 at 5:41 am; viz.

If the same method was applied to data just through 1980, would it accurately predict the warming in the 80′s and 90′s?

I note the strong caveat in the above essay concerning the possible inappropriateness of extrapolation of the obtained curve. However, if it is assumed that the curve fit does capture the form of the temperature changes, then failure of the method to predict the temperature curve after 1980 would imply the temperature changes are internal variability of the climate system and are not externally forced.
Richard

Jeff Patterson
September 11, 2013 6:17 am

IanE says:
September 11, 2013 at 5:52 am
‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.
As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.

zentgraf2
September 11, 2013 6:18 am

A good intellectual exercise and perhaps more valid than the 73 worldwide GCM’s. If by 2040 the forecast pans out, we can save a lot of tax money. No need for an understanding of climate physics or chemistry. Just have a small group of DSP mathematics folks hidden away in a skunk works.
I am joking ,of course. Nevertheless I yearn for a creditable effort that would create severe competition for the circular thinking climate science community to reorient their thinking towards a comprehensive understanding of the natural forces in climate.

John Day
September 11, 2013 6:27 am

@Me

Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

As soon as I hit the “Post Comment” button I realized that my reponse was wrong because the 170-year wave in question is indeed “over-sampled”, not under-sampled.
But the reason I was so sure this couldn’t be done is that I’ve tried to reconstruct these kinds of “over-sampled partial waves” myself and have never been successful at it. But perhaps I going at it in the wrong way.
😐

kb
September 11, 2013 6:29 am

“extracting spurious signals from noisy data”
Hopefully this means “extracting meaningful signals” or “removing spurious signals”…

Editor
September 11, 2013 6:29 am

John Day says:
September 11, 2013 at 6:08 am

@Bruckner8
>Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?
Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

This all applies to periodic sampling of a periodic signal. That’s not what we have here, we have many samples of part of a single cycle.
Had this been similar to Nyquist sampling, there would be either one or two samples of the entire 113 years. Instead, the sampling rate is far higher than that and there are hundreds of samples to work with.

Greg Goodman
September 11, 2013 6:31 am

I’ll parse this in more detail later but initial reactions:
1. HadCruft is a land + ocean amalgam. Rate of change of land is twice that of SST, how will this affect freq decomposition.
2. “To remove seasonal variations while avoiding spectral smearing and aliasing effects, the data was box-car averaged over a 12-month period and decimated by 12 ” Running mean filter _introduce_ spurious signals. http://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/
3. A 12m filter it NOT an appropriate anti-aliasing filter for 12m decimation
4. HadCruftX is probably aliased already even in the monthly data series further filtering will not fix it once it’s done: http://judithcurry.com/2011/10/18/does-the-aliasing-beast-feed-the-uncertainty-monster/
5. Series of add harmonics suggests a triangular waveform likely due to non stationary data. The first and last point trick is new to me and I can’t see this being any guarantee of the conditions needed for FFT. Linear detrending of whole dataset or taking first diff would be more usual.
It’s an interesting idea and I don’t want to totally dismiss the result without a closer look, but this seems like a fair list of processing errors so far.

RC Saumarez
September 11, 2013 6:36 am

This appears to be an example of the well known fallacy of Fourier Extrapolation.

Luther Wu
September 11, 2013 6:39 am

Thank you, Mr. Patterson for your fine work.
Fortunately, you’ll need spend no effort trying to construct a compensatory microcode.
Bit of a busman’s holiday, is that it?

G. Karst
September 11, 2013 6:42 am

If the above statistical analysis is valid ( as it appears)… why not submit it for publication and peer review. Surely the gatekeeping practices, of climate research, have somewhat abated since the climategate revelations. Or is this, just a pre-review before submission. GK

Jeff Patterson
September 11, 2013 6:46 am

Here’s a hopefully approachable summary:
A time record can be represented in the time domain (like the Hadcrut plot) or the frequency domain (like the power spectral density plots in the post). The frequency domain looks like an old fashioned radio dial, with the high frequencies to the right and the low frequencies to the left. A radio station is assigned a frequency, unique from all others in the area, so that its signal only appears one place on the dial and is not interfered with by other stations. If we were do do a spectral analysis (i.e. a PSD plot) of the AM band, we would see a peak at each frequency where a station is broadcasting and a valley where no station was present.
We can transform from one domain to the other without distortion via the Fourier Transform. It uses every possible frequency which are all harmonically related to the frequency at which the data was sampled (i.e. monthly in the case of Hadcrut3) to exactly represent the data in the frequency domain. To reconstruct the 113 samples in my decimated record exactly would require 226 variables (113 frequencies and their phases). We show that we excellent results using just 5 harmonically related frequencies where instead of using the lowest frequency of the FFT, we use the frequency with the highest peak as the fundamental. Actually, the 5th harmonic doesn’t add much and most of the fit is captured with just 3 harmonics.
The great thing about sine waves is that they repeat which means if they are really present, the model can be used to predict the future trend. The bad thing is that it is easy to get fooled if the data is noisy. The second half of the paper examines the likelihood that we were fooled. By generating some random sequences from a process designed to produce data that in character (peak-to-peak changes, intermediate slopes etc.) matches the temperature record, we can perform the same harmonic decomposition we did to the temperature data and see how many times the fit is as good as we achieved with the actual data. In this case, only .6% of the 500 trials were as good, which places the probability that we were fooled at about 8%

September 11, 2013 6:47 am

Hmmm, based on the data, we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome. Oh boy, who would have guessed?

Joe Chang
September 11, 2013 6:49 am

Lets assume that the periodicities are fixed as opposed to variable. Given the 113 years of data, the longest periodicity extracted was 170 years. Lets us also assume that the shorter periodicities have been correctly extracted. So now, instead of projecting forward to take a stab at the future, how about projecting backward, past the LIA, MWP, RWP. Then with an estimate as to the magnitude and time of these events without concern on the decade scale wiggles, guess what the longer periodicities are? Sure this is torturing data …

Jeff Patterson
September 11, 2013 6:49 am

Greg Goodman says:
September 11, 2013 at 6:31 am
… Running mean filter _introduce_ spurious signals.
Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

pochas
September 11, 2013 6:53 am

That fundamental of 170.7 years speaks strongly of the Jose cycle. If Patterson knew about it (I suspect he did) there is some danger of confirmation bias. I have the same bias. Fortunately the projected cooling is modest. I hope it stays that way. As for the usual comment that this is a curve-fitting exercise, in a world of junk “physics-based” math models, faulty theory, manipulated data and agenda-driven science, it’s really all we have.

September 11, 2013 6:54 am

The 170 year cycle length is suspiciously close to the approximate cycle length of the solar system barycenter. the 170 year cycle continues to show up in the works of many different authors. as does the 55-60 year harmonic of 170, which closely approximates the observed length of the PDO.
Based on what we know about tidal prediction, DSP is likely to be at least as accurate as the global climate models in predicting future climate, and quite likely a whole lot more accurate.
if one tries to model the ocean tides using first principles, as is done with the climate models, the results are garbage. earth’s tides would be miniscule based on gravitational attraction alone.
the earth’s tides are predicted to a high degree of accuracy by the motion of the celestial bodies. this is because the earth’s oceans have over thousands and millions of years, come to oscillate in phase with the motion of the celestial bodies.
it is completely likely that s that climate oscillates similar to the tides, only on a longer timescale, as a result of a small but regular harmonic forcings based on the motion of the celestial bodies over thousands and millions of years.
this suggests that we can predict climate with some hope of accuracy using methods similar to what the author has proposed here today.

Greg Goodman
September 11, 2013 6:54 am

This is similar in some ways to what I did 18 ago on Judith Curry’s site where I fitted just three , non harmonic frequencies to examine how Met Office processing had changed the data. (NB this was not intended as a prediction tool, just an evaluation of changes to the existing data).
: http://judithcurry.com/2012/03/15/on-the-adjustments-to-the-hadsst3-data-set-2/
see middle panels in figures 9 and 10.
It is interesting that I found hadSST (sea temp only) had _removed_ a 160 year periodicity. Maybe the combined use of Had + CRUT here, puts it back ;?

Rod Everson
September 11, 2013 6:57 am

The relevant line from a very long article: “Even if we accept the premise of statistical significance, without knowledge of the underlying mechanism producing the periodicity, forecasting becomes a suspect endeavor. ”
Let’s not duplicate the errors of our adversaries and assume we know more than we know.

Jeff Patterson
September 11, 2013 6:58 am

RC Saumarez says: This appears to be an example of the well known fallacy of Fourier Extrapolation.
HArmonic decomposition is much more constrained than fourier analysis but subject to some of same concerns about seeing periodicity that isn’t there. The last half of the paper address this issue directly.

Greg Goodman
September 11, 2013 6:59 am

JP “Exactly why I used box care (i.e. non overlapping) filtering, not running mean.”
Could you explain what you mean by “box car” in more engineering/mathematical language , maybe I misunderstood what you mean by railway analogy. 😉

rabbit
September 11, 2013 6:59 am

Jeffrey Patterson:
Have you considered Singular Spectrum Analysis (SSA) to extract the dominant modes of the temperature graph? This does not suffer from the windowing effects that a DFT suffers from, nor is it limited to a prescribed set of basis functions.

Jeff Patterson
September 11, 2013 7:01 am

pochas says:
September 11, 2013 at 6:53 am
That fundamental of 170.7 years speaks strongly of the Jose cycle.If Patterson knew about it …
I do not. I did a cursory google search for “170 year climate cycle” but couldn’t find anything relevant, I would appreciate a reference if you have one.

pochas
September 11, 2013 7:05 am

If we’re dealing with harmonics of 170 years, there is a millenial signal that is missing.

Jeff Patterson
September 11, 2013 7:06 am

@Greg Goodman – Box car decimation is simply dividing the recorded into N equal length segments and taking the average of each segment as the new sample (thus decimating by N). Unlike MA averaging, it introduces no correlation between samples.

RC Saumarez
September 11, 2013 7:09 am

I agree with Greg Goodman.
There are some rather astonishing propositions here.
The first seems to be a fundamental misubderstatnding of the relationship between the Fourier Transform and the Discrete (complex exponential) discrete Fourier series i.e. the FFT. In the former case there is an infinite resolution in frequency but requires the signal to be observed from – to + infinity. In the case of the FFT, if the fundamental frequency is determined by the length of the record. One might try to infer the presence of lower frequencies, but this lacks any formal justification.
The anti-aliasing processing does not seem to be correct and is using data that may be aliased.
The processes involved are almost certainly non-stationary.
This approach says that because the earlier record looks like a sine wave, we assume that is what will happen in the future. If you perturb a low-pass system, such as an ARMA system, you will get trends whose magnitudes depend on the persistance of the process. I am very suspicious of the autocorrelation function presented here – I suspect that it is artefactually widened due to prior filtering – an elementary mistake and that persistance is overestimated:
http://judithcurry.com/2012/02/19/autocorrelation-and-trends/

pochas
September 11, 2013 7:10 am

Jeff Patterson says:
September 11, 2013 at 7:01 am
“I do not [know about the Jose cycle]. I did a cursory google search for “170 year climate cycle” but couldn’t find anything relevant, I would appreciate a reference if you have one.”
http://www.ann-geophys.net/18/399/2000/angeo-18-399-2000.pdf
or simply Google (Jose cycle)

Kasuha
September 11, 2013 7:11 am

I am really curious how well would this method “predict” second half of the record from the first half.

John Day
September 11, 2013 7:15 am

@Jeff Patterson

September 11, 2013 at 6:49 am
Greg Goodman says:
September 11, 2013 at 6:31 am
… Running mean filter _introduce_ spurious signals.
Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

So the artifacts will not overlap, but they’re still there. The only window filter that I know of that has no artifacts is a Gaussian window. It produces no sidebands.
Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.
😐

Greg Goodman
September 11, 2013 7:16 am

Suggested improvements :
1. triple RM filter at 24m before 12m decimation would be correct anti-alias.
2. why decimate anyway? data is not cumbersome and any aliasing that in the monthly data will not go away by further filtering.
3. Use land or sea (or both separately) not amalgam dataset.
4. work on rate of change dT/dt (from first difference of data) since what we want to know is temp change anyway. This will remove the slope (and likely much of the odd harmonics) whilst preserving any true 170y or other cycles.

Jeff Patterson
September 11, 2013 7:16 am

@Greg Goodman – I’ve addressed your 1st concern elsewhere. The others are concerns about aliasing which is of course always a concern. I disagree with your statement regarding box care filtering – the zeros of the filters transfer function lie exactly on the aliasing spectral lines. It is a very effective re-sampling filter. Of course it can’t do anything for the aliases already present in the data, but as you can see from the PSD plots (e.g. Figure 9), the high frequencies are rolled of by some 40dB so any aliasing will be insignificant.

Bruce Cobb
September 11, 2013 7:18 am

richardscourtney says:
September 11, 2013 at 6:14 am
Friends:
I second the request of Mike Lewis at September 11, 2013 at 5:41 am; viz.
If the same method was applied to data just through 1980, would it accurately predict the warming in the 80′s and 90′s?
I note the strong caveat in the above essay concerning the possible inappropriateness of extrapolation of the obtained curve. However, if it is assumed that the curve fit does capture the form of the temperature changes, then failure of the method to predict the temperature curve after 1980 would imply the temperature changes are internal variability of the climate system and are not externally forced.
Richard

Yes, but step 1 should be to determine how accurately those temperature changes have been recorded. We know that there has been a warm bias, and that needs to be subtracted.

September 11, 2013 7:19 am

the sun’s motion around the solar system barycenter roughly repeats every 171.4 years as measured by its angular momentum. this is remarkably close to the author’s 170.7 year cycle length in the Hadcrut3 temperature data.
if one sets aside the argument over the mechanism by which climate oscillates with a period of 170 years, and simply accept that there is evidence for such an oscillation in the observed data, then one can go forward and accept that prediction is possible.
humans learned to predict the seasons long before we understood the mechanism. why then should we assume that we cannot predict climate without understanding the mechanism?
history shows that almost always the prediction comes BEFORE the understanding. when the prediction is shown to be correct by observation, this suggest the mechanism. and from there we investigate to discover the mechanism that governs the prediction.
for example, the motion of the planets in the heavens. we learned to predict this with amazing accuracy long before we discovered that gravity was the mechanism. modern science would tell us that such a thing was impossible, you cannot predict without a mechanism.
the facts are that humans invent mechanisms to explain predictions, and more often than not these mechanism are later shown to be false. this in no way affects the accuracy of the prediction. right answer, wrong reason is still a right answer.

Jeff Patterson
September 11, 2013 7:20 am

@Greg Goodman says work on rate of change dT/dt (from first difference of data) since what we want to know is temp change anyway. This will remove the slope (and likely much of the odd harmonics) whilst preserving any true 170y or other cycles.
First difference processing suppresses low frequencies and greatly extenuates high frequencies (it’s transfer function match a continuous-time differentiator across most of the Nyquist band). It also introduces frequency dependent phase shifts which would adversely effect the reconstruction.

Greg Goodman
September 11, 2013 7:26 am

John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”
The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.
No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.
Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. 🙁 )

RC Saumarez
September 11, 2013 7:28 am

You miss the point about Fourier Extrapolation.
Figure 13 is simply a re-run of the 20th century – i.e.: a periodicity. There is no theoretical justification for this approach – it simply says that the future will be a repeat of the past. What you have done here is to apply some extremely artificial constraints on the system by assuming that:
a) You have characterised the system correctly
b) The inputs to the system are stationary.

Jeff Patterson
September 11, 2013 7:29 am

RC Saumarez says:
The first seems to be a fundamental misubderstatnding of the relationship between the Fourier Transform and the Discrete (complex exponential) discrete Fourier series i.e. the FFT. In the former case there is an infinite resolution in frequency but requires the signal to be observed from – to + infinity. In the case of the FFT, if the fundamental frequency is determined by the length of the record. One might try to infer the presence of lower frequencies, but this lacks any formal justification.
I don’t think I misubderstatnd 🙂 anything. I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).
I addressed your record-length concern in the paper (see the section on record periodization).

Greg Goodman
September 11, 2013 7:39 am

JP “First difference processing suppresses low frequencies and greatly extenuates high frequencies (it’s transfer function match a continuous-time differentiator across most of the Nyquist band). It also introduces frequency dependent phase shifts which would adversely effect the reconstruction.”
It’s true that differential attenuates as 1/f but this does not remove the 170 as I found in my Curry article linked above, from which this comes. (I found 164 years)
http://curryja.files.wordpress.com/2012/03/icoads_monthly_adj0_40-triple1.png
Working on monthly resolution would make the phase shift minimal but you could use a three point diff to retain a symmetrical kernel and avoid the phase shift problem. (-0.5, 0, +0.5)
A 3-sig gaussian of sigma=12m should remove most of the annual and could be combined with the above into a gauss-diff kernel to do it all in one operation with better approximation of the true (non discrete) differential.

Jeff Patterson
September 11, 2013 7:39 am

RC Saumarez says:
September 11, 2013 at 7:28 am
You miss the point about Fourier Extrapolation.
>>Figure 13 is simply a re-run of the 20th century – i.e.: a periodicity.
Figure 13 contains 500 paths. They can’t all match the single path of the 20th century,
>>There is no theoretical justification for this approach – it simply says that the future will be a repeat of the past.
Which will be partially true in a the mean sense if the periodicity is present. As I took pains to point out though, that’s a big if and any projection depends on the major climatic parameters remaining constant over the projection period. I thought I was pretty clear on that.
What you have done here is to apply some extremely artificial constraints on the system by assuming that:
a) You have characterised the system correctly
b) The inputs to the system are stationary.
I am confused by (b). The projection assumes a periodic input process (and stationary climate process). Periodic signal + noise is decidedly a non-stationary process. Interesting though, the residual of the fit passes the Unit Root Test with a p vaule of ~ 10^-12. And the decimated record is at least weakly stationary (p = 10^-6).

Jeff Patterson
September 11, 2013 7:45 am

If you perturb a low-pass system, such as an ARMA system, you will get trends whose magnitudes depend on the persistance of the process.
Exactly the point of the second half of the paper where I shown that the probability of achieving similar goodness of fit results to an ensemble of AR process outputs is small (but not insignificant).
Note that I get even better results (i.e. lower probability of fluke) with an ARMA process although I can’t explain why.

Greg Goodman
September 11, 2013 7:46 am

JP: “I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).”
But in fitting your five harmonic model to the time series with no linear component you are just modeling the general slope with a truncated FS based on 170. RC point about repetition still stands. Plot your model out over 1000 years , it will be 170 year saw-tooth , more or less.

Jeff Patterson
September 11, 2013 7:47 am

Greg Goodman says: It’s true that differential attenuates as 1/f
Actually it’s 1/f^2. Low frequency attentuation is extreme.

stevefitzpatrick
September 11, 2013 7:48 am

“The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period”
.
And that is where the essay should have stopped. You simply can’t make any meaningful long term projection of periodic behavior based on too little data. Sorry, but this analysis is 100% unconvincing.

richardscourtney
September 11, 2013 7:48 am

Ken Coffman:
Your post at September 11, 2013 at 6:47 am says in total

Hmmm, based on the data, we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome. Oh boy, who would have guessed?

Eh? What gives you that idea? Please explain?
Richard

September 11, 2013 7:52 am

@Jeff Patterson
To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique is called record periodization. The data is regressed about a line connecting the record endpoints, dropping the last point in the resulting residual. This process eliminates the endpoint discontinuity while preserving the position of the spectral peaks (although it does extenuate the amplitudes at higher frequencies and modifies the phase of the spectral components).
There is a lot going on behind the scenes in this paragraph that I thinks needs more explanation.
“The data is” What data? The time series? [I think] Or the resulting PSD?
“regressed” how?
“about a line connecting the record endpoints”.
Did you draw a line from Point 1 to point 113? Did you do a least squares regression line across the [time-based?] data?
Figure 3, to 4
“Dropping the last point”. Fourier analysis has to assume that the time based data record repeats. You must think of the time series as a chart recorder plot where you wrap the end to the beginning as a cylindrical plot. There will be a splice where Point 113 meets Point 1. Dropping the last point doesn’t remove the splice. I don’t see what good that does.
The peaks are harmonically related, with odd harmonics dominating until the eighth. Since spectral resolution increases with frequency, we use the eighth harmonic of the periodized PSD to estimate the fundamental
Huh? Why should we conclude the peaks [in the Fig. 4 PSD] are harmonically related? Why use the eighth to estimate the fundamental harmonic? What if we used the 5th, 3rd, of 7th? This is not obvious.
The x-axis of Fig 3, Fig 4, Fig 7, Fig 9a, Fig 9b need some labels and units. They leave too much work for the reader.
From inspection of the PSD we form the harmonic model
I need more.
fit the model to the original (unperiodized) data
I give you credit for putting the 6 line parameter function of this step in the paper, but an illustration would have been nice.
Fig. 10 spaghetti graph.
What was allowed to vary between trials? What stayed the same?

Greg Goodman
September 11, 2013 7:56 am

Bearing mind also that much of the 60y cycle was inserted by Hadley ‘corrections’. (Rightly or wrongly).
http://curryja.files.wordpress.com/2012/03/hadsst3-cosine-fit1.png

September 11, 2013 7:58 am

Mr. Courtney, you don’t see a span that includes a cooling anomaly in the projection? I do. Watch and see, this is a theme you’ll see pop up here more and more: the recognition that that data does not exclude the fact that the overall effect of adding CO2 to the atmosphere could be cooling, not heating.

richardscourtney
September 11, 2013 8:00 am

Bruce Cobb:
In response to my supporting the view (shared by several in this thread) to assess if the method applied to half the time series can predict the other half, you have replied September 11, 2013 at 7:18 am

Yes, but step 1 should be to determine how accurately those temperature changes have been recorded. We know that there has been a warm bias, and that needs to be subtracted.

Sorry, but no. We are discussing an ability to model the data set and not its ability to model whatever you, I or anyone else thinks the data set should be.
However, if you are saying the data is dubious, then I agree; see
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc0102.htm
Whilst I agree that point, I stress that it is not relevant to considerations in this thread.
Richard

John Day
September 11, 2013 8:00 am

Greg Goodman says:
September 11, 2013 at 7:26 am
John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”
The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.
No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.
Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. 🙁 )

Greg,
Of course moving-average (“boxcar”) filters are very intuitive and simple to implement. But that doesn’t change the fact that they also produce spurious artifacts in their output.
Gaussian filters are “ideal” in the sense that they do not produce these artifacts. But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.
http://en.wikipedia.org/wiki/Gaussian_filter
“Gaussian filters have the properties of having no overshoot to a step function input while minimizing the rise and fall time. This behavior is closely connected to the fact that the Gaussian filter has the minimum possible group delay. It is considered the ideal time domain filter,…”

Greg Goodman
September 11, 2013 8:00 am

Jeff Patterson says:
Greg Goodman says: It’s true that differential attenuates as 1/f
Actually it’s 1/f^2. Low frequency attentuation is extreme.
===
In the power spectrum maybe but in FT or your fitted model 1/f . d/dt(cos(wt)) ….

RC Saumarez
September 11, 2013 8:01 am

I am very surprised to see this.
You are imposing highly artificial constraints on the system. Your projected results (fig 13) are simply a periodic rerun of the 20th century. What you have done, in essense it to assume an impulse response based on auto-correlation, and perturbed this with a periodic input to get a result.
The point is that you cannot gain a statistical characterisation of your model because the persistance of the climate system is too long, compared to the record length, to allow segmentation of the record and improve estimated accuracy. It is inevitable that random inputs into a system with persistence will produce trends. The estimation of that persistence is very difficult and distinction between competing physical models almost impossible with current data. Nevertheless, trends that are comparable to that seen in the modern record are just possible with a randomly excited Hurst system. See a tutorial post:
http://judithcurry.com/2012/02/19/autocorrelation-and-trends/
The model assumes a periodic defined input and noise with some statistical properties. This is a stationary input, in the commonly accepted sense, because it has constant parameters with time. This is a huge assumption.
I would comment that this is the blind application of signal processing without giving sufficient thought to underlying physical assumptions: I wrote here, in a tutorial post
http://wattsupwiththat.com/2013/04/09/correlation-filtering-systems-and-degrees-of-freedom/
“Modern scripting programs such as “R” allow one to perform many signal processing calculations very easily. The use of these programs lies in not applying them blindly to data but in deciding how to use them appropriately. Speaking from bitter experience, it is very easy to make mistakes in signal processing and it is difficult to recognise them. These mistakes fall into three categories, programming errors, procedural errors in handling the signals and not understanding the theory as well as one should. While modern scripting languages are robust, and may largely eliminate straight programming errors, they most certainly do not protect one from making the others!”

Jeff Patterson
September 11, 2013 8:04 am

John Day says:
In AR modeling, the predictive modeling results depend on the time series being stationary, i.e. constant mean and variance. But historical temperature records are not stationary, and in fact it is the non-stationary part of the record that you are trying to model. So results of predicting values in the future based on values in the past will generally not be credible.
The residual _is_ stationary. That’s the point. We can certainly drive an AR process with stationary (and in this case also nearly Gaussian) noise + sine waves and get a valid projection. This of course, as I pointed out, depends on the assumptions the the observed periodicity is a forcing function (i.e.. not internal) and the AR process itself is time invariant. As I mentioned, this might not be the case and so the projection should be taken with the appropriate grain of salt.
The main point (beside the surprisingly good fit that can be achieved with such few harmonically related elements) was not the projection but rather that there exists plausible explanations for the observed data which do not require significant forcing from anthropogenic sources.

September 11, 2013 8:09 am

I wonder about the futility in taking apart the frequency response of all the combined signals from all the sensors. It reminds me of signal frequency response testing (IE hearing aids, brainwaves, musical instruments, etc). Isolating instruments producing a combined sound by filtering out the known frequency response of each note of the various instruments is possible. But in temperature signals, the instruments producing the combined frequency response are many more and they are not playing from the same sheet of music. Each instrument (each sensor) produces a series of sounds in its little music room which has separate acoustics to the one down the road that is producing its own sounds in its little music room which has separate acoustics to the next one down the road, and so on. Logically, at best you could filter out all the separate acoustics back to a single one and say something about that instrument’s data and its particular music room and its particular sheet of music. Putting them all together and saying something about the resulting frequency response is probably not very useful and would lead to a high degree of spurious conclusions.

Onion
September 11, 2013 8:10 am

So I’d like to see this analysis applied up to 1990 to observe how its predictions from 1990 fared compared to reality.

Jim G
September 11, 2013 8:13 am

joshv says:
September 11, 2013 at 5:12 am
“Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.”
Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out. As I have said before, one can obtain any result one desires with enough data manipulation and statistical analysis.

richardscourtney
September 11, 2013 8:14 am

kencoffman:
Thankyou for your answer to me at September 11, 2013 at 7:58 am in reply to my request for clarification.
I understand your reply to say you have invented from whole cloth your notion that

we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome.

I fail to see how your invention has any relevance to this thread whether or not you “can see a span that includes a cooling anomaly in the projection”.
Richard

Greg Goodman
September 11, 2013 8:14 am

JD: “But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.”
If you add the proviso “for the same length filter” you are totally correct. I work from the spec I need so I compare two filters with similar FWHM and conclude I need a longer window to do the same job without unacceptable defects ( be it gaussian or triple-RM).
I actually favour 3RM for this kind of stuff since it has a zero in frequency. It’s marginally better than gauss for removing annual, for example.
http://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/

Jim G
September 11, 2013 8:17 am

Pamela Gray says:
September 11, 2013 at 8:09 am
“Putting them all together and saying something about the resulting frequency response is probably not very useful and would lead to a high degree of spurious conclusions.”
As one who plays guitar regularly I would say such methods also destroy the melody which is what one is looking for in this analogy.

September 11, 2013 8:21 am

Jeff Patterson on September 11, 2013 at 6:46 am
Here’s a hopefully approachable summary:

– – – – – – – –
Jeff Patterson,
Thanks for your effort.
I particularly appreciate your care in describing the limitations and shortcomings of your methods.
John

commieBob
September 11, 2013 8:23 am

Folks, we have been schooled on digital signal processing here. Nicely done Jeff.
Anyone interested in the topic should check out The Scientist and Engineer’s Guide to Digital Signal Processing. The book is a free download and is written for people who might actually want do practical signal processing. There are pitfalls that await anyone who naively attempts to use dsp techniques in real life (as opposed to a classroom exercise) and many undergrad textbooks don’t mention them.

September 11, 2013 8:23 am

Is this anything to do with Mr.T.Pratchett or the unseen university? There is certainly a whiff of Hex in the methodology and written conclusions.

Gary Pearse
September 11, 2013 8:24 am

Chuck Nolan says:
September 11, 2013 at 5:46 am
WillR says:
September 11, 2013 at 5:11 am
After figure 3:
“To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.
Note the word [that] — it appears to be missing…
———————————————-
Don’t add words, delete the word “is”.”
cn
Grammatically it is okay to eliminate the first “that”, too. Also, the “the” before ‘spectral’. There, Chuck Nolan, WillR and I have made a statement on our full understanding of the mathematical process!

crosspatch
September 11, 2013 8:26 am

The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen. Would an analysis of the opening of the 1812 Overture predict the finale? About the only thing I am prepared to say with any certainty is that the next 1000 years is more likely to be cooler than the last 1000 years than it is likely to be warmer.

Phil C
September 11, 2013 8:31 am

Thanks Jeff for a clear and concise post. I have more than a smattering of exposure to digital signal processing and Fourier analysis(although not enough to do it myself!) so I could understand what you were trying to do. You carefully noted and explained the limitations(non-linear externally forced vs. internally generated) and showed that statistically the result is unlikely to be a random error.
Now I’d like to see some publications examing the hypthesis that the climate is an exteranally forced linear process, or showing that it is an interanlly forced randlom process.

Greg Goodman
September 11, 2013 8:32 am

JP: “The main point (beside the surprisingly good fit that can be achieved with such few harmonically related elements) was not the projection but rather that there exists plausible explanations for the observed data which do not require significant forcing from anthropogenic sources.”
Sorry, maybe I missed something in the article. What plausible explanation did this present?

September 11, 2013 8:33 am

Crosspatch, actually that is likely. There is a musicality test, standardized no less, that in its simplest form, has the examinee predict what note comes next after a short series of notes. Because the muical scale is mathematical and orderly one can if one has an ear for the “language” predict what comes next.

Phil C
September 11, 2013 8:33 am

sorry, “hypothesis”. My right ring fingertip is numb from a cut, hard to feel where the upper rlow lf keys is.

IanE
September 11, 2013 8:34 am

‘Jeff Patterson says: September 11, 2013 at 6:17 am
IanE says:
September 11, 2013 at 5:52 am
‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.
As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.’
But of course he could choose any sort of function – he chose something that worked, he did not have to use the approach he ended up with. That is the fundamental problem with data-mining exercises.

milodonharlani
September 11, 2013 8:34 am

crosspatch says:
September 11, 2013 at 8:26 am
Right you are. The long-term temperature trend for the second half (past 5700 years) or third third (3800 years) of the Holocene Interglacial is down. Simply extrapolating that trend gets us into the next glacial sooner rather than later. The trend could reverse, but hasn’t done so in any prior interglacial, once established.
Too bad that man-made GHGs can’t do that trick.

Bruce Cobb
September 11, 2013 8:37 am

richardscourtney says:
That occurred to me afterwards. The entire exercise is pointless, then. Proper science demands accuracy from the start. If you are starting with data that are flawed, the result is going to be flawed as well.

Jim G
September 11, 2013 8:41 am

Pamela Gray says:
September 11, 2013 at 8:33 am
“Crosspatch, actually that is likely. There is a musicality test, standardized no less, that in its simplest form, has the examinee predict what note comes next after a short series of notes. Because the muical scale is mathematical and orderly one can if one has an ear for the “language” predict what comes next.”
This is very true in 1,4,5 chords in simple tunes such as blue grass and old time rock and roll, not so much in classical, at least not that I can tell. Too complex and many times have less repetitive sequences. But then I do not play classical. But torture the “data” enough and you will lose the tune.

DayHay
September 11, 2013 8:42 am

Nice science, you put it out there, with all the data, your assumptions, process, limitations, admission of doubts, etc and let everyone rip it apart. Then you defend. All publicly. I think a few other “climate scientists” could take a lesson here, Great post and discussion.

DesertYote
September 11, 2013 8:47 am

Yay Agilent! (E44xx) I was wondering when someone would apply modern Information Theory or Signal Processing techniques to this problem, let alone modern statistical methods. I have yet to see any study doing bootstrap 🙁

DirkH
September 11, 2013 8:58 am

I like it a lot. A quantification of the likelihood that the model has predictive skill. That’s more than the IPCC gives you.

DirkH
September 11, 2013 9:01 am

Jim G says:
September 11, 2013 at 8:13 am
“Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out.”
That’s completely unfair. Jeff has argued convincingly why there is reason to believe that the periodicity is real.

Pippen Kool
September 11, 2013 9:10 am

Can you “predict” the last 25 years of your data set analyzing just the years before ?
I mean, using your same methods , can you predict the warming in the last few decades?

Slartibartfast
September 11, 2013 9:11 am

Get off that tangent.

8/

September 11, 2013 9:15 am

In the last year in a series of posts at
http://climatesense-norpag.blogspot.com
I have laid out a method of climate forecasting based on recognising quasi repetitive- quasi cyclic patterns in the temperature and other relevant climate-driver data time series.Pattersons post illustrates a useful approach to recognising possible patterns.Interestingly his forecast cooling until 2040 followed by warming until about 2070 is generally similar to my projections..However I forecast further cooling beyond 2070 based on incorporating the 1000 year solar millennial cycle into the forecast.
We should not expect mathematical precision in this type of forecast because of the changing resonances between the quasi cyclic rate processes which integtrate into the temperature climate metric.It would however be very useful if Patterson could extend his analysis to the 2000 year proxy record of Christiansen and Ljundqvist 2012 which is the basis for most of my projections for the next several hundred years .The following simple relationships should be noticed.Pattersons 171 year cycle = 3x PDO+/- 3X 171 =513, 6 x 171 = 1026. These intervals relate to the Roman Climate Optimum ,the Dark Ages ,the MWP,the LIA and the Modern Peak.The shape of the curve is not a simple sine curve. The Christiansen record of the last 1000 years is the most obvious go-by for the future trends
.Here is part of my original post on this subject at the above link on 1/22/13
“e) The real controlling factor for the immediate future is where the present day stands relative to the approximate 1000 year solar cycle peak
f) Having looked at numerous reconstructions I suggest that that of Christiansen and Ljungqvist 2012 is the most likely to be closest to reality. : http://www.clim-past.net/8/765/2012/cp-8-765-2012.pdf
Discussion.
The key working hypothesis is that the solar cycle from 1000- 2000 may repeat and we may see a pattern of temperatures from 2000 – 3000 which is similar to that from 1000 – 2000. Fig.5 from the Christiansen paper is shown above.The solid lines are the 50 year moving averages and the dashed red lines are the upper and lower quantiles.
Inspection of Figure 5 – both the moving average and the annual data suggests the following.
1) The millennial peak is sharp – perhaps 18 years +/-. We have now had 16 years since 1997 with no net warming – and so might expect a sharp drop in a year or two – 2014/16 -with a net cooling by 2035 of about 0.35.Within that time frame however there could well be some exceptional years with NH temperatures +/- 0.25 degrees colder than that.
2) The cooling gradient might be fairly steep down to the Oort minimum equivalent which would occur about 2100. (about 1100 on Fig 5) with a total cooling in 2100 from the present estimated at about 1.2 +/-
3) From 2100 on through the Wolf and Sporer minima equivalents with intervening highs to the Maunder Minimum equivalent which could occur from about 2600 – 2700 a further net cooling of about 0.7 degrees could occur for a total drop of 1.9 +/- degrees
4)The time frame for the significant cooling in 2014 – 16 is strengthened by recent developments already seen in solar activity. With a time lag of about 12 years between the solar driver proxy and climate -see:
http://adsabs.harvard.edu/full/2005ESASP.560…19U we should see the effects of the sharp drop in the Ap Index which took place in 2004/5 in 2016-17. This estimate is quite independent from the estimate made from Fig5.
Conclusions1) It seems reasonably probable – say 60/40 that the NH will cool by about .35 degrees by 2035.
2) We should be able to check the accuracy of this forecast by 2018 -20.
3)The forecast of a 1.2 degree drop by 2100 is little more than a mildly interesting suggestion at this time.
4)The idea of a Maunder Minimum equivalent at 2600 – 2700 is highly speculative.
5)Contrary to the forecasts made here, the Livingston and Penn solar data are suggesting a possible Maunder type Minimum in the near future.Given our ignorance of solar physics this is entirely possible. In this case a much more rapid cooling would occur with very serious consequences to the global food supply and the world economy.
6) Global cooling will take place concurrently with that of the NH but because of the great extent of the southern oceans the global cooling will be significantly less – maybe +/- 50 % and there will also be considerable regional variability. in both hemispheres.
7) There is no reason to expect damaging global warming.Cooling is more likely .To prepare for it, all ethanol and biofuel subsidies and mandates should be abolished.Renewable energy and electric car subsidies are economically wasteful and accomplish nothing.There is no reason to control CO2 emissions, indeed some extra CO2, while having little effect on temperature, might aid farm productivity . 25% of the increased crop yields in the 20th century was due to the CO2 increase

lemiere jacques
September 11, 2013 9:16 am

the problem is you can use this method with so many curves…
so take any kind of data. price of wheat oranges or .., well anything..cut the last part of the data do the same analysis and..see if it can “foresee ” anything…i am pretty sure no…
i don’t buy that..because you don’t have any idea if the harmonics are related to something ,corespondind to a oscillating phenomena ,or just …a way to fit the curve.
May be this method can be interesting to find out if some frequancy you find smell “physical”

Jim G
September 11, 2013 9:21 am

DirkH says:
September 11, 2013 at 9:01 am
Jim G says:
September 11, 2013 at 8:13 am
“Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out.”
“That’s completely unfair. Jeff has argued convincingly why there is reason to believe that the periodicity is real.”
Define “unfair”. Extreme data pre-massage is evident to the extent of data torture. We cannot be skeptical of warmist models and not be skeptical of this. Time will tell if it predicts well as we know that the AGW models do not.

JerryC
September 11, 2013 9:34 am

So if we believe the modeling at all, then this is telling us that from 2020 – 2075 we can expect below average temperatures on average. How exactly are they planning to explain that away?

Steven Devijver
September 11, 2013 9:35 am

@jeffery, two questions:
1/ why not apply these techniques in hindcast mode? That would be really interesting.
2/ would it be possible to provide this code in a stand-alone form so that others could play with it. I’m a developer, I would be happy to help out.

John Day
September 11, 2013 9:39 am

@Jeff Patterson

The residual _is_ stationary. That’s the point. We can certainly drive an AR process with stationary (and in this case also nearly Gaussian) noise + sine waves and get a valid projection.

Well, ideally, the residuals should be pure white noise, with mean=0
Overall I really like your experiment. Well done. A good teaching moment for us all!
But I have used similar ARIMA-based estimators myself to predict these kinds of time-series (stock-market) and managed to find a lot of “useless” predictability (i.e. in the linear sense). It always turned out that the event sequences that I couldn’t predict (the “innovations”) were mostly the same events that would have made me a rich man if my predictor worked the way I hoped it would.
To make these models perform better you need to know a lot more about the hidden processes which generate the observable outputs.
😐

MikeN
September 11, 2013 9:42 am

170 year period data concluded from a time period of less than a full cycle?
All that means is that the increase in temperatures is then turned into a decrease on the other side of the wave.

mpainter
September 11, 2013 9:45 am

It has been said that if it is good science, it does not require a statistical massage. I heartily endorse that sentiment.

September 11, 2013 9:46 am

People are confused about “decimation” — think down-sampling for an analogy.
See the documentation in GNU Octave and Matlab for an explanation of decimation… Here is one link.
http://octave.sourceforge.net/signal/function/decimate.html
It might make more sense than — or less perhaps if you don’t follow this type of stuff..

RC Saumarez
September 11, 2013 9:49 am

@ Jim G and others.
I can take broadband noise and then pass it through a low-pass system. Provided the system has persistance, I will get a signal containing trends and this may appear quasi-periodic. I can fit a Fourier series to this signal and calculate an “input” that will create the output for a short length of recorded signal.
However, the input is noise and therefore the small segment of “input” is simply one snapshot of its underlying process. It is inevitable that if you do this you will find want appear to be cycles. If you repeat the “experiment”, you will find different periodicities in each repeat.
What one cannot do is assume that this snapshot will repeat – it is simply one sample of a noise record.
If you look at figure 13, it is the 21st century prediction and is simply a recapitulation of the 20th century with a basic triangular waveform so that the 20th and 21st centuries connect.
The logic underlying this post is, to my mind, hopelessly flawed.

Greg Goodman
September 11, 2013 9:51 am

“Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead”
Let’s see a plot of the next 170 years to see what this method suggests.
My guess is that it’s going to ramp down to about -0.4 and rise steadily upwards with a couple of bumps in the middle.
I promise I have not try plotting it, just guessing…..

September 11, 2013 9:51 am

crosspatch says:
September 11, 2013 at 8:26 am
The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen. Would an analysis of the opening of the 1812 Overture predict the finale? About the only thing I am prepared to say with any certainty is that the next 1000 years is more likely to be cooler than the last 1000 years than it is likely to be warmer.

Sure it could! And in 1812’s case (as with many [most?] other pieces of classical music) the intro comes back in the finale in the tonic key. This is the entire point of prediction, and human nature, to find (and create) patterns/expectation/reconciliation….then do it again.

September 11, 2013 9:54 am

If I understand correctly what he is doing the “BoxCar and the Decimation” are more about creating yearly “buckets” (or average/smoothed values) so that yearly values can be determined and examined. The same techniques are in R and widely used there for the same purposes — sometimes named differently.
Probably better explained in “The little book of Time Series” than I can do
http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/
hth

September 11, 2013 9:57 am

I don’t have time this week. If I remember this discussion next week I will run it through a somewhat different set of processes and see If I get the same periodic repeats.

Greg Goodman
September 11, 2013 9:58 am

oops, hadn’t seen the caption on fig 13, I must have started skimming well before I got that far. Guess what if the previous 170 years isn’t strangely similar.
As RC Saumarez says, this is nothing more than the extrapolation fallacy.

Jeff Patterson
September 11, 2013 10:03 am

I appreciate the response posted here. rather than address each in turn, let me make some general comments on the method.
First, the concerns about aliasing are IMHO unwarranted. The monthly data record is highly low-passed, being averaged in both space and time. Also, presumably the climate process, containing as it does many integrative elements, imposes its own filtering of high frequency input variances. The power spectral density of the unaltered Hadcrut data shows the high frequency floor to be down some 60dB (1/1000) from the spectral peaks we are trying to extract. If one understands the aliasing process in the time domain, a moment’s reflection should reveal the mathematically provable theorem that the amplitude of an alias cannot exceed the amplitude of it source. Hence aliasing effects here are insignificant.
On filtering: In harmonic decomposition, filtering must be avoided like the plague because it alters the amplitudes and phases of the periodicity we’re attempting to extract. Loss of spectral resolution is also an issue, especially when trying to extract cycles whose periods exceed the record length. However, the monthly data has a strong annual component that must be removed prior to decomposition because the underlying assumption of the method is that all periodicities are harmonically related. Fortunely, if you know the period of the offending cycle there is a solution, Box care filtering as described above. This filter is ideal in that it places its transfer function zeros precisely on top of the spectral peak being removed. It is in essence a high-Q notch filter which has the least impact on the remaining spectrum. It is true that this filter has high sidelobes but in this case that’s fine because the power spectral density at the lobe peaks is extremely low as evidenced above.
On record periodization: First note this is only a test to verify that observed spectral peaking is not due to the discontinuity at the record end points- the harmonic decomposition is performed on the original, unperiodized data).
Fourier analysis (which is used here only to perform the PSD estimate) gets around the required infinite length record by simple assuming the record repeats ad infinitum. If the endpoints don’t line up (withing the average step-size of the sample), the discontinuity created by stitching the records together (that’s not how an FFT works but in principle the effect is the same) creates a periodic error that shows up as spectral peaks. We can test for this effect by subtracting a line that connects the data record endpoints and dropping the last point from the result to avoid repeating the zero-error point at the ends. The result is a record which has no discontinuity distinguishable from the noise present in the data. If the PSD peaks remain after this process, it indicates the spectral peaks are not record-length artifacts.
One reader asked “why use the eight harmonic?”. Because there are more cycles present of the eight harmonic and so it provides a better estimate of the fundamental.
Fig. 10 spaghetti graph.
What was allowed to vary between trials? What stayed the same?
Figure 10 is output of the AR process used to create “climate-like”. Each path is a 113 point sample of the process for a new randomized state and random, white, input function of the variance derived by the AR process estimation function.

September 11, 2013 10:15 am

Jeff Patterson
Best Regards and thanks for your post. I repeat the request made in my post at 9:15 Could you possibly find the time to do a similar analysis of the 2000 year proxy record referred to in that comment.I think all the basic data is available on the web It might be a very illuminating exercise.

James at 48
September 11, 2013 10:19 am

If we started to fairly and objectively consider ringing, overshoot, beat frequency and other of these sorts of behaviors with respect to the global climate system, some interesting and perhaps frightening possibilities might start to become more apparent and would open the door for a whole new tone in the conversation.

Jeff Patterson
September 11, 2013 10:25 am

I forget to add a comment on forecasting:
In retrospect I wish I had left this section out, even though I thought I had presented the appropriate caveats from the start. Unsurprisingly, readers naturally gravitate to projections and by doing so miss the point of the exercise which was not to try to predict the future but rather to show that a plausible alternative hypothesis (actually two, harmonic forcing and/or resonances in the climate system) provides a good fit to the observed data and that such a good fit is unusual. I have neither the time or expertise to complete the exercise – namely providing a physical explanation for the hypothesis and empirically testing the result.
Some general comments about modeling:
Information theory tells us nothing can be learned from a calculation. Knowledge is created by surprise (entropy) and no calculation should surprise us. All modeling, including GCMs, no matter how complex are simply calculations. The results are completely determined by the input and initial state. They are thus only useful in the hypothesis-forming portion of the scientific method (something the climate modelers seem to have forgotten). Once formed, the hypothesis must be verified by empirical observation.
Again, thanks to all for the useful comments. I’ll try and follow up with the suggestion to run the method on a subset of the data and post the results. My thanks to Mr. Watts for providing this site and to all for the interesting conversation.

Jonathan Abbott
September 11, 2013 10:28 am

I think DayHay makes the most important point.

September 11, 2013 10:31 am

We can test for this effect by subtracting a line that connects the data record endpoints and dropping the last point from the result to avoid repeating the zero-error point at the ends. The result is a record which has no discontinuity distinguishable from the noise present in the data. If the PSD peaks remain after this process, it indicates the spectral peaks are not record-length artifacts.
Ok. But then your line that you subtract from the time-based data is exquisitely sensitive to just two sample points, Point 1 and Point 113. I’ll go along with the method, if and only if, you use it on at least 9 other subsets of the data: Points 1-112, 1-111, 1-110, 2-113, 2-112, 2-111, 3-113, 3-112, 3-111. Now you have 10 different base lines to remove from the data and 10 different resulting PSD’s. What are the consistent peaks, what are the inconsistant peaks? You gotta admit, it add robutness and “fold” to the results.
One reader asked “why use the eight harmonic?”. Because there are more cycles present of the eight harmonic and so it provides a better estimate of the fundamental.
There is no free lunch. There might be more cycles, but fewer samples per cycle. Frankly it makes it more sensitive to the choice of data length.
Your response to the Fig. 10 question didn’t answer what was asked. What was randomized? Just the phase of the spectral components?
Like a few others above, I am highly skeptical of a fundamental cycle length longer than the original time series. From what I learned in geophysics, that is a violation of the Fourier theorems. Your support of this long period seems to come from “harmonic decomposition”. I am very skeptical. It sounds like circular logic. Are you in effect padding the time series with a variable numbe of zeros until you get your peaks to resonate? 32 years out of my Ph.D. [you listening, Gail?] I’m still willing to learn, but like I said, “I need to see more.”

Tetragrammaton
September 11, 2013 10:41 am

Climate scientists got off on the wrong foot three decades ago, by prematurely assuming that there were just two important drivers of climate – namely the sun (insolation) and a CO2-driven greenhouse effect. These, most reckoned, were the “first-order” influences and all of the other factors were second-order or third-order and therefore could be largely ignored. CO2, they said, was the “control knob” of climate variability.
Mother Nature has unkindly demonstrated their basic error, by providing fifteen years of almost unchanged world average temperatures while the atmospheric CO2 level, for whatever reason, continued to increase significantly. The control knob was turned, and almost nothing happened! The most bizarre and ingenious ecclesiastical apologia forthcoming from the Jesuit intelligentsia would pale by comparison with some of the climate establishment’s amazing “explanations” for this hiatus. We wait with bated breath for the IPCC’s upcoming summary to see if it “explains” or simply ignores what nature has done to them.
This is why Jeff Patterson’s post is so exciting. It assumes absolutely nothing about the theories or explanations of climate scientists – or anything else – and simply mines the available data for plausible patterns which could provide some insight into future trends or actual mechanisms. And, unlike most other such attempts at naive analysis, it is performed by somebody who clearly understands the false moire and other mirages which can emerge from this type of statistical treatment. Such trend analysis is surely the best starting point for the badly-needed “reboot” of climate science.

September 11, 2013 10:56 am

Allow me to join the several other commenters who asked that you perform the last part of the analysis again. No, I’m not asking that you again seek the fundamental frequency. I’m just asking that, taking the fundamental you’ve already identified, you use just the part of the record prior to, say, 1945, to determine the harmonics’ amplitudes and phases. Then, using those amplitudes and phases, project forward through the present. (Although my guess is that your computer time required to identify what by my count are ten different parameters will be significant, you wouldn’t have to rewrite the program, of which even just the part we’re requesting you re-run would probably take plodders like this writer more than an hour to write and debug.)
I recognize that this a slight diversion, but it is indeed only slight, and the results could support the following interesting question: If the climate’s behavior in the record’s latter portion, which represents a time when CO2 concentration was rising significantly, is predictable, without any adjustment for CO2–and on what’s little more than a the-trend’s-your-friend basis–from the record’s former portion, which represents a period when little CO2 increase occurred, shouldn’t the proposition that CO2’s effects are significant bear a heavy burden of proof?
Allow me also to join others who have thanked you for a clear post and responsive follow-ups.

September 11, 2013 10:58 am

Henry@all
I really feel sorry for you all because you are all missing an important point:
where would be without the sun? We’d be dead, would we not?
We know that earth has a number of factors (built-in) that confuse matters,
on mean average temp.
like
its own volcanic action,
turning of the iron core,
magnetic fields/ clouds
lunar interaction
to name but a few.
I am sure there a lot more.
So why do you all keep staring at those stupid figures for mean average temps. & their anomalies?
Surely anyone with brains must have figured out by now that maximum temperatures are a much better proxy to look at, as I found. But don’t look at only one station, because if you were to look at CET maxima alone you might get confused: they have a lot of weather…..there in London.
http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/
http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/

Jeff Patterson
September 11, 2013 10:58 am

crosspatch says: The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen.
But showing what has happened is important, is it not? The AWG proponents are convinced what _has_ happened is due to CO2 emissions. I’ve forwarded an alternate hypothesis which, if it could be varied, would alleviate these concerns. In that case we could confidently predict that the future climate will change, just as it has in the past, and these natural changes will likely be benign, just as they have been in the past.

Chris Schoneveld
September 11, 2013 11:04 am

So, richardscourtney, how does this jive with Akasofu’s linear “recovery” from the LIA?

Pete
September 11, 2013 11:08 am

Echoing others: Thanks for this article – for writing clearly and explaining, for acknowledging problems, for being bold and trying stuff out, for providing the code, and for being around to discuss issues with others. Even if your post were complete nonsense (Im not in a position to judge) it’s still a good example of what we should all be doing here.

Beta Blocker
September 11, 2013 11:10 am

Eliza says: September 11, 2013 at 5:15 am
Try same with CET (reliable rural station) you will probably get a much more pronounced cooling in the future

Take a look at: CET Patterns, 1659-2007, Wikipedia Graph
If we were to guess about the next two centuries based upon historical patterns in the CET record, we could guess that the current pause in rising temperatures will be merely a pause; and that over the next 200 years, GMT will continue its long-term rise while following a jagged pattern of localized up-and-down trends, doing so at a linearized trend over a period of two centuries of about + 0.3 C per century.
Certainly, one or more localized downward trends in GMT could occur within that 200 year period, and probably will.
But in the mode of saying that when in doubt, predict that past trends will continue, we could also guess that a jagged up and down pattern — one with a gradually rising long-term linearized trend — will continue until the maximum of the Medievel Warm Period is reached.
A reasonable estimate of GMT at the height of the Medievel Warm Period is needed. Is there anyone with the appropriate credentials working on that estimate?
In the meantime, what about the next 100 years, as opposed to the next 200? Make your own guess about GMT in the year 2100 using Beta Blocker’s CET Pattern Picker.

Jeff Patterson
September 11, 2013 11:12 am

Stephen Rasey says:
September 11, 2013 at 10:31 am
“There is no free lunch. There might be more cycles, but fewer samples per cycle. Frankly it makes it more sensitive to the choice of data length.”
True but even the eight harmonic is highly oversampled. Nyquists proves that increasing the number of samples per cycle (beyond two per the shortest period) adds no information.
“Your response to the Fig. 10 question didn’t answer what was asked. What was randomized? Just the phase of the spectral components?”
As was stated in the paper and elsewhere, figure 10 is the output of an AR process designed to provide random sequences which mimic (in character) the observed climate record. An AR process is just an IIR filter. The input to the filter in this case is random noise with the specified variance. Each path in figure 10 is the “filter’s” output with a different, random input sequence (and initial state). We then perform the same harmonic decomposition on these “climate-like” paths to test the null hypothesis that any old randomly produced sequence which looks kinda like temperature data would yield a goodness-of-fit similar to that achieved with the real data. While we cannot quite reject the null hypotheis (p=.081), we can state that getting the achieved GOF is not to be expected and in fact the achieved residual variance is near the 3-sigma of the distribution of the randomly generated results.

September 11, 2013 11:17 am

@Sam The First says: September 11, 2013 at 5:57 am
Menahwile in the broadsheet papers here in the UK, notably The Independent and The Guardian which are read by liberal / left opinion formers (inc teachers at all levels), the comments to this article below demonstrate that none of their readers is taking any notice of the long pause in warming and the overall cyclical record.
Will someone with the time and expertise please add a comment or two to explain what is really going on?
========================================================================
I’ve long been banned from CiF for dissent. The Indy, well, it’s best for me not to go there as it enrages me in no time at all. Occasionally, I will employ my rapier like wit to puncture some balloons, but ultimately, it’s a waste of time talking to such as comment there.

September 11, 2013 11:17 am

Jeff Patterson says
(future climate change) will likely be benign, just as they have been in the past.
henry says
Dear Jeff, unfortunately I think it will not be benign
Under normal circumstances, like you, I would have let things rest and just be happy to know the truth for myself. Indeed, I let things lie a bit. However, chances are that humanity will fall in the pit of global cooling and later me blaming myself for not having done enough to try to safeguard food production for 7 billion people and counting.
It really was very cold in 1940′s….The Dust Bowl drought 1932-1939 was one of the worst environmental disasters of the Twentieth Century anywhere in the world. Three million people left their farms on the Great Plains during the drought and half a million migrated to other states, almost all to the West. Please see here:
http://www.ldeo.columbia.edu/res/div/ocp/drought/dust_storms.shtml
I found confirmation in certain other graphs, that as we are moving back, up, from the deep end of the 88 year sine wave,
http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/
there will be standstill in the speed of cooling, on the bottom of the wave, and therefore naturally, there will also be a lull in pressure difference at that > [40 latitude], where the Dust Bowl droughts took place, meaning: no wind and no weather (read: rain). According to my calculations, which includes certain planetary positions, this will start around 2020 or 2021….and last until 2028.
Danger from global cooling is documented and provable. We have only ca. 7 “fat” years left……
1) We urgently need to develop and encourage more agriculture at lower latitudes, like in Africa and/or South America. This is where we can expect to find warmth and more rain during a global cooling period.
2) We need to tell the farmers living at the higher latitudes (>40) who already suffered poor crops due to the cold and/ or due to the droughts that things are not going to get better there for the next few decades. It will only get worse as time goes by.
3) We also have to provide more protection against more precipitation at certain places of lower latitudes (FLOODS!),

Andyj
September 11, 2013 11:19 am

Reading 170 year cycles… I knew I read about this before!
People, try a quick blast on Google and you will be decimated by the amount of similar findings from differing sources. From war to famine. Even the 10Be records have something to say

jono1066
September 11, 2013 11:21 am

How can one decimate by 12 ?

September 11, 2013 11:22 am

I’m delighted to see that the head posting confirms – and by similar methodology – work being done elsewhere which has come to very much the same conclusion – that there may be a 0.5 K drop in global temperatures within the next few years. I mentioned this briefly in an earlier posting, and the usual suspects elsewhere began sniffing and sneering and offering me bets.
Fourier analysis, and particularly the subset of it that is used here, is an appropriate method. But best results are obtained not so much by looking at the past temperature record in isolation and over little more than a century, as here, but in association with possible aetiologies and their datasets, and over all timescales. Murry Salby’s lecture takes this approach, and is worthy of study. The Akasofu and Scafetta approach, extrapolating from previous cycles, is also interesting, as is Tsonis on the ocean oscillations, whose 58-year cycles are close enough to be harmonics of the 171-year period mentioned in the head posting as having been detected by the author’s harmonic decomposition.
If the IPCC did science this way, openly and inviting public scrutiny, it would have been out of business long ago: it is only by concealment that they get away with continuing this scam. If Mann, Bradley and Hughes had done science this way, they would never have dared to foist the “hockey stick” on the world.
Finally, at Bishop Hill there is a hilarious debate in one of the House of Commons committee rooms about global warming, in which the true-believers take quite a pasting.

DavidG
September 11, 2013 11:27 am

There is a 170 year cycle that is based upon the conjunctions of Uranus and Neptune, 1992, 1821,1648,1470,1300 etc.

phlogiston
September 11, 2013 11:29 am

This paper by Coughlin and Tung 2004 employs the nonlinear EMD (empirical mode decomposition) analysis to stratospheric temperatures, and finds a strong 11 year solar cycle.
This is the right kind of analysis to look for nonlinear forcing of the climate system by solar and other astrophysical cycles. Check out figure 3.

Jeff Patterson
September 11, 2013 11:31 am

Stephen Rasey says:September 11, 2013 at 10:31 am
“Ok. But then your line that you subtract from the time-based data is exquisitely sensitive to just two sample points,”
Actually quite the opposite. This is because the Fourier Transform of a sum is equal to the sum of the FTs. Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the line (ramp), irregardless of the slope of the line. Here this compensation step was unnecessary as we were only interested in determining that the observed peaks were not record-length artifacts.

richardscourtney
September 11, 2013 11:36 am

Chris Schoneveld:
At September 11, 2013 at 11:04 am you ask me

So, richardscourtney, how does this jive with Akasofu’s linear “recovery” from the LIA?

Among the many possible meanings of your question, one is,
‘How do the studies of Patterson and Akasofu relate?’
And, of course, the answer to that question is,
‘They don’t relate because they are different analyses conducted for different purposes although they are conducted on the same data’.
Patterson attempts to use signal processing to determine underlying frequencies in the global temperature time series. It remains to be seen if his analysis can provide useful information, but if he has identified frequencies which are ‘real’ then consideration of those frequencies may induce studies for provision of information concerning significant climate processes.
Akasofu observes that the temperature global time series can be matched by assuming a linear recovery from the LIA combined with a sinusoidal oscillation (possibly related to ocean behaviour) which provides alternate periods of warming and cooling each of 30 years duration. He extrapolates that pattern with a view to determining when it changes, and upon observation of a change (which will surely eventuate) then it will be reasonable to infer whatever has induced that pattern has changed.
I don’t know how either analysis could jive or perform any other dance. But I am pleased with my jiving and have medals for my ability on the dance floor.
Richard

Jeff Patterson
September 11, 2013 11:46 am

RC Saumarez says:
September 11, 2013 at 9:49 am
“I can take broadband noise and then pass it through a low-pass system. Provided the system has persistance, I will get a signal containing trends and this may appear quasi-periodic. I can fit a Fourier series to this signal and calculate an “input” that will create the output for a short length of recorded signal.”
But that is precisely what I’ve done in the section entitled “Stochastic Analysis”! The AR process used is a low-pass filter with coefficients adjusted to produce “climate-like” outputs from white noise inputs. When we do harmonic decomp on these random sequences we see that in general we cannot get close to the goodness-of-fit (as measured by the variance of the residual error) achieved with the actual climate data.

Reed Coray
September 11, 2013 12:00 pm

Mr. Patterson,
The quality of a five-harmonically-related-frequency model fit to the data surprises me. I’d be interested in seeing the quality of a similar fit using a five frequency model whose frequencies are not required to be harmonically related. That is, if the software routine you used to determined the values of harmonic sinewave amplitudes and phases that best fit the data can accommodate sinewave frequency as well, I’d be interested in seeing (a) how the best-fit estimated frequencies differ from their harmonic counterparts, and (b) the change, if any, of the residuals and residual spectral content. If the software you employed uses a weighted-least-squares cost function, my guess would be that giving that software an initial guess corresponding to your harmonic analysis results, the software would converge to a solution with frequencies, amplitudes, and phases differing only slightly from their “harmonic analysis” equivalents.

Greg Goodman
September 11, 2013 12:01 pm

JP: ” forget to add a comment on forecasting: In retrospect I wish I had left this section out”
Yep, forgetting the title of the post may be the best approach.
“All modeling, including GCMs, no matter how complex are simply calculations. The results are completely determined by the input and initial state. They are thus only useful in the hypothesis-forming portion of the scientific method (something the climate modelers seem to have forgotten). Once formed, the hypothesis must be verified by empirical observation.”
A very good point.
JP: “First, the concerns about aliasing are IMHO unwarranted. The monthly data record is highly low-passed, being averaged in both space and time. ….The power spectral density of the unaltered Hadcrut data shows the high frequency floor to be down some 60dB (1/1000) from the spectral peaks we are trying to extract.”
That is precisely where the problem lies. It _had been_ heavily low-pass filtered and the evidence is that it was not correctly anti-aliased before that was done.
http://judithcurry.com/2011/10/18/does-the-aliasing-beast-feed-the-uncertainty-monster/
The fact that there is NOW very little H.F. tells us _nothing_ about how much this was in the data that is now aliased elsewhere in the spectrum. And that could multi-year or decadal in scale.
The papers on Hadley SST processing sketch out the process to build the global ‘climatology’ grid, which involves running averages in adjacent grid cells across latitude and longitude. This is repeated in a loop until the result ‘converges’. These 5×5 degree grid cells of 5 day averages are then processed into calendar monthly means.
The papers do not make any mention of anti-alias filtering prior to re-sampling .
So, yes there is lots of low pass filtering going on , that accounts for the low amplitude of h.f. signals that you comment on. That in no way indicates that the power in those poorly filtered and re-sampled frequencies are not now lying in the multi-year to decadal frequency bands.
RC Saumarez showed that the frequency spectrum of that data suggests rather clearly the presence of aliasing.
my comparison of ICOADS to hadSST :
http://climategrog.files.wordpress.com/2013/03/icoad_v_hadsst3_ddt_n_pac_chirp.png
http://climategrog.wordpress.com/2013/03/01/61/
…shows some limited but significant changes to the frequency content of the data due to Hadley processing. Whether this is an improvement or a degradation of the data is not my job to establish but I am unaware of any account that this was even measured as part of Hadley’s QA on their data processing.
Maybe it was , maybe I missed it. Maybe it wasn’t.
Caveat Emptor.

September 11, 2013 12:04 pm

@Jeff Patterson at 11:31 am
Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the [known] line (ramp),
You don’t know the line. 113 points in the dataset is just an artifact of how long someone was recording data. When the data starts and ends is similarly an artifact. Different starts and different ends give different lines and therefore different transforms. I expect differences in the high frequencies and power at the low frequencies.
Also, the choice of a base “line” is arbitrary. Simplistic, but not necessarily the best choice. That base line might be a linear function of CO2 concentration, logarithmic function of CO2, or a non-linear function of sunspot number.

Matthew R Marler
September 11, 2013 12:05 pm

The observed temperature anomaly since 1900 can be well modeled with a simple harmonic decomposition of the temperature record based on a fundamental period of 170.7 years.
That’s dear: 4 significant figures with a time series less than one full period.
Anthony: This DSP engineer is often tasked with extracting spurious signals from noisy data.
Is this “signal” spurious?
His model does not fit the data as well as Vaughan Pratt’s model. It should be clear by now that experienced data analysts can get just about any result they want from the extant data. Now that he has published his model, it can be tested by future data.

See - owe to Rich
September 11, 2013 12:06 pm

Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?
I also think that some (albeit minor) global cooling is going to occur, because of the Sun, but I do not trust this model to tell me why or by how much. But I did enjoy the mathematics!
Rich.

Jeff Patterson
September 11, 2013 12:06 pm

For those interested, a fifty year hindcast back to 1850 is available here (the model was created using only data from 1900 to present because of missing data samples prior to that time )
http://montpeliermonologs.wordpress.com/2013/09/11/hindcast-of-model-back-to-1850/
The plotted data is the unaltered monthly Hadcrut data.

george e. smith
September 11, 2013 12:09 pm

“””””…….Ric Werme says:
September 11, 2013 at 5:43 am
A few comments:
1) decimated by 12
I know that “decimate” came from the Roman practice of inspiring soldiers after battles where the commanders felt were uninspired. They lined up the soldiers and killed every tenth man, leaving the remaining 90% inspired to win the next battle. At first blush, this sound like torturing the data until it confesses or perhaps this is “dodecimated” – killing every 12th measurement. :-)…….””””””
Ric, In this instance “decimated is a highly specific technical term from digital signal processing; so don’t go looking for a highway lines with occupied crosses.
And no adherence to the ten implication is required.
And I know about enough about it to be a hazard to myself; but if an Agilent Cowboy, doesn’t know Digital Signal Processing, then nobody does.
My concern with the exercise, is that I wonder what result one could pull out of the phone numbers in the Manhattan telephone directory.
It’s not clear to me that the Hadcrud numbers actually comprise data that relates to planet earth.
And having seen in the past, a mathematical exercise, that recovered the value of one of the fundamental physical constants of science (the fine structure constant), to about 8 significant digits of precision (about 1/3 of the standard deviation of the best experimental value, at that time); yet the model had precisely zero input from the physical universe; it was literally just fetzing around with numbers; so I don’t go for all new things, as an early adopter. I’m often the last to see the light.
I have quite recently seen a movie called “Gone with the Wind”, for the very first time.

Jeff Patterson
September 11, 2013 12:21 pm

Stephen Rasey says:
September 11, 2013 at 12:04 pm
@Jeff Patterson at 11:31 am
Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the [known] line (ramp),
“You don’t know the line. 113 points in the dataset is just an artifact of how long someone was recording data. When the data starts and ends is similarly an artifact. Different starts and different ends give different lines and therefore different transforms. I expect differences in the high frequencies and power at the low frequencies.”
You know the line exactly because you create it! More precisely, it is a periodic sawtooth that returns to zero foreach record. You know its slope, amplitude and period and thus it’s Fourier (not DFT) transform exactly. The (complex) coefficients of this transform can be evaluated at the DFT bins subtracted from the periodized DFT to get the spectrum of the unperiodized record. Do the math, try and experiment or two. It works (and we’ve been shipping it in products for years!)

Matthew R Marler
September 11, 2013 12:23 pm

I should say that I enjoyed the post. Why start the series at year = 1900?
The following is well-said: If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.
The author estimates 10 parameters, after estimating the fundamental period. Why not just fit a 10th order polynomial or a Bayesian Adaptive Regression Spline model, or some well-selected set of 10 wavelets — as long as you are acknowledging that this is merely another curve-fitting exercise with no claim to predictive power, any old and new functions are just as good. If the fundamental frequency is evolving with time, in what sense is it a “fundamental” frequency.

Matthew R Marler
September 11, 2013 12:23 pm

oh nuts. I omitted the “?” on my last sentence.

Greg Goodman
September 11, 2013 12:25 pm

The harmonic technique used is useful in freeing the process of reproducing a ramp determined by the length of the dataset.
The first second and third harmonics are just recreating the longer ramp. The most notable is perhaps the strong 8th.
170/8= 21.25 years. Also one of the three periods my 3 cosine analysis latched on to.
The suggestion of Schwabe solar cycle is obvious.

Michael J. Dunn
September 11, 2013 12:25 pm

mpainter says:
September 11, 2013 at 9:45 am
It has been said that if it is good science, it does not require a statistical massage.
Yeah, usually by non-scientists. It is not science if there is no measurement of phenomena (signal). There is no measurement without error (noise). There is no way to sort out the signal from the noise unless you have an insight into, and a command of, the mathematics describing the noise statistics. Try it sometime. It ain’t so easy as you might imagine.

Matthew R Marler
September 11, 2013 12:32 pm

Pamela Gray: Comment 1: While data massaging here is extreme, it does demonstrate the futility in ordinary least squares statistics so adored by AGWing scientists.
This is non-linear least squares, after finding the fundamental period.

Greg Goodman
September 11, 2013 12:39 pm

“I have quite recently seen a movie called “Gone with the Wind”, for the very first time.”
Yeah, I should find to time to watch too, one day. 😉

Matthew R Marler
September 11, 2013 12:40 pm

Jeff Patterson: As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.
That’s smart.

Jeff Patterson
September 11, 2013 12:43 pm

Greg Goodman says: That is precisely where the problem lies. It _had been_ heavily low-pass filtered and the evidence is that it was not correctly anti-aliased before that was done.
I looked over the referenced paper and while I certainly agree that aliased data is irretrievably corrupted, the question is by how much? The fundamental of the extracted model has a peak-to-peak amplitude of .6 degrees. If this were an alias (remembering that the amplitude of the alias is equal to the high frequency periodicity being aliased) surely someone would have detected the .6 degree high frequency signal in the data. It would only require a few weeks of hourly data to find it!

Jeff Patterson
September 11, 2013 12:46 pm

Greg Goodman says:
September 11, 2013 at 12:01 pm
JP: ” forget to add a comment on forecasting: In retrospect I wish I had left this section out”
Yep, forgetting the title of the post may be the best approach.
I didn’t choose the post title which I agree was unfortunate.

Greg Goodman
September 11, 2013 12:52 pm

See – owe to Rich says:
Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?
170.7/3*2=113 = data length, oo-errr.
Well spotted.
“This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data.”
-Anthony
Freudian slip?

Matthew R Marler
September 11, 2013 12:53 pm

Nick Stokes says: How can you establish a 170 year periodicity from 113 years of data?
Basically, all he did was estimate a nonlinear function from the set of parameterized (phase and amplitude) family of sine curves. Using that estimated period, he then chose harmonics of it as the additional terms in a multiple non-linear regression. In harmonic regression, which is what this is, there is no non-arbitrary way to select the fundamental frequency: the standard in Fourier analysis is to let the full data series represent 1 full period, which may not have any relevance if you have managed to sample, say, over 1.25 times the period.

nc
September 11, 2013 12:58 pm

This is off topic somewhat and is a criticism of computer run climate models. Picked this up in the comments section over at JoNova and think its hilarious,,
“Roy Hogue
September 12, 2013 at 5:42 am · Reply

…a foundation of bullshit coming from computers, not bovines.
Then let’s call it what it is — bitshit.
And you’re right about it without any doubt too. 🙂 “

September 11, 2013 12:59 pm

@Jeff Patterson at 12:21
You know the line exactly because you create it!
YES!
YOU create it. Not Nature. It is an artifact of the measurement time window and processing.
That is why you must try different subsets and get different lines to explore the changes in the resulting transform. Then look for similarities that are more likely in the signal and not an artifact of the processing.
It’s like trying different stacking velocities in seismic processing to improve the signal-noise ratio.

Matthew R Marler
September 11, 2013 1:05 pm

rabbit: Have you considered Singular Spectrum Analysis (SSA) to extract the dominant modes of the temperature graph? This does not suffer from the windowing effects that a DFT suffers from, nor is it limited to a prescribed set of basis functions.
I second the question. After choosing the base frequency, why did you limit yourself to harmonics of the base frequency? Is there relevant physical theory that the important frequencies are multiples of the base frequency?

Greg Goodman
September 11, 2013 1:07 pm

JP: “I looked over the referenced paper and while I certainly agree that aliased data is irretrievably corrupted, the question is by how much? The fundamental of the extracted model has a peak-to-peak amplitude of .6 degrees.”
No , I was not suggesting the whole long term trend is an alias. Neither did RC’s article on Climate etc suggest that degree of error. But if there is a lack of filtering it will not just be one frequency effected. RC demonstrated that this can lead to spurious decadal length trends (did you read the link?).
If there is error in the long term it is just as likely ‘bias correction’. I had the fortunate opportunity to discuss this at some length with John Kennedy in the comments. He was very forthcoming and helpful, though obviously defending their work.
What he told me of the validation process showed it was based on circular logic and geographically anecdotal “validation” from japanese data also using _buckets_. (Apparently japanese buckets don’t ‘bias’ like british and american ones!).
For the title , that’s editors for you . They’re always out for a catchy headline. 😉

Greg Goodman
September 11, 2013 1:19 pm

PS the japanese paper was full of bias confirmation too. They only retained the data which suggested Hadley corrections went in the right direction. (The ‘validation’ was not any stronger then that). But all those going the other way were deemed “not suitable” and removed from consideration.
This is the problem with that state of the literature after 30 years of biased pseudo-science and political gate-keeping. The garbage is stacked several palettes high now.

MikeN
September 11, 2013 1:36 pm

Jeff, is there any sort of robustness tests for this method? Can the time series be analyzed on a shorter interval, and see if the predicted results for the out of sample period match?

Steve from Rockwood
September 11, 2013 1:57 pm

I used to program DSPs and I can assure everyone they do not predict the future.

TimB
September 11, 2013 2:12 pm

Correct me if I’m wrong, but any non-random data points will show harmonics. I’m not sure what AGW would show but I would think it would simply be a S/N ratio of an AGW harmonic on top of natural harmonics. In fact, mixing the two (natural and AGW) would result in seeing the AGW mix in even pre-AGW data as it would be unable to blend out. I think the assumption using DSP is that temperature is the convolution of linear functions and that may not be the case.
My gut tells me that DSP methods will treat random steps as non-harmonic noise and will not detect random and non-linear changes in DC offset that is characteristic of AGW. My gut is that a rise in temperature that is a sequence of step changes that are random in time and magnitude will be washed out in the deconvolution and the individual residual components that make up long term AGW would be below the noise level. Put another way, AGW may manifest itself as a sequence of below Nyquist rate events that all lie below the noise level. If the residual magnitudes look like “black body radiation chart” with a peak that is below the noise level, and a frequency longer than Nyquist, I would suspect you could have AGW in that signal and not be accounting for it as non-random and non-periodic and quite possibly attributing it to natural variation. The fact that you can model the past as a harmonic is from the math that will always solve, less a residual. It has no predictive value, though, because a driving component is non-linear addition to data. The math is good at figuring out how many “mixers and sources” are present in the existing data (or rather how many it needs to reach a certain S/N ratio) but not so good at predicting how many mixers and sources will be present in the future. I believe if it’s random in time and magnitude, and small at each step, no amount of oversampling and decimation will detect a fingerprint to predict a future value. For example, ENSO might be a large harmonic source that describes a long variation it temperature. If reality is that AGW is tiny, independent, sources and mixer to inject AGW at random points (i.e. say in 0.01C in June of 2007 and then 0.012C for January 2008, and then another 0.005C in September 2009), the analysis of those “below the nyquist and noise level will either leave them out or model them as a harmonic that has a lower residual error even if it’s really an accumulated error that will continue. The method will always add a harmonic term to reduce the error if it can whether it’s physical or not. I think the only way to overcome it is massive oversampling and decimation but you may still run into the noise floor of the data before you can pick out small, random, non-periodic residual signals. It’s like describing something as a finite series and every time your error gets too large, you add another term to the series that reduces the error. That’s what DSP algos will do and it’s not discriminate as to whether the term is physically descriptive of the underlying phenomena. It’s a fit and the number of terms is a function of the tolerable error and the technique. It works well to compare two different datasets but it’s not predictive and I’m not sure you can lump all the “fit” into natural variation and claim only the residual is a man-made artifact.
An interesting exercise would be to break the data up into ranges always starting at 0. 0-11 years, 0-23 years, 0-37 years, 0-61 years (maybe just prime intervals) see what the solution was at the end of each of those periods and see how those functions compare to the full dataset. The PSD harmonics should not change as long as the dataset is longer than the fundamental mode nyquist rate. If the PSD harmonics are different depending on dataset choices, I think there is a problem with the method and it’s assumptions.

Jeff Patterson
September 11, 2013 2:22 pm

Greg Goodman says: But if there is a lack of filtering it will not just be one frequency effected. RC demonstrated that this can lead to spurious decadal length trends (did you read the link?).
Color me dubious. The human body is about the closest thing I can think of to continuous time thermometer. Are we fooled into sensing a temperature trend that doesn’t exist due to aliasing? Of course not, because the earth’s temperature itself does not experience high frequency fluctuations. All the more so with a real thermometer because it couldn’t react to rapid changes even if they were there. If we then average equally spaced (in time) temperature readings, we are averaging already smoothed (by the slow reaction of the climate and the thermometer) data so again no aliasing occurs. Sure there are errors in the readings which may look like white noise, but these could not produce a measurable trend unless integrated to produce a random walk. What process are you proposing that would produce such a result?

Jeff Patterson
September 11, 2013 2:28 pm

Matthew R Marler says:
September 11, 2013 at 1:05 pm
After choosing the base frequency, why did you limit yourself to harmonics of the base frequency? Is there relevant physical theory that the important frequencies are multiples of the base frequency?
Because the PSD indicated a harmonic relation between the spectral peaks and because harmonics are generated anytime sinusoids drive a non-linear response (thermodynamic equilibrium involves a quartic relationship to temperature if I remember correctly ) .

September 11, 2013 2:29 pm

Why do you smooth and desample to annual averages? High frequency cutoff filtering is better done in de frequency domain, as you are doing an fft anyways.

Bart
September 11, 2013 2:30 pm

I do not see any spectral peaks here of any distinctive character. Which surprises me, because i would expect to see something distinctive in the neighborhood of 0.016 years^-1 (60-65 year cycle evident in the data). Rounded peaks are generally artifacts of finite record length, and not representative of actual underlying processes.

Jeff Patterson
September 11, 2013 2:32 pm

Greg Goodman says:
September 11, 2013 at 12:52 pm
“This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data.”
-Anthony
Freudian slip?
No, my day job is signal generation via direct digital synthesis. In such a system, the carrier is clear, unambiguous and easy to measure. Very small (< -90 dBc) spurious tones are also generated due to system non-idealities. We need to extract these spurs and characterize their amplitudes from the noise produced by DACs and measurement floors.

Editor
September 11, 2013 2:59 pm

I have a couple of large objections to this procedure.
The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data. This is extremely basic testing, and the fact it wasn’t done is very worrisome, particularly given the claimed expertise of the author.
The second is that as far as anyone has every determined, the climate is chaotic … how does he reconcile this with his claim that it is deterministic? Extraordinary claims (such as the climate being deterministic in nature) require extraordinary evidence … and he hasn’t even provided the most basic of tests, using out-of-sample data.
The third is that his “Monte Carlo” analysis is extremely simplistic. I have provided a lot of evidence that the climate is governed (regulated) by a variety of emergent phenomena (thunderstorms, El Nino/La Nina, PDO, etc.). Accordingly, to investigate the system, among other approaches you need to generate synthetic data for such a governed system to use in the Monte Carlo analysis.
Monte Carlo analysis is easy to do … but very difficult to do well. The problem is that you are begging the question.
For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.
I am sure that folks can see the circularity in that argument … to do a Monte Carlo analysis, you have to either:
a) Know the underlying form of the data (which we don’t), OR
b) Use a variety of assumptions about the data to try to cover the actual possibilities.
He has done neither.
Finally, I am completely unconvinced that you can determine a 170.7 year cycle (including the ludicrous .7 year decimal part) from a hundred and sixty years of data. The mere fact that he has included the .7 is a huge danger sign to me, it indicates that he is used to dealing with definitive numbers, not error-filled data like we have in climate science.
So, despite his claimed expertise … color me completely unimpressed by his claims. He has not done even a small part of the work necessary to substantiate his claims, he has not tested his method on out-of-sample data, he is claiming that the climate is deterministic (Mandelbrot among others disagrees) and he has done a childishly simplistic Monte Carlo analysis.
Note that these are all correctable errors … it remains to be seen if he corrects them.
w.

megawati
September 11, 2013 3:23 pm

Testing the model´s ability to fore-/backcast, i.e., its usefulness, is easy: just use a subset of the time-series and see how the model´s extension matches the rest.
I suspect that if this is done, it will produce a completely different model for each subset, and all of them wil fail miserably at modelling anything outside of their respective known boxes.

Peter Foster
September 11, 2013 4:07 pm

We know that the hadcru, giss and noaa data is severely adjusted to reduce the late 30’s early 40’s warming, to reduce the cooling from 1945 to 1975 and to enhance the warming from 1980.
The data sets also fail to account properly for urban heat island effect, for station deterioration and for the removal of many cooler stations from the dataset.
The question is; do these massive questionable adjustments coupled with the failure to account for the factors mentioned above, affect the sort of analysis you have performed here.

U. Torvaldsson
September 11, 2013 4:11 pm

Ok, now what if you tried again, but this time with unadjusted data ?

Reed Coray
September 11, 2013 4:12 pm

See – owe to Rich says: September 11, 2013 at 12:06 pm
Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?

Using the five sinusoids (amplitude, frequency, phase) given in JP’s paper, the match (sinusoids versus measured data) going back to 1850 is not very good. You ask a good question, and I’d like to hear JP’s answer. I’d put my plot of that data in this comment, but I don’t know how to include figures in a comment.

1sky1
September 11, 2013 4:44 pm

Patterson presents yet another simplistic modeling study of Hadcrut3 in which DSP methods are applied to make predictions with scant comprehension of the stochastic nature of geophysical signals. There should be little doubt that,once the diurnal and seasonal cycles are effectively removed in decimating the data to yearly averages, what remains should be treated as a wide-band stochastic process. Unlike the aforementioned cycles, there no known physical forcings that are strictly periodic aside from the astronomical tides. it is only in the contrary case that any model consisting of a finite set of pure sinusoids can be expected to be effective in making predictions.
Although a ~170yr oscillation does indeed appear, inter alia, in the power density of GISP2 oxygen isotope data, Kalman-Bucy filters fail to produce close, practically useful predictions. The author’s computational domonstration of the poorer fit of independent realizations of nominally the same process is ipso fact irrelevant when only tested against the realization present in the manufactured Hadcrut3 data series. It is on this point, rather than the many other objections raised by others, that Patterson’s predictions are likely to fail.

September 11, 2013 4:47 pm

I think that this is a most useful, reasoned and practical contribution to the overall debate. Sure, it hasn’t ‘proved’ anything one way or the other (my math is far too weak and my neurons retiring too rapidly to follow the fine grain of it, so use the salt shaker here…).
But the attempt is praiseworthy in itself: sticking one’s head over the parapet is always a risky move, and I congratulate Jeff for having done so.

September 11, 2013 5:28 pm

one way to end the debate about the merits of this technique is to divide the data into two 65 year time periods and see if one predicts the other without changing the methodology.
if the method can predict the full time series with 1/2 the data from either end, without some magical adjustable “aerosol” parameters, then you either have the basis of a successful prediction or you should be buying lottery tickets.
However, if the method cannot predict the missing 1/2, then it is time to look for a new methodology.
If the methodology can predict the full time series from 1/2 the series, then it is worthy of a second, separate WUWT article because that would be strong evidence the author is on the right track.

September 11, 2013 5:35 pm

Willis Eschenbach says:
September 11, 2013 at 2:59 pm
The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data.
========
I’d also like to see this and encourage the author to post the results.

September 11, 2013 5:55 pm

Discerning a 170 year period from HadCRUT alone sounds to me far-fetched considering past global temperature approximations.
What I see more is that HadCRUT, especially HadCRUT3 (which I see as correlating with UAH and RSS better than HadCRUT4), has a visible periodic cycle.
http://www.metoffice.gov.uk/hadobs/hadcrut3/diagnostics/global/nh+sh/
The periodic cycle there, assuming it’s sinusoidal, appears to me to have a period of 64 years, with peaks in 2005, 1941, and 1877. Its amplitude appears to me as peaking at nearly .11 degree C above or below the longer-term trend. I got this by trying my hand at Fourier for several trials of 2 cycles, cosine component only, with start dates around 1877 and end dates around 2005. (I did not do well with figuring the sine component, due to contamination by the linear trend, due to the simple methods I used.)
The nearly +/- .11 degree C periodic factor explains about 40% of the warming in HadCRUT3 from the early 1970s to 2005. Manmade direct increase of greenhouse gases other than CO2, which was largely stalled in the 1990s, appears to me to explain a little less than 10% of the early 1970s to 2005 warming. That means the anthropogenic part of the reported warming rate from the early 1970s to 2005 is about or slightly over half the total in that time period.
The anthropogenic part includes effects of bias in generating the HadCRUT3 index, such as adjustments with insufficient consideration for growth of urban effects around surface stations – but that appears to me fairly small – in light of HadCRUT4, current and recent past versions of GISS, and NCDC, and their divergences from UAH and RSS.
Thankfully HadCRUT, especially HadCRUT3, has sea surface temperatures fully considered. The sea surface consideration accounts for about 2/3 of HadCRUT and is essentially uncontaminated by growth of locally nearby airports or cities.
Overall, it appears to me that anthropogenic global warming is for real, but degree of its existence appears to me as something like 35% of the “center track” reported by IPCC in AR4.

Ulric Lyons
September 11, 2013 6:00 pm

Willis Eschenbach says:
“The second is that as far as anyone has every determined, the climate is chaotic …”
You should have dropped in on me when you were in the West Country, I could have shown you how I do deterministic weather forecasts. At the noise level it is not chaotic, so the sum of weather at the climatic scale cannot be either.

September 11, 2013 6:10 pm

Donald, you focus on the linear trend which is an artifact of the statistical analysis. As such linear trend cannot be proof of cause, nor even suggest it. In addition, a linear trend line has yet to be shown to be an effective method of demonstrating warming or cooling. If it were, the models would be no different than observation (the models use that statistic). Because they are not at all the same, the trend line itself is suspect in terms of calculating a meaningful trend (except for very short ones), regardless of its cause, either in the past and certainly not in the future.

Jeff Alberts
September 11, 2013 6:18 pm

Nick Stokes says:
September 11, 2013 at 5:11 am
How can you establish a 170 year periodicity from 113 years of data?

Dunno. Maybe it’s like saying todays temperatures are unprecedented for 1000 years while hiding the decline.

September 11, 2013 6:32 pm

Willis Exchenbach: “The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data.”
I chimed in with others to ask for this, and, although he did not do it in precisely the way in which I asked, he did hindcast to the pre-1900 period. To these old eyes the results weren’t impressive. I’m perfectly open to others’ convincing me otherwise, but otherwise I don’t find the results very interesting from a climate perspective.
That said, I’m not sure that all the criticisms are well taken. Those based on a obtaining a long fundamental period from a short record, for instance, are understandable but perhaps misapprehend what he actually did. He used the DFT to look for a clue as to what the fundamental might be, by observing peaks at frequencies that were integer multiples of a common fundamental. But it was only after he determined how much of the record could be “explained” by a periodic signal having that fundamental that he adopted it. Of course, the predictive value of the fact that such an explanation was achieved for that single record is not apparent.
And the criticism that “he is claiming that the climate is deterministic” is puzzling. Perhaps someone could point out where he claims that–or where the cite for Mandelbrot contradicts it.
Whatever the post’s other shortcomings are, I for one am grateful for not having had to grope through yet another fog of impressionism; the author actually laid out what he did.

Jeff Patterson
September 11, 2013 7:53 pm

Willis Eschenbach says:
September 11, 2013 at 2:59 pm
Hmm, I’ve seem to have struck some nerve with my hobby horse, that was not my intent. I long ago lost interest in the climate debate, knowing from the outset that model-based science is an oxymoron. My interest was re-sparked recently when I stumbled here while looking for some work related info on Wiener processes. There was an interesting post speculating on whether the climate was a random walk process. There was a link to the dataset so I pulled it in and ran a unit root test which failed with p=.7. The plot by eye looked to have some periodic patterns and knowing that periodic forcing functions can make a stationary process appear non-stationary, I looked at the PSD. I noticed the harmonic relationship between the spectral peaks so it seemed like a good candidate for harmonic decomposition, a technique we use widely for analyzing non-linear processes. After HD, the residual passed the URT at p=10^-12. I was also surprised by how few terms it took to get a good fit and that set me wondering if the periodicity was real or simply a curve fitting artifact.
Although I recognized the short record problem, it’s not one I normally encounter (if we need longer records we simply acquire more data – at 10 GSa/sec it doesn’t take long :>) I am an engineer and not a statistician so I used the tools I am familiar with to devise a significance test. The reasoning was simply. Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results. I do not see the flaw in the logic, nor in the method for testing the assertion. A s”simplistic” test is not necessarily a flawed test.
Perhaps the problem is revealed in your mischaracterization of my use of the AR model here:
“For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.”
Nowhere do I assume that it is AR data that we are seeing, nor do I anywhere attempt to prove that it is AR data we are seeing. The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record. If the HD fit were insignificant, it seems to me that one should easily achieve similar GOF results on the AR process data. As was shown in my post, this is not the case. It seemed an interesting result so I shared it.
As for halving the data, extracting a 170+ year of data from a 55 point record looks pretty hopeless. The HD algorithm fails to converge, as does the AR generated sequences 95% of the time. I may fool around with it some more if I get some time. As for hindcasting, I posted a link to a fifty year hindcast on the monthly data prior to your post.
Anyway, take it for what its worth (perhaps nothing). I need to get back to my day job.
Cheers.

John Andrews
September 11, 2013 9:26 pm

I, for one, am not going to bet the house on the “predictions”. For me, I think that tomorrow is going to be pretty much like today, and this year is going to be pretty much like last year… So far, this seems to fit the data in my lifetime (79 years, thank you.).

LdB
September 11, 2013 9:37 pm

@ Jeffery S. Patterson
If you happen to be reading this could you reprocess it after first passing it thru a high pass filter set at zero to remove the offset .. I would love to see what it shows?
It’s the same trick you do in your field to remove a DC voltage offset if my language is confusing.

Neil Jordan
September 11, 2013 10:44 pm

A useful reference that might bridge the gap between electrical engineering digital signal processing and climatology (hydrology and hydraulics) is “Stochastic Processes in Hydrology” by Vujica Yevjevich. The book can be obtained from several sources, including these:
http://www.amazon.com/Stochastic-Processes-Hydrology-Vujica-Yevjevich/dp/0918334012
http://books.google.com/books/about/Stochastic_processes_in_hydrology.html?id=uPJOAAAAMAAJ
http://trove.nla.gov.au/work/21245724?q&versionId=25362456
http://www.barnesandnoble.com/w/stochastic-processes-in-hydrology-vujica-m-yevjevich/1001110582?ean=9780918334015
A tribute to Prof. Yevjevich is located here:
http://www.engr.colostate.edu/ce/facultystaff/yevjevich/Yevjevich_index.htm

Editor
September 11, 2013 11:58 pm

Ulric Lyons says:
September 11, 2013 at 6:00 pm

Willis Eschenbach says:

“The second is that as far as anyone has every determined, the climate is chaotic …”

You should have dropped in on me when you were in the West Country, I could have shown you how I do deterministic weather forecasts. At the noise level it is not chaotic, so the sum of weather at the climatic scale cannot be either.

Are you a rich man, Ulric?
Because if not … why not? Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions.
Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.
w.

Editor
September 12, 2013 12:00 am

Jeff Patterson says:
September 11, 2013 at 7:53 pm

Willis Eschenbach says:
September 11, 2013 at 2:59 pm
Hmm, I’ve seem to have struck some nerve with my hobby horse, that was not my intent.

Struck a nerve? No, you’ve just done a poor and incomplete analysis, for the reasons I stated. If you want to pretend that somehow irritated people, sorry … the objections were scientific.
w.

Editor
September 12, 2013 12:18 am

Jeff Patterson says:
September 11, 2013 at 7:53 pm

Although I recognized the short record problem, it’s not one I normally encounter (if we need longer records we simply acquire more data – at 10 GSa/sec it doesn’t take long :>) I am an engineer and not a statistician so I used the tools I am familiar with to devise a significance test. The reasoning was simply. Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results. I do not see the flaw in the logic, nor in the method for testing the assertion. A s”simplistic” test is not necessarily a flawed test.

Indeed, you are correct—you’re not a statistician. While your reasoning is “simplistic”, it’s also wrong.

Perhaps the problem is revealed in your mischaracterization of my use of the AR model here:

“For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.”

Nowhere do I assume that it is AR data that we are seeing, nor do I anywhere attempt to prove that it is AR data we are seeing. The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record. If the HD fit were insignificant, it seems to me that one should easily achieve similar GOF results on the AR process data. As was shown in my post, this is not the case. It seemed an interesting result so I shared it.

Huh? You’ve admitted you used an AR model for your Monte Carlo test … but since you are not a statistician, you failed to realize that the choice of the model for the Monte Carlo test is a make-or-break decision for the validity of the test. You can’t just grab any data with similar bandwidth and variance as you have done and claim you’ve established your claims, that’s a joke.

As for halving the data, extracting a 170+ year of data from a 55 point record looks pretty hopeless.

Bad news on that front, Jeff. Extracting a 170+ year cycle from a 110 year dataset is equally hopeless. Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.

The HD algorithm fails to converge, as does the AR generated sequences 95% of the time. I may fool around with it some more if I get some time. As for hindcasting, I posted a link to a fifty year hindcast on the monthly data prior to your post.

Sorry, not impressed. Do it properly or not at all.

Anyway, take it for what its worth (perhaps nothing).

I see you have a keen eye for the estimation, not only of your own statistical abilities, but of the value of your work.

I need to get back to my day job.
Cheers.

Thanks for the reply, Jeff. Truly, you seem to be totally at sea in this particular corner of signal processing. It’s a recondite corner, with many pitfalls for the unwary, and full of poor data.
In addition, there is general agreement that the signal is chaotic in nature (I gave a reference above). If your analysis is correct, it is deterministic … which is a huge claim that seems to have escaped your notice. To show that it is not chaotic, I fear you need more than an un-testable (by your own admission) extraction of a 170-year cycle from 110 years of data.
All the best,
w.

Greg Goodman
September 12, 2013 12:23 am

JP: “Although I recognized the short record problem, it’s not one I normally encounter …”
This is a key problem with using DSP techniques and experience from other fields like electrical engineering and acoustics, where adequate samples can usually be made available. (And if not this is recognized. )
Most climate data is fundamentally inadequate to determine the nature of longer term variability. Whether century scale change is periodic, linear AGW , randow walk, or some other stochastic process can not be determined from the available data.
The big lie is that the IPCC is now, apparently, 95% certain that it can.
The last 17 years is the proof that they can’t.
That they become more certain in the face of increasing discrepancy shows how detached they have become from scientific reality and how dominated this intergovernmental body is preconceived policy objectives.

george e. smith
September 12, 2013 1:49 am

In the end, Digital signal processing, is just following a mathematical algorithm; applying it to a set of numbers.
So it differs from computing an average, only in that the mathematical algorithm is different.
You can take the very same set of numbers, and calculate the average, using that particular algorithm, or you can do the more complex DSP algorithms, and get something else from exactly the same set of numbers.
Well of course, you can also apply the same mathematics to the set of numbers you can find in your local telephone directory and get an average, or whatever the DSP algorithms produce.
The mathematics is quite rigorous; but that doesn’t mean the result has any meaning.
In the case of DSP, there usually is a starting assumption that there IS a “signal” there to be found in the set of numbers. Some procedures make use of prior knowledge of what the signal is supposed to be, and when it is supposed to be present, so you aren’t looking for something at times when it is not there, so you just gather noise.; such as locking onto a Loran C signal or these days, a set of GPS signals.
But if you apply these methods to sets of numbers which are acquired without regard to the rules of sampled data systems (which all our measurements are), then there may be no signal there to acquire.
That’s the problem with “climate data”. It doesn’t even come close to satisfying the Nyquist criterion, particular for spatial sampling of Temperatures, and the common method of Temporal sampling, also fails on that score. A daily min max Temperature sampling, only suffices, if the diurnal Temperature variation is strictly sinusoidal with a 24 hour period. Now in climatism, the aim is not to recover the continuous Temperature function for processing, but just to get the average, which is the zero frequency component of the spectrum. You can’t even recover the average, if the daily Temperature cycle has even a second harmonic component, let alone any higher frequency components, such as the results of cloud cover variations during the day.
So the mathematics may be quite elegant; but the input “data” is just garbage.

gnomish
September 12, 2013 2:26 am

chaotic != whimsical
ever seen a fetus with ultrasound?
dsp is abstruse. snarking in a corner is obtuse.

Sensorman
September 12, 2013 2:40 am

Jeffrey – when I look at the FFT result, one thing strikes me – there is obviously periodicity, but the picture looks reminiscent of a response containng reverberation or echoes. Unfortunately, I don’t have easy access to cepstral processing, but I would be fascinated to see whether there were peaks in the cepstrum. There may be “impulse responses” present that are triggered by discrete “events”, and there may be many overlaid so that they convolve in the spectrum. Just my feeling, as there is no real trace of actual peaks in the spectrum. There are maxima, but relatively broad. It’s the minima that interest me…

Sensorman
September 12, 2013 2:41 am

And apologies for mis-spelling your name, Jeffery!

September 12, 2013 3:21 am

Jonathan Abbott says: September 11, 2013 at 6:07 am “Straw man? Valid?”
Jeffery has given adequate qualifications and caveats for the treatment of data this way.
He notes that it could be mere curve fitting, then adds an extra test to show, to put it another way, that there is a high probability that this approach, used by others, would give a similar result.
One could agree with you that curve fitting has some weaknesses, because in the future case it cannot cope well with large, random events perhaps like the 1998 anomaly in temperature. But that is the case with all projections. For a short-term projection, this analysis beats most weaknesses of a linear least squares as is often used.
Personally, I would not push the analysis as far as he did, but that is a preference. Like Nick Stokes, I have a problem with projecting partial cycles; but Jeffery has incorporated this point in his caveats.
My worry would be that the temperature record has been adjusted in arbitrary and capricious ways that have the ability to distort the properties of the calculated curve. For instance, the ‘adjusted’ warming of the early years of the record is bound to have an effect.

Ulric Lyons
September 12, 2013 3:36 am

Willis Eschenbach says:
“Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions. Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.”
Help me market it and I’ll cut you in then. I know I have the product without any doubt.

LdB
September 12, 2013 5:54 am

@george e. smith says:
September 12, 2013 at 1:49 am

So the mathematics may be quite elegant; but the input “data” is just garbage.

You are right but you cheated and gave him the answer I wanted him to work it out.
I will give him credit he is at least thinking outside the usual box that climate science seems to be stuck in.
JP there is a range of techniques for non linear time invariant systems which would are far more suitable for doing the sort of analysis you are trying 🙂

LdB
September 12, 2013 5:55 am

Sigh … mods can you please fix the end of blockquote above .. sorry
[Fixed … -w.]

September 12, 2013 6:09 am

I talked to a thermo engineer from Intel yesterday and he surprised me when he told me that 40-50% of the cooling of a small phone in his pocket was via radiation (hence a surface optimized for emissivity). That’s a lot, generally we don’t see more than about 10-15% of the cooling effect via radiation. Of course, outside the package, we don’t get much benefit from conduction and the smaller the enclosure…well, of course, the less area there is for heat spreading. When it’s your job to cool something (or, conversely, to heat something) and radiation becomes an important part of toolbox, that’s truly a game-changer. That’s why guys like me get the big bucks, I guess.
It’s unfair to expect thoughtful colleagues like Mr. Courtney to consider the balance between CO2 backradiation heating and CO2 cooling via convection…or to realize that if a projected temperature anomaly includes negative numbers, that indicates the theoretical span includes cooling correlated with increasing CO2 concentration. It’s all good, carry on, my friends.

Pete Brown
September 12, 2013 6:51 am

This implies that we should have experienced a period of significant cooling over the last decade. Whereas in fact temperatures have been flat. Is this where the missing heat went?

Wayne
September 12, 2013 6:58 am

@Rabbit @Matthew R Marler: Done:

Wayne
September 12, 2013 7:04 am

@Rabbit @Matthew R Marler: Oops, it cuts out HTML. I did an SSA for GISS NH and SH separately. (There’s an R package for this: Rssa.) The Trend graph is based on the first two eigenvalues, and the Season1 graph is based on the second two eigenvalues.
http://i1090.photobucket.com/albums/i371/WayneAnom/GISSssaNH_zpsf67f65f6.png
http://i1090.photobucket.com/albums/i371/WayneAnom/GISSssaSH_zpsb04724ae.png
Interestingly, both hemisphere’s Season1 appear to be damped oscillations. Perhaps artifacts of GISS processing. (I’ve had discussions about GISS in other forums, and especially the SH is whacky, since they smooth over 1200km. When Antarctica came online in the 1950’s you can detect a large change in volatility.)

richardscourtney
September 12, 2013 7:08 am

Pete Brown:
Your post at September 12, 2013 at 6:51 am asks

This implies that we should have experienced a period of significant cooling over the last decade. Whereas in fact temperatures have been flat. Is this where the missing heat went?

Nice try, but no coconut.
The hypothesis of discernible warming is wrong and Trenberth’s “missing heat” is one of the many reasons we know it is wrong. Clutching at straws about why it is missing is merely desperation by warmunists trying to stay afloat when they are attached to the sinking AGW-scare.
There is no sign of Trenberth’s missing heat in the depths or how it could have got there. I think that if the “missing heat” ever existed then probably went the other way and by now it has passed Alpha Centauri. But so what? The important point is that it is missing and not why it cannot be seen.
Richard

Wayne
September 12, 2013 7:20 am

@Willis: “Indeed, you are correct—you’re not a statistician.”
Neither are you.
Neither am I, for that matter, though you and I are both avid students of statistics. Jeff’s method has issues, but I’d point out that he’s more knowledgeable and has been more rigorous than the average WUWT poster. Considering your confusion on basic time series analysis a while back (“linear trend”), I’d suggest that you should be more gracious in your criticisms. Jeff has certainly been reasonable in his tone, and deserves the same from you.
As a high school physics teacher once told me, it gets frustrating teaching the same course year after year and yet every single year the students still come into the class not yet understanding it. 😉 Curve-fitting is a problem, but don’t take out your frustration for past curvologists on Jeff.

Matthew R Marler
September 12, 2013 9:30 am

Jeff Patterson: I’ve seem to have struck some nerve with my hobby horse,
Not so. Willis Eschenbach is a vigorous critic of most modeling that gets presented here. Like me, he mocked the pretended precision of the estimated period of “170.7”. I don’t think data halving is as useful a technique as waiting for the next 20 years of out-of-sample data. I think he missed your qualification that if the system is chaotic your model was useless for prediction. I think your defense of your choice of null distribution for null hypothesis testing is adequate for now.
I think it will be a miracle if your model proves to be reasonably accurate over the next 20 years, and I think on the whole it is less credible than Vaughan Pratt’s model (which also will, in my guess, require a “miracle” to be accurate over the next 20 years), but I put in in what I call “the pile of active models”. Because you provided the code, anyone who is interested can test.

September 12, 2013 9:35 am

@Geoff Sherrington 3:21 am. +1 A balanced evaluation.
@Greg Goodman 12:23 am +5 Succinct. A reposte to remember for AR5.

Most climate data is fundamentally inadequate to determine the nature of longer term variability. Whether century scale change is periodic, linear AGW , random walk, or some other stochastic process cannot be determined from the available data.
The big lie is that the IPCC is now, apparently, 95% certain that it can.
The last 17 years is the proof that they can’t.

@Willis Eschenbach 12:18am
Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.
Amen! That’s why the BEST scalpel is folly. It turns a low pass signal into a band pass signal. At worst, it preserves drift as signal and discards vital recalibration events.

Matthew R Marler
September 12, 2013 9:39 am

Wayne: @Rabbit @Matthew R Marler:
Cute. What was the reference to “Science”? Did that article describe the method?

September 12, 2013 9:44 am

@Willis Eschenbach 12:18am
Extracting a 170+ year cycle from a 110 year dataset is equally hopeless.
You cannot do it with an FFT. You can and should recognize the possibility there is a dominant low frequency in the data that the tool cannot adequately detect.
Take an Excel function describing a partial sine wave with a linear ramp.
=AmpS*SIN(Phase)+LinSlope*(Year-2000)+Y2KIntercept
With Year=(1900:2012)
AmpS = 0.4 (deg C)
Phase =(Year-PhaseOrig)/CycLen*2*PI()
PhaseOrig = 1965
CycLen = 175 (years)
LinSlope=0.001 (deg C/yr) (= 0.1 deg C/ century)
Y2KIntercept = 0.3 (deg C)
If you plot it up, (thick blue line on)
Excel chart picture: http://i43.tinypic.com/16j3ndj.jpg
If you do get a least squares linear fit you get.
Y = 0.0098*(Year-2000) + 0.3 (almost 1.0 deg C / century)
And an R^2 of 0.9328. (thin blue line)
In truth, there is only a 0.1 deg C / century linear trend super imposed upon a clean sinusoidal -0.4 to +0.4 deg with a wave length of 175 years that the measurements caught between phase (-2.33 and 1.68 radians) (-133 and 96 deg) (thick blue curve.)
If you remove the baseline between the 1900 and 2012 end points (maroon dashed line.), as per Jeff, you will get a thick maroon residual that you would do FFT and harmonic analysis. By visual inspection, I think most people can see the problem. The removal of the baseline has the following effects on the residual curve:
1. It shortened the cycle length to maybe 120 years instead of 175,
2. greatly reduced the amplitude of the Sinusoid to about 0.2 instead of 0.4,
3. introduced high frequency ringing caused by the mismatch of slopes at the end points,
4. and finally, baked in a 0.71 deg/century linear trend created by the choice of baseline, seven times higher than what is in the original [known], under-measured signal.
Such pitfalls make me want to give the pit wide berth.

September 12, 2013 10:39 am

Clarification: my 9:44am is in support of Willis’s 12:18am
Extracting a 170+ year cycle from a 110 year dataset is equally hopeless
I just wanted to point out that just because a tool cannot see (like the FFT) a longer wave length or cannot uniquely determine a long wave length (harmonic decomp), doesn’t mean we should blind ourselves to the possibility of long wave length cycles in the system response of the climate. Indeed, at geologic scales with mutiple ice ages and interglacial periods, there likely are some. But our temperature records are too short, too noisy, and too adjusted to have any hope of determining any century scale or longer wave lengths in the earth’s system’s response.

george e. smith
September 12, 2013 11:03 am

“””””””…….LdB says:
September 12, 2013 at 5:54 am
@george e. smith says:
September 12, 2013 at 1:49 am
So the mathematics may be quite elegant; but the input “data” is just garbage.
You are right but you cheated and gave him the answer I wanted him to work it out………””””””””
So “I” cheated ??
No, and I also did not assume, that Jeff was in any need of being given any answer. My post on this thread, in no way conveys any criticism of Jeff or his methods. It is always refreshing to listen to someone tell what he does in his cubicle.
Having been an Agilent (read REAL HP) employee myself, in a past life, I fully understand, that those blokes have to know that their stuff works.
But if you send a bunch of researchers out to a bunch of places, and ask them to count and tabulate the number of animals per square meter, or maybe Hectare; animal being restricted to only those bigger than an ant, and you record that data for 150 years; then Jeff could apply his methods to your data set, and come up with some output that you could feed to the WWF (no not the Hulk Hogan crowd); and they could all go gaga over it, and ask the UN for funding to rectify the problems discovered.
But don’t blame Jeff’s analytical tools; he didn’t know a priori (perhaps), that your data is garbage.
Of course the number of animals per hectare; regardless; or irregardless, as the case may be, of the size, and species of such critters; contains about as much information as the Temperatures, taken at quite random, unrelated times, at quite randomly placed, and also unrelated sparse locations around the globe.
Yes you can do a FFT on the numbers, you could even expand them as an orthogonal set of Tchebychev polynomials, or Bessel functions. Well there’s nothing wrong with the maths; make it as simple as calculating the average, or maybe the median. The result is valid, but still garbage, because the data was.
Now Mother Gaia does it right; even proper, or properly; because she has in effect a Thermometer in each and every molecule, so she always knows what the Temperature is; and it always is what it is supposed to be. But she is not going to communicate the answer to us.

Steven Groeneveld
September 12, 2013 11:32 am

This is an interesting analysis and of course there are a lot of reasonable rational caviats that are mostly valid and necessary for rigorous analysis. Still that should not blind us to the possiblity of being able to make some sense of imperfect data.
Jeff Patterson says:
In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data. if the condition is not met, the high frequency components masquerade as low frequency components in a process referred to as aliasing. That is why we remove the high frequency components by filtering before decimating (re-sampling) the data.
Willis Eschenbach says:
“The second is that as far as anyone has every determined, the climate is chaotic
@Willis Eschenbach Even Chaos can have an underlying harmonic structure (for example, fractals, Strange attractors, etc.)
Willis Eschenbach says:
Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.
That is correct and valid when you can get the data. For dynamic analysis of flight test data for flutter prediction we use a sample rate of at least 5 times the highest vibration frequency expected, so this is in line with standard practice.
“And as also pointed out proper signal filtering before sampling is necessary to avoid problems like aliasing. Also I have very little confidence in the manner in which global temperatures are averaged. (Since the earth loses heat only by radiation an average of the 4th power of (absolute scale) temeratures makes more sense but that makes a very small difference compared to the distribution of source stations, or time of day of recording or using only min and max etc.). So the whole data set is suspect from the start but there can still be a harmonic signal in it.
However it has been useful to me in the past to have data that was not correctly processed. I was involved in the investigation of an aircraft accident that lost pitch control due to flutter and, as luck would have it there was a primitive recording device on board the aircraft. Signals from the AHRS (Attitude and Heading reference system) were recorded without pre-filtering onto a laptop computer at a relatively low sample rate (about 16 Hz I think). When analysing the data at the point of the failure there was a 5.5Hz signal that indicated the flutter that caused the failure. By calculating the frequences that could alias to 5.5 Hz the possibilities could have been 11 Hz 22 Hz or some higher Frequency. By putting the pilots on a seat on a shaker (that we use for ground vibration test excitation) at the possible and plausible frequencies the pilots could identify the shaking they felt as the 11 Hz. This was also in line with one of the resonant frequencies or the tailplane identified in a ground vibration test of the aircraft Having an external confirmation is necessary to identifying which frequency is aliased (although , as also pointed out, confirmation bias is always a possibility)”
Looking for high frequencies in an aliased signal is at the opposite end to the spectrum to looking for low frequencies in too short a period may not be exactly similar but I do recognise the fact that there could be a signal and I have spent some time myself curve fitting cycles to the satalite temperature record so this analysis intrigues me. I am also intriqued that the 170 point whatever cycle is close the the 179 year solar baricentre cycle. Even though a recent WUWT post discounted any planetary cycle influence on the solar cycle, It may be, as Willis pointed out, there is not enough data length to cover enough cycles sufficiently to recognise the signals properly, or that there is more noise than signal. Anyway I am biassed in appreciation for Jeff Patterson’s analysis since it is an approach that is familar to me.

Wayne
September 12, 2013 11:38 am

@Matthew R Marler says: “Cute. What was the reference to ‘Science’? Did that article describe the method?”
? I didn’t make any reference to “Science”, and I’m not sure what’s “cute”. @Rabbit asked about doing an SSA and I found a library to do so and did so. I haven’t managed to pull out frequencies yet — its internal data structure is obtuse and requires a couple of extra layers of unwrapping — but I thought the graphs would be interesting to compare to Jeff’s straight FFT approach.

george e. smith
September 12, 2013 11:53 am

Speaking about giving the answers; there’s a “Science” web site out there, where consensus reigns supreme.
Anyone can ask a question. “Are the ends of a wormhole polarized the same or opposite ?”
Anyone can answer the question. No ! the two ends of a wormhole have the same polarity; but it can be up or down.
Well Wikipedia says they are always opposite !
If you give an answer different from Wikipedia, you are in trouble.
Then anybody can vote for or against your answer; to determine its popularity. No restrictions on the credentials or lack thereof, of the critics. And so you get a score.
Now all the kids taking 4-H club Quantum Chromo-dynamics classes, like to post their homework problems there so they don’t have to look it up in Wikipedia.
You are not aloud to answer someone’s homework problem; which some referee designates as an “hw” problem. Well you can tell the “OP” which is code for the chap who asked the question; go to the Stanford Science library, and look it up in Born and Wolfe; but you can’t tell him/er that the answer is 12.
So someone; the “OP” put up a network of resistors, and asked for the equivalent resistance between two nodes. So the correct answer is; go to Wikipedia and look up Kirchoff”s Laws.
But this network has a problem. The values for the separate resistors, are not industry standard (RMA) values. So it had numbers, like 40, 120, 100 Ohms, instead of 39, 120 and 100, which are good numbers.
It was quite obvious from the numbers in the problem that the examiner, fully intended the student (New Zealand white rabbit farmer), to figure it out in his head, not putting pen to paper, till it came time to write down 32 Ohms.
I did that and gave him the answer; 32 Ohms. That drew the condemnation of some twirp younger than my children, who gave some elaborate algebra, that rabbit farmers wouldn’t understand; and the consensus fan base voted me a -2 popularity score.
I once set an exam question not unlike that one, for a pre-med first year Physics class (of 200 student doctors to be).
Part (a) Write down Kirchoff’s laws.
Part (b) Use Kirchoff’s laws, to derive the balance condition for a Wheatstone Bridge.
Part (c) Determine the current flowing in the resistor R7 in this network.
Now “this network” consisted of seven resistors connected between a total of four nodes. In two instances, there were simply two resistors in parallel between a pair of nodes. A Voltage of 10 Volts, was applied between nodes (1) and (3) and resistor R7 of 1,000 Ohms, was between nodes (2) and (4).
So if you figured out the parallel pairs (in your head) reducing the network to five resistors, and imagined the four nodes, to be in a diamond configuration, instead of the rectangular boxy shape I drew them, you would immediately see that four of the resistors comprised a Wheatstone Bridge; (please do part (b) of the problem before doing part (c). and a simple mental note would show that with those values, the Wheatstone bridge was indeed balanced; Please do Part (a) of the problem before doing part (c).
So the correct answer was zero current.
Out of the 200 students who sat the exam, not a single one figured out that the network was indeed a Wheatstone bridge, let alone that it was balanced.
About 50 students laboriously worked through the Kirchoff’s laws equations, about 45 got the arithmetic correct to give the correct answer; the other five worked the equations properly, and goofed on the arithmetic.
150 of them got parts (a) and (b) correct . Nobody received the full 20 points for the question, because none of them seemed to understand that the theory part might be related to the simple problem.
Anyhow, this consensus science site is a gas, just reading some of the goofy questions.
But you can get your favorite string theory questions answered there.
On bright student, asked if anybody had ever studied whether the universe was moving; and if so, in which direction.
So these folks are the future of science, unless some rationality eventually prevails in our education systems.
Nine (9) of my class of 200 students of excellent doctor/dentist/veterinarian candidates, got accepted by the medical school, which at the time, was at another campus, in another (smaller) town.
I elected to transfer over to Industry, from academia.

September 12, 2013 12:18 pm

Correction to: 10:39 am
… just because a tool cannot see … a longer wave length or cannot uniquely determine a long wave length (harmonic decomp), doesn’t mean we shouldn’t should blind ourselves to the possibility of long wave length cycles in the system response of the climate. ….
In other words, always be aware of what you cannot (or did not) detect.
[Fixed. -w.]

1sky1
September 12, 2013 2:26 pm

The oft-repeated idea that a noisy 113-yr temperature record doesn’t contain sufficient information to determine the parameters of a 170-yr periodic cycle is somewhat misguided. Simple heterodyning with the available record is entirely adequate for that purpose–provided that the fundamental period is known a priori on the basis of physics. That is the proven basis of Munk’s “Super-resolution of the Tides.” Without such a priori knowledge, however, even a 170-yr record would not be adequate, since the resulting raw periodogram would vary as chi-squared with only two degrees of freedom. In the face of spectral frequency smearing by the the DFT, there would be scant grounds for any reliable identifcation of any periodic components
The real culprit.in various simplistic models of yearly average temperature series is the wholly unjustified presumption that there ARE strictly periodic components (spectral lines) to be found in the empirical record That is the common blunder made not only here, but with Pratt’s multi-decadal “sawtooth” or Scafetta& Lohle’s two-sinusoid oscillation. In fact, what we have is stochastic oscillations with various bandwidths, which give the appearance of “trends” and “periodicities” in records too short for scientific purposes.

Matthew R Marler
September 12, 2013 3:58 pm

Wayne: ? I didn’t make any reference to “Science”, and I’m not sure what’s “cute”.
One of the links that you supplied had a series of ppt-type pages displaying graphs and such, and one of those pages was a header from a page of “Science”. The other link was a single page.

September 12, 2013 6:45 pm

I did not read the code in entirety, so I wonder: Does it consider the squares of temperature anomalies from a horizontal line or a best-fit whatever, or does it consider unsquared anomalies?
It appears to me that Fourier works with unsquared data. Using squared anomalies from a best-fit linear trend or whatever would give more weight to the the 1878 and 1998 El Ninos than Fourier does, and less weight to the broader peak centered around 2004-2005. The ~1910 dip and the 1950s dip, if they are squared, could cause a shorter period determination for appears to me as determination of the 3rd harmonic.
I have more faith in Fourier for spectral analysis.
My eyeballs tell me that the strongest periodic component in HadCRUT3, of periods near or less than 200 years, has a period around 60-70 years.
I am expecting the warming from the early 1970s to ~2005 to be roughly repeated, and to only slightly greater extent, after about 30 years of 5-year-smoothed global temperature roughly stagnating or very slightly cooling. (Or trend of 1-year or shorter period temperature going maybe 34-35 years without warming, if the start time is sometime in 1997 when a century-class El Nino was kicking up or about to kick up.)

Wayne
September 12, 2013 6:50 pm

@ Matthew R Marler: I supplied two links to two graphs of the results of SSA. Evidently there are back and forward arrows to other pictures I’ve linked to on this site before, which I didn’t know you’d see. The other “slides” had nothing to do with the current discussion.
I’ll have to delete old slides each time I post new ones. Stupid site.

September 12, 2013 8:12 pm

For anyone looking for a simple assessment.
Average global temperature history since 1975 is like a hill. We went up the hill from 1975 to 2001 where the average global temperature trend reached a plateau (per the average of the five government agencies that publicly report average global temperature anomalies). The average global temperature trend since 2001 has been flat to slightly declining but is on the plateau at the top of the hill. Claiming that the hill is highest at its top is not very profound. The temperature trend has started to decline but the decline will be slow; about 0.1 K per decade for the planet, approximately twice that fast for land areas.
A licensed mechanical engineer (retired) who has been researching this issue (unfunded) for 6 years, and in the process discovered what actually caused global warming and why it ended, has four papers on the web that you may find of interest. They provide some eye-opening insight on the cause of change to average global temperature and why it has stopped warming. The papers are straight-forward calculations (not just theory) using readily available data up to May, 2013. (data through July made no significant difference)
The first one is ‘Global warming made simple’ at http://lowaltitudeclouds.blogspot.com It shows, with simple thermal radiation calculations, how a tiny change in the amount of low-altitude clouds could account for half of the average global temperature change in the 20th century, and what could have caused that tiny cloud change. (The other half of the temperature change is from net average natural ocean oscillation which is dominated by the PDO)
The second paper is ‘Natural Climate change has been hiding in plain sight’ at http://climatechange90.blogspot.com/2013/05/natural-climate-change-has-been.html . This paper presents a simple equation that, using a single external forcing, calculates average global temperatures since they have been accurately measured world wide (about 1895) with an accuracy of 90%, irrespective of whether the influence of CO2 is included or not. The equation uses a proxy which is the time-integral of sunspot numbers (the external forcing). A graph is included which shows the calculated trajectory overlaid on measurements.
Change to the level of atmospheric CO2 has had no significant effect on average global temperature.
The time-integral of sunspot numbers since 1610 which is shown at http://hockeyschtick.blogspot.com/2010/01/blog-post_23.html corroborates the significance of this factor.
A third paper, ‘The End of Global Warming’ at http://endofgw.blogspot.com/ expands recent (since 1996) measurements and includes a graph showing the growing separation between the rising CO2 and not-rising average global temperature.
The fourth paper http://consensusmistakes.blogspot.com/ exposes some of the mistakes that have been made by the ‘Consensus’ and the IPCC

Mike Wryley
September 12, 2013 8:20 pm

Dang DSPs have taken all the eye candy glamor out of electronics. Instead of little silver plated coils, silver mica capacitors, crystal filters, cavity resonators, lots of little things to tune and a bunch of gold plated goodies to tie everything together in an artful manner, all you see now is a black epoxy stamp size wafer with a bunch of wires coming out of all four sides, processing signals in the digital domain in ways that are nearly impossible in analog.

Pete Brown
September 13, 2013 12:43 am

richardscourtney:
Regarding your post at September 12, 2013 at 7:08 am
Thanks for your response. To be fair, I was asking a question though, not making a point. I don’t have any particular need to grasp any straws. I notice though that you don’t seem to have quite addressed the point, despite seeming very confident about it. So I wonder if we could revisit this?
My thinking is this:
Let’s assume everyone agrees that CO2 is a greenhouse gas, and will have a slight warming effect, all other things being equal.
Let’s also assume that everyone agrees that all other things are NOT equal, and that natural variability plays a significant role in global climate relative to the impact of CO2.
One upshot of this is that it is much harder to detect the impact of CO2 against natural variability than climate extremists would have us imagine. The impact of CO2 relative to natural variability is much smaller than claimed.
But just how reassuring is that? No-one can say that it is not warmer now than it would otherwise have been, can they? How would we possibly begin to establish that empirically?
Again, I’m asking a question here, not trying to make a point.
Also, if it is accepted that CO2 is a greenhouse gas, and will have a slight warming effect, all other things being equal, then any theory which fails to take account of this – by only including natural variability for example – is also wrong. Isn’t that true?
Grateful for any comments on this…

richardscourtney
September 13, 2013 1:28 am

Pete Brown:
I am replying to your post addressed to me at September 13, 2013 at 12:43 am
http://wattsupwiththat.com/2013/09/11/digital-signal-processing-analysis-of-global-temperature-data-suggests-global-cooling-ahead/#comment-1415817
If I understand you correctly, then what you are really asking about is climate sensitivity. If so, then the magnitude of climate sensitivity is of importance and – in the context of your interest – the precise details of WHY climate sensitivity has that magnitude are secondary.
If I have misunderstood you then please say because I am replying to what I think you are saying and not avoiding anything.
I provide two answers. Firstly, if you use the WUWT Search function then you will find much on climate sensitivity. Secondly, my view on the matter is as follows.
I am convinced that increased atmospheric CO2 concentration will result in some rise in global temperature, but I am also convinced any such temperature rise would be too small for it to be discernible and, therefore, it would only have an abstract existence. I explain this as follows.
Before presenting my argument, I point out I remain to be convinced that human emissions are or are not the cause – in part or in whole – of the observed recent CO2 rise. However, the cause of a rise in atmospheric CO2 concentration is not relevant to the effect on global temperature of that rise.
My view is simple and can be summarised as follows. The feedbacks in the climate system are negative and, therefore, any effect of increased CO2 will be too small to discern. This concurs with the empirically determined values of low climate sensitivity obtained by Idso, by Lindzen&Choi, etc..
In other words, the man-made global warming from man’s emissions of greenhouse gases (GHG) would be much smaller than natural fluctuations in global temperature so it would be physically impossible to detect the man-made global warming.
Of course, human activities have some effect on global temperature for several reasons. For example, cities are warmer than the land around them, so cities cause some warming. But the temperature rise from cities is too small to be detected when averaged over the entire surface of the planet, although this global warming from cities can be estimated by measuring the warming of all cities and their areas.
Similarly, the global warming from man’s GHG emissions would be too small to be detected. Indeed, because climate sensitivity is less than 1.0°C for a doubling of CO2 equivalent, it is physically impossible for the man-made global warming to be large enough to be detected. If something exists but is too small to be detected then it only has an abstract existence; it does not have a discernible existence that has effects (observation of the effects would be its detection).
I hold this view because I am an empiricist so I accept whatever is indicated by data obtained from observation of the real world.
Empirical – n.b. not model-derived – determinations indicate climate sensitivity is less than 1.0°C for a doubling of atmospheric CO2 equivalent. This is indicated by the studies of
Idso from surface measurements
http://www.warwickhughes.com/papers/Idso_CR_1998.pdf
and Lindzen & Choi from ERBE satellite data
http://www.drroyspencer.com/Lindzen-and-Choi-GRL-2009.pdf
and Gregory from balloon radiosonde data
http://www.friendsofscience.org/assets/documents/OLR&NGF_June2011.pdf
Climate sensitivity is less than 1.0°C for a doubling of atmospheric CO2 concentration and, therefore, any effect on global temperature of increase to atmospheric CO2 concentration only has an abstract existence; it does not have a discernible existence that has observable effects.
In the context of this thread, the implication of small climate sensitivity is that atmospheric CO2 concentration has very small effect and, therefore, may not be capable of discrimination when analysing temperature data.
Please get back to me if I have failed to address what you wanted or I have not been clear.
Richard

richardscourtney
September 13, 2013 1:32 am

Pete Brown:
re your comment to me at September 13, 2013 at 12:43 am.
I have provided a reply but (for some reason) it is stuck in moderation and should appear in an hour or so. Whatever the reason for this delay, it is not that I am avoiding provision of an answer to you. Please be patient.
Richard

Pete Brown
September 13, 2013 1:50 am

Richard
Thanks, I am grateful.

September 13, 2013 3:42 am

nice piece of work. what it shows of course is that the data supports very little correlation with rising CO2 levels at all..sop co2 sensitivity is very low indeed, and almost all the warming is down to ‘something else’

richardscourtney
September 13, 2013 4:26 am

Pete Brown:
It has now appeared. Hopefully this will appear in your inbox.
Richard

Pete Brown
September 13, 2013 7:26 am

richardscourtney:
Richard
Thanks. I am familiar with the concept of climate sensitivity, but very interesting to hear your views. Thanks for your time in responding.
Pete

Editor
September 13, 2013 7:27 am

Ulric Lyons says:
September 12, 2013 at 3:36 am

Willis Eschenbach says:

“Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions. Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.”

Help me market it and I’ll cut you in then. I know I have the product without any doubt.

And I greatly doubt that you have the product. What a team!
w.

Editor
September 13, 2013 7:32 am

Wayne says:
September 12, 2013 at 7:20 am

@Willis: “Indeed, you are correct—you’re not a statistician.”
Neither are you.

Hey, I’m quoting him, not expressing a second opionion.
w.

Henry Galt
September 13, 2013 7:46 am

Willis Eschenbach says:
September 13, 2013 at 7:27 am
You are missing out twice Willis.
Scientifically. Monetarily.
I have seen Ulric’s work from its inception.
I predict much egg. On many, many faces.

Editor
September 13, 2013 7:48 am

Jeff Patterson says:
September 11, 2013 at 6:11 am

Nick Stokes says:

How can you establish a 170 year periodicity from 113 years of data?

In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data.

I’m not understanding this one at all. Suppose we have two points, one at say (0,0) and one at (1, 0.3).
My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.
Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that. For example … what is the shortest period sine wave in the HadCRUT data? And how does that relate to a 170 year cycle?
Thanks in advance for your answers,
w.

Wayne
September 13, 2013 8:16 am

@ Matthew R Marler: I’ve removed all but the two graphs of the results of the SSA. As I mentioned, the “Trend” is the first two components, and the “Season” is the second two components, which I chose based on a scree plot. The Season for both NH and SH are very similar, and appear to be some kind of damped signal with a roughly 50-year frequency — perhaps a ringing of some sort — which may simply be an artifact of GISS’ processing.
@Willis: Yes, yes, you were quoting Jeff, but in a way that is clearly a personal judgement. Maybe it’s just me, but you come across as harsh rather than disagreeing. Which might be justified if he were vociferous in his claims, which he wasn’t.

pochas
September 13, 2013 8:20 am

Since a sine wave has two variables, amplitude and phase, all you need to determine it is two points giving two equations which you solve simultaneously. You can in principle generate a sine wave with a period of one year from two points days apart.

Editor
September 13, 2013 9:18 am

Hmmm… There are a load of other cycles seen in longer term weather (including the Bond Event cycles). Several of them look to have a lunar periodicity / connection. I lean somewhat toward the simple mechanism of lunar / tidal ocean mixing modulation (that I’ve linked to here many times in a peer reviewed paper / link). So I wonder if there is a lunar period “close” to this?
Draconic Month is when the Moon crosses from above to below the ecliptic (or the other way). That period is 18.6 years.
https://en.wikipedia.org/wiki/Month#Draconic_month
“Sometimes written ‘draconitic’ month, and also called the nodical month. The orbit of the moon lies in a plane that is tilted with respect to the plane of the ecliptic: it has an inclination of about five degrees. The line of intersection of these planes defines two points on the celestial sphere: the ascending node, when the moon’s path crosses the ecliptic as the moon moves into the northern hemisphere, and descending node when the moon’s path crosses the ecliptic as the moon moves into the southern hemisphere. The draconic or nodical month is the average interval between two successive transits of the moon through its ascending node. Because of the sun’s gravitational pull on the moon, the moon’s orbit gradually rotates westward on its axis, which means the nodes gradually rotate around the earth. As a result, the time it takes the moon to return to the same node is shorter than a sidereal month. It lasts 27.212220 days (27 d 5 h 5 min 35.8 s). The plane of the moon’s orbit precesses over a full circle in about 18.6 years.”
3 x that is 55.8 years (very close to the 56 year period often called a “60 year cycle” seen in weather cycles – when the Saros cycle returns to over the same 1/3 of the globe …) Now what is 3 x 56? 167.4 years. Or very close to that 170.7 year period found.
I think what we are finding is the lunar tidal effect on ocean depth and mixing as the moon changes where it is relative to the continents. Above and below the ecliptic. And in line with the sun when particular continents are underfoot. Essentially an interaction of tides with the continents as the moon makes different strength tides with different land lined up.
Folks often talk about natural ocean oscillations, but could not there be a tidal metronome?…

Matthew R Marler
September 13, 2013 9:36 am

Willis Eschenbach: My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.
Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that.

Right on both counts.
pochas: Since a sine wave has two variables, amplitude and phase,
A sine wave has three variables: amplitude, phase and period. The claim that you only need 2 points to estimate a sine wave depends on the assumption that one of those is known.
To go back to the modeling of Jeffrey S. Patterson, he started by focusing on sine curves. After some smoothing, he used his 112 years of data to estimate the amplitude, period and phase of the fundamental, then nonlinear least squares to estimate the amplitude and phase of each harmonic. His decision to focus on sine curves is totally unprincipled, unless there is strong external evidence that the system has to be periodic: he could have used orthogonal polynomials of high order, wavelets, b-splines, etc. He wrote appropriate caveats: if the system is chaotic instead of periodic, there is no reason to expect his result to have any predictive value. And Willis Eschenbach appropriately critiqued Patterson’s null hypothesis test by pointing out that the “correct” null distribution is not known (that is, the “true” background variation is not known.) But Patterson explained that his choice of a null distribution, which in this field is about all one can be expected to do, unless we require an infinite set of null distributions.
In my mind, the value of such work is in what comes after: perhaps someone will be stimulated to find, and will find, a physical basis for the fundamental period, and maybe analysis will show that some features of the system produce the harmonics from the fundamental. In statistics, some people refer to work such as Patterson’s as “hypothesis generation”. At least with Vaughan Pratt’s model he had a physical basis for the main monotonic trend, but his estimates of the natural variation were suspect. Nicola Scaffetta is seeking physical drivers in the geometry and gravitation of the solar system. And so on.

Juan
September 13, 2013 9:39 am

This is just curve fitting. No matter how tricky you present it and how many tricks you use, the result is is the same: just curve fitting. Curve fitting CANNOT be used to make predictions. The misconception here is that you assumed that there are some cycles composing the evolution of temperature contained in the data you are using. But that’s just an assumption and is based in nothing.
You’re just playing with the data and the math, but there’s no physics on your work. Whenever you look for cycles in a data set, you need to be ready to explain the periods you get from it. But for this case you even forced the results and, therefore, the periods of the cycles you obtain are a result of the temporal length of the data set and the methodology you use.
Sorry, very impressive tricky math but the result is just a nonsense.
Ask the tide predictors if they can predict tides the way you’re trying to predict temperatures. Of course not.

Matthew R Marler
September 13, 2013 9:43 am

E. M. Smith: The plane of the moon’s orbit precesses over a full circle in about 18.6 years.”
3 x that is 55.8 years (very close to the 56 year period often called a “60 year cycle” seen in weather cycles – when the Saros cycle returns to over the same 1/3 of the globe …) Now what is 3 x 56? 167.4 years. Or very close to that 170.7 year period found.

So a harmonic model with sines of periods 18.6, 55.8, and 167.4 years would produce a result not unlike what Patterson found (that is, appx the same sum of squared residuals), and might be predictive of the future. It’s a mere conjecture, but what isn’t at this point?

Ulric Lyons
September 13, 2013 10:18 am

Willis Eschenbach says:
“And I greatly doubt that you have the product. What a team!”
I agree, there is no currency in it with your attitude, it would be like flogging a dead horse. The last type of person I need to team up with is someone who has looked and failed to find any connections and who also thinks that their opinion on the subject is superior. I am more interested in showing you a set of highly interesting and meaningful connections that can change your mind on the matter. If you refuse to look, it’s not the end of the World for me.

Jeff Patterson
September 13, 2013 10:24 am

Willis Eschenbach says:
September 13, 2013 at 7:48 am
“My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.”
Nyquist would be very sorry to hear that. Fortunately for us (especially for me – otherwise I’d be digging ditches) he is correct. The bandwidth limitation constrains the maximum slope and amplitude, squeezing your infinite solution space down to just one. Actually sampling at exactly the Nyquist rate splits the spectral energy between the Nyquist bin and d.c. in a manner that depends on the phase of the signal relative to your sample clock and is ill-advised.
“Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that?
Because I low-pass filtered the data prior to decimation.
Thanks in advance for your answers,
Your welcome

Jeff Patterson
September 13, 2013 10:44 am

“Huh? You’ve admitted you used an AR model for your Monte Carlo test … but since you are not a statistician, you failed to realize that the choice of the model for the Monte Carlo test is a make-or-break decision for the validity of the test. You can’t just grab any data with similar bandwidth and variance as you have done and claim you’ve established your claims, that’s a joke.”
If by Monte Carlo test you refer to section entitled “Stochastic Analysis” and not to the section on forecast (where I also use a AR process), please show me where “[I am] ASSUMING that AR data is what we are actually looking at” or where “I attempt to prove that if it is AR data we are seeing”. That section makes no such assumption. The assumption is the one I stated earlier. To wit:
“Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results.” You call this a joke but fail to reveal the punch line. Come on, give it up. I could use a good laugh even at my own expense.

Jeff Patterson
September 13, 2013 10:46 am

Sorry, the post above should have referenced Willis Eschenbach post of September 12, 2013 at 12:18 am