Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead

This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data. Digital Signal Processing analysis suggests cooling ahead in the immediate future with no significant probability of a positive anomaly exceeding .5°C between 2023 and 2113. See figures 13 and 14. Code and data is made available for replication. – Anthony

Guest essay by Jeffery S. Patterson, DSP Design Architect, Agilent Technologies

Harmonic Decomposition of the Modern Temperature Anomaly Record

Abstract: The observed temperature anomaly since 1900 can be well modeled with a simple harmonic decomposition of the temperature record based on a fundamental period of 170.7 years. The goodness-of-fit of the resulting model significantly exceeds the expected fit to a stochastic AR sequence matching the general characteristic of the modern temperature record.

Data

I’ve used the monthly Hadcrut3 temperature anomaly data available from http://woodfortrees.org/data/hadcrut3vgl/every as plotted in Figure 1.

clip_image002

Figure 1 – Hadcrut3 Temperature Record 1850-Present

To remove seasonal variations while avoiding spectral smearing and aliasing effects, the data was box-car averaged over a 12-month period and decimated by 12 to obtain the average annual temperature plotted in Figure 2.

clip_image004

Figure 2 – Monthly data decimated to yearly average

A Power Spectral Density (PSD) plot of the decimated data reveals harmonically related spectral peaks.

clip_image006

Figure 3 – PSD of annual temperature anomaly in dB

To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique is called record periodization. The data is regressed about a line connecting the record endpoints, dropping the last point in the resulting residual. This process eliminates the endpoint discontinuity while preserving the position of the spectral peaks (although it does extenuate the amplitudes at higher frequencies and modifies the phase of the spectral components). The PSD of the residual is plotted in Figure 4.

clip_image008

Figure 4 – PSD of the periodized record

Since the spectral peaking is still present we conclude these are not record-length artifacts. The peaks are harmonically related, with odd harmonics dominating until the eighth. Since spectral resolution increases with frequency, we use the eighth harmonic of the periodized PSD to estimate the fundamental. The following Mathematica (Mma) code finds the 5th peak (8th harmonic) and estimates the fundamental.

wpkY1=Abs[ArgMax[{psdY,w>.25},w]]/8
0.036811

The units are radian frequency across the Nyquist band, mapped to ±p (the plots are zoomed to 0 < w < 1 to show the area of interest). To convert to years, invert wpkY1 and multiply by 2p, which yields a fundamental period of 170.7 years.

From inspection of the PSD we form the harmonic model (note all of the radian frequencies are harmonically related to the fundamental):

(*Define the 5th order harmonic model used in curve fit*)

model=AY1*Sin[wpkY1 t+phiY1]+AY2*Sin[2*wpkY1* t+phiY2]+AY3*
Sin[3*wpkY1* t+phiY3]+AY4*Sin[4*wpkY1* t+phiY4]+AY5*
Sin[5*wpkY1* t+phiY5]];
vars= {AY1,phiY1,AY2,phiY2,AY3,phiY3,AY4,phiY4,AY5,phiY5 }

and fit the model to the original (unperiodized) data to find the unknown amplitudes, AYx, and phases, phiYx.

fitParms1=FindFit[yearly,model,vars,t]
fit1=Table[model/.fitParms1,{t,0,112}];
residualY1= yearly- fit1;{AY1→-0.328464,phiY1→1.44861,AY2→-
0.194251,phiY2→3.03246,AY3→0.132514,phiY3→2.26587,AY4→0.0624932,
phiY4→-3.42662,AY5→-0.0116186,phiY5→-1.36245,AY8→0.0563983,phiY8→
1.97142,wpkY1→0.036811}

The fit is shown in Figure 5 and the residual error in Figure 6.

clip_image010

Figure 5 – Harmonic model fit to annual data

clip_image012 clip_image014

Figure 6 – Residual Error Figure 7 – PSD of the residual error

The residual is nearly white, as evidenced by Figure 7, justifying use of the Hodric-Prescott filter on the decimated data. This filter is designed to separate cyclical, non-stationary components from data. Figure 8 shows an excellent fit with a smoothing factor of 15.

clip_image016

Figure 8 – Model vs. HP Filtered data (smoothing factor=3)

Stochastic Analysis

The objection that this is simple curve fitting can be rightly raised. After all, harmonic decomposition is a highly constrained form of Fourier analysis, which is itself a curve fitting exercise that yields the harmonic coefficients (where the fundamental is the sample rate) which recreate the sequence exactly in the sample domain. That does not mean however, that any periodicity found by Fourier analysis (or by implication, harmonic decomposition) are not present in the record. Nor, as will be shown below, is it true that harmonic decomposition on an arbitrary sequence would be expected to yield the goodness-of-fit achieved here.

The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period, although others have found significance in the 57-year (here 56.9 year) third harmonic. We can however, estimate the probability that the results are a statistical fluke.

To do so, we use the data record to estimate an AR process.

procY=ARProcess[{a1,a2,a3,a4,a5},v];
procParamsY = FindProcessParameters[yearlyTD["States"],procY]
estProcY= procY /. procParamsY
WeakStationarity[estProcY]
{a1→0.713,a2→0.0647,a3→0.0629,a4→0.181,a5→0.0845,v→0.0124391}

As can be seen in Figure 9 below, the process estimate yields a reasonable match to observed power spectral density and covariance function.

clip_image018 clip_image020

Figure 9 – PSD of estimated AR process (red) vs. data Figure 9b – Correlation function (model in blue)

clip_image022 clip_image024

Figure 10 – 500 trial spaghetti plot Figure 10b – Three paths chosen at random

As shown in 10b, the AR process produces sequences which in general character match the temperature record. Next we perform a fifth-order harmonic decomposition on all 500 paths, taking the variance of the residual as a goodness-of-fit metric. Of the 500 trials, harmonic decomposition failed to converge 74 times, meaning that no periodicity could be found which reduced the variance of the residual (this alone disproves the hypothesis that any arbitrary AR sequences can be decomposed). To these failed trials we assigned the variance of the original sequence. The scattergram of results are plotted in Figure 11 along with a dashed line representing the variance of the model residual found above.

clip_image026

Figure 11 – Variance of residual; fifth order HC (Harmonic Coefficients), residual 5HC on climate record shown in red

We see that the fifth-order fit to the actual climate record produces an unusually good result. Of the 500 trials, 99.4% resulted in residual variance exceeding that achieved on the actual temperature data. Only 1.8% of the trials came within 10% and 5.2% within 20%. We can estimate the probability of achieving this result by chance by examining the cumulative distribution of the results plotted in Figure 12.

clip_image028

Figure 12 – CDF (Cumulative Distribution Function) of trial variances

The CDF estimates the probability of achieving these results by chance at ~8.1%.

Forecast

Even if we accept the premise of statistical significance, without knowledge of the underlying mechanism producing the periodicity, forecasting becomes a suspect endeavor. If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.

Nevertheless, having come thus far, who could resist a peek into the future?

We assume the periodicity is externally forced and the climate response remains constant. We are interested in modeling the remaining variance so we fit a stochastic model to the residual. Empirically, we found that again, a 5th order AR (autoregressive) process matches the residual well.

tDataY=TemporalData[residualY1-Mean[residualY1],Automatic];
yearTD=TemporalData[residualY1,{ DateRange[{1900},{2012},"Year"]}]
procY=ARProcess[{a1,a2,a3,a4,a5},v];
procParamsY = FindProcessParameters[yearTD["States"],procY]
estProcY= procY /. procParamsY
WeakStationarity[estProcY]

clip_image030 clip_image032

A 100-path, 100-year run combining the paths of the AR model with the harmonic model derived above is shown in Figure 13.

clip_image034

Figure 13 – Projected global mean temperature anomaly (centered 1950-1965 mean)

clip_image036

Figure 14 – Survivability at 10 (Purple), 25 (Orange), 50 (Red), 75 (Blue) and 100 (Green) years

The survivability plots predict no significant probability of a positive anomaly exceeding .5°C between 2023 and 2113.

Discussion

With a roughly one-in-twelve chance that the model obtained above is the manifestation of a statistical fluke, these results are not definitive. They do however show that a reasonable hypothesis for the observed record can be established independent of any significant contribution from greenhouse gases or other anthropogenic effects.

About these ads

313 thoughts on “Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead

  1. After figure 3:

    To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.

    Note the word [that] — it appears to be missing…

  2. Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.

  3. Unfortunately Hadcrut etc not reliable and even this would be showing cooling haha. BTW BBC is shouting NH ice lowest volume this PAST winter on its running red tape news please someone advise them.

  4. Not being a statistician I understand about one term in ten but I think I get the gist. It all looks very impressive but appears to boil down to more curve fitting. At least the author admits to fundamental reasons why the forecast made could be entirely wrong.

    Could anyone supply a summary of the the process used in plain English?

  5. Evaluating the math and the assumptions are way beyond me, but I would be more likely to accept the likelihood of this exercise being reasonable if a proven example of the method was demonstrated — something in signal processing which is well-understood both theoretically and experimentally.

  6. As an electrical engineer, I’ve been wondering about using this approach for sometime. Mr. Patterson has done all the right things, so I don’t doubt the numbers per se. We don’t know is if the forecast portion is valid, but now that we have a model we can at least determine that going forward. What is interesting is the relative simplicity of the final result when compared with the climate models.

  7. Jonathan Abbott says: Could anyone supply a summary of the the process used in plain English?
    ……………………
    It is.

  8. Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?

    Because they lie in the same harmonic series (in a 3/2 ratio). The other harmonic periodicity noted (56.9) was in 3:1 ratio. (That’s what I gathered anyway.)

  9. As Geofff says — it is in plain English. ;-)

    Plausible? Why not? I am doing similar work with other climate data using a similar technique. You can indeed spot these periodic form with less than the (normally required) full data stream.

  10. The maths is way over my head but if I understand the gist correctly you are saying that using standard DSP analysis techniques you can state with greater than 90% confidence that global temperature is cyclical and is heading down.

    Unlike the IPCC you at least provide the maths!

  11. Compliments Jeffrey on a thorough naive evaluation of the observed temperature data updating us on the latest techniques, with estimates of the probabilities involved.

    For comparison:
    Armstrong & Kesten Green evaluate a naive no-change model and several simple models. See Scientific Forecasting eg for Public Policy
    The near term projections are similar to a linear rise from the Little Ice Age plus a 50-60 year PDO model. See Syun Akasofu’s work provokes journal resignation
    Nicola Scafetta developed an astronomical climate model using harmonic analysis to identify natural cycles plus an anthropogenic contribution. Scafetta recently published the invited review:
    Solar and Planetary Oscillation Control on Climate Change: Hind-cast, Forecast and a Comparison with the CMIP5 GCMs

    All of these appear to give global temperature predictions with higher near term probability than the current IPCC global climate models.

    Jeffrey, It would be interesting to see who well your analysis techniques perform to hindcast/forecast from half the data to the other half.

  12. If the same method was applied to data just through 1980, would it accurately predict the warming in the 80’s and 90’s?

  13. A few comments:

    1) decimated by 12

    I know that “decimate” came from the Roman practice of inspiring soldiers after battles where the commanders felt were uninspired. They lined up the soldiers and killed every tenth man, leaving the remaining 90% inspired to win the next battle. At first blush, this sound like torturing the data until it confesses or perhaps this is “dodecimated” – killing every 12th measurement. :-)

    2) The obvious counter is that the relatively recent surge in CO2 levels and (possible) concomitant heating may be messing up any nice periodicities.

    3) Would these techniques return similar results to the 2,450 year record described in http://wattsupwiththat.com/2011/12/07/in-china-there-are-no-hockey-sticks/ ?

    And a free, bonus rant:

    Ever since hurricane Katerina (and weak defenses) devastated New Orleans, new reporters have been saying that and other disasters have “decimated” the area. Bugs the heck out of me. I keep retorting that the reporter is a stupid idiot, but they keep on talking as though I’m not in the same room as said idiot.

  14. A lot of work and for that many thanks. BUT, there is always a but with temperature, the data set used is altered to reflect current theories that temperatures must increase with rising CO2 levels. Raw temperature data is notoriously variable even if all collections are from correctly sited and using the same calibrated instruments there will be temperature differences even a few yards apart. So what is the real temperature of the earth’s surface? we really do not know because temperature can only be accurate if the item being measured is at temperature equilibrium which the earth never is.

  15. WillR says:
    September 11, 2013 at 5:11 am
    After figure 3:

    To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.

    Note the word [that] — it appears to be missing…
    ———————————————-

    Don’t add words, delete the word “is”.
    cn

  16. ‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.

  17. Eliza, what the BBC are doing (http://www.bbc.co.uk/news/science-environment-23964372) is to shift to ‘winter’ ice VOLUME. They don’t mention that it has recovered (http://psc.apl.washington.edu/wordpress/wp-content/uploads/schweiger/ice_volume/BPIOMASIceVolumeAnomalyCurrentV2_CY.png). But then this is the BBC now. As Jeremy Paxman said, they gave up the pretence of being impartial some time ago. This is what warmist organisations and individuals are doing now. As we find things not happening as planned, focus will be shifted to areas that somehow impart the feeling that we’re still doomed. It’s only when the global temperature actually starts to properly go into freefall that we will be able to prosecute warmists for their absurdity and ignorance. Just be patient (although none of us welcome cooler world) the Earth will show up the nonsense of AGW without sceptics having to do or say anything.

  18. Nick Stokes says:
    September 11, 2013 at 5:11 am

    > How can you establish a 170 year periodicity from 113 years of data?

    From the post:

    The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period, although others have found significance in the 57-year (here 56.9 year) third harmonic.

    So he found a 56.9 year cycle and extended it looking for the fundamental.

    Here’s a mental exercise:

    Draw a one cycle sine wave, marvel that it’s continuously differentiable and the results are more sinusoids. Get off that tangent.

    Trim the ends so that only 113/170 of the cycle remains. Observe that “Hey, it looks like a clipped sine wave.” Apply appropriate techniques to determine the phase, amplitude, and period.

    Back in my EE days, we barely had decent enough computers to do DSP, except in the CompSci dept, and I was focused on operating system and ARPAnet code. So I don’t know beans about DSP. Barring bothering to learn anything about it, it would be easy enough to take the digital data for that 113/170 th of a sine wave and convolve with with various test sines looking for the best match. So I’m sure there are mathematical techniques beyong Fourier transforms that can handle those fractional sines.

  19. Menahwile in the broadsheet papers here in the UK, notably The Independent and The Guardian which are read by liberal / left opinion formers (inc teachers at all levels), the comments to this article below demonstrate that none of their readers is taking any notice of the long pause in warming and the overall cyclical record.

    Will someone with the time and expertise please add a comment or two to explain what is really going on?

    http://www.independent.co.uk/news/science/woolly-mammoths-became-extinct-because-of-climate-change–not-hunting-8808684.html

  20. Hadcrut 3 is not a balanced sample of weather stations. It may give you some indication of periodicity but with bias.
    Just take a random sample, of weather stations, balanced by latitude and 70/30 @sea/inland,

    http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/

    pay particular attention to maxima, and plot:

    http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/

    I don’t understand why people want to do so difficult.

  21. Comment 1: While data massaging here is extreme, it does demonstrate the futility in ordinary least squares statistics so adored by AGWing scientists.

    Comment 2: We know from the outset that the signal is representative of a complicated internal system (regardless of external possibilities) in its own right, with teleconnected yet varying unbalanced short and long term internal oscillations (ie decidedly non sine-waveish and very discordant). Therefore the assumption should always be that the variations are entirely internal, quite resistant to demonstrating affects of small external stimulus, and nearly impossible to predict going forward.

  22. To Geoff Sherrington:
    It’s in plain English if you already understand the technical terminology.
    *resigns self to spending a few hours reading up on signal processing techniques*

    To Ian E:
    You have it backwards. This is observing the wriggling trunk and trying to work out what the elephant looks like.

    To all WUWTers:
    I have seen doubts expressed in some places on the applicability of techniques developed for electronic signal processing when applied to the climate. Straw man? Valid?

  23. @Bruckner8
    >Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?

    Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

    The sequence of odd harmonics in the processed waves suggests that windowing artifacts remain, as the convolution of sine waves (which are always infinite in extend from an abstract mathematical viewpoint) multiplied by a sampling window (“square wave”) would suggest.http://en.wikipedia.org/wiki/Convolution_theorem

    In AR modeling, the predictive modeling results depend on the time series being stationary, i.e. constant mean and variance. But historical temperature records are not stationary, and in fact it is the non-stationary part of the record that you are trying to model. So results of predicting values in the future based on values in the past will generally not be credible.

    http://en.wikipedia.org/wiki/Wold's_theorem

    :-|

  24. Nick Stokes says: How can you establish a 170 year periodicity from 113 years of data?

    In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data. if the condition is not met, the high frequency components masquerade as low frequency components in a process referred to as aliasing. That is why we remove the high frequency components by filtering before decimating (re-sampling) the data.

  25. Ric, don’t ya love “back in my day” times? Want to do an Anova, or Coanova? Take an MSDOS computer with a blank screen and program it (I programmed mine to say good morning to me every time I turned it on). Or grab a calculator, the clunky dark cream colored kind with sharp square corners, and do it that way. Hell, just use a pencil and paper. I remember getting all twitterpated when I got a little Mac SE with Statview installed on it. I thought it was Christmas every day.

  26. Friends:

    I second the request of Mike Lewis at September 11, 2013 at 5:41 am; viz.

    If the same method was applied to data just through 1980, would it accurately predict the warming in the 80′s and 90′s?

    I note the strong caveat in the above essay concerning the possible inappropriateness of extrapolation of the obtained curve. However, if it is assumed that the curve fit does capture the form of the temperature changes, then failure of the method to predict the temperature curve after 1980 would imply the temperature changes are internal variability of the climate system and are not externally forced.

    Richard

  27. IanE says:
    September 11, 2013 at 5:52 am
    ‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.

    As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.

  28. A good intellectual exercise and perhaps more valid than the 73 worldwide GCM’s. If by 2040 the forecast pans out, we can save a lot of tax money. No need for an understanding of climate physics or chemistry. Just have a small group of DSP mathematics folks hidden away in a skunk works.
    I am joking ,of course. Nevertheless I yearn for a creditable effort that would create severe competition for the circular thinking climate science community to reorient their thinking towards a comprehensive understanding of the natural forces in climate.

  29. @Me

    Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

    As soon as I hit the “Post Comment” button I realized that my reponse was wrong because the 170-year wave in question is indeed “over-sampled”, not under-sampled.

    But the reason I was so sure this couldn’t be done is that I’ve tried to reconstruct these kinds of “over-sampled partial waves” myself and have never been successful at it. But perhaps I going at it in the wrong way.
    :-|

  30. “extracting spurious signals from noisy data”

    Hopefully this means “extracting meaningful signals” or “removing spurious signals”…

  31. John Day says:
    September 11, 2013 at 6:08 am

    @Bruckner8
    >Nick Stokes said How can you establish a 170 year periodicity from 113 years of data?

    Short answer: you can’t. You need at least two samples per period from a sinewave, in order to reconstruct it from a sampled set of measurements. (In the same sense that you need two sample from a line to reconstruct it) http://en.wikipedia.org/wiki/Shannon%E2%80%93Hartley_theorem

    This all applies to periodic sampling of a periodic signal. That’s not what we have here, we have many samples of part of a single cycle.

    Had this been similar to Nyquist sampling, there would be either one or two samples of the entire 113 years. Instead, the sampling rate is far higher than that and there are hundreds of samples to work with.

  32. I’ll parse this in more detail later but initial reactions:

    1. HadCruft is a land + ocean amalgam. Rate of change of land is twice that of SST, how will this affect freq decomposition.
    2. “To remove seasonal variations while avoiding spectral smearing and aliasing effects, the data was box-car averaged over a 12-month period and decimated by 12 ” Running mean filter _introduce_ spurious signals. http://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/

    3. A 12m filter it NOT an appropriate anti-aliasing filter for 12m decimation
    4. HadCruftX is probably aliased already even in the monthly data series further filtering will not fix it once it’s done: http://judithcurry.com/2011/10/18/does-the-aliasing-beast-feed-the-uncertainty-monster/
    5. Series of add harmonics suggests a triangular waveform likely due to non stationary data. The first and last point trick is new to me and I can’t see this being any guarantee of the conditions needed for FFT. Linear detrending of whole dataset or taking first diff would be more usual.

    It’s an interesting idea and I don’t want to totally dismiss the result without a closer look, but this seems like a fair list of processing errors so far.

  33. Thank you, Mr. Patterson for your fine work.
    Fortunately, you’ll need spend no effort trying to construct a compensatory microcode.

    Bit of a busman’s holiday, is that it?

  34. If the above statistical analysis is valid ( as it appears)… why not submit it for publication and peer review. Surely the gatekeeping practices, of climate research, have somewhat abated since the climategate revelations. Or is this, just a pre-review before submission. GK

  35. Here’s a hopefully approachable summary:

    A time record can be represented in the time domain (like the Hadcrut plot) or the frequency domain (like the power spectral density plots in the post). The frequency domain looks like an old fashioned radio dial, with the high frequencies to the right and the low frequencies to the left. A radio station is assigned a frequency, unique from all others in the area, so that its signal only appears one place on the dial and is not interfered with by other stations. If we were do do a spectral analysis (i.e. a PSD plot) of the AM band, we would see a peak at each frequency where a station is broadcasting and a valley where no station was present.

    We can transform from one domain to the other without distortion via the Fourier Transform. It uses every possible frequency which are all harmonically related to the frequency at which the data was sampled (i.e. monthly in the case of Hadcrut3) to exactly represent the data in the frequency domain. To reconstruct the 113 samples in my decimated record exactly would require 226 variables (113 frequencies and their phases). We show that we excellent results using just 5 harmonically related frequencies where instead of using the lowest frequency of the FFT, we use the frequency with the highest peak as the fundamental. Actually, the 5th harmonic doesn’t add much and most of the fit is captured with just 3 harmonics.

    The great thing about sine waves is that they repeat which means if they are really present, the model can be used to predict the future trend. The bad thing is that it is easy to get fooled if the data is noisy. The second half of the paper examines the likelihood that we were fooled. By generating some random sequences from a process designed to produce data that in character (peak-to-peak changes, intermediate slopes etc.) matches the temperature record, we can perform the same harmonic decomposition we did to the temperature data and see how many times the fit is as good as we achieved with the actual data. In this case, only .6% of the 500 trials were as good, which places the probability that we were fooled at about 8%

  36. Hmmm, based on the data, we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome. Oh boy, who would have guessed?

  37. Lets assume that the periodicities are fixed as opposed to variable. Given the 113 years of data, the longest periodicity extracted was 170 years. Lets us also assume that the shorter periodicities have been correctly extracted. So now, instead of projecting forward to take a stab at the future, how about projecting backward, past the LIA, MWP, RWP. Then with an estimate as to the magnitude and time of these events without concern on the decade scale wiggles, guess what the longer periodicities are? Sure this is torturing data …

  38. Greg Goodman says:
    September 11, 2013 at 6:31 am
    … Running mean filter _introduce_ spurious signals.

    Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

  39. That fundamental of 170.7 years speaks strongly of the Jose cycle. If Patterson knew about it (I suspect he did) there is some danger of confirmation bias. I have the same bias. Fortunately the projected cooling is modest. I hope it stays that way. As for the usual comment that this is a curve-fitting exercise, in a world of junk “physics-based” math models, faulty theory, manipulated data and agenda-driven science, it’s really all we have.

  40. The 170 year cycle length is suspiciously close to the approximate cycle length of the solar system barycenter. the 170 year cycle continues to show up in the works of many different authors. as does the 55-60 year harmonic of 170, which closely approximates the observed length of the PDO.

    Based on what we know about tidal prediction, DSP is likely to be at least as accurate as the global climate models in predicting future climate, and quite likely a whole lot more accurate.

    if one tries to model the ocean tides using first principles, as is done with the climate models, the results are garbage. earth’s tides would be miniscule based on gravitational attraction alone.

    the earth’s tides are predicted to a high degree of accuracy by the motion of the celestial bodies. this is because the earth’s oceans have over thousands and millions of years, come to oscillate in phase with the motion of the celestial bodies.

    it is completely likely that s that climate oscillates similar to the tides, only on a longer timescale, as a result of a small but regular harmonic forcings based on the motion of the celestial bodies over thousands and millions of years.

    this suggests that we can predict climate with some hope of accuracy using methods similar to what the author has proposed here today.

  41. This is similar in some ways to what I did 18 ago on Judith Curry’s site where I fitted just three , non harmonic frequencies to examine how Met Office processing had changed the data. (NB this was not intended as a prediction tool, just an evaluation of changes to the existing data).

    : http://judithcurry.com/2012/03/15/on-the-adjustments-to-the-hadsst3-data-set-2/

    see middle panels in figures 9 and 10.

    It is interesting that I found hadSST (sea temp only) had _removed_ a 160 year periodicity. Maybe the combined use of Had + CRUT here, puts it back ;?

  42. The relevant line from a very long article: “Even if we accept the premise of statistical significance, without knowledge of the underlying mechanism producing the periodicity, forecasting becomes a suspect endeavor. ”

    Let’s not duplicate the errors of our adversaries and assume we know more than we know.

  43. RC Saumarez says: This appears to be an example of the well known fallacy of Fourier Extrapolation.

    HArmonic decomposition is much more constrained than fourier analysis but subject to some of same concerns about seeing periodicity that isn’t there. The last half of the paper address this issue directly.

  44. JP “Exactly why I used box care (i.e. non overlapping) filtering, not running mean.”

    Could you explain what you mean by “box car” in more engineering/mathematical language , maybe I misunderstood what you mean by railway analogy. ;)

  45. Jeffrey Patterson:

    Have you considered Singular Spectrum Analysis (SSA) to extract the dominant modes of the temperature graph? This does not suffer from the windowing effects that a DFT suffers from, nor is it limited to a prescribed set of basis functions.

  46. pochas says:
    September 11, 2013 at 6:53 am
    That fundamental of 170.7 years speaks strongly of the Jose cycle.If Patterson knew about it …

    I do not. I did a cursory google search for “170 year climate cycle” but couldn’t find anything relevant, I would appreciate a reference if you have one.

  47. @Greg Goodman – Box car decimation is simply dividing the recorded into N equal length segments and taking the average of each segment as the new sample (thus decimating by N). Unlike MA averaging, it introduces no correlation between samples.

  48. I agree with Greg Goodman.

    There are some rather astonishing propositions here.

    The first seems to be a fundamental misubderstatnding of the relationship between the Fourier Transform and the Discrete (complex exponential) discrete Fourier series i.e. the FFT. In the former case there is an infinite resolution in frequency but requires the signal to be observed from – to + infinity. In the case of the FFT, if the fundamental frequency is determined by the length of the record. One might try to infer the presence of lower frequencies, but this lacks any formal justification.

    The anti-aliasing processing does not seem to be correct and is using data that may be aliased.

    The processes involved are almost certainly non-stationary.

    This approach says that because the earlier record looks like a sine wave, we assume that is what will happen in the future. If you perturb a low-pass system, such as an ARMA system, you will get trends whose magnitudes depend on the persistance of the process. I am very suspicious of the autocorrelation function presented here – I suspect that it is artefactually widened due to prior filtering – an elementary mistake and that persistance is overestimated:

    http://judithcurry.com/2012/02/19/autocorrelation-and-trends/

  49. @Jeff Patterson

    September 11, 2013 at 6:49 am
    Greg Goodman says:
    September 11, 2013 at 6:31 am
    … Running mean filter _introduce_ spurious signals.

    Exactly why I used box care (i.e. non overlapping) filtering, not running mean.

    So the artifacts will not overlap, but they’re still there. The only window filter that I know of that has no artifacts is a Gaussian window. It produces no sidebands.

    Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.
    :-|

  50. Suggested improvements :
    1. triple RM filter at 24m before 12m decimation would be correct anti-alias.
    2. why decimate anyway? data is not cumbersome and any aliasing that in the monthly data will not go away by further filtering.
    3. Use land or sea (or both separately) not amalgam dataset.
    4. work on rate of change dT/dt (from first difference of data) since what we want to know is temp change anyway. This will remove the slope (and likely much of the odd harmonics) whilst preserving any true 170y or other cycles.

  51. @Greg Goodman – I’ve addressed your 1st concern elsewhere. The others are concerns about aliasing which is of course always a concern. I disagree with your statement regarding box care filtering – the zeros of the filters transfer function lie exactly on the aliasing spectral lines. It is a very effective re-sampling filter. Of course it can’t do anything for the aliases already present in the data, but as you can see from the PSD plots (e.g. Figure 9), the high frequencies are rolled of by some 40dB so any aliasing will be insignificant.

  52. richardscourtney says:
    September 11, 2013 at 6:14 am

    Friends:

    I second the request of Mike Lewis at September 11, 2013 at 5:41 am; viz.

    If the same method was applied to data just through 1980, would it accurately predict the warming in the 80′s and 90′s?

    I note the strong caveat in the above essay concerning the possible inappropriateness of extrapolation of the obtained curve. However, if it is assumed that the curve fit does capture the form of the temperature changes, then failure of the method to predict the temperature curve after 1980 would imply the temperature changes are internal variability of the climate system and are not externally forced.

    Richard

    Yes, but step 1 should be to determine how accurately those temperature changes have been recorded. We know that there has been a warm bias, and that needs to be subtracted.

  53. the sun’s motion around the solar system barycenter roughly repeats every 171.4 years as measured by its angular momentum. this is remarkably close to the author’s 170.7 year cycle length in the Hadcrut3 temperature data.

    if one sets aside the argument over the mechanism by which climate oscillates with a period of 170 years, and simply accept that there is evidence for such an oscillation in the observed data, then one can go forward and accept that prediction is possible.

    humans learned to predict the seasons long before we understood the mechanism. why then should we assume that we cannot predict climate without understanding the mechanism?

    history shows that almost always the prediction comes BEFORE the understanding. when the prediction is shown to be correct by observation, this suggest the mechanism. and from there we investigate to discover the mechanism that governs the prediction.

    for example, the motion of the planets in the heavens. we learned to predict this with amazing accuracy long before we discovered that gravity was the mechanism. modern science would tell us that such a thing was impossible, you cannot predict without a mechanism.

    the facts are that humans invent mechanisms to explain predictions, and more often than not these mechanism are later shown to be false. this in no way affects the accuracy of the prediction. right answer, wrong reason is still a right answer.

  54. @Greg Goodman says work on rate of change dT/dt (from first difference of data) since what we want to know is temp change anyway. This will remove the slope (and likely much of the odd harmonics) whilst preserving any true 170y or other cycles.

    First difference processing suppresses low frequencies and greatly extenuates high frequencies (it’s transfer function match a continuous-time differentiator across most of the Nyquist band). It also introduces frequency dependent phase shifts which would adversely effect the reconstruction.

  55. John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”

    The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
    The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.

    No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.

    Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. :( )

  56. You miss the point about Fourier Extrapolation.

    Figure 13 is simply a re-run of the 20th century – i.e.: a periodicity. There is no theoretical justification for this approach – it simply says that the future will be a repeat of the past. What you have done here is to apply some extremely artificial constraints on the system by assuming that:

    a) You have characterised the system correctly
    b) The inputs to the system are stationary.

  57. RC Saumarez says:
    The first seems to be a fundamental misubderstatnding of the relationship between the Fourier Transform and the Discrete (complex exponential) discrete Fourier series i.e. the FFT. In the former case there is an infinite resolution in frequency but requires the signal to be observed from – to + infinity. In the case of the FFT, if the fundamental frequency is determined by the length of the record. One might try to infer the presence of lower frequencies, but this lacks any formal justification.

    I don’t think I misubderstatnd :) anything. I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).

    I addressed your record-length concern in the paper (see the section on record periodization).

  58. JP “First difference processing suppresses low frequencies and greatly extenuates high frequencies (it’s transfer function match a continuous-time differentiator across most of the Nyquist band). It also introduces frequency dependent phase shifts which would adversely effect the reconstruction.”

    It’s true that differential attenuates as 1/f but this does not remove the 170 as I found in my Curry article linked above, from which this comes. (I found 164 years)

    Working on monthly resolution would make the phase shift minimal but you could use a three point diff to retain a symmetrical kernel and avoid the phase shift problem. (-0.5, 0, +0.5)

    A 3-sig gaussian of sigma=12m should remove most of the annual and could be combined with the above into a gauss-diff kernel to do it all in one operation with better approximation of the true (non discrete) differential.

  59. RC Saumarez says:
    September 11, 2013 at 7:28 am
    You miss the point about Fourier Extrapolation.

    >>Figure 13 is simply a re-run of the 20th century – i.e.: a periodicity.
    Figure 13 contains 500 paths. They can’t all match the single path of the 20th century,

    >>There is no theoretical justification for this approach – it simply says that the future will be a repeat of the past.
    Which will be partially true in a the mean sense if the periodicity is present. As I took pains to point out though, that’s a big if and any projection depends on the major climatic parameters remaining constant over the projection period. I thought I was pretty clear on that.

    What you have done here is to apply some extremely artificial constraints on the system by assuming that:

    a) You have characterised the system correctly
    b) The inputs to the system are stationary.

    I am confused by (b). The projection assumes a periodic input process (and stationary climate process). Periodic signal + noise is decidedly a non-stationary process. Interesting though, the residual of the fit passes the Unit Root Test with a p vaule of ~ 10^-12. And the decimated record is at least weakly stationary (p = 10^-6).

  60. If you perturb a low-pass system, such as an ARMA system, you will get trends whose magnitudes depend on the persistance of the process.

    Exactly the point of the second half of the paper where I shown that the probability of achieving similar goodness of fit results to an ensemble of AR process outputs is small (but not insignificant).
    Note that I get even better results (i.e. lower probability of fluke) with an ARMA process although I can’t explain why.

  61. JP: “I’m only using Fourier analysis (indirectly via the PSD function) to guess the fundamental. Given the fundamental we assume only the presences of its harmonics and solve for the unknown amplitude and phases to a given order (in this case 5).”

    But in fitting your five harmonic model to the time series with no linear component you are just modeling the general slope with a truncated FS based on 170. RC point about repetition still stands. Plot your model out over 1000 years , it will be 170 year saw-tooth , more or less.

  62. Greg Goodman says: It’s true that differential attenuates as 1/f

    Actually it’s 1/f^2. Low frequency attentuation is extreme.

  63. “The 113 sample record examined above is not long enough to attribute statistical significance to the fundamental 170.7 year period”
    .
    And that is where the essay should have stopped. You simply can’t make any meaningful long term projection of periodic behavior based on too little data. Sorry, but this analysis is 100% unconvincing.

  64. Ken Coffman:

    Your post at September 11, 2013 at 6:47 am says in total

    Hmmm, based on the data, we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome. Oh boy, who would have guessed?

    Eh? What gives you that idea? Please explain?

    Richard

  65. @Jeff Patterson
    To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique is called record periodization. The data is regressed about a line connecting the record endpoints, dropping the last point in the resulting residual. This process eliminates the endpoint discontinuity while preserving the position of the spectral peaks (although it does extenuate the amplitudes at higher frequencies and modifies the phase of the spectral components).

    There is a lot going on behind the scenes in this paragraph that I thinks needs more explanation.
    “The data is” What data? The time series? [I think] Or the resulting PSD?

    “regressed” how?

    “about a line connecting the record endpoints”.
    Did you draw a line from Point 1 to point 113? Did you do a least squares regression line across the [time-based?] data?

    Figure 3, to 4

    “Dropping the last point”. Fourier analysis has to assume that the time based data record repeats. You must think of the time series as a chart recorder plot where you wrap the end to the beginning as a cylindrical plot. There will be a splice where Point 113 meets Point 1. Dropping the last point doesn’t remove the splice. I don’t see what good that does.

    The peaks are harmonically related, with odd harmonics dominating until the eighth. Since spectral resolution increases with frequency, we use the eighth harmonic of the periodized PSD to estimate the fundamental

    Huh? Why should we conclude the peaks [in the Fig. 4 PSD] are harmonically related? Why use the eighth to estimate the fundamental harmonic? What if we used the 5th, 3rd, of 7th? This is not obvious.

    The x-axis of Fig 3, Fig 4, Fig 7, Fig 9a, Fig 9b need some labels and units. They leave too much work for the reader.

    From inspection of the PSD we form the harmonic model

    I need more.

    fit the model to the original (unperiodized) data
    I give you credit for putting the 6 line parameter function of this step in the paper, but an illustration would have been nice.

    Fig. 10 spaghetti graph.
    What was allowed to vary between trials? What stayed the same?

  66. Bearing mind also that much of the 60y cycle was inserted by Hadley ‘corrections’. (Rightly or wrongly).

  67. Mr. Courtney, you don’t see a span that includes a cooling anomaly in the projection? I do. Watch and see, this is a theme you’ll see pop up here more and more: the recognition that that data does not exclude the fact that the overall effect of adding CO2 to the atmosphere could be cooling, not heating.

  68. Bruce Cobb:

    In response to my supporting the view (shared by several in this thread) to assess if the method applied to half the time series can predict the other half, you have replied September 11, 2013 at 7:18 am

    Yes, but step 1 should be to determine how accurately those temperature changes have been recorded. We know that there has been a warm bias, and that needs to be subtracted.

    Sorry, but no. We are discussing an ability to model the data set and not its ability to model whatever you, I or anyone else thinks the data set should be.

    However, if you are saying the data is dubious, then I agree; see

    http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc0102.htm

    Whilst I agree that point, I stress that it is not relevant to considerations in this thread.

    Richard

  69. Greg Goodman says:
    September 11, 2013 at 7:26 am
    John Day:” Then why don’t we always use a Gaussian window instead of Boxcar? Because its spectral resolution is less than a boxcar. So we generally put up with the sidebands to get that extra resolving power.”

    The transition is a little slower on gaussian but not so as it’s going to change “resolving power”.
    The main disadvantage is it requires a wider window to make a better filter (whatever one you chose). But that’s cost of not screwing up the data.

    No, It’s mostly because people that use running mean (which is commonly called boxcar because of the rectangular weighting) don’t even know what a filter is, how to specify one or what is a good or a bad one.

    Everyone understands an average, right? So a running average must be OK. ( Plus it’s nice and easy to calculate and I get frightened that I might not know how to program a gaussian , sound complicated. :( )

    Greg,
    Of course moving-average (“boxcar”) filters are very intuitive and simple to implement. But that doesn’t change the fact that they also produce spurious artifacts in their output.
    Gaussian filters are “ideal” in the sense that they do not produce these artifacts. But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.

    http://en.wikipedia.org/wiki/Gaussian_filter

    “Gaussian filters have the properties of having no overshoot to a step function input while minimizing the rise and fall time. This behavior is closely connected to the fact that the Gaussian filter has the minimum possible group delay. It is considered the ideal time domain filter,…”

  70. Jeff Patterson says:
    Greg Goodman says: It’s true that differential attenuates as 1/f
    Actually it’s 1/f^2. Low frequency attentuation is extreme.
    ===

    In the power spectrum maybe but in FT or your fitted model 1/f . d/dt(cos(wt)) ….

  71. I am very surprised to see this.

    You are imposing highly artificial constraints on the system. Your projected results (fig 13) are simply a periodic rerun of the 20th century. What you have done, in essense it to assume an impulse response based on auto-correlation, and perturbed this with a periodic input to get a result.

    The point is that you cannot gain a statistical characterisation of your model because the persistance of the climate system is too long, compared to the record length, to allow segmentation of the record and improve estimated accuracy. It is inevitable that random inputs into a system with persistence will produce trends. The estimation of that persistence is very difficult and distinction between competing physical models almost impossible with current data. Nevertheless, trends that are comparable to that seen in the modern record are just possible with a randomly excited Hurst system. See a tutorial post:

    http://judithcurry.com/2012/02/19/autocorrelation-and-trends/

    The model assumes a periodic defined input and noise with some statistical properties. This is a stationary input, in the commonly accepted sense, because it has constant parameters with time. This is a huge assumption.

    I would comment that this is the blind application of signal processing without giving sufficient thought to underlying physical assumptions: I wrote here, in a tutorial post

    http://wattsupwiththat.com/2013/04/09/correlation-filtering-systems-and-degrees-of-freedom/

    “Modern scripting programs such as “R” allow one to perform many signal processing calculations very easily. The use of these programs lies in not applying them blindly to data but in deciding how to use them appropriately. Speaking from bitter experience, it is very easy to make mistakes in signal processing and it is difficult to recognise them. These mistakes fall into three categories, programming errors, procedural errors in handling the signals and not understanding the theory as well as one should. While modern scripting languages are robust, and may largely eliminate straight programming errors, they most certainly do not protect one from making the others!”

  72. John Day says:
    In AR modeling, the predictive modeling results depend on the time series being stationary, i.e. constant mean and variance. But historical temperature records are not stationary, and in fact it is the non-stationary part of the record that you are trying to model. So results of predicting values in the future based on values in the past will generally not be credible.

    The residual _is_ stationary. That’s the point. We can certainly drive an AR process with stationary (and in this case also nearly Gaussian) noise + sine waves and get a valid projection. This of course, as I pointed out, depends on the assumptions the the observed periodicity is a forcing function (i.e.. not internal) and the AR process itself is time invariant. As I mentioned, this might not be the case and so the projection should be taken with the appropriate grain of salt.

    The main point (beside the surprisingly good fit that can be achieved with such few harmonically related elements) was not the projection but rather that there exists plausible explanations for the observed data which do not require significant forcing from anthropogenic sources.

  73. I wonder about the futility in taking apart the frequency response of all the combined signals from all the sensors. It reminds me of signal frequency response testing (IE hearing aids, brainwaves, musical instruments, etc). Isolating instruments producing a combined sound by filtering out the known frequency response of each note of the various instruments is possible. But in temperature signals, the instruments producing the combined frequency response are many more and they are not playing from the same sheet of music. Each instrument (each sensor) produces a series of sounds in its little music room which has separate acoustics to the one down the road that is producing its own sounds in its little music room which has separate acoustics to the next one down the road, and so on. Logically, at best you could filter out all the separate acoustics back to a single one and say something about that instrument’s data and its particular music room and its particular sheet of music. Putting them all together and saying something about the resulting frequency response is probably not very useful and would lead to a high degree of spurious conclusions.

  74. So I’d like to see this analysis applied up to 1990 to observe how its predictions from 1990 fared compared to reality.

  75. joshv says:
    September 11, 2013 at 5:12 am
    “Though I am extremely skeptical of model projections in general, I see no reason to believe the projections of this model any less than the projections of GCMs.”

    Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out. As I have said before, one can obtain any result one desires with enough data manipulation and statistical analysis.

  76. kencoffman:

    Thankyou for your answer to me at September 11, 2013 at 7:58 am in reply to my request for clarification.

    I understand your reply to say you have invented from whole cloth your notion that

    we must embrace the possibility that increasing the concentration of atmospheric CO2 includes cooling as a possible outcome.

    I fail to see how your invention has any relevance to this thread whether or not you “can see a span that includes a cooling anomaly in the projection”.

    Richard

  77. JD: “But, for the same length filter, produce thicker spectral output lines, thus have less frequency resolving power.”

    If you add the proviso “for the same length filter” you are totally correct. I work from the spec I need so I compare two filters with similar FWHM and conclude I need a longer window to do the same job without unacceptable defects ( be it gaussian or triple-RM).

    I actually favour 3RM for this kind of stuff since it has a zero in frequency. It’s marginally better than gauss for removing annual, for example.

    http://climategrog.wordpress.com/2013/05/19/triple-running-mean-filters/

  78. Pamela Gray says:
    September 11, 2013 at 8:09 am
    “Putting them all together and saying something about the resulting frequency response is probably not very useful and would lead to a high degree of spurious conclusions.”

    As one who plays guitar regularly I would say such methods also destroy the melody which is what one is looking for in this analogy.

  79. Jeff Patterson on September 11, 2013 at 6:46 am

    Here’s a hopefully approachable summary:

    – – – – – – – –

    Jeff Patterson,

    Thanks for your effort.

    I particularly appreciate your care in describing the limitations and shortcomings of your methods.

    John

  80. Folks, we have been schooled on digital signal processing here. Nicely done Jeff.

    Anyone interested in the topic should check out The Scientist and Engineer’s Guide to Digital Signal Processing. The book is a free download and is written for people who might actually want do practical signal processing. There are pitfalls that await anyone who naively attempts to use dsp techniques in real life (as opposed to a classroom exercise) and many undergrad textbooks don’t mention them.

  81. Is this anything to do with Mr.T.Pratchett or the unseen university? There is certainly a whiff of Hex in the methodology and written conclusions.

  82. Chuck Nolan says:
    September 11, 2013 at 5:46 am

    WillR says:
    September 11, 2013 at 5:11 am
    After figure 3:

    “To eliminate the possibility that these are FFT (Fast Fourier Transform) artifacts while avoiding the spectral leakage associated with data windowing, we use a technique [that] is called record periodization.

    Note the word [that] — it appears to be missing…
    ———————————————-

    Don’t add words, delete the word “is”.”
    cn

    Grammatically it is okay to eliminate the first “that”, too. Also, the “the” before ‘spectral’. There, Chuck Nolan, WillR and I have made a statement on our full understanding of the mathematical process!

  83. The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen. Would an analysis of the opening of the 1812 Overture predict the finale? About the only thing I am prepared to say with any certainty is that the next 1000 years is more likely to be cooler than the last 1000 years than it is likely to be warmer.

  84. Thanks Jeff for a clear and concise post. I have more than a smattering of exposure to digital signal processing and Fourier analysis(although not enough to do it myself!) so I could understand what you were trying to do. You carefully noted and explained the limitations(non-linear externally forced vs. internally generated) and showed that statistically the result is unlikely to be a random error.

    Now I’d like to see some publications examing the hypthesis that the climate is an exteranally forced linear process, or showing that it is an interanlly forced randlom process.

  85. JP: “The main point (beside the surprisingly good fit that can be achieved with such few harmonically related elements) was not the projection but rather that there exists plausible explanations for the observed data which do not require significant forcing from anthropogenic sources.”

    Sorry, maybe I missed something in the article. What plausible explanation did this present?

  86. Crosspatch, actually that is likely. There is a musicality test, standardized no less, that in its simplest form, has the examinee predict what note comes next after a short series of notes. Because the muical scale is mathematical and orderly one can if one has an ear for the “language” predict what comes next.

  87. sorry, “hypothesis”. My right ring fingertip is numb from a cut, hard to feel where the upper rlow lf keys is.

  88. ‘Jeff Patterson says: September 11, 2013 at 6:17 am

    IanE says:
    September 11, 2013 at 5:52 am
    ‘With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.’ John von Neumann.

    As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.’

    But of course he could choose any sort of function – he chose something that worked, he did not have to use the approach he ended up with. That is the fundamental problem with data-mining exercises.

  89. crosspatch says:
    September 11, 2013 at 8:26 am

    Right you are. The long-term temperature trend for the second half (past 5700 years) or third third (3800 years) of the Holocene Interglacial is down. Simply extrapolating that trend gets us into the next glacial sooner rather than later. The trend could reverse, but hasn’t done so in any prior interglacial, once established.

    Too bad that man-made GHGs can’t do that trick.

  90. richardscourtney says:
    That occurred to me afterwards. The entire exercise is pointless, then. Proper science demands accuracy from the start. If you are starting with data that are flawed, the result is going to be flawed as well.

  91. Pamela Gray says:
    September 11, 2013 at 8:33 am
    “Crosspatch, actually that is likely. There is a musicality test, standardized no less, that in its simplest form, has the examinee predict what note comes next after a short series of notes. Because the muical scale is mathematical and orderly one can if one has an ear for the “language” predict what comes next.”

    This is very true in 1,4,5 chords in simple tunes such as blue grass and old time rock and roll, not so much in classical, at least not that I can tell. Too complex and many times have less repetitive sequences. But then I do not play classical. But torture the “data” enough and you will lose the tune.

  92. Nice science, you put it out there, with all the data, your assumptions, process, limitations, admission of doubts, etc and let everyone rip it apart. Then you defend. All publicly. I think a few other “climate scientists” could take a lesson here, Great post and discussion.

  93. Yay Agilent! (E44xx) I was wondering when someone would apply modern Information Theory or Signal Processing techniques to this problem, let alone modern statistical methods. I have yet to see any study doing bootstrap :(

  94. I like it a lot. A quantification of the likelihood that the model has predictive skill. That’s more than the IPCC gives you.

  95. Jim G says:
    September 11, 2013 at 8:13 am
    “Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out.”

    That’s completely unfair. Jeff has argued convincingly why there is reason to believe that the periodicity is real.

  96. Can you “predict” the last 25 years of your data set analyzing just the years before ?
    I mean, using your same methods , can you predict the warming in the last few decades?

  97. In the last year in a series of posts at

    http://climatesense-norpag.blogspot.com

    I have laid out a method of climate forecasting based on recognising quasi repetitive- quasi cyclic patterns in the temperature and other relevant climate-driver data time series.Pattersons post illustrates a useful approach to recognising possible patterns.Interestingly his forecast cooling until 2040 followed by warming until about 2070 is generally similar to my projections..However I forecast further cooling beyond 2070 based on incorporating the 1000 year solar millennial cycle into the forecast.
    We should not expect mathematical precision in this type of forecast because of the changing resonances between the quasi cyclic rate processes which integtrate into the temperature climate metric.It would however be very useful if Patterson could extend his analysis to the 2000 year proxy record of Christiansen and Ljundqvist 2012 which is the basis for most of my projections for the next several hundred years .The following simple relationships should be noticed.Pattersons 171 year cycle = 3x PDO+/- 3X 171 =513, 6 x 171 = 1026. These intervals relate to the Roman Climate Optimum ,the Dark Ages ,the MWP,the LIA and the Modern Peak.The shape of the curve is not a simple sine curve. The Christiansen record of the last 1000 years is the most obvious go-by for the future trends
    .Here is part of my original post on this subject at the above link on 1/22/13
    “e) The real controlling factor for the immediate future is where the present day stands relative to the approximate 1000 year solar cycle peak
    f) Having looked at numerous reconstructions I suggest that that of Christiansen and Ljungqvist 2012 is the most likely to be closest to reality. : http://www.clim-past.net/8/765/2012/cp-8-765-2012.pdf
    Discussion.
    The key working hypothesis is that the solar cycle from 1000- 2000 may repeat and we may see a pattern of temperatures from 2000 – 3000 which is similar to that from 1000 – 2000. Fig.5 from the Christiansen paper is shown above.The solid lines are the 50 year moving averages and the dashed red lines are the upper and lower quantiles.
    Inspection of Figure 5 – both the moving average and the annual data suggests the following.

    1) The millennial peak is sharp – perhaps 18 years +/-. We have now had 16 years since 1997 with no net warming – and so might expect a sharp drop in a year or two – 2014/16 -with a net cooling by 2035 of about 0.35.Within that time frame however there could well be some exceptional years with NH temperatures +/- 0.25 degrees colder than that.
    2) The cooling gradient might be fairly steep down to the Oort minimum equivalent which would occur about 2100. (about 1100 on Fig 5) with a total cooling in 2100 from the present estimated at about 1.2 +/-
    3) From 2100 on through the Wolf and Sporer minima equivalents with intervening highs to the Maunder Minimum equivalent which could occur from about 2600 – 2700 a further net cooling of about 0.7 degrees could occur for a total drop of 1.9 +/- degrees
    4)The time frame for the significant cooling in 2014 – 16 is strengthened by recent developments already seen in solar activity. With a time lag of about 12 years between the solar driver proxy and climate -see:
    http://adsabs.harvard.edu/full/2005ESASP.560…19U we should see the effects of the sharp drop in the Ap Index which took place in 2004/5 in 2016-17. This estimate is quite independent from the estimate made from Fig5.
    Conclusions1) It seems reasonably probable – say 60/40 that the NH will cool by about .35 degrees by 2035.
    2) We should be able to check the accuracy of this forecast by 2018 -20.
    3)The forecast of a 1.2 degree drop by 2100 is little more than a mildly interesting suggestion at this time.
    4)The idea of a Maunder Minimum equivalent at 2600 – 2700 is highly speculative.
    5)Contrary to the forecasts made here, the Livingston and Penn solar data are suggesting a possible Maunder type Minimum in the near future.Given our ignorance of solar physics this is entirely possible. In this case a much more rapid cooling would occur with very serious consequences to the global food supply and the world economy.
    6) Global cooling will take place concurrently with that of the NH but because of the great extent of the southern oceans the global cooling will be significantly less – maybe +/- 50 % and there will also be considerable regional variability. in both hemispheres.
    7) There is no reason to expect damaging global warming.Cooling is more likely .To prepare for it, all ethanol and biofuel subsidies and mandates should be abolished.Renewable energy and electric car subsidies are economically wasteful and accomplish nothing.There is no reason to control CO2 emissions, indeed some extra CO2, while having little effect on temperature, might aid farm productivity . 25% of the increased crop yields in the 20th century was due to the CO2 increase

  98. the problem is you can use this method with so many curves…
    so take any kind of data. price of wheat oranges or .., well anything..cut the last part of the data do the same analysis and..see if it can “foresee ” anything…i am pretty sure no…

    i don’t buy that..because you don’t have any idea if the harmonics are related to something ,corespondind to a oscillating phenomena ,or just …a way to fit the curve.

    May be this method can be interesting to find out if some frequancy you find smell “physical”

  99. DirkH says:
    September 11, 2013 at 9:01 am
    Jim G says:
    September 11, 2013 at 8:13 am
    “Or any more. Statistical masturbation, at its best, in both cases. Should have left it in the “boxcar” and shipped it out.”

    “That’s completely unfair. Jeff has argued convincingly why there is reason to believe that the periodicity is real.”

    Define “unfair”. Extreme data pre-massage is evident to the extent of data torture. We cannot be skeptical of warmist models and not be skeptical of this. Time will tell if it predicts well as we know that the AGW models do not.

  100. So if we believe the modeling at all, then this is telling us that from 2020 – 2075 we can expect below average temperatures on average. How exactly are they planning to explain that away?

  101. @jeffery, two questions:

    1/ why not apply these techniques in hindcast mode? That would be really interesting.

    2/ would it be possible to provide this code in a stand-alone form so that others could play with it. I’m a developer, I would be happy to help out.

  102. @Jeff Patterson

    The residual _is_ stationary. That’s the point. We can certainly drive an AR process with stationary (and in this case also nearly Gaussian) noise + sine waves and get a valid projection.

    Well, ideally, the residuals should be pure white noise, with mean=0

    Overall I really like your experiment. Well done. A good teaching moment for us all!

    But I have used similar ARIMA-based estimators myself to predict these kinds of time-series (stock-market) and managed to find a lot of “useless” predictability (i.e. in the linear sense). It always turned out that the event sequences that I couldn’t predict (the “innovations”) were mostly the same events that would have made me a rich man if my predictor worked the way I hoped it would.

    To make these models perform better you need to know a lot more about the hidden processes which generate the observable outputs.
    :-|

  103. 170 year period data concluded from a time period of less than a full cycle?

    All that means is that the increase in temperatures is then turned into a decrease on the other side of the wave.

  104. It has been said that if it is good science, it does not require a statistical massage. I heartily endorse that sentiment.

  105. @ Jim G and others.

    I can take broadband noise and then pass it through a low-pass system. Provided the system has persistance, I will get a signal containing trends and this may appear quasi-periodic. I can fit a Fourier series to this signal and calculate an “input” that will create the output for a short length of recorded signal.

    However, the input is noise and therefore the small segment of “input” is simply one snapshot of its underlying process. It is inevitable that if you do this you will find want appear to be cycles. If you repeat the “experiment”, you will find different periodicities in each repeat.

    What one cannot do is assume that this snapshot will repeat – it is simply one sample of a noise record.

    If you look at figure 13, it is the 21st century prediction and is simply a recapitulation of the 20th century with a basic triangular waveform so that the 20th and 21st centuries connect.

    The logic underlying this post is, to my mind, hopelessly flawed.

  106. “Digital Signal Processing analysis of global temperature data time series suggests global cooling ahead”

    Let’s see a plot of the next 170 years to see what this method suggests.

    My guess is that it’s going to ramp down to about -0.4 and rise steadily upwards with a couple of bumps in the middle.

    I promise I have not try plotting it, just guessing…..

  107. crosspatch says:
    September 11, 2013 at 8:26 am

    The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen. Would an analysis of the opening of the 1812 Overture predict the finale? About the only thing I am prepared to say with any certainty is that the next 1000 years is more likely to be cooler than the last 1000 years than it is likely to be warmer.

    Sure it could! And in 1812’s case (as with many [most?] other pieces of classical music) the intro comes back in the finale in the tonic key. This is the entire point of prediction, and human nature, to find (and create) patterns/expectation/reconciliation….then do it again.

  108. If I understand correctly what he is doing the “BoxCar and the Decimation” are more about creating yearly “buckets” (or average/smoothed values) so that yearly values can be determined and examined. The same techniques are in R and widely used there for the same purposes — sometimes named differently.

    Probably better explained in “The little book of Time Series” than I can do

    http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/

    hth

  109. I don’t have time this week. If I remember this discussion next week I will run it through a somewhat different set of processes and see If I get the same periodic repeats.

  110. oops, hadn’t seen the caption on fig 13, I must have started skimming well before I got that far. Guess what if the previous 170 years isn’t strangely similar.

    As RC Saumarez says, this is nothing more than the extrapolation fallacy.

  111. I appreciate the response posted here. rather than address each in turn, let me make some general comments on the method.

    First, the concerns about aliasing are IMHO unwarranted. The monthly data record is highly low-passed, being averaged in both space and time. Also, presumably the climate process, containing as it does many integrative elements, imposes its own filtering of high frequency input variances. The power spectral density of the unaltered Hadcrut data shows the high frequency floor to be down some 60dB (1/1000) from the spectral peaks we are trying to extract. If one understands the aliasing process in the time domain, a moment’s reflection should reveal the mathematically provable theorem that the amplitude of an alias cannot exceed the amplitude of it source. Hence aliasing effects here are insignificant.

    On filtering: In harmonic decomposition, filtering must be avoided like the plague because it alters the amplitudes and phases of the periodicity we’re attempting to extract. Loss of spectral resolution is also an issue, especially when trying to extract cycles whose periods exceed the record length. However, the monthly data has a strong annual component that must be removed prior to decomposition because the underlying assumption of the method is that all periodicities are harmonically related. Fortunely, if you know the period of the offending cycle there is a solution, Box care filtering as described above. This filter is ideal in that it places its transfer function zeros precisely on top of the spectral peak being removed. It is in essence a high-Q notch filter which has the least impact on the remaining spectrum. It is true that this filter has high sidelobes but in this case that’s fine because the power spectral density at the lobe peaks is extremely low as evidenced above.

    On record periodization: First note this is only a test to verify that observed spectral peaking is not due to the discontinuity at the record end points- the harmonic decomposition is performed on the original, unperiodized data).

    Fourier analysis (which is used here only to perform the PSD estimate) gets around the required infinite length record by simple assuming the record repeats ad infinitum. If the endpoints don’t line up (withing the average step-size of the sample), the discontinuity created by stitching the records together (that’s not how an FFT works but in principle the effect is the same) creates a periodic error that shows up as spectral peaks. We can test for this effect by subtracting a line that connects the data record endpoints and dropping the last point from the result to avoid repeating the zero-error point at the ends. The result is a record which has no discontinuity distinguishable from the noise present in the data. If the PSD peaks remain after this process, it indicates the spectral peaks are not record-length artifacts.

    One reader asked “why use the eight harmonic?”. Because there are more cycles present of the eight harmonic and so it provides a better estimate of the fundamental.

    Fig. 10 spaghetti graph.
    What was allowed to vary between trials? What stayed the same?

    Figure 10 is output of the AR process used to create “climate-like”. Each path is a 113 point sample of the process for a new randomized state and random, white, input function of the variance derived by the AR process estimation function.

  112. Jeff Patterson
    Best Regards and thanks for your post. I repeat the request made in my post at 9:15 Could you possibly find the time to do a similar analysis of the 2000 year proxy record referred to in that comment.I think all the basic data is available on the web It might be a very illuminating exercise.

  113. If we started to fairly and objectively consider ringing, overshoot, beat frequency and other of these sorts of behaviors with respect to the global climate system, some interesting and perhaps frightening possibilities might start to become more apparent and would open the door for a whole new tone in the conversation.

  114. I forget to add a comment on forecasting:

    In retrospect I wish I had left this section out, even though I thought I had presented the appropriate caveats from the start. Unsurprisingly, readers naturally gravitate to projections and by doing so miss the point of the exercise which was not to try to predict the future but rather to show that a plausible alternative hypothesis (actually two, harmonic forcing and/or resonances in the climate system) provides a good fit to the observed data and that such a good fit is unusual. I have neither the time or expertise to complete the exercise – namely providing a physical explanation for the hypothesis and empirically testing the result.

    Some general comments about modeling:
    Information theory tells us nothing can be learned from a calculation. Knowledge is created by surprise (entropy) and no calculation should surprise us. All modeling, including GCMs, no matter how complex are simply calculations. The results are completely determined by the input and initial state. They are thus only useful in the hypothesis-forming portion of the scientific method (something the climate modelers seem to have forgotten). Once formed, the hypothesis must be verified by empirical observation.

    Again, thanks to all for the useful comments. I’ll try and follow up with the suggestion to run the method on a subset of the data and post the results. My thanks to Mr. Watts for providing this site and to all for the interesting conversation.

  115. We can test for this effect by subtracting a line that connects the data record endpoints and dropping the last point from the result to avoid repeating the zero-error point at the ends. The result is a record which has no discontinuity distinguishable from the noise present in the data. If the PSD peaks remain after this process, it indicates the spectral peaks are not record-length artifacts.

    Ok. But then your line that you subtract from the time-based data is exquisitely sensitive to just two sample points, Point 1 and Point 113. I’ll go along with the method, if and only if, you use it on at least 9 other subsets of the data: Points 1-112, 1-111, 1-110, 2-113, 2-112, 2-111, 3-113, 3-112, 3-111. Now you have 10 different base lines to remove from the data and 10 different resulting PSD’s. What are the consistent peaks, what are the inconsistant peaks? You gotta admit, it add robutness and “fold” to the results.

    One reader asked “why use the eight harmonic?”. Because there are more cycles present of the eight harmonic and so it provides a better estimate of the fundamental.

    There is no free lunch. There might be more cycles, but fewer samples per cycle. Frankly it makes it more sensitive to the choice of data length.

    Your response to the Fig. 10 question didn’t answer what was asked. What was randomized? Just the phase of the spectral components?

    Like a few others above, I am highly skeptical of a fundamental cycle length longer than the original time series. From what I learned in geophysics, that is a violation of the Fourier theorems. Your support of this long period seems to come from “harmonic decomposition”. I am very skeptical. It sounds like circular logic. Are you in effect padding the time series with a variable numbe of zeros until you get your peaks to resonate? 32 years out of my Ph.D. [you listening, Gail?] I’m still willing to learn, but like I said, “I need to see more.”

  116. Climate scientists got off on the wrong foot three decades ago, by prematurely assuming that there were just two important drivers of climate – namely the sun (insolation) and a CO2-driven greenhouse effect. These, most reckoned, were the “first-order” influences and all of the other factors were second-order or third-order and therefore could be largely ignored. CO2, they said, was the “control knob” of climate variability.

    Mother Nature has unkindly demonstrated their basic error, by providing fifteen years of almost unchanged world average temperatures while the atmospheric CO2 level, for whatever reason, continued to increase significantly. The control knob was turned, and almost nothing happened! The most bizarre and ingenious ecclesiastical apologia forthcoming from the Jesuit intelligentsia would pale by comparison with some of the climate establishment’s amazing “explanations” for this hiatus. We wait with bated breath for the IPCC’s upcoming summary to see if it “explains” or simply ignores what nature has done to them.

    This is why Jeff Patterson’s post is so exciting. It assumes absolutely nothing about the theories or explanations of climate scientists – or anything else – and simply mines the available data for plausible patterns which could provide some insight into future trends or actual mechanisms. And, unlike most other such attempts at naive analysis, it is performed by somebody who clearly understands the false moire and other mirages which can emerge from this type of statistical treatment. Such trend analysis is surely the best starting point for the badly-needed “reboot” of climate science.

  117. Allow me to join the several other commenters who asked that you perform the last part of the analysis again. No, I’m not asking that you again seek the fundamental frequency. I’m just asking that, taking the fundamental you’ve already identified, you use just the part of the record prior to, say, 1945, to determine the harmonics’ amplitudes and phases. Then, using those amplitudes and phases, project forward through the present. (Although my guess is that your computer time required to identify what by my count are ten different parameters will be significant, you wouldn’t have to rewrite the program, of which even just the part we’re requesting you re-run would probably take plodders like this writer more than an hour to write and debug.)

    I recognize that this a slight diversion, but it is indeed only slight, and the results could support the following interesting question: If the climate’s behavior in the record’s latter portion, which represents a time when CO2 concentration was rising significantly, is predictable, without any adjustment for CO2–and on what’s little more than a the-trend’s-your-friend basis–from the record’s former portion, which represents a period when little CO2 increase occurred, shouldn’t the proposition that CO2’s effects are significant bear a heavy burden of proof?

    Allow me also to join others who have thanked you for a clear post and responsive follow-ups.

  118. Henry@all
    I really feel sorry for you all because you are all missing an important point:
    where would be without the sun? We’d be dead, would we not?
    We know that earth has a number of factors (built-in) that confuse matters,
    on mean average temp.
    like
    its own volcanic action,
    turning of the iron core,
    magnetic fields/ clouds
    lunar interaction
    to name but a few.
    I am sure there a lot more.

    So why do you all keep staring at those stupid figures for mean average temps. & their anomalies?
    Surely anyone with brains must have figured out by now that maximum temperatures are a much better proxy to look at, as I found. But don’t look at only one station, because if you were to look at CET maxima alone you might get confused: they have a lot of weather…..there in London.

    http://blogs.24.com/henryp/2013/02/21/henrys-pool-tables-on-global-warmingcooling/

    http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/

  119. crosspatch says: The problem with this sort of analysis, while interesting, is that it can only show what HAS happened and not what WILL happen.

    But showing what has happened is important, is it not? The AWG proponents are convinced what _has_ happened is due to CO2 emissions. I’ve forwarded an alternate hypothesis which, if it could be varied, would alleviate these concerns. In that case we could confidently predict that the future climate will change, just as it has in the past, and these natural changes will likely be benign, just as they have been in the past.

  120. Echoing others: Thanks for this article – for writing clearly and explaining, for acknowledging problems, for being bold and trying stuff out, for providing the code, and for being around to discuss issues with others. Even if your post were complete nonsense (Im not in a position to judge) it’s still a good example of what we should all be doing here.

  121. Eliza says: September 11, 2013 at 5:15 am
    Try same with CET (reliable rural station) you will probably get a much more pronounced cooling in the future

    Take a look at: CET Patterns, 1659-2007, Wikipedia Graph

    If we were to guess about the next two centuries based upon historical patterns in the CET record, we could guess that the current pause in rising temperatures will be merely a pause; and that over the next 200 years, GMT will continue its long-term rise while following a jagged pattern of localized up-and-down trends, doing so at a linearized trend over a period of two centuries of about + 0.3 C per century.

    Certainly, one or more localized downward trends in GMT could occur within that 200 year period, and probably will.

    But in the mode of saying that when in doubt, predict that past trends will continue, we could also guess that a jagged up and down pattern — one with a gradually rising long-term linearized trend — will continue until the maximum of the Medievel Warm Period is reached.

    A reasonable estimate of GMT at the height of the Medievel Warm Period is needed. Is there anyone with the appropriate credentials working on that estimate?

    In the meantime, what about the next 100 years, as opposed to the next 200? Make your own guess about GMT in the year 2100 using Beta Blocker’s CET Pattern Picker.

  122. Stephen Rasey says:
    September 11, 2013 at 10:31 am
    “There is no free lunch. There might be more cycles, but fewer samples per cycle. Frankly it makes it more sensitive to the choice of data length.”

    True but even the eight harmonic is highly oversampled. Nyquists proves that increasing the number of samples per cycle (beyond two per the shortest period) adds no information.

    “Your response to the Fig. 10 question didn’t answer what was asked. What was randomized? Just the phase of the spectral components?”

    As was stated in the paper and elsewhere, figure 10 is the output of an AR process designed to provide random sequences which mimic (in character) the observed climate record. An AR process is just an IIR filter. The input to the filter in this case is random noise with the specified variance. Each path in figure 10 is the “filter’s” output with a different, random input sequence (and initial state). We then perform the same harmonic decomposition on these “climate-like” paths to test the null hypothesis that any old randomly produced sequence which looks kinda like temperature data would yield a goodness-of-fit similar to that achieved with the real data. While we cannot quite reject the null hypotheis (p=.081), we can state that getting the achieved GOF is not to be expected and in fact the achieved residual variance is near the 3-sigma of the distribution of the randomly generated results.

  123. @Sam The First says: September 11, 2013 at 5:57 am
    Menahwile in the broadsheet papers here in the UK, notably The Independent and The Guardian which are read by liberal / left opinion formers (inc teachers at all levels), the comments to this article below demonstrate that none of their readers is taking any notice of the long pause in warming and the overall cyclical record.

    Will someone with the time and expertise please add a comment or two to explain what is really going on?
    ========================================================================
    I’ve long been banned from CiF for dissent. The Indy, well, it’s best for me not to go there as it enrages me in no time at all. Occasionally, I will employ my rapier like wit to puncture some balloons, but ultimately, it’s a waste of time talking to such as comment there.

  124. Jeff Patterson says
    (future climate change) will likely be benign, just as they have been in the past.

    henry says
    Dear Jeff, unfortunately I think it will not be benign
    Under normal circumstances, like you, I would have let things rest and just be happy to know the truth for myself. Indeed, I let things lie a bit. However, chances are that humanity will fall in the pit of global cooling and later me blaming myself for not having done enough to try to safeguard food production for 7 billion people and counting.

    It really was very cold in 1940′s….The Dust Bowl drought 1932-1939 was one of the worst environmental disasters of the Twentieth Century anywhere in the world. Three million people left their farms on the Great Plains during the drought and half a million migrated to other states, almost all to the West. Please see here:

    http://www.ldeo.columbia.edu/res/div/ocp/drought/dust_storms.shtml

    I found confirmation in certain other graphs, that as we are moving back, up, from the deep end of the 88 year sine wave,

    http://blogs.24.com/henryp/2012/10/02/best-sine-wave-fit-for-the-drop-in-global-maximum-temperatures/

    there will be standstill in the speed of cooling, on the bottom of the wave, and therefore naturally, there will also be a lull in pressure difference at that > [40 latitude], where the Dust Bowl droughts took place, meaning: no wind and no weather (read: rain). According to my calculations, which includes certain planetary positions, this will start around 2020 or 2021….and last until 2028.

    Danger from global cooling is documented and provable. We have only ca. 7 “fat” years left……

    1) We urgently need to develop and encourage more agriculture at lower latitudes, like in Africa and/or South America. This is where we can expect to find warmth and more rain during a global cooling period.

    2) We need to tell the farmers living at the higher latitudes (>40) who already suffered poor crops due to the cold and/ or due to the droughts that things are not going to get better there for the next few decades. It will only get worse as time goes by.

    3) We also have to provide more protection against more precipitation at certain places of lower latitudes (FLOODS!),

  125. Reading 170 year cycles… I knew I read about this before!
    People, try a quick blast on Google and you will be decimated by the amount of similar findings from differing sources. From war to famine. Even the 10Be records have something to say

  126. I’m delighted to see that the head posting confirms – and by similar methodology – work being done elsewhere which has come to very much the same conclusion – that there may be a 0.5 K drop in global temperatures within the next few years. I mentioned this briefly in an earlier posting, and the usual suspects elsewhere began sniffing and sneering and offering me bets.

    Fourier analysis, and particularly the subset of it that is used here, is an appropriate method. But best results are obtained not so much by looking at the past temperature record in isolation and over little more than a century, as here, but in association with possible aetiologies and their datasets, and over all timescales. Murry Salby’s lecture takes this approach, and is worthy of study. The Akasofu and Scafetta approach, extrapolating from previous cycles, is also interesting, as is Tsonis on the ocean oscillations, whose 58-year cycles are close enough to be harmonics of the 171-year period mentioned in the head posting as having been detected by the author’s harmonic decomposition.

    If the IPCC did science this way, openly and inviting public scrutiny, it would have been out of business long ago: it is only by concealment that they get away with continuing this scam. If Mann, Bradley and Hughes had done science this way, they would never have dared to foist the “hockey stick” on the world.

    Finally, at Bishop Hill there is a hilarious debate in one of the House of Commons committee rooms about global warming, in which the true-believers take quite a pasting.

  127. There is a 170 year cycle that is based upon the conjunctions of Uranus and Neptune, 1992, 1821,1648,1470,1300 etc.

  128. This paper by Coughlin and Tung 2004 employs the nonlinear EMD (empirical mode decomposition) analysis to stratospheric temperatures, and finds a strong 11 year solar cycle.

    This is the right kind of analysis to look for nonlinear forcing of the climate system by solar and other astrophysical cycles. Check out figure 3.

  129. Stephen Rasey says:September 11, 2013 at 10:31 am
    “Ok. But then your line that you subtract from the time-based data is exquisitely sensitive to just two sample points,”

    Actually quite the opposite. This is because the Fourier Transform of a sum is equal to the sum of the FTs. Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the line (ramp), irregardless of the slope of the line. Here this compensation step was unnecessary as we were only interested in determining that the observed peaks were not record-length artifacts.

  130. Chris Schoneveld:

    At September 11, 2013 at 11:04 am you ask me

    So, richardscourtney, how does this jive with Akasofu’s linear “recovery” from the LIA?

    Among the many possible meanings of your question, one is,
    ‘How do the studies of Patterson and Akasofu relate?’
    And, of course, the answer to that question is,
    ‘They don’t relate because they are different analyses conducted for different purposes although they are conducted on the same data’.

    Patterson attempts to use signal processing to determine underlying frequencies in the global temperature time series. It remains to be seen if his analysis can provide useful information, but if he has identified frequencies which are ‘real’ then consideration of those frequencies may induce studies for provision of information concerning significant climate processes.

    Akasofu observes that the temperature global time series can be matched by assuming a linear recovery from the LIA combined with a sinusoidal oscillation (possibly related to ocean behaviour) which provides alternate periods of warming and cooling each of 30 years duration. He extrapolates that pattern with a view to determining when it changes, and upon observation of a change (which will surely eventuate) then it will be reasonable to infer whatever has induced that pattern has changed.

    I don’t know how either analysis could jive or perform any other dance. But I am pleased with my jiving and have medals for my ability on the dance floor.

    Richard

  131. RC Saumarez says:
    September 11, 2013 at 9:49 am

    “I can take broadband noise and then pass it through a low-pass system. Provided the system has persistance, I will get a signal containing trends and this may appear quasi-periodic. I can fit a Fourier series to this signal and calculate an “input” that will create the output for a short length of recorded signal.”

    But that is precisely what I’ve done in the section entitled “Stochastic Analysis”! The AR process used is a low-pass filter with coefficients adjusted to produce “climate-like” outputs from white noise inputs. When we do harmonic decomp on these random sequences we see that in general we cannot get close to the goodness-of-fit (as measured by the variance of the residual error) achieved with the actual climate data.

  132. Mr. Patterson,
    The quality of a five-harmonically-related-frequency model fit to the data surprises me. I’d be interested in seeing the quality of a similar fit using a five frequency model whose frequencies are not required to be harmonically related. That is, if the software routine you used to determined the values of harmonic sinewave amplitudes and phases that best fit the data can accommodate sinewave frequency as well, I’d be interested in seeing (a) how the best-fit estimated frequencies differ from their harmonic counterparts, and (b) the change, if any, of the residuals and residual spectral content. If the software you employed uses a weighted-least-squares cost function, my guess would be that giving that software an initial guess corresponding to your harmonic analysis results, the software would converge to a solution with frequencies, amplitudes, and phases differing only slightly from their “harmonic analysis” equivalents.

  133. JP: ” forget to add a comment on forecasting: In retrospect I wish I had left this section out”

    Yep, forgetting the title of the post may be the best approach.

    “All modeling, including GCMs, no matter how complex are simply calculations. The results are completely determined by the input and initial state. They are thus only useful in the hypothesis-forming portion of the scientific method (something the climate modelers seem to have forgotten). Once formed, the hypothesis must be verified by empirical observation.”

    A very good point.

    JP: “First, the concerns about aliasing are IMHO unwarranted. The monthly data record is highly low-passed, being averaged in both space and time. ….The power spectral density of the unaltered Hadcrut data shows the high frequency floor to be down some 60dB (1/1000) from the spectral peaks we are trying to extract.”

    That is precisely where the problem lies. It _had been_ heavily low-pass filtered and the evidence is that it was not correctly anti-aliased before that was done.

    http://judithcurry.com/2011/10/18/does-the-aliasing-beast-feed-the-uncertainty-monster/

    The fact that there is NOW very little H.F. tells us _nothing_ about how much this was in the data that is now aliased elsewhere in the spectrum. And that could multi-year or decadal in scale.

    The papers on Hadley SST processing sketch out the process to build the global ‘climatology’ grid, which involves running averages in adjacent grid cells across latitude and longitude. This is repeated in a loop until the result ‘converges’. These 5×5 degree grid cells of 5 day averages are then processed into calendar monthly means.

    The papers do not make any mention of anti-alias filtering prior to re-sampling .

    So, yes there is lots of low pass filtering going on , that accounts for the low amplitude of h.f. signals that you comment on. That in no way indicates that the power in those poorly filtered and re-sampled frequencies are not now lying in the multi-year to decadal frequency bands.

    RC Saumarez showed that the frequency spectrum of that data suggests rather clearly the presence of aliasing.

    my comparison of ICOADS to hadSST :

    http://climategrog.wordpress.com/2013/03/01/61/

    …shows some limited but significant changes to the frequency content of the data due to Hadley processing. Whether this is an improvement or a degradation of the data is not my job to establish but I am unaware of any account that this was even measured as part of Hadley’s QA on their data processing.

    Maybe it was , maybe I missed it. Maybe it wasn’t.

    Caveat Emptor.

  134. @Jeff Patterson at 11:31 am
    Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the [known] line (ramp),

    You don’t know the line. 113 points in the dataset is just an artifact of how long someone was recording data. When the data starts and ends is similarly an artifact. Different starts and different ends give different lines and therefore different transforms. I expect differences in the high frequencies and power at the low frequencies.

    Also, the choice of a base “line” is arbitrary. Simplistic, but not necessarily the best choice. That base line might be a linear function of CO2 concentration, logarithmic function of CO2, or a non-linear function of sunspot number.

  135. The observed temperature anomaly since 1900 can be well modeled with a simple harmonic decomposition of the temperature record based on a fundamental period of 170.7 years.

    That’s dear: 4 significant figures with a time series less than one full period.

    Anthony: This DSP engineer is often tasked with extracting spurious signals from noisy data.

    Is this “signal” spurious?

    His model does not fit the data as well as Vaughan Pratt’s model. It should be clear by now that experienced data analysts can get just about any result they want from the extant data. Now that he has published his model, it can be tested by future data.

  136. Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?

    I also think that some (albeit minor) global cooling is going to occur, because of the Sun, but I do not trust this model to tell me why or by how much. But I did enjoy the mathematics!

    Rich.

  137. “””””…….Ric Werme says:

    September 11, 2013 at 5:43 am

    A few comments:

    1) decimated by 12

    I know that “decimate” came from the Roman practice of inspiring soldiers after battles where the commanders felt were uninspired. They lined up the soldiers and killed every tenth man, leaving the remaining 90% inspired to win the next battle. At first blush, this sound like torturing the data until it confesses or perhaps this is “dodecimated” – killing every 12th measurement. :-)…….””””””

    Ric, In this instance “decimated is a highly specific technical term from digital signal processing; so don’t go looking for a highway lines with occupied crosses.

    And no adherence to the ten implication is required.

    And I know about enough about it to be a hazard to myself; but if an Agilent Cowboy, doesn’t know Digital Signal Processing, then nobody does.

    My concern with the exercise, is that I wonder what result one could pull out of the phone numbers in the Manhattan telephone directory.

    It’s not clear to me that the Hadcrud numbers actually comprise data that relates to planet earth.

    And having seen in the past, a mathematical exercise, that recovered the value of one of the fundamental physical constants of science (the fine structure constant), to about 8 significant digits of precision (about 1/3 of the standard deviation of the best experimental value, at that time); yet the model had precisely zero input from the physical universe; it was literally just fetzing around with numbers; so I don’t go for all new things, as an early adopter. I’m often the last to see the light.

    I have quite recently seen a movie called “Gone with the Wind”, for the very first time.

  138. Stephen Rasey says:
    September 11, 2013 at 12:04 pm
    @Jeff Patterson at 11:31 am
    Thus we can reconstruct the _exact_ spectrum of the unperiodized record by subtracting out the known transform of the [known] line (ramp),

    “You don’t know the line. 113 points in the dataset is just an artifact of how long someone was recording data. When the data starts and ends is similarly an artifact. Different starts and different ends give different lines and therefore different transforms. I expect differences in the high frequencies and power at the low frequencies.”

    You know the line exactly because you create it! More precisely, it is a periodic sawtooth that returns to zero foreach record. You know its slope, amplitude and period and thus it’s Fourier (not DFT) transform exactly. The (complex) coefficients of this transform can be evaluated at the DFT bins subtracted from the periodized DFT to get the spectrum of the unperiodized record. Do the math, try and experiment or two. It works (and we’ve been shipping it in products for years!)

  139. I should say that I enjoyed the post. Why start the series at year = 1900?

    The following is well-said: If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.

    The author estimates 10 parameters, after estimating the fundamental period. Why not just fit a 10th order polynomial or a Bayesian Adaptive Regression Spline model, or some well-selected set of 10 wavelets — as long as you are acknowledging that this is merely another curve-fitting exercise with no claim to predictive power, any old and new functions are just as good. If the fundamental frequency is evolving with time, in what sense is it a “fundamental” frequency.

  140. The harmonic technique used is useful in freeing the process of reproducing a ramp determined by the length of the dataset.

    The first second and third harmonics are just recreating the longer ramp. The most notable is perhaps the strong 8th.

    170/8= 21.25 years. Also one of the three periods my 3 cosine analysis latched on to.

    The suggestion of Schwabe solar cycle is obvious.

  141. mpainter says:
    September 11, 2013 at 9:45 am
    It has been said that if it is good science, it does not require a statistical massage.

    Yeah, usually by non-scientists. It is not science if there is no measurement of phenomena (signal). There is no measurement without error (noise). There is no way to sort out the signal from the noise unless you have an insight into, and a command of, the mathematics describing the noise statistics. Try it sometime. It ain’t so easy as you might imagine.

  142. Pamela Gray: Comment 1: While data massaging here is extreme, it does demonstrate the futility in ordinary least squares statistics so adored by AGWing scientists.

    This is non-linear least squares, after finding the fundamental period.

  143. “I have quite recently seen a movie called “Gone with the Wind”, for the very first time.”
    Yeah, I should find to time to watch too, one day. ;)

  144. Jeff Patterson: As the stochastic fitting exercise showed, JvN could only only do this .6% of the time if all he had to work with was harmonically related sine waves.

    That’s smart.

  145. Greg Goodman says: That is precisely where the problem lies. It _had been_ heavily low-pass filtered and the evidence is that it was not correctly anti-aliased before that was done.

    I looked over the referenced paper and while I certainly agree that aliased data is irretrievably corrupted, the question is by how much? The fundamental of the extracted model has a peak-to-peak amplitude of .6 degrees. If this were an alias (remembering that the amplitude of the alias is equal to the high frequency periodicity being aliased) surely someone would have detected the .6 degree high frequency signal in the data. It would only require a few weeks of hourly data to find it!

  146. Greg Goodman says:
    September 11, 2013 at 12:01 pm
    JP: ” forget to add a comment on forecasting: In retrospect I wish I had left this section out”

    Yep, forgetting the title of the post may be the best approach.

    I didn’t choose the post title which I agree was unfortunate.

  147. See – owe to Rich says:

    Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?

    170.7/3*2=113 = data length, oo-errr.

    Well spotted.

    “This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data.”
    -Anthony

    Freudian slip?

  148. Nick Stokes says: How can you establish a 170 year periodicity from 113 years of data?

    Basically, all he did was estimate a nonlinear function from the set of parameterized (phase and amplitude) family of sine curves. Using that estimated period, he then chose harmonics of it as the additional terms in a multiple non-linear regression. In harmonic regression, which is what this is, there is no non-arbitrary way to select the fundamental frequency: the standard in Fourier analysis is to let the full data series represent 1 full period, which may not have any relevance if you have managed to sample, say, over 1.25 times the period.

  149. This is off topic somewhat and is a criticism of computer run climate models. Picked this up in the comments section over at JoNova and think its hilarious,,
    “Roy Hogue

    September 12, 2013 at 5:42 am · Reply

    …a foundation of bullshit coming from computers, not bovines.

    Then let’s call it what it is — bitshit.

    And you’re right about it without any doubt too. :-) “

  150. @Jeff Patterson at 12:21
    You know the line exactly because you create it!
    YES!
    YOU create it. Not Nature. It is an artifact of the measurement time window and processing.
    That is why you must try different subsets and get different lines to explore the changes in the resulting transform. Then look for similarities that are more likely in the signal and not an artifact of the processing.

    It’s like trying different stacking velocities in seismic processing to improve the signal-noise ratio.

  151. rabbit: Have you considered Singular Spectrum Analysis (SSA) to extract the dominant modes of the temperature graph? This does not suffer from the windowing effects that a DFT suffers from, nor is it limited to a prescribed set of basis functions.

    I second the question. After choosing the base frequency, why did you limit yourself to harmonics of the base frequency? Is there relevant physical theory that the important frequencies are multiples of the base frequency?

  152. JP: “I looked over the referenced paper and while I certainly agree that aliased data is irretrievably corrupted, the question is by how much? The fundamental of the extracted model has a peak-to-peak amplitude of .6 degrees.”

    No , I was not suggesting the whole long term trend is an alias. Neither did RC’s article on Climate etc suggest that degree of error. But if there is a lack of filtering it will not just be one frequency effected. RC demonstrated that this can lead to spurious decadal length trends (did you read the link?).

    If there is error in the long term it is just as likely ‘bias correction’. I had the fortunate opportunity to discuss this at some length with John Kennedy in the comments. He was very forthcoming and helpful, though obviously defending their work.

    What he told me of the validation process showed it was based on circular logic and geographically anecdotal “validation” from japanese data also using _buckets_. (Apparently japanese buckets don’t ‘bias’ like british and american ones!).

    For the title , that’s editors for you . They’re always out for a catchy headline. ;)

  153. PS the japanese paper was full of bias confirmation too. They only retained the data which suggested Hadley corrections went in the right direction. (The ‘validation’ was not any stronger then that). But all those going the other way were deemed “not suitable” and removed from consideration.

    This is the problem with that state of the literature after 30 years of biased pseudo-science and political gate-keeping. The garbage is stacked several palettes high now.

  154. Jeff, is there any sort of robustness tests for this method? Can the time series be analyzed on a shorter interval, and see if the predicted results for the out of sample period match?

  155. Correct me if I’m wrong, but any non-random data points will show harmonics. I’m not sure what AGW would show but I would think it would simply be a S/N ratio of an AGW harmonic on top of natural harmonics. In fact, mixing the two (natural and AGW) would result in seeing the AGW mix in even pre-AGW data as it would be unable to blend out. I think the assumption using DSP is that temperature is the convolution of linear functions and that may not be the case.

    My gut tells me that DSP methods will treat random steps as non-harmonic noise and will not detect random and non-linear changes in DC offset that is characteristic of AGW. My gut is that a rise in temperature that is a sequence of step changes that are random in time and magnitude will be washed out in the deconvolution and the individual residual components that make up long term AGW would be below the noise level. Put another way, AGW may manifest itself as a sequence of below Nyquist rate events that all lie below the noise level. If the residual magnitudes look like “black body radiation chart” with a peak that is below the noise level, and a frequency longer than Nyquist, I would suspect you could have AGW in that signal and not be accounting for it as non-random and non-periodic and quite possibly attributing it to natural variation. The fact that you can model the past as a harmonic is from the math that will always solve, less a residual. It has no predictive value, though, because a driving component is non-linear addition to data. The math is good at figuring out how many “mixers and sources” are present in the existing data (or rather how many it needs to reach a certain S/N ratio) but not so good at predicting how many mixers and sources will be present in the future. I believe if it’s random in time and magnitude, and small at each step, no amount of oversampling and decimation will detect a fingerprint to predict a future value. For example, ENSO might be a large harmonic source that describes a long variation it temperature. If reality is that AGW is tiny, independent, sources and mixer to inject AGW at random points (i.e. say in 0.01C in June of 2007 and then 0.012C for January 2008, and then another 0.005C in September 2009), the analysis of those “below the nyquist and noise level will either leave them out or model them as a harmonic that has a lower residual error even if it’s really an accumulated error that will continue. The method will always add a harmonic term to reduce the error if it can whether it’s physical or not. I think the only way to overcome it is massive oversampling and decimation but you may still run into the noise floor of the data before you can pick out small, random, non-periodic residual signals. It’s like describing something as a finite series and every time your error gets too large, you add another term to the series that reduces the error. That’s what DSP algos will do and it’s not discriminate as to whether the term is physically descriptive of the underlying phenomena. It’s a fit and the number of terms is a function of the tolerable error and the technique. It works well to compare two different datasets but it’s not predictive and I’m not sure you can lump all the “fit” into natural variation and claim only the residual is a man-made artifact.

    An interesting exercise would be to break the data up into ranges always starting at 0. 0-11 years, 0-23 years, 0-37 years, 0-61 years (maybe just prime intervals) see what the solution was at the end of each of those periods and see how those functions compare to the full dataset. The PSD harmonics should not change as long as the dataset is longer than the fundamental mode nyquist rate. If the PSD harmonics are different depending on dataset choices, I think there is a problem with the method and it’s assumptions.

  156. Greg Goodman says: But if there is a lack of filtering it will not just be one frequency effected. RC demonstrated that this can lead to spurious decadal length trends (did you read the link?).

    Color me dubious. The human body is about the closest thing I can think of to continuous time thermometer. Are we fooled into sensing a temperature trend that doesn’t exist due to aliasing? Of course not, because the earth’s temperature itself does not experience high frequency fluctuations. All the more so with a real thermometer because it couldn’t react to rapid changes even if they were there. If we then average equally spaced (in time) temperature readings, we are averaging already smoothed (by the slow reaction of the climate and the thermometer) data so again no aliasing occurs. Sure there are errors in the readings which may look like white noise, but these could not produce a measurable trend unless integrated to produce a random walk. What process are you proposing that would produce such a result?

  157. Matthew R Marler says:
    September 11, 2013 at 1:05 pm
    After choosing the base frequency, why did you limit yourself to harmonics of the base frequency? Is there relevant physical theory that the important frequencies are multiples of the base frequency?

    Because the PSD indicated a harmonic relation between the spectral peaks and because harmonics are generated anytime sinusoids drive a non-linear response (thermodynamic equilibrium involves a quartic relationship to temperature if I remember correctly ) .

  158. I do not see any spectral peaks here of any distinctive character. Which surprises me, because i would expect to see something distinctive in the neighborhood of 0.016 years^-1 (60-65 year cycle evident in the data). Rounded peaks are generally artifacts of finite record length, and not representative of actual underlying processes.

  159. Greg Goodman says:
    September 11, 2013 at 12:52 pm
    “This DSP engineer is often tasked with extracting spurious signals from noisy data. He submits this interesting result of applying these techniques to the HadCRUT temperature anomaly data.”
    -Anthony

    Freudian slip?

    No, my day job is signal generation via direct digital synthesis. In such a system, the carrier is clear, unambiguous and easy to measure. Very small (< -90 dBc) spurious tones are also generated due to system non-idealities. We need to extract these spurs and characterize their amplitudes from the noise produced by DACs and measurement floors.

  160. I have a couple of large objections to this procedure.

    The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data. This is extremely basic testing, and the fact it wasn’t done is very worrisome, particularly given the claimed expertise of the author.

    The second is that as far as anyone has every determined, the climate is chaotic … how does he reconcile this with his claim that it is deterministic? Extraordinary claims (such as the climate being deterministic in nature) require extraordinary evidence … and he hasn’t even provided the most basic of tests, using out-of-sample data.

    The third is that his “Monte Carlo” analysis is extremely simplistic. I have provided a lot of evidence that the climate is governed (regulated) by a variety of emergent phenomena (thunderstorms, El Nino/La Nina, PDO, etc.). Accordingly, to investigate the system, among other approaches you need to generate synthetic data for such a governed system to use in the Monte Carlo analysis.

    Monte Carlo analysis is easy to do … but very difficult to do well. The problem is that you are begging the question.

    For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.

    I am sure that folks can see the circularity in that argument … to do a Monte Carlo analysis, you have to either:

    a) Know the underlying form of the data (which we don’t), OR

    b) Use a variety of assumptions about the data to try to cover the actual possibilities.

    He has done neither.

    Finally, I am completely unconvinced that you can determine a 170.7 year cycle (including the ludicrous .7 year decimal part) from a hundred and sixty years of data. The mere fact that he has included the .7 is a huge danger sign to me, it indicates that he is used to dealing with definitive numbers, not error-filled data like we have in climate science.

    So, despite his claimed expertise … color me completely unimpressed by his claims. He has not done even a small part of the work necessary to substantiate his claims, he has not tested his method on out-of-sample data, he is claiming that the climate is deterministic (Mandelbrot among others disagrees) and he has done a childishly simplistic Monte Carlo analysis.

    Note that these are all correctable errors … it remains to be seen if he corrects them.

    w.

  161. Testing the model´s ability to fore-/backcast, i.e., its usefulness, is easy: just use a subset of the time-series and see how the model´s extension matches the rest.
    I suspect that if this is done, it will produce a completely different model for each subset, and all of them wil fail miserably at modelling anything outside of their respective known boxes.

  162. We know that the hadcru, giss and noaa data is severely adjusted to reduce the late 30’s early 40’s warming, to reduce the cooling from 1945 to 1975 and to enhance the warming from 1980.
    The data sets also fail to account properly for urban heat island effect, for station deterioration and for the removal of many cooler stations from the dataset.
    The question is; do these massive questionable adjustments coupled with the failure to account for the factors mentioned above, affect the sort of analysis you have performed here.

  163. See – owe to Rich says: September 11, 2013 at 12:06 pm

    Goodman and Suamarez have many criticisms on this article, but no-one has raised the things which immediately troubled me. First, given that HadCRUT3 starts in 1850, why did JP throw away 50 years and start at 1900? This is not explained, and is fishy. Second, if the model is good at predicting forwards, how well does it do in predicting backwards (hindcasting) the said 1850-1899 period. Third, since the data length is 113 years, is not a harmonic of 56.9 years (almost half of the 113) extremely fishy?

    Using the five sinusoids (amplitude, frequency, phase) given in JP’s paper, the match (sinusoids versus measured data) going back to 1850 is not very good. You ask a good question, and I’d like to hear JP’s answer. I’d put my plot of that data in this comment, but I don’t know how to include figures in a comment.

  164. Patterson presents yet another simplistic modeling study of Hadcrut3 in which DSP methods are applied to make predictions with scant comprehension of the stochastic nature of geophysical signals. There should be little doubt that,once the diurnal and seasonal cycles are effectively removed in decimating the data to yearly averages, what remains should be treated as a wide-band stochastic process. Unlike the aforementioned cycles, there no known physical forcings that are strictly periodic aside from the astronomical tides. it is only in the contrary case that any model consisting of a finite set of pure sinusoids can be expected to be effective in making predictions.

    Although a ~170yr oscillation does indeed appear, inter alia, in the power density of GISP2 oxygen isotope data, Kalman-Bucy filters fail to produce close, practically useful predictions. The author’s computational domonstration of the poorer fit of independent realizations of nominally the same process is ipso fact irrelevant when only tested against the realization present in the manufactured Hadcrut3 data series. It is on this point, rather than the many other objections raised by others, that Patterson’s predictions are likely to fail.

  165. I think that this is a most useful, reasoned and practical contribution to the overall debate. Sure, it hasn’t ‘proved’ anything one way or the other (my math is far too weak and my neurons retiring too rapidly to follow the fine grain of it, so use the salt shaker here…).

    But the attempt is praiseworthy in itself: sticking one’s head over the parapet is always a risky move, and I congratulate Jeff for having done so.

  166. one way to end the debate about the merits of this technique is to divide the data into two 65 year time periods and see if one predicts the other without changing the methodology.

    if the method can predict the full time series with 1/2 the data from either end, without some magical adjustable “aerosol” parameters, then you either have the basis of a successful prediction or you should be buying lottery tickets.

    However, if the method cannot predict the missing 1/2, then it is time to look for a new methodology.

    If the methodology can predict the full time series from 1/2 the series, then it is worthy of a second, separate WUWT article because that would be strong evidence the author is on the right track.

  167. Willis Eschenbach says:
    September 11, 2013 at 2:59 pm
    The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data.
    ========
    I’d also like to see this and encourage the author to post the results.

  168. Discerning a 170 year period from HadCRUT alone sounds to me far-fetched considering past global temperature approximations.

    What I see more is that HadCRUT, especially HadCRUT3 (which I see as correlating with UAH and RSS better than HadCRUT4), has a visible periodic cycle.

    http://www.metoffice.gov.uk/hadobs/hadcrut3/diagnostics/global/nh+sh/

    The periodic cycle there, assuming it’s sinusoidal, appears to me to have a period of 64 years, with peaks in 2005, 1941, and 1877. Its amplitude appears to me as peaking at nearly .11 degree C above or below the longer-term trend. I got this by trying my hand at Fourier for several trials of 2 cycles, cosine component only, with start dates around 1877 and end dates around 2005. (I did not do well with figuring the sine component, due to contamination by the linear trend, due to the simple methods I used.)

    The nearly +/- .11 degree C periodic factor explains about 40% of the warming in HadCRUT3 from the early 1970s to 2005. Manmade direct increase of greenhouse gases other than CO2, which was largely stalled in the 1990s, appears to me to explain a little less than 10% of the early 1970s to 2005 warming. That means the anthropogenic part of the reported warming rate from the early 1970s to 2005 is about or slightly over half the total in that time period.

    The anthropogenic part includes effects of bias in generating the HadCRUT3 index, such as adjustments with insufficient consideration for growth of urban effects around surface stations – but that appears to me fairly small – in light of HadCRUT4, current and recent past versions of GISS, and NCDC, and their divergences from UAH and RSS.

    Thankfully HadCRUT, especially HadCRUT3, has sea surface temperatures fully considered. The sea surface consideration accounts for about 2/3 of HadCRUT and is essentially uncontaminated by growth of locally nearby airports or cities.

    Overall, it appears to me that anthropogenic global warming is for real, but degree of its existence appears to me as something like 35% of the “center track” reported by IPCC in AR4.

  169. Willis Eschenbach says:

    “The second is that as far as anyone has every determined, the climate is chaotic …”

    You should have dropped in on me when you were in the West Country, I could have shown you how I do deterministic weather forecasts. At the noise level it is not chaotic, so the sum of weather at the climatic scale cannot be either.

  170. Donald, you focus on the linear trend which is an artifact of the statistical analysis. As such linear trend cannot be proof of cause, nor even suggest it. In addition, a linear trend line has yet to be shown to be an effective method of demonstrating warming or cooling. If it were, the models would be no different than observation (the models use that statistic). Because they are not at all the same, the trend line itself is suspect in terms of calculating a meaningful trend (except for very short ones), regardless of its cause, either in the past and certainly not in the future.

  171. Nick Stokes says:
    September 11, 2013 at 5:11 am

    How can you establish a 170 year periodicity from 113 years of data?

    Dunno. Maybe it’s like saying todays temperatures are unprecedented for 1000 years while hiding the decline.

  172. Willis Exchenbach: “The first is that he has not divided the data in half, used his whizbang method to determine cycles much longer than the data itself, and then shown that when extended, his procedure closely matches the other half of the data.”

    I chimed in with others to ask for this, and, although he did not do it in precisely the way in which I asked, he did hindcast to the pre-1900 period. To these old eyes the results weren’t impressive. I’m perfectly open to others’ convincing me otherwise, but otherwise I don’t find the results very interesting from a climate perspective.

    That said, I’m not sure that all the criticisms are well taken. Those based on a obtaining a long fundamental period from a short record, for instance, are understandable but perhaps misapprehend what he actually did. He used the DFT to look for a clue as to what the fundamental might be, by observing peaks at frequencies that were integer multiples of a common fundamental. But it was only after he determined how much of the record could be “explained” by a periodic signal having that fundamental that he adopted it. Of course, the predictive value of the fact that such an explanation was achieved for that single record is not apparent.

    And the criticism that “he is claiming that the climate is deterministic” is puzzling. Perhaps someone could point out where he claims that–or where the cite for Mandelbrot contradicts it.

    Whatever the post’s other shortcomings are, I for one am grateful for not having had to grope through yet another fog of impressionism; the author actually laid out what he did.

  173. Willis Eschenbach says:
    September 11, 2013 at 2:59 pm

    Hmm, I’ve seem to have struck some nerve with my hobby horse, that was not my intent. I long ago lost interest in the climate debate, knowing from the outset that model-based science is an oxymoron. My interest was re-sparked recently when I stumbled here while looking for some work related info on Wiener processes. There was an interesting post speculating on whether the climate was a random walk process. There was a link to the dataset so I pulled it in and ran a unit root test which failed with p=.7. The plot by eye looked to have some periodic patterns and knowing that periodic forcing functions can make a stationary process appear non-stationary, I looked at the PSD. I noticed the harmonic relationship between the spectral peaks so it seemed like a good candidate for harmonic decomposition, a technique we use widely for analyzing non-linear processes. After HD, the residual passed the URT at p=10^-12. I was also surprised by how few terms it took to get a good fit and that set me wondering if the periodicity was real or simply a curve fitting artifact.

    Although I recognized the short record problem, it’s not one I normally encounter (if we need longer records we simply acquire more data – at 10 GSa/sec it doesn’t take long :>) I am an engineer and not a statistician so I used the tools I am familiar with to devise a significance test. The reasoning was simply. Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results. I do not see the flaw in the logic, nor in the method for testing the assertion. A s”simplistic” test is not necessarily a flawed test.

    Perhaps the problem is revealed in your mischaracterization of my use of the AR model here:
    “For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.”

    Nowhere do I assume that it is AR data that we are seeing, nor do I anywhere attempt to prove that it is AR data we are seeing. The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record. If the HD fit were insignificant, it seems to me that one should easily achieve similar GOF results on the AR process data. As was shown in my post, this is not the case. It seemed an interesting result so I shared it.

    As for halving the data, extracting a 170+ year of data from a 55 point record looks pretty hopeless. The HD algorithm fails to converge, as does the AR generated sequences 95% of the time. I may fool around with it some more if I get some time. As for hindcasting, I posted a link to a fifty year hindcast on the monthly data prior to your post.

    Anyway, take it for what its worth (perhaps nothing). I need to get back to my day job.
    Cheers.

  174. I, for one, am not going to bet the house on the “predictions”. For me, I think that tomorrow is going to be pretty much like today, and this year is going to be pretty much like last year… So far, this seems to fit the data in my lifetime (79 years, thank you.).

  175. @ Jeffery S. Patterson

    If you happen to be reading this could you reprocess it after first passing it thru a high pass filter set at zero to remove the offset .. I would love to see what it shows?

    It’s the same trick you do in your field to remove a DC voltage offset if my language is confusing.

  176. A useful reference that might bridge the gap between electrical engineering digital signal processing and climatology (hydrology and hydraulics) is “Stochastic Processes in Hydrology” by Vujica Yevjevich. The book can be obtained from several sources, including these:

    http://books.google.com/books/about/Stochastic_processes_in_hydrology.html?id=uPJOAAAAMAAJ

    http://trove.nla.gov.au/work/21245724?q&versionId=25362456

    http://www.barnesandnoble.com/w/stochastic-processes-in-hydrology-vujica-m-yevjevich/1001110582?ean=9780918334015

    A tribute to Prof. Yevjevich is located here:

    http://www.engr.colostate.edu/ce/facultystaff/yevjevich/Yevjevich_index.htm

  177. Ulric Lyons says:
    September 11, 2013 at 6:00 pm

    Willis Eschenbach says:

    “The second is that as far as anyone has every determined, the climate is chaotic …”

    You should have dropped in on me when you were in the West Country, I could have shown you how I do deterministic weather forecasts. At the noise level it is not chaotic, so the sum of weather at the climatic scale cannot be either.

    Are you a rich man, Ulric?

    Because if not … why not? Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions.

    Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.

    w.

  178. Jeff Patterson says:
    September 11, 2013 at 7:53 pm

    Willis Eschenbach says:
    September 11, 2013 at 2:59 pm

    Hmm, I’ve seem to have struck some nerve with my hobby horse, that was not my intent.

    Struck a nerve? No, you’ve just done a poor and incomplete analysis, for the reasons I stated. If you want to pretend that somehow irritated people, sorry … the objections were scientific.

    w.

  179. Jeff Patterson says:
    September 11, 2013 at 7:53 pm

    Although I recognized the short record problem, it’s not one I normally encounter (if we need longer records we simply acquire more data – at 10 GSa/sec it doesn’t take long :>) I am an engineer and not a statistician so I used the tools I am familiar with to devise a significance test. The reasoning was simply. Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results. I do not see the flaw in the logic, nor in the method for testing the assertion. A s”simplistic” test is not necessarily a flawed test.

    Indeed, you are correct—you’re not a statistician. While your reasoning is “simplistic”, it’s also wrong.

    Perhaps the problem is revealed in your mischaracterization of my use of the AR model here:

    “For example, in his case he is ASSUMING that AR data is what we are actually looking at, so he uses AR data to test his theory, and then he proves that if it is AR data we are seeing, then his result is significant.”

    Nowhere do I assume that it is AR data that we are seeing, nor do I anywhere attempt to prove that it is AR data we are seeing. The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record. If the HD fit were insignificant, it seems to me that one should easily achieve similar GOF results on the AR process data. As was shown in my post, this is not the case. It seemed an interesting result so I shared it.

    Huh? You’ve admitted you used an AR model for your Monte Carlo test … but since you are not a statistician, you failed to realize that the choice of the model for the Monte Carlo test is a make-or-break decision for the validity of the test. You can’t just grab any data with similar bandwidth and variance as you have done and claim you’ve established your claims, that’s a joke.

    As for halving the data, extracting a 170+ year of data from a 55 point record looks pretty hopeless.

    Bad news on that front, Jeff. Extracting a 170+ year cycle from a 110 year dataset is equally hopeless. Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.

    The HD algorithm fails to converge, as does the AR generated sequences 95% of the time. I may fool around with it some more if I get some time. As for hindcasting, I posted a link to a fifty year hindcast on the monthly data prior to your post.

    Sorry, not impressed. Do it properly or not at all.

    Anyway, take it for what its worth (perhaps nothing).

    I see you have a keen eye for the estimation, not only of your own statistical abilities, but of the value of your work.

    I need to get back to my day job.
    Cheers.

    Thanks for the reply, Jeff. Truly, you seem to be totally at sea in this particular corner of signal processing. It’s a recondite corner, with many pitfalls for the unwary, and full of poor data.

    In addition, there is general agreement that the signal is chaotic in nature (I gave a reference above). If your analysis is correct, it is deterministic … which is a huge claim that seems to have escaped your notice. To show that it is not chaotic, I fear you need more than an un-testable (by your own admission) extraction of a 170-year cycle from 110 years of data.

    All the best,

    w.

  180. JP: “Although I recognized the short record problem, it’s not one I normally encounter …”

    This is a key problem with using DSP techniques and experience from other fields like electrical engineering and acoustics, where adequate samples can usually be made available. (And if not this is recognized. )

    Most climate data is fundamentally inadequate to determine the nature of longer term variability. Whether century scale change is periodic, linear AGW , randow walk, or some other stochastic process can not be determined from the available data.

    The big lie is that the IPCC is now, apparently, 95% certain that it can.

    The last 17 years is the proof that they can’t.

    That they become more certain in the face of increasing discrepancy shows how detached they have become from scientific reality and how dominated this intergovernmental body is preconceived policy objectives.

  181. In the end, Digital signal processing, is just following a mathematical algorithm; applying it to a set of numbers.
    So it differs from computing an average, only in that the mathematical algorithm is different.

    You can take the very same set of numbers, and calculate the average, using that particular algorithm, or you can do the more complex DSP algorithms, and get something else from exactly the same set of numbers.

    Well of course, you can also apply the same mathematics to the set of numbers you can find in your local telephone directory and get an average, or whatever the DSP algorithms produce.

    The mathematics is quite rigorous; but that doesn’t mean the result has any meaning.

    In the case of DSP, there usually is a starting assumption that there IS a “signal” there to be found in the set of numbers. Some procedures make use of prior knowledge of what the signal is supposed to be, and when it is supposed to be present, so you aren’t looking for something at times when it is not there, so you just gather noise.; such as locking onto a Loran C signal or these days, a set of GPS signals.

    But if you apply these methods to sets of numbers which are acquired without regard to the rules of sampled data systems (which all our measurements are), then there may be no signal there to acquire.

    That’s the problem with “climate data”. It doesn’t even come close to satisfying the Nyquist criterion, particular for spatial sampling of Temperatures, and the common method of Temporal sampling, also fails on that score. A daily min max Temperature sampling, only suffices, if the diurnal Temperature variation is strictly sinusoidal with a 24 hour period. Now in climatism, the aim is not to recover the continuous Temperature function for processing, but just to get the average, which is the zero frequency component of the spectrum. You can’t even recover the average, if the daily Temperature cycle has even a second harmonic component, let alone any higher frequency components, such as the results of cloud cover variations during the day.

    So the mathematics may be quite elegant; but the input “data” is just garbage.

  182. Jeffrey – when I look at the FFT result, one thing strikes me – there is obviously periodicity, but the picture looks reminiscent of a response containng reverberation or echoes. Unfortunately, I don’t have easy access to cepstral processing, but I would be fascinated to see whether there were peaks in the cepstrum. There may be “impulse responses” present that are triggered by discrete “events”, and there may be many overlaid so that they convolve in the spectrum. Just my feeling, as there is no real trace of actual peaks in the spectrum. There are maxima, but relatively broad. It’s the minima that interest me…

  183. Jonathan Abbott says: September 11, 2013 at 6:07 am “Straw man? Valid?”
    Jeffery has given adequate qualifications and caveats for the treatment of data this way.
    He notes that it could be mere curve fitting, then adds an extra test to show, to put it another way, that there is a high probability that this approach, used by others, would give a similar result.
    One could agree with you that curve fitting has some weaknesses, because in the future case it cannot cope well with large, random events perhaps like the 1998 anomaly in temperature. But that is the case with all projections. For a short-term projection, this analysis beats most weaknesses of a linear least squares as is often used.
    Personally, I would not push the analysis as far as he did, but that is a preference. Like Nick Stokes, I have a problem with projecting partial cycles; but Jeffery has incorporated this point in his caveats.
    My worry would be that the temperature record has been adjusted in arbitrary and capricious ways that have the ability to distort the properties of the calculated curve. For instance, the ‘adjusted’ warming of the early years of the record is bound to have an effect.

  184. Willis Eschenbach says:

    “Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions. Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.”

    Help me market it and I’ll cut you in then. I know I have the product without any doubt.

  185. @george e. smith says:
    September 12, 2013 at 1:49 am

    So the mathematics may be quite elegant; but the input “data” is just garbage.

    You are right but you cheated and gave him the answer I wanted him to work it out.

    I will give him credit he is at least thinking outside the usual box that climate science seems to be stuck in.

    JP there is a range of techniques for non linear time invariant systems which would are far more suitable for doing the sort of analysis you are trying :-)

  186. I talked to a thermo engineer from Intel yesterday and he surprised me when he told me that 40-50% of the cooling of a small phone in his pocket was via radiation (hence a surface optimized for emissivity). That’s a lot, generally we don’t see more than about 10-15% of the cooling effect via radiation. Of course, outside the package, we don’t get much benefit from conduction and the smaller the enclosure…well, of course, the less area there is for heat spreading. When it’s your job to cool something (or, conversely, to heat something) and radiation becomes an important part of toolbox, that’s truly a game-changer. That’s why guys like me get the big bucks, I guess.

    It’s unfair to expect thoughtful colleagues like Mr. Courtney to consider the balance between CO2 backradiation heating and CO2 cooling via convection…or to realize that if a projected temperature anomaly includes negative numbers, that indicates the theoretical span includes cooling correlated with increasing CO2 concentration. It’s all good, carry on, my friends.

  187. This implies that we should have experienced a period of significant cooling over the last decade. Whereas in fact temperatures have been flat. Is this where the missing heat went?

  188. @Rabbit @Matthew R Marler: Oops, it cuts out HTML. I did an SSA for GISS NH and SH separately. (There’s an R package for this: Rssa.) The Trend graph is based on the first two eigenvalues, and the Season1 graph is based on the second two eigenvalues.

    Interestingly, both hemisphere’s Season1 appear to be damped oscillations. Perhaps artifacts of GISS processing. (I’ve had discussions about GISS in other forums, and especially the SH is whacky, since they smooth over 1200km. When Antarctica came online in the 1950’s you can detect a large change in volatility.)

  189. Pete Brown:

    Your post at September 12, 2013 at 6:51 am asks

    This implies that we should have experienced a period of significant cooling over the last decade. Whereas in fact temperatures have been flat. Is this where the missing heat went?

    Nice try, but no coconut.

    The hypothesis of discernible warming is wrong and Trenberth’s “missing heat” is one of the many reasons we know it is wrong. Clutching at straws about why it is missing is merely desperation by warmunists trying to stay afloat when they are attached to the sinking AGW-scare.

    There is no sign of Trenberth’s missing heat in the depths or how it could have got there. I think that if the “missing heat” ever existed then probably went the other way and by now it has passed Alpha Centauri. But so what? The important point is that it is missing and not why it cannot be seen.

    Richard

  190. @Willis: “Indeed, you are correct—you’re not a statistician.”

    Neither are you.

    Neither am I, for that matter, though you and I are both avid students of statistics. Jeff’s method has issues, but I’d point out that he’s more knowledgeable and has been more rigorous than the average WUWT poster. Considering your confusion on basic time series analysis a while back (“linear trend”), I’d suggest that you should be more gracious in your criticisms. Jeff has certainly been reasonable in his tone, and deserves the same from you.

    As a high school physics teacher once told me, it gets frustrating teaching the same course year after year and yet every single year the students still come into the class not yet understanding it. ;-) Curve-fitting is a problem, but don’t take out your frustration for past curvologists on Jeff.

  191. Jeff Patterson: I’ve seem to have struck some nerve with my hobby horse,

    Not so. Willis Eschenbach is a vigorous critic of most modeling that gets presented here. Like me, he mocked the pretended precision of the estimated period of “170.7”. I don’t think data halving is as useful a technique as waiting for the next 20 years of out-of-sample data. I think he missed your qualification that if the system is chaotic your model was useless for prediction. I think your defense of your choice of null distribution for null hypothesis testing is adequate for now.

    I think it will be a miracle if your model proves to be reasonably accurate over the next 20 years, and I think on the whole it is less credible than Vaughan Pratt’s model (which also will, in my guess, require a “miracle” to be accurate over the next 20 years), but I put in in what I call “the pile of active models”. Because you provided the code, anyone who is interested can test.

  192. @Geoff Sherrington 3:21 am. +1 A balanced evaluation.

    @Greg Goodman 12:23 am +5 Succinct. A reposte to remember for AR5.

    Most climate data is fundamentally inadequate to determine the nature of longer term variability. Whether century scale change is periodic, linear AGW , random walk, or some other stochastic process cannot be determined from the available data.

    The big lie is that the IPCC is now, apparently, 95% certain that it can.

    The last 17 years is the proof that they can’t.

    @Willis Eschenbach 12:18am
    Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.

    Amen! That’s why the BEST scalpel is folly. It turns a low pass signal into a band pass signal. At worst, it preserves drift as signal and discards vital recalibration events.

  193. Wayne: @Rabbit @Matthew R Marler:

    Cute. What was the reference to “Science”? Did that article describe the method?

  194. @Willis Eschenbach 12:18am
    Extracting a 170+ year cycle from a 110 year dataset is equally hopeless.

    You cannot do it with an FFT. You can and should recognize the possibility there is a dominant low frequency in the data that the tool cannot adequately detect.

    Take an Excel function describing a partial sine wave with a linear ramp.
    =AmpS*SIN(Phase)+LinSlope*(Year-2000)+Y2KIntercept
    With Year=(1900:2012)
    AmpS = 0.4 (deg C)
    Phase =(Year-PhaseOrig)/CycLen*2*PI()
    PhaseOrig = 1965
    CycLen = 175 (years)
    LinSlope=0.001 (deg C/yr) (= 0.1 deg C/ century)
    Y2KIntercept = 0.3 (deg C)

    If you plot it up, (thick blue line on)
    Excel chart picture: http://i43.tinypic.com/16j3ndj.jpg
    If you do get a least squares linear fit you get.
    Y = 0.0098*(Year-2000) + 0.3 (almost 1.0 deg C / century)
    And an R^2 of 0.9328. (thin blue line)

    In truth, there is only a 0.1 deg C / century linear trend super imposed upon a clean sinusoidal -0.4 to +0.4 deg with a wave length of 175 years that the measurements caught between phase (-2.33 and 1.68 radians) (-133 and 96 deg) (thick blue curve.)

    If you remove the baseline between the 1900 and 2012 end points (maroon dashed line.), as per Jeff, you will get a thick maroon residual that you would do FFT and harmonic analysis. By visual inspection, I think most people can see the problem. The removal of the baseline has the following effects on the residual curve:

    1. It shortened the cycle length to maybe 120 years instead of 175,
    2. greatly reduced the amplitude of the Sinusoid to about 0.2 instead of 0.4,
    3. introduced high frequency ringing caused by the mismatch of slopes at the end points,
    4. and finally, baked in a 0.71 deg/century linear trend created by the choice of baseline, seven times higher than what is in the original [known], under-measured signal.

    Such pitfalls make me want to give the pit wide berth.

  195. Clarification: my 9:44am is in support of Willis’s 12:18am
    Extracting a 170+ year cycle from a 110 year dataset is equally hopeless

    I just wanted to point out that just because a tool cannot see (like the FFT) a longer wave length or cannot uniquely determine a long wave length (harmonic decomp), doesn’t mean we should blind ourselves to the possibility of long wave length cycles in the system response of the climate. Indeed, at geologic scales with mutiple ice ages and interglacial periods, there likely are some. But our temperature records are too short, too noisy, and too adjusted to have any hope of determining any century scale or longer wave lengths in the earth’s system’s response.

  196. “””””””…….LdB says:

    September 12, 2013 at 5:54 am

    @george e. smith says:
    September 12, 2013 at 1:49 am

    So the mathematics may be quite elegant; but the input “data” is just garbage.

    You are right but you cheated and gave him the answer I wanted him to work it out………””””””””

    So “I” cheated ??

    No, and I also did not assume, that Jeff was in any need of being given any answer. My post on this thread, in no way conveys any criticism of Jeff or his methods. It is always refreshing to listen to someone tell what he does in his cubicle.

    Having been an Agilent (read REAL HP) employee myself, in a past life, I fully understand, that those blokes have to know that their stuff works.

    But if you send a bunch of researchers out to a bunch of places, and ask them to count and tabulate the number of animals per square meter, or maybe Hectare; animal being restricted to only those bigger than an ant, and you record that data for 150 years; then Jeff could apply his methods to your data set, and come up with some output that you could feed to the WWF (no not the Hulk Hogan crowd); and they could all go gaga over it, and ask the UN for funding to rectify the problems discovered.

    But don’t blame Jeff’s analytical tools; he didn’t know a priori (perhaps), that your data is garbage.

    Of course the number of animals per hectare; regardless; or irregardless, as the case may be, of the size, and species of such critters; contains about as much information as the Temperatures, taken at quite random, unrelated times, at quite randomly placed, and also unrelated sparse locations around the globe.

    Yes you can do a FFT on the numbers, you could even expand them as an orthogonal set of Tchebychev polynomials, or Bessel functions. Well there’s nothing wrong with the maths; make it as simple as calculating the average, or maybe the median. The result is valid, but still garbage, because the data was.

    Now Mother Gaia does it right; even proper, or properly; because she has in effect a Thermometer in each and every molecule, so she always knows what the Temperature is; and it always is what it is supposed to be. But she is not going to communicate the answer to us.

  197. This is an interesting analysis and of course there are a lot of reasonable rational caviats that are mostly valid and necessary for rigorous analysis. Still that should not blind us to the possiblity of being able to make some sense of imperfect data.

    Jeff Patterson says:
    In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data. if the condition is not met, the high frequency components masquerade as low frequency components in a process referred to as aliasing. That is why we remove the high frequency components by filtering before decimating (re-sampling) the data.

    Willis Eschenbach says:
    “The second is that as far as anyone has every determined, the climate is chaotic

    @Willis Eschenbach Even Chaos can have an underlying harmonic structure (for example, fractals, Strange attractors, etc.)

    Willis Eschenbach says:
    Me, I don’t trust any cycle I find in climate data that’s more than a third of the length of my dataset, and I would strongly encourage you to do the same.

    That is correct and valid when you can get the data. For dynamic analysis of flight test data for flutter prediction we use a sample rate of at least 5 times the highest vibration frequency expected, so this is in line with standard practice.

    “And as also pointed out proper signal filtering before sampling is necessary to avoid problems like aliasing. Also I have very little confidence in the manner in which global temperatures are averaged. (Since the earth loses heat only by radiation an average of the 4th power of (absolute scale) temeratures makes more sense but that makes a very small difference compared to the distribution of source stations, or time of day of recording or using only min and max etc.). So the whole data set is suspect from the start but there can still be a harmonic signal in it.

    However it has been useful to me in the past to have data that was not correctly processed. I was involved in the investigation of an aircraft accident that lost pitch control due to flutter and, as luck would have it there was a primitive recording device on board the aircraft. Signals from the AHRS (Attitude and Heading reference system) were recorded without pre-filtering onto a laptop computer at a relatively low sample rate (about 16 Hz I think). When analysing the data at the point of the failure there was a 5.5Hz signal that indicated the flutter that caused the failure. By calculating the frequences that could alias to 5.5 Hz the possibilities could have been 11 Hz 22 Hz or some higher Frequency. By putting the pilots on a seat on a shaker (that we use for ground vibration test excitation) at the possible and plausible frequencies the pilots could identify the shaking they felt as the 11 Hz. This was also in line with one of the resonant frequencies or the tailplane identified in a ground vibration test of the aircraft Having an external confirmation is necessary to identifying which frequency is aliased (although , as also pointed out, confirmation bias is always a possibility)”

    Looking for high frequencies in an aliased signal is at the opposite end to the spectrum to looking for low frequencies in too short a period may not be exactly similar but I do recognise the fact that there could be a signal and I have spent some time myself curve fitting cycles to the satalite temperature record so this analysis intrigues me. I am also intriqued that the 170 point whatever cycle is close the the 179 year solar baricentre cycle. Even though a recent WUWT post discounted any planetary cycle influence on the solar cycle, It may be, as Willis pointed out, there is not enough data length to cover enough cycles sufficiently to recognise the signals properly, or that there is more noise than signal. Anyway I am biassed in appreciation for Jeff Patterson’s analysis since it is an approach that is familar to me.

  198. @Matthew R Marler says: “Cute. What was the reference to ‘Science’? Did that article describe the method?”

    ? I didn’t make any reference to “Science”, and I’m not sure what’s “cute”. @Rabbit asked about doing an SSA and I found a library to do so and did so. I haven’t managed to pull out frequencies yet — its internal data structure is obtuse and requires a couple of extra layers of unwrapping — but I thought the graphs would be interesting to compare to Jeff’s straight FFT approach.

  199. Speaking about giving the answers; there’s a “Science” web site out there, where consensus reigns supreme.
    Anyone can ask a question. “Are the ends of a wormhole polarized the same or opposite ?”
    Anyone can answer the question. No ! the two ends of a wormhole have the same polarity; but it can be up or down.
    Well Wikipedia says they are always opposite !

    If you give an answer different from Wikipedia, you are in trouble.

    Then anybody can vote for or against your answer; to determine its popularity. No restrictions on the credentials or lack thereof, of the critics. And so you get a score.

    Now all the kids taking 4-H club Quantum Chromo-dynamics classes, like to post their homework problems there so they don’t have to look it up in Wikipedia.

    You are not aloud to answer someone’s homework problem; which some referee designates as an “hw” problem. Well you can tell the “OP” which is code for the chap who asked the question; go to the Stanford Science library, and look it up in Born and Wolfe; but you can’t tell him/er that the answer is 12.

    So someone; the “OP” put up a network of resistors, and asked for the equivalent resistance between two nodes. So the correct answer is; go to Wikipedia and look up Kirchoff”s Laws.

    But this network has a problem. The values for the separate resistors, are not industry standard (RMA) values. So it had numbers, like 40, 120, 100 Ohms, instead of 39, 120 and 100, which are good numbers.

    It was quite obvious from the numbers in the problem that the examiner, fully intended the student (New Zealand white rabbit farmer), to figure it out in his head, not putting pen to paper, till it came time to write down 32 Ohms.

    I did that and gave him the answer; 32 Ohms. That drew the condemnation of some twirp younger than my children, who gave some elaborate algebra, that rabbit farmers wouldn’t understand; and the consensus fan base voted me a -2 popularity score.

    I once set an exam question not unlike that one, for a pre-med first year Physics class (of 200 student doctors to be).

    Part (a) Write down Kirchoff’s laws.

    Part (b) Use Kirchoff’s laws, to derive the balance condition for a Wheatstone Bridge.

    Part (c) Determine the current flowing in the resistor R7 in this network.

    Now “this network” consisted of seven resistors connected between a total of four nodes. In two instances, there were simply two resistors in parallel between a pair of nodes. A Voltage of 10 Volts, was applied between nodes (1) and (3) and resistor R7 of 1,000 Ohms, was between nodes (2) and (4).

    So if you figured out the parallel pairs (in your head) reducing the network to five resistors, and imagined the four nodes, to be in a diamond configuration, instead of the rectangular boxy shape I drew them, you would immediately see that four of the resistors comprised a Wheatstone Bridge; (please do part (b) of the problem before doing part (c). and a simple mental note would show that with those values, the Wheatstone bridge was indeed balanced; Please do Part (a) of the problem before doing part (c).

    So the correct answer was zero current.

    Out of the 200 students who sat the exam, not a single one figured out that the network was indeed a Wheatstone bridge, let alone that it was balanced.

    About 50 students laboriously worked through the Kirchoff’s laws equations, about 45 got the arithmetic correct to give the correct answer; the other five worked the equations properly, and goofed on the arithmetic.

    150 of them got parts (a) and (b) correct . Nobody received the full 20 points for the question, because none of them seemed to understand that the theory part might be related to the simple problem.

    Anyhow, this consensus science site is a gas, just reading some of the goofy questions.

    But you can get your favorite string theory questions answered there.

    On bright student, asked if anybody had ever studied whether the universe was moving; and if so, in which direction.

    So these folks are the future of science, unless some rationality eventually prevails in our education systems.

    Nine (9) of my class of 200 students of excellent doctor/dentist/veterinarian candidates, got accepted by the medical school, which at the time, was at another campus, in another (smaller) town.

    I elected to transfer over to Industry, from academia.

  200. Correction to: 10:39 am
    … just because a tool cannot see … a longer wave length or cannot uniquely determine a long wave length (harmonic decomp), doesn’t mean we shouldn’t should blind ourselves to the possibility of long wave length cycles in the system response of the climate. ….
    In other words, always be aware of what you cannot (or did not) detect.

    [Fixed. -w.]

  201. The oft-repeated idea that a noisy 113-yr temperature record doesn’t contain sufficient information to determine the parameters of a 170-yr periodic cycle is somewhat misguided. Simple heterodyning with the available record is entirely adequate for that purpose–provided that the fundamental period is known a priori on the basis of physics. That is the proven basis of Munk’s “Super-resolution of the Tides.” Without such a priori knowledge, however, even a 170-yr record would not be adequate, since the resulting raw periodogram would vary as chi-squared with only two degrees of freedom. In the face of spectral frequency smearing by the the DFT, there would be scant grounds for any reliable identifcation of any periodic components

    The real culprit.in various simplistic models of yearly average temperature series is the wholly unjustified presumption that there ARE strictly periodic components (spectral lines) to be found in the empirical record That is the common blunder made not only here, but with Pratt’s multi-decadal “sawtooth” or Scafetta& Lohle’s two-sinusoid oscillation. In fact, what we have is stochastic oscillations with various bandwidths, which give the appearance of “trends” and “periodicities” in records too short for scientific purposes.

  202. Wayne: ? I didn’t make any reference to “Science”, and I’m not sure what’s “cute”.

    One of the links that you supplied had a series of ppt-type pages displaying graphs and such, and one of those pages was a header from a page of “Science”. The other link was a single page.

  203. I did not read the code in entirety, so I wonder: Does it consider the squares of temperature anomalies from a horizontal line or a best-fit whatever, or does it consider unsquared anomalies?

    It appears to me that Fourier works with unsquared data. Using squared anomalies from a best-fit linear trend or whatever would give more weight to the the 1878 and 1998 El Ninos than Fourier does, and less weight to the broader peak centered around 2004-2005. The ~1910 dip and the 1950s dip, if they are squared, could cause a shorter period determination for appears to me as determination of the 3rd harmonic.

    I have more faith in Fourier for spectral analysis.

    My eyeballs tell me that the strongest periodic component in HadCRUT3, of periods near or less than 200 years, has a period around 60-70 years.

    I am expecting the warming from the early 1970s to ~2005 to be roughly repeated, and to only slightly greater extent, after about 30 years of 5-year-smoothed global temperature roughly stagnating or very slightly cooling. (Or trend of 1-year or shorter period temperature going maybe 34-35 years without warming, if the start time is sometime in 1997 when a century-class El Nino was kicking up or about to kick up.)

  204. @ Matthew R Marler: I supplied two links to two graphs of the results of SSA. Evidently there are back and forward arrows to other pictures I’ve linked to on this site before, which I didn’t know you’d see. The other “slides” had nothing to do with the current discussion.

    I’ll have to delete old slides each time I post new ones. Stupid site.

  205. For anyone looking for a simple assessment.

    Average global temperature history since 1975 is like a hill. We went up the hill from 1975 to 2001 where the average global temperature trend reached a plateau (per the average of the five government agencies that publicly report average global temperature anomalies). The average global temperature trend since 2001 has been flat to slightly declining but is on the plateau at the top of the hill. Claiming that the hill is highest at its top is not very profound. The temperature trend has started to decline but the decline will be slow; about 0.1 K per decade for the planet, approximately twice that fast for land areas.

    A licensed mechanical engineer (retired) who has been researching this issue (unfunded) for 6 years, and in the process discovered what actually caused global warming and why it ended, has four papers on the web that you may find of interest. They provide some eye-opening insight on the cause of change to average global temperature and why it has stopped warming. The papers are straight-forward calculations (not just theory) using readily available data up to May, 2013. (data through July made no significant difference)

    The first one is ‘Global warming made simple’ at http://lowaltitudeclouds.blogspot.com It shows, with simple thermal radiation calculations, how a tiny change in the amount of low-altitude clouds could account for half of the average global temperature change in the 20th century, and what could have caused that tiny cloud change. (The other half of the temperature change is from net average natural ocean oscillation which is dominated by the PDO)

    The second paper is ‘Natural Climate change has been hiding in plain sight’ at http://climatechange90.blogspot.com/2013/05/natural-climate-change-has-been.html . This paper presents a simple equation that, using a single external forcing, calculates average global temperatures since they have been accurately measured world wide (about 1895) with an accuracy of 90%, irrespective of whether the influence of CO2 is included or not. The equation uses a proxy which is the time-integral of sunspot numbers (the external forcing). A graph is included which shows the calculated trajectory overlaid on measurements.

    Change to the level of atmospheric CO2 has had no significant effect on average global temperature.

    The time-integral of sunspot numbers since 1610 which is shown at http://hockeyschtick.blogspot.com/2010/01/blog-post_23.html corroborates the significance of this factor.

    A third paper, ‘The End of Global Warming’ at http://endofgw.blogspot.com/ expands recent (since 1996) measurements and includes a graph showing the growing separation between the rising CO2 and not-rising average global temperature.

    The fourth paper http://consensusmistakes.blogspot.com/ exposes some of the mistakes that have been made by the ‘Consensus’ and the IPCC

  206. Dang DSPs have taken all the eye candy glamor out of electronics. Instead of little silver plated coils, silver mica capacitors, crystal filters, cavity resonators, lots of little things to tune and a bunch of gold plated goodies to tie everything together in an artful manner, all you see now is a black epoxy stamp size wafer with a bunch of wires coming out of all four sides, processing signals in the digital domain in ways that are nearly impossible in analog.

  207. richardscourtney:
    Regarding your post at September 12, 2013 at 7:08 am

    Thanks for your response. To be fair, I was asking a question though, not making a point. I don’t have any particular need to grasp any straws. I notice though that you don’t seem to have quite addressed the point, despite seeming very confident about it. So I wonder if we could revisit this?

    My thinking is this:

    Let’s assume everyone agrees that CO2 is a greenhouse gas, and will have a slight warming effect, all other things being equal.
    Let’s also assume that everyone agrees that all other things are NOT equal, and that natural variability plays a significant role in global climate relative to the impact of CO2.
    One upshot of this is that it is much harder to detect the impact of CO2 against natural variability than climate extremists would have us imagine. The impact of CO2 relative to natural variability is much smaller than claimed.
    But just how reassuring is that? No-one can say that it is not warmer now than it would otherwise have been, can they? How would we possibly begin to establish that empirically?

    Again, I’m asking a question here, not trying to make a point.

    Also, if it is accepted that CO2 is a greenhouse gas, and will have a slight warming effect, all other things being equal, then any theory which fails to take account of this – by only including natural variability for example – is also wrong. Isn’t that true?

    Grateful for any comments on this…

  208. Pete Brown:

    I am replying to your post addressed to me at September 13, 2013 at 12:43 am

    http://wattsupwiththat.com/2013/09/11/digital-signal-processing-analysis-of-global-temperature-data-suggests-global-cooling-ahead/#comment-1415817

    If I understand you correctly, then what you are really asking about is climate sensitivity. If so, then the magnitude of climate sensitivity is of importance and – in the context of your interest – the precise details of WHY climate sensitivity has that magnitude are secondary.

    If I have misunderstood you then please say because I am replying to what I think you are saying and not avoiding anything.

    I provide two answers. Firstly, if you use the WUWT Search function then you will find much on climate sensitivity. Secondly, my view on the matter is as follows.

    I am convinced that increased atmospheric CO2 concentration will result in some rise in global temperature, but I am also convinced any such temperature rise would be too small for it to be discernible and, therefore, it would only have an abstract existence. I explain this as follows.

    Before presenting my argument, I point out I remain to be convinced that human emissions are or are not the cause – in part or in whole – of the observed recent CO2 rise. However, the cause of a rise in atmospheric CO2 concentration is not relevant to the effect on global temperature of that rise.

    My view is simple and can be summarised as follows. The feedbacks in the climate system are negative and, therefore, any effect of increased CO2 will be too small to discern. This concurs with the empirically determined values of low climate sensitivity obtained by Idso, by Lindzen&Choi, etc..

    In other words, the man-made global warming from man’s emissions of greenhouse gases (GHG) would be much smaller than natural fluctuations in global temperature so it would be physically impossible to detect the man-made global warming.

    Of course, human activities have some effect on global temperature for several reasons. For example, cities are warmer than the land around them, so cities cause some warming. But the temperature rise from cities is too small to be detected when averaged over the entire surface of the planet, although this global warming from cities can be estimated by measuring the warming of all cities and their areas.

    Similarly, the global warming from man’s GHG emissions would be too small to be detected. Indeed, because climate sensitivity is less than 1.0°C for a doubling of CO2 equivalent, it is physically impossible for the man-made global warming to be large enough to be detected. If something exists but is too small to be detected then it only has an abstract existence; it does not have a discernible existence that has effects (observation of the effects would be its detection).

    I hold this view because I am an empiricist so I accept whatever is indicated by data obtained from observation of the real world.

    Empirical – n.b. not model-derived – determinations indicate climate sensitivity is less than 1.0°C for a doubling of atmospheric CO2 equivalent. This is indicated by the studies of
    Idso from surface measurements

    http://www.warwickhughes.com/papers/Idso_CR_1998.pdf

    and Lindzen & Choi from ERBE satellite data

    http://www.drroyspencer.com/Lindzen-and-Choi-GRL-2009.pdf

    and Gregory from balloon radiosonde data

    http://www.friendsofscience.org/assets/documents/OLR&NGF_June2011.pdf

    Climate sensitivity is less than 1.0°C for a doubling of atmospheric CO2 concentration and, therefore, any effect on global temperature of increase to atmospheric CO2 concentration only has an abstract existence; it does not have a discernible existence that has observable effects.

    In the context of this thread, the implication of small climate sensitivity is that atmospheric CO2 concentration has very small effect and, therefore, may not be capable of discrimination when analysing temperature data.

    Please get back to me if I have failed to address what you wanted or I have not been clear.

    Richard

  209. Pete Brown:

    re your comment to me at September 13, 2013 at 12:43 am.

    I have provided a reply but (for some reason) it is stuck in moderation and should appear in an hour or so. Whatever the reason for this delay, it is not that I am avoiding provision of an answer to you. Please be patient.

    Richard

  210. nice piece of work. what it shows of course is that the data supports very little correlation with rising CO2 levels at all..sop co2 sensitivity is very low indeed, and almost all the warming is down to ‘something else’

  211. richardscourtney:

    Richard

    Thanks. I am familiar with the concept of climate sensitivity, but very interesting to hear your views. Thanks for your time in responding.

    Pete

  212. Ulric Lyons says:
    September 12, 2013 at 3:36 am

    Willis Eschenbach says:

    “Anyone on this planet who could make weather predictions with the accuracy you claim, and at the distance out that you claim, could make millions. Given that you haven’t done so, I fear that I greatly doubt the claims that you so confidently put forward.”

    Help me market it and I’ll cut you in then. I know I have the product without any doubt.

    And I greatly doubt that you have the product. What a team!

    w.

  213. Wayne says:
    September 12, 2013 at 7:20 am

    @Willis: “Indeed, you are correct—you’re not a statistician.”

    Neither are you.

    Hey, I’m quoting him, not expressing a second opionion.

    w.

  214. Willis Eschenbach says:
    September 13, 2013 at 7:27 am

    You are missing out twice Willis.

    Scientifically. Monetarily.

    I have seen Ulric’s work from its inception.

    I predict much egg. On many, many faces.

  215. Jeff Patterson says:
    September 11, 2013 at 6:11 am

    Nick Stokes says:

    How can you establish a 170 year periodicity from 113 years of data?

    In theory, you need just two samples per period. Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data.

    I’m not understanding this one at all. Suppose we have two points, one at say (0,0) and one at (1, 0.3).

    My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.

    Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that. For example … what is the shortest period sine wave in the HadCRUT data? And how does that relate to a 170 year cycle?

    Thanks in advance for your answers,

    w.

  216. @ Matthew R Marler: I’ve removed all but the two graphs of the results of the SSA. As I mentioned, the “Trend” is the first two components, and the “Season” is the second two components, which I chose based on a scree plot. The Season for both NH and SH are very similar, and appear to be some kind of damped signal with a roughly 50-year frequency — perhaps a ringing of some sort — which may simply be an artifact of GISS’ processing.

    @Willis: Yes, yes, you were quoting Jeff, but in a way that is clearly a personal judgement. Maybe it’s just me, but you come across as harsh rather than disagreeing. Which might be justified if he were vociferous in his claims, which he wasn’t.

  217. Since a sine wave has two variables, amplitude and phase, all you need to determine it is two points giving two equations which you solve simultaneously. You can in principle generate a sine wave with a period of one year from two points days apart.

  218. Hmmm… There are a load of other cycles seen in longer term weather (including the Bond Event cycles). Several of them look to have a lunar periodicity / connection. I lean somewhat toward the simple mechanism of lunar / tidal ocean mixing modulation (that I’ve linked to here many times in a peer reviewed paper / link). So I wonder if there is a lunar period “close” to this?

    Draconic Month is when the Moon crosses from above to below the ecliptic (or the other way). That period is 18.6 years.

    https://en.wikipedia.org/wiki/Month#Draconic_month

    “Sometimes written ‘draconitic’ month, and also called the nodical month. The orbit of the moon lies in a plane that is tilted with respect to the plane of the ecliptic: it has an inclination of about five degrees. The line of intersection of these planes defines two points on the celestial sphere: the ascending node, when the moon’s path crosses the ecliptic as the moon moves into the northern hemisphere, and descending node when the moon’s path crosses the ecliptic as the moon moves into the southern hemisphere. The draconic or nodical month is the average interval between two successive transits of the moon through its ascending node. Because of the sun’s gravitational pull on the moon, the moon’s orbit gradually rotates westward on its axis, which means the nodes gradually rotate around the earth. As a result, the time it takes the moon to return to the same node is shorter than a sidereal month. It lasts 27.212220 days (27 d 5 h 5 min 35.8 s). The plane of the moon’s orbit precesses over a full circle in about 18.6 years.”

    3 x that is 55.8 years (very close to the 56 year period often called a “60 year cycle” seen in weather cycles – when the Saros cycle returns to over the same 1/3 of the globe …) Now what is 3 x 56? 167.4 years. Or very close to that 170.7 year period found.

    I think what we are finding is the lunar tidal effect on ocean depth and mixing as the moon changes where it is relative to the continents. Above and below the ecliptic. And in line with the sun when particular continents are underfoot. Essentially an interaction of tides with the continents as the moon makes different strength tides with different land lined up.
    Folks often talk about natural ocean oscillations, but could not there be a tidal metronome?…

  219. Willis Eschenbach: My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.

    Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that.

    Right on both counts.

    pochas: Since a sine wave has two variables, amplitude and phase,

    A sine wave has three variables: amplitude, phase and period. The claim that you only need 2 points to estimate a sine wave depends on the assumption that one of those is known.

    To go back to the modeling of Jeffrey S. Patterson, he started by focusing on sine curves. After some smoothing, he used his 112 years of data to estimate the amplitude, period and phase of the fundamental, then nonlinear least squares to estimate the amplitude and phase of each harmonic. His decision to focus on sine curves is totally unprincipled, unless there is strong external evidence that the system has to be periodic: he could have used orthogonal polynomials of high order, wavelets, b-splines, etc. He wrote appropriate caveats: if the system is chaotic instead of periodic, there is no reason to expect his result to have any predictive value. And Willis Eschenbach appropriately critiqued Patterson’s null hypothesis test by pointing out that the “correct” null distribution is not known (that is, the “true” background variation is not known.) But Patterson explained that his choice of a null distribution, which in this field is about all one can be expected to do, unless we require an infinite set of null distributions.

    In my mind, the value of such work is in what comes after: perhaps someone will be stimulated to find, and will find, a physical basis for the fundamental period, and maybe analysis will show that some features of the system produce the harmonics from the fundamental. In statistics, some people refer to work such as Patterson’s as “hypothesis generation”. At least with Vaughan Pratt’s model he had a physical basis for the main monotonic trend, but his estimates of the natural variation were suspect. Nicola Scaffetta is seeking physical drivers in the geometry and gravitation of the solar system. And so on.

  220. This is just curve fitting. No matter how tricky you present it and how many tricks you use, the result is is the same: just curve fitting. Curve fitting CANNOT be used to make predictions. The misconception here is that you assumed that there are some cycles composing the evolution of temperature contained in the data you are using. But that’s just an assumption and is based in nothing.
    You’re just playing with the data and the math, but there’s no physics on your work. Whenever you look for cycles in a data set, you need to be ready to explain the periods you get from it. But for this case you even forced the results and, therefore, the periods of the cycles you obtain are a result of the temporal length of the data set and the methodology you use.
    Sorry, very impressive tricky math but the result is just a nonsense.
    Ask the tide predictors if they can predict tides the way you’re trying to predict temperatures. Of course not.

  221. E. M. Smith: The plane of the moon’s orbit precesses over a full circle in about 18.6 years.”

    3 x that is 55.8 years (very close to the 56 year period often called a “60 year cycle” seen in weather cycles – when the Saros cycle returns to over the same 1/3 of the globe …) Now what is 3 x 56? 167.4 years. Or very close to that 170.7 year period found.

    So a harmonic model with sines of periods 18.6, 55.8, and 167.4 years would produce a result not unlike what Patterson found (that is, appx the same sum of squared residuals), and might be predictive of the future. It’s a mere conjecture, but what isn’t at this point?

  222. Willis Eschenbach says:

    “And I greatly doubt that you have the product. What a team!”

    I agree, there is no currency in it with your attitude, it would be like flogging a dead horse. The last type of person I need to team up with is someone who has looked and failed to find any connections and who also thinks that their opinion on the subject is superior. I am more interested in showing you a set of highly interesting and meaningful connections that can change your mind on the matter. If you refuse to look, it’s not the end of the World for me.

  223. Willis Eschenbach says:
    September 13, 2013 at 7:48 am
    “My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.”

    Nyquist would be very sorry to hear that. Fortunately for us (especially for me – otherwise I’d be digging ditches) he is correct. The bandwidth limitation constrains the maximum slope and amplitude, squeezing your infinite solution space down to just one. Actually sampling at exactly the Nyquist rate splits the spectral energy between the Nyquist bin and d.c. in a manner that depends on the phase of the signal relative to your sample clock and is ill-advised.

    “Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that?

    Because I low-pass filtered the data prior to decimation.

    Thanks in advance for your answers,
    Your welcome

  224. “Huh? You’ve admitted you used an AR model for your Monte Carlo test … but since you are not a statistician, you failed to realize that the choice of the model for the Monte Carlo test is a make-or-break decision for the validity of the test. You can’t just grab any data with similar bandwidth and variance as you have done and claim you’ve established your claims, that’s a joke.”

    If by Monte Carlo test you refer to section entitled “Stochastic Analysis” and not to the section on forecast (where I also use a AR process), please show me where “[I am] ASSUMING that AR data is what we are actually looking at” or where “I attempt to prove that if it is AR data we are seeing”. That section makes no such assumption. The assumption is the one I stated earlier. To wit:
    “Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results.” You call this a joke but fail to reveal the punch line. Come on, give it up. I could use a good laugh even at my own expense.

  225. Juan says:
    September 13, 2013 at 9:39 am
    “This is just curve fitting. No matter how tricky you present it and how many tricks you use, the result is is the same: just curve fitting. Curve fitting CANNOT be used to make predictions. The misconception here is that you assumed that there are some cycles composing the evolution of temperature contained in the data you are using. But that’s just an assumption and is based in nothing.”

    Curve fitting can be used to make predictions given a set of assumptions about the underlying system. The prediction is only as good as those assumptions, which were explicitly stated. The analysis in no way addressed the validity of those assumptions and so the projection should be taken with a grain of salt, unless and until those assumptions (stated simply, that the climate can be modeled as a non-linear response to a period external forcing function). What curve fitting can provide, is a clue as to where to look for validation. In other words, as I have stated elsewhere, any model (curve fit, GCM or WAG) is properly used only in forming a hypothesis, hypothesis which must be validated empirically.

  226. Matthew R Marler says:
    September 13, 2013 at 9:36 am
    “His decision to focus on sine curves is totally unprincipled,”
    Was Fourier equally unprincipled when he showed that any arbitrary waveform can be decomposed into sine waves?

    “unless there is strong external evidence that the system has to be periodic”
    The PSD suggested a harmonic relationship between spectral components. A non-linear system (such as the climate) produces harmonics when driven by a periodic signal. That was the motivating principle.

  227. Matthew R Marler says:
    September 13, 2013 at 9:36 am

    “pochas: Since a sine wave has two variables, amplitude and phase,

    A sine wave has three variables: amplitude, phase and period. The claim that you only need 2 points to estimate a sine wave depends on the assumption that one of those is known.”

    Right. You do need 3 points to specify a sine wave.

  228. “””””…..pochas says:

    September 13, 2013 at 8:20 am

    Since a sine wave has two variables, amplitude and phase, all you need to determine it is two points giving two equations which you solve simultaneously. You can in principle generate a sine wave with a period of one year from two points days apart…….”””””

    So my two points are exactly 24 hours apart in time. First point has a value of +1, second point has a value of -1.

    So what is the frequency (or period) of this sine wave; what is the peak to peak amplitude of the sine wave, and what is the next point in time when the value will be exactly zero ???

    You are the only person I know of, who can answer any one of those three questions.

  229. Jeff Patterson: Was Fourier equally unprincipled when he showed that any arbitrary waveform can be decomposed into sine waves?

    No.

    That was a couple centuries ago. Since then lots of orthonormal bases have been invented. You have provided not a single reason to believe that sines are better for this problem than Chebysheff polynomials or anything else.

    The following is well-expressed: Curve fitting can be used to make predictions given a set of assumptions about the underlying system. The prediction is only as good as those assumptions, which were explicitly stated. The analysis in no way addressed the validity of those assumptions and so the projection should be taken with a grain of salt, unless and until those assumptions (stated simply, that the climate can be modeled as a non-linear response to a period external forcing function). What curve fitting can provide, is a clue as to where to look for validation. In other words, as I have stated elsewhere, any model (curve fit, GCM or WAG) is properly used only in forming a hypothesis, hypothesis which must be validated empirically.

    E. M. Smith has tried to relate your results to lunar cycles. Maybe that will work out.

  230. Jeff Patterson: The bandwidth limitation constrains the maximum slope and amplitude, squeezing your infinite solution space down to just one.

    A sine curve can only be estimated from 2 data points if there is a strong constraint on at least one of the parameters. Here you constrain maximum slope and amplitude. Nyquist sampling is a necessary condition in a constrained problem, not a sufficient condition generally.

  231. Jeff Patterson says:
    September 13, 2013 at 10:24 am

    Willis Eschenbach says:
    September 13, 2013 at 7:48 am

    “My first objection is that I can only draw one straight line between the two points … but I can draw an infinite number of sine waves between those two points.”

    Nyquist would be very sorry to hear that. Fortunately for us (especially for me – otherwise I’d be digging ditches) he is correct. The bandwidth limitation constrains the maximum slope and amplitude, squeezing your infinite solution space down to just one. Actually sampling at exactly the Nyquist rate splits the spectral energy between the Nyquist bin and d.c. in a manner that depends on the phase of the signal relative to your sample clock and is ill-advised.

    Thanks for the reply, Jeff. I fear I still don’t see it, likely my error. To recap, you say that if we have two points, and we know that they are separated by less than half of the shortest cycle in the data, we can only put one sine wave through them. In your words:

    Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data.

    OK, so say we have two points at (0,0), and at (1.3, 0.9), and we know that the shortest period in the data is say 4.2. Here’s a graph of the situation:

    As you can see, I’ve drawn two sine waves through the two specified points, and per your specs, they are separated in time by less than half the shortest period in the data.

    What am I missing?

    “Next, you say “if we know the points are separated in time by no more than half the shortest period” … I don’t understand how on earth we could possibly know that?

    Because I low-pass filtered the data prior to decimation.

    OK. So I assume the shortest period in the data is a year? And you are saying that you can figure out a 170-year cycle from two points that are four months apart?

    Again, what am I missing here?

    w.

  232. Willis:

    I write with affection. It is nearly 1 am here and you are blogging while on holiday with your ladies.

    I repectfully suggest you have other priorities. Your ladies are more important than writing for us. And they will need to to be awake througout the day.

    Richard

  233. Jeff Patterson says:
    September 13, 2013 at 10:44 am

    “Huh? You’ve admitted you used an AR model for your Monte Carlo test … but since you are not a statistician, you failed to realize that the choice of the model for the Monte Carlo test is a make-or-break decision for the validity of the test. You can’t just grab any data with similar bandwidth and variance as you have done and claim you’ve established your claims, that’s a joke.”

    If by Monte Carlo test you refer to section entitled “Stochastic Analysis” and not to the section on forecast (where I also use a AR process), please show me where “[I am] ASSUMING that AR data is what we are actually looking at” or where “I attempt to prove that if it is AR data we are seeing”.

    Sure. It starts when you say:

    To do so, we use the data record to estimate an AR process.

    You are generating a large number of random realizations, but not just of any process—of an AR process … but the problem is, we have no guarantee that the climate data is well-represented by an AR process. That is only an assumption.

    That section makes no such assumption. The assumption is the one I stated earlier. To wit:
    “Asserting that there is no significance to the goodness of fit achieved is equivalent to asserting that HD on any sequence of similar BW and variance would yield similar results.” You call this a joke but fail to reveal the punch line. Come on, give it up. I could use a good laugh even at my own expense.

    The assumption is made when you generated the random pseudo-data to use in the Monte Carlo analysis. For example, you could also “use the data record to estimate” an MA process, or an ARMA process, or an ARIMA process, or a white-noise process … that’s the joke, that you think you’ve exhausted the possibilities by generating an AR process.

    What you call a “stochastic analysis” is known in climate science as a “Monte Carlo” analysis, where you test your assumptions against random pseudo-data … but for the test to be valid, the pseudo-data must be of the same nature as the real data. And for that, your choice of model for the pseudo-data is of critical importance. You can’t just assume it’s say white noise, and claim you’ve proven your case.

    And you haven’t even begun to exhaust the possibilities by just looking at AR data, nor have you justified your choice of AR data as opposed to the many other models that you could use for your Monte Carlo analysis …

    All the best,

    w.

  234. pochas says:
    September 13, 2013 at 8:20 am (Edit)

    Since a sine wave has two variables, amplitude and phase, all you need to determine it is two points giving two equations which you solve simultaneously. You can in principle generate a sine wave with a period of one year from two points days apart.

    Huh? A sine wave has THREE variables—frequency, amplitude and phase.

    w.

  235. Matthew R Marler says:
    September 13, 2013 at 1:05 pm
    “A sine curve can only be estimated from 2 data points if there is a strong constraint on at least one of the parameters. Here you constrain maximum slope and amplitude. ”

    If we’re getting just two samples per period (the nyquist criteria) we know the frequency is the nyquist frequency (i.e.the sample frequency /2). The amplitude and phase can then be derived from the fourier coefficients (assuming just a sine wave, no d.c. term)
    e.g.
    DFT[{1,-1}] = {0,Sqrt[2]}
    D.C. term =0 so phase = 90degs, Freq = Nyquist, Amplitude is correct
    DFT[{1,.9}] = {1.3435, .0707}
    D.C. term =1.34, Freq = Nyquist, Amplitude is the sum of the two terms 1.3+.07=Sqrt[2]
    phase is 2*pi*(d.c. term)/Sqrt[2]

  236. There is a profound reason for preferring sinusoids as an orthonormal basis in modeling geophysical time series: they constitute the first-order analytic solution to many physical problems, especiaily those involving oscillatory behavior. In fact, they often appear in the solution of a very wide class of differential equations known as Sturm-Liouville problems. Offhand, no example of successful modeling of any physical time series by, say, Chebysheff polynomials comes to mind. The essence of the problem here is the presumption of a DISCRETE set of finite-amplitude sinusoids, which individually have U-shaped ordinate histograms, instead of an CONTINUUM of infinitesimally small-amplitude sinusoids that, upon integration over a frequency-band, produce a gaussian random process.

    There are also profound physical reasons for rejecting ideas that the dracionian or Saros lunar cycles produce any harmonic effects upon surface temperatures. In particular, suharmonics require not only a nonlinear system, but one that is resonant with a high Q-factor. The differential eqations of thermodynamics are invariably first-order, which makes any resonance well-nigh impossible. It requires far more than the approximate matching of “periodicities” to make a persuasive physical case for causation.

  237. Jeff Patterson:

    The issue at hand is not the mathematical possibility of determining the amplitude and phase at Nyquist frequency from two sample points. It is the identification of the 170-yr “fundamental” as a DISCRETE sinusoid, rather than (a fairly narrow-band) gaussian process whose power density peaks near that period.

    BTW, the PSD of the GISP2 proxy data, which I’ve analyzed over the last ~8000 years, does not show any truly harmonic sequence of spectral peaks

  238. 1sky1 says:
    September 13, 2013 at 5:10 pm

    “In particular, suharmonics require not only a nonlinear system, but one that is resonant with a high Q-factor.”
    Parmetric non-linearities can produce subharmonics (i.e. an oscillator whose resonance is a function of output voltage). But in any case, couldn’t a sinusoidal driving function produce harmonics (as opposed to sub-harmonics?

    “The differential eqations of thermodynamics are invariably first-order, which makes any resonance well-nigh impossible.” This is interesting and surprising to me. I would have thought transport delays in the many feedback loops could easily cause resonances.

    This is probably naive (when all you have is hammer…) but has anyone ever explored the possibility that the climate is an unstable system injection-locked to the celestial forcing? Such systems are highly stable in the phase domain (phase-locked in essence), exhibit high sensitivity (i.e. gain) to the forcing function and do produce sub-harmonics (this feature is exploited in mixer based frequency dividers which are among the cleanest know to man). Would the climate physics preclude this?

  239. 1sky1 says:
    September 13, 2013 at 5:23 pm
    Jeff Patterson:

    “The issue at hand is not the mathematical possibility of determining the amplitude and phase at Nyquist frequency from two sample points.”

    Yes, we got off into the weeds over my attempt to provide an approachable explanation of the Nyquist criteria. The discussion you referenced has nothing to do with the analysis, or the 170 year cycle which is highly oversampled.

  240. Willis Eschenbach says:
    September 13, 2013 at 4:44 pm

    “What am I missing”

    You actually have three sample points (they also cross at zero) and note that they are not equally spaced. If you picked your intersection points s.t. they were equally spaced in time (we’re assuming uniform sampling here) you’d see that the sine waves would have to be harmonically related (no beat) and only one would satisfy the nyquist criteria.

    BTW, how did you embed the graph? That’s cool.

  241. Willis Eschenbach says:
    September 13, 2013 at 5:04 pm

    “For example, you could also “use the data record to estimate” an MA process, or an ARMA process, or an ARIMA process, or a white-noise process”

    I actually did use an ARMA process and got even better results (lower probaility of fluke), I posted the more conservative of the two. But I really don’t see your point. An AR process is just an IIR filter driven with noise, a MA process is just an FIR filter, an ARMA is the cascade of the two and ARIMA has a pure integration (which doesn’t match the data at all). In any case, they are all just filters. What other effect could the process have that wouldn’t be reflected in the PSD and covariance of the output? If these closely match the observed data, why wouldn’t they make good test cases?

  242. @Willis

    I see you are a Mathematica user. You wouldn’t happen to have an SSA routine coded up by chance? I’m struggling a bit with the grouping part of the algorithm.

    JP

  243. Jeff Patterson says:
    September 13, 2013 at 5:45 pm

    Willis Eschenbach says:
    September 13, 2013 at 4:44 pm

    “What am I missing”

    You actually have three sample points (they also cross at zero) and note that they are not equally spaced. If you picked your intersection points s.t. they were equally spaced in time (we’re assuming uniform sampling here) you’d see that the sine waves would have to be harmonically related (no beat) and only one would satisfy the nyquist criteria.

    No, I do NOT have three sample points, I only have two points specified—(0,0), and (1.3, 0.9). I have neither sampled nor specified a third point. Yes, they happen to cross again, but that’s not a sampled point.

    So my objection still stands. Given two and only two points, which fit your criterion of being closer together than half the shortest period, I have fitted two sine waves to them.

    Again … what am I missing?

    w.

  244. Jeff Patterson says:
    September 13, 2013 at 5:45 pm

    BTW, how did you embed the graph? That’s cool.

    I fear I can only do that because I am a official site contributor approved by Anthony. However, if you need to add graphs, post them up on the web somewhere (I use Photobucket), and then put them in as links, and when I see them, I’ll convert them to the underlying graphs.

    w.

  245. Jeff Patterson says:
    September 13, 2013 at 6:03 pm

    Willis Eschenbach says:
    September 13, 2013 at 5:04 pm

    “For example, you could also “use the data record to estimate” an MA process, or an ARMA process, or an ARIMA process, or a white-noise process”

    I actually did use an ARMA process and got even better results (lower probaility of fluke), I posted the more conservative of the two.

    A good choice … the important issue, however, is that you got different results with the two.

    But I really don’t see your point. An AR process is just an IIR filter driven with noise, a MA process is just an FIR filter, an ARMA is the cascade of the two and ARIMA has a pure integration (which doesn’t match the data at all). In any case, they are all just filters. What other effect could the process have that wouldn’t be reflected in the PSD and covariance of the output? If these closely match the observed data, why wouldn’t they make good test cases?

    If the processes were actually equivalent, you would not have gotten different results … therefore we can assume that, contrary to your claim, they are NOT equivalent.

    My point is that in a Monte Carlo analysis, you are assuming a model of the data. AR, ARIMA, white noise, whatever, it is an assumed model of what is actually going on. Now this is neither good nor bad … it’s just a fact.

    And sadly, the validity of the Monte Carlo analysis depends on whether your data actually IS a realization of the process that you have chosen.

    Above, you say that:

    The AR process was just a convenient way of generating random sequences with bandwidths and variances similar to the temperature record.

    This is a necessary but not sufficient condition. In particular, climate data often contains cycles which appear and disappear for no apparent reason. The rate of sea level change, for example, contains clear cycles for the last fifty years … but not for the fifty years before that. Why? No one knows, but there they are.

    And unless your Monte Carlo data is similarly structured and has such appearing and disappearing cycles, it will give you erroneous results.

    Best regards, and as before, my thanks for your replies and explanations, which have been more gracious than I likely deserve. Keep’m coming …

    w.

    PS—I suspect that part of the difficulty is that you are used to using Fourier Transforms on a signal of essentially infinite length. But in climate, we’re forced to use the Discrete Fourier Transforms on very short samples of the signal. In those, accuracy drops as the cycle length increases. As I said above, I do not trust any results (from DFT or any other method, I often use Periodicity Analysis) which have periods longer than about a third of the length of the dataset, and I would recommend this limit to others as well.

    And I am absolutely skeptical about the possibility of extracting a 170-year signal (or even more improbably a 170.7 year signal) from a 110-year dataset. I think you may be mistaking the Nyquist limit of sampling a long signal at twice the frequency desired to be found, with the current condition of sampling a short dataset twice … but that’s just a guess.

  246. Jeff Patterson says:
    September 13, 2013 at 6:11 pm

    @Willis

    I see you are a Mathematica user. You wouldn’t happen to have an SSA routine coded up by chance? I’m struggling a bit with the grouping part of the algorithm.

    JP

    Sadly, I fear that I struggle with Mathematica. I can do the simpler stuff … but more than that is something I tackle rarely.

    I actually use R a lot in preference to Mathematica. R has a package, Rssa, for doing singular spectrum analysis.

    R is free, handles matrices and arrays effortlessly, runs on all platforms, and has packages for just about anything. I learned it at the urging of Steve McIntyre six or eight years ago, and it was one of the best moves I’ve made in a while.

    w.

  247. Jeff Patterson: If we’re getting just two samples per period (the nyquist criteria) we know the frequency is the nyquist frequency (i.e.the sample frequency /2).

    Well, ,,, , that is what I wrote: if you know 1 of the 3 parameters — here the frequency is assumed known — then you can estimate the amplitude and phase.

  248. Some comments suggested I do a Singular Spectrum Analysis (SSA) on the data. I have done so with the results available at http://montpeliermonologs.wordpress.com/2013/09/14/singular-spectrum-analysis-of-hadcrut3/ . I am new to SSA but am very impressed with its capability.

    The results seem consistent with those found above. The fundamental frequency came out at 180 years with a strong 3rd at 60 years (note to Willis, I’ve taken your suggestion re significant figures to heart). The hindcast is good for 25 years or so. Interestingly, deviations from the model seem coincident with impulsive events in the climate record (e.g. the spike at 1878, the rapid cooling in the 1940’s, and the ENSO event of 1997). Whether this indicates an issue with the methodology of indicative of climate phase-state changes I cannot say.

    Comments and suggestions are welcome as usual.

  249. Specifically, the [IPCC Summary] draft report says that “equilibrium climate sensitivity” (ECS)—eventual warming induced by a doubling of carbon dioxide in the atmosphere, which takes hundreds of years to occur—is…… “

    This statement leads to a condundrum from the IPCC.

    Any concept that it “takes hundreds of years” requires that the earth has a predictable system response that is not chaotic. If so, then the system response to any impulse is centuries long and predictions can be made from historical impulses already in the system. Fourier Analysis can be done on such a system —- provided you have the data of long enough to deconvolve a system response from known inputs. We don’t, not if the system response is centuries long.

    On the other hand, if this were true, then missing the pause is doubly embarrassing. The only explanation is that the pause is the result of a system response that derives from impulses as far back as the LIA. The further means that any action taken by us today to mitigate climate change won’t have much effect for hundreds of years. Try selling that to the world.

    Conversely, if the climate is chaotic, largely unpredictable, and Fourier Analysis is a waste of time, then statements like “takes hundreds of years to occur” are blather. The climate can turn on a dime. Or not. No matter what the world does.

    Other possibilities present the IPCC with other problems. The system response is decades long, but GHG is not a primary driver, it is something else. Oops. Or the system response is predictable and short … again the pause shows that CO2 sensitivity is tiny to insignificant with other significant factors unrecognized.

  250. Stephen Rasey says:
    September 14, 2013 at 5:12 pm

    I take that sentence to read the CO2 doubling will take hundreds of years to occur. Is that in line with their current projections?

    I agree with “again the pause shows that CO2 sensitivity is tiny to insignificant with other significant factors unrecognized.” I recently posted a follow up to the SSA analysis I posted earlier (http://montpeliermonologs.wordpress.com/2013/09/14/ssa-of-the-hardcrut3-record-part-2/) which examines the first two SSA eigenmodes. It shows the trend has been losing steam since 1965, while CO2 rose exponentially. The residual from just first two modes is trendless and noise-like. Guess the AWG is all in the ocean, just below the Argo buoy’s maximum depth. Rats, and after we spent all that money deploying them. AWG is weally a Wascally wabbit

  251. Jeff Patterson:

    Active man-made systems often incorporate elements, such as op amps, transducers, various modulators and true feedback loops, that can produce all sorts of harmonics when driven by a sinusoidal input. Those elements, however, are simply not found in the inanimate natural world. In passive natural thermodynamic systems, whatever harmonics one finds are usually the product of direct forcing, That is certainly the case with the diurnal temperature cyle that is driven by the cosine-arch of insolation, with zero values at night.

    Although it’s not clear what sort of “transport delay” you had in mind, the workings of entropy–which always disperse thermal energy rather than concentrate it–militate against anything similar to the mechanical resonance found in some second-order systems.

    If you ignore physics completely and concentrate solely on DSP algorithms applied to woefully short records, you can find all sorts of “cycles” that have no embodyment in reality. Without any frequency decimation, the raw periodogram that you present in your post is not even a consistent estimator of the PSD. And you’ll find that that the results of SSA depend quite critically upon some rather abitrarily chosen analysis paramters The case for strictly periodic comnponents in the “climate” signal reamis flimsy at best.

  252. @1sky1

    Thank you for your thoughtful reply. The transports I had in mind were things like the delay in ocean heat transport to the poles, the delay in albedo changes from rising temps, delays in the carbon cycle etc. It seems to me that to the extent the output of any delayed mechanism contributes to the forcing, it seems to me it could result in frequency selective peaking in the climate response, even if passive.

    You said “In passive natural thermodynamic systems, whatever harmonics one finds are usually the product of direct forcing”. Does this lend credence to the notion that the harmonic relationship seen above in in the SSA analysis are due to external forcing or are you of the opinion that they are a statistical fluke?

    I’d be very interested in your thoughts about the analysis outlined here. http://montpeliermonologs.wordpress.com/2013/09/16/global-warming-a-la-mode/

    It makes no assumption about underlying physical mechanisms nor whether they are persistent or transitory. It does use SSA to decompose the climate record but its conclusion (I maintain) is independent of the means of decomposition.

    All the best

  253. @1sky1 9/16 3:12 pm
    Let me start off by saying that I do not believe in a linear system response of the earth that is over a decade long. I recognize the possibility, but I do not believe it is real. Hiding decades of climate forcings in 0.04 deg C warming of the deep oceans is theater not thermodynamics.

    I do believe that the earth has a profound NON-linear systems response that can run decades, or centuries in duration. It explains causes of ice ages, of warm periods, and of hot houses. It is chaos as seen in the geologic record. This hypothesis has the ramification that the sign of a change in “forcing” is not mathematically symmetrical as we would expect in a linear system.

    It is quite possible that a +1 W/m^2 (If I choose to get trapped in a questionable paradigm) has a short system response, where as a -1 W/m^2 has a much longer response.

    As you point out, 1sky1, it means little to talk of response functions until you can point to physical processes that can animate these responses. Two of the processes are snow and Life. The positive feedback of increasing snow begetting colder temperatures begetting more snow leading to ice ages is accepted. Likewise, the retreat of ice sheets causes a positive feedback of warming that is also well accepted. Less accepted are the mechanisms for the turn — what flips to start the warming or cooling if it isn’t astronomical (orbits, inclination, impacts, or solar super cycle) or volcanic?

    Less accepted, I think, is the non-linearity of Life. Life it seems is a powerful negative feedback mechanism on this plant. It is the core of the Gaia Hypothesis. When one thinks of a typical feedback look, we often model it as if it were a linear system in a known range: if a +x change in input yields a +y output, then a -x change in input yields a -y change in output.

    The non-linearity comes into play when we realize that it is much easier to destroy than to create. A bountiful spring and summer can result in much seed for the next year. This would be a quick cycle feedback. Conversely, a late spring, dry fiery summer, or early autumn can harm the life cycle causing many years of recovery – a lengthy system response. Timberlines should fall faster than climb. Tundra lines should move equator-ward faster than pole-ward. Forests are felled or burned faster than they can regrow. Reef’s are bleached faster than they repopulate. Plagues wipe out populations taking generations to grow back, if ever.

    The mathematics to model such a world are difficult – provided you don’t worry about accuracy. It makes for interesting science fiction, but intractable science. Fourier Analysis is pointless here. I argue that it is impossible to realistically model such a world with any sort of consensus. There are too many non-linear, non-symmetric system responses to nail down and agree upon; too many degrees of freedom. In such a system, the very concept of an Equilibrium Climate sensitivity to [CO2] has little meaning; a linear system concept made irrelevant by non-linear system dynamics.

  254. Jeff Patterson:

    Regrettably, demands on my time force my responses to be very brief.

    The delays that you mention essentially constitute either a straightforward spatial redistribution of thermal energy around the globe or, in the case of albedo changes, an adaptive change in the forcing and/or response characteristics of the global system. Certainly clouds are the gatekeepers of insolation capable of modulating the nearly constant TSI seen at TOA. Unfortunately, we know precious little about the their long-term variability on a global scale. Nevertheless, being a product of chaotic convective development, it’s highly unlikely that such modulation would be a strictly periodic function of any kind.

    I actually suspect that the “harmonic periodicities” found in Hadcrut3 are an artifact of the not-very-transparent construction of that index, which relies greatly upon manufacturing time-series from highly scattered observations by ships of opportunity. It differs very materially in that regard from any genuine long-term temperature time-series that I’ve ever analyzed over decades of work with such series. Lacking any physical basis for a strictly periodic driver, one should look for stochastic forcing of appropriate bandwidth instead. After all, it is signal bandwidth that determines its predictability.

  255. Steven Rasey:
    Having worked extensively on dynamical-system problems in a geophysical setting throughout the mature portion of my career, I don’t rely upon Fourier analysis blindly. The principal nonlinearities in the climate system, beyond the fourth-root of temperature dependence upon energy, are found in the Navier-Stokes equations governing convective and advective transport. Important as it is, that transport doesn’t directly affect the thermalization of insolation on a global yearly-average basis. Thus much is to be learned about year-to year variations from a linear-system standpoint. It turns out that, at stations where long, vetted records are available, nothing coheres as consistently with temperature throughout the entire spectral baseband than total sunshine hours experienced. Bi-spectrum analysis seldom shows any indication of strong quadratic interaction within such records. The imprint of solar forcing is clearly evident even in locations, such as Ross-on-Wye, where advective effects dominate during overcast winters.
    The nonlinearities that you point to are, of course, real. But for many practical purposes, they can be effectively linearized, quite irrespective of their time scale. What distrust I have of multidecadal and longer components manifest in many manufactured indices stems from the fact that they have been heavily corrupted by one means or another. Nor do the “feedbacks” that you mention constitute a true looping back of system output into system input; they are more akin to changes in response characteristics, which are ordinarily assumed to be time-invariant. This indeed makes their mathematical modeling much less easily tractable, but by no means impossible. And in such modeling Fourier analysis (though, of course, not the simplistic periodogram) continues to play a vital role.
    Finally, let’s not forget that solar energy ultimately powers not only thermal processes, but mechanical ones (winds) and biological ones (plant growth) as well. As seen in the Milankovitch-cycle explanation of the ice ages, it is variations in solar forcing, rather than nonlinear internal responses, that dominate the record.

  256. Jeff Patterson:

    A brief addendum, based upon reviewing the results of SSA that you linked to:

    It would be instructive to break Hadcrut3 into three disjoint 54-year segments and do identical SSAs on each. Such an exercise might dissuade you from the view that such analysis (which is essentially an empirical orthogonal function decomposition based on the Karhunen-Loeve expansion) reveals in its lowest modes some quantitatively consistent, physically meaningful “components” of the time-series, “independent of the means of decomposition.”

  257. @1sky1
    It turns out that, at stations where long, vetted records are available, nothing coheres as consistently with temperature throughout the entire spectral baseband than total sunshine hours experienced.
    I’ll by that. But it begs the question of what drives total sunshine hours?
    (assuming that is the driver of temperature and not the other way around)
    That’s more of a retorical question getting back to my non-linear life-based drivers.

    The nonlinearities that you point to are, of course, real. But for many practical purposes, they can be effectively linearized, quite irrespective of their time scale.
    Maybe. Yes, non-linear functions are commonly linearized within narrow domains to make the math easier. I was drawing attention to that common assumption that here may not be valid. How do you linearize an effect that has properties that resembles is a hysteresis? How do you linearize a process who’s system response is path dependent?

    Let me give you an economic example. Back in 1982 at the SEG conference, I gave a panel paper that for argument’s sake began with an oscillating Oil Price curve from $35/bbl to $15/bbl as kind of a joke. I next put up the traditional Supply-Demand curve. “We are explorationists. We find tomorrow’s oil. We know today were there are a trillion bbls of oil. — It’s just a question of price to get it out. If we feel the price will be high enough, we build our Alaskan Pipeline, we develop our North Sea, we build our deep water platforms, we develop Athabasca. Working our way up the Supply price curve to greater supply. But then, demand adjusts to the higher prices, lowering the clearing price.

    “So do we turn off the TAPS? Do we turn off North Sea? Do we shut in our platforms? Do we work our way down the SAME supply curve? WE DO NOT. We accept a lower price at the same supply rate. Instead of sliding back down that one long-term supply curve, we depart the curve, dropping to a steeper curve, with lower prices for the same previoous supply points.

    The Long Term Oil Supply curve is a hysteresis! Changes in prices are met by delayed restoring forces to supply. In any system that involves delayed restoring forces you get (first price projection slide) an oscillator! ” The slide that started out as a joke returns as a dead serious possibility. And it turned out I was being optimistic — the bottom was below $12/bbl about 5 years later.

    So, linearizing a non-linear curve is not the key issue. It is recognizing that some systems performance is a path-dependent non-symmetric hysteresis. That is exactly the way much of Life works. Maybe… so does the Climate — at least that part of the Climate influenced by Life.

  258. Well, Mr. Rasey, perhaps, you intuited something significant in your “hysteresis” supposition. If Lady Moon’s 18.6-year odysseys, gently but firmly shaping the ocean tides, are a key driver of ocean heat oscillations, her “hysterical” [from Greek] influence may, as others above posited, be a key mechanism.

    Silly, huh? Well, I, the night janitor, had fun grabbing the chalk and writing that up on the board, here, in the dark university classroom, after everyone’s gone home.

    And………… IT’S A FULL MOON. #(:)) Heh, heh, heh.

    Even though you likely won’t read this, THANK YOU, Jeff Patterson, for all that first class (albeit obscure, sometimes, for a non-scientist) straightforward, pure science, honest analysis. I LOVE THIS SITE — one learns so much. And, thanks, too, Mr. Patterson, for persevering in answering all the (sometimes worthwhile, sometimes obtuse or off-base, and, SOMETIMES, just plain obnoxiously rude) challenges to your methods, above. I hope you are given the right to post graphics — YOU certainly should!

  259. Jeff, I’m still waiting for your explanation of my falsification of your claim that

    Just as two points define a line uniquely, there is only one way to draw a sine wave between two points as long as you know the points are separated in time by no more than 1/2 the shortest period (or equivalently twice the highest frequency) sine wave present in the data.

    I said that wasn’t true, and provided the following graphic showing two distinct sine waves going through the two points (0,0) and (1.3, 0.9). Here’s the graphic again:

    Your first response was to claim I’d used three points, and I pointed out that no, only two points were specified, and they fit your criteria.

    At that point, you disappeared from that topic and went on to discuss a host of other things … interesting things to be sure.

    But when a man does that, makes a claim and then runs away when people object (I wasn’t the only one), I fear that their credibility goes down the tubes.

    Now, you certainly appear to have fooled some of the people some of the time. For example, Janice gushes:

    Even though you likely won’t read this, THANK YOU, Jeff Patterson, for all that first class (albeit obscure, sometimes, for a non-scientist) straightforward, pure science, honest analysis. I LOVE THIS SITE — one learns so much. And, thanks, too, Mr. Patterson, for persevering in answering all the (sometimes worthwhile, sometimes obtuse or off-base, and, SOMETIMES, just plain obnoxiously rude) challenges to your methods, above. I hope you are given the right to post graphics — YOU certainly should!

    Sorry, Janice, he didn’t persevere in the slightest. He ran away from that question and never returned. In addition, he did only the most simplistic type of out-of-sample analysis, he claims he can determine a cycle of a most precise length (170.7 years) from 110 years of data, he did not do anything resembling a comprehensive Monte Carlo analysis. His response to my pointing out that the climate is chaotic was also most puzzling …

    Even Chaos can have an underlying harmonic structure (for example, fractals, Strange attractors, etc.)

    Overall, I’d give his work about a C+, maybe a B-, but more for effort than results. However, his running away from from my falsification of a very important claim, a claim that is the basis of his idea that you can detect a 170.7 year cycle in 110 years of data, takes it down to an F.

    Still waiting …

    w.

  260. Let me also say that I strongly support the analysis of 1sky1, who says above:

    If you ignore physics completely and concentrate solely on DSP algorithms applied to woefully short records, you can find all sorts of “cycles” that have no embodyment in reality. Without any frequency decimation, the raw periodogram that you present in your post is not even a consistent estimator of the PSD. And you’ll find that that the results of SSA depend quite critically upon some rather abitrarily chosen analysis paramters The case for strictly periodic comnponents in the “climate” signal reamis flimsy at best.

    What he said …

    w.

  261. Janice Moore:

    The moon’s 18.6yr nodal precession cycle is a very far-fetched driver for any long-period thermal oscillations. To begin with, the tides cannot change either the total volume or effective area of water on our planet, nor its thermal capacity. Their amplitude is measured in centimeters in the deep ocean and the precession cycle doesn’t even make the top 20 of tidal constituents. Furthermore, while hysteresis alters the wave form by introducing harmonics, it doesn’t produce sub-harmonics. What multi-decadal and longer oscillations are evident in SST data are much more likely the result of little-explored modulations of surface insolation by clouds, i.e. induced oscillations in the hydrological cycle.

  262. Steven Rasey:

    Instantaneous (memory-less) nonlinearities are stochastically linearized by so-called “describing function” methods, which allow a spectral viewpoint to be maintained in analysis, albeit without the assurance of independence of different frequencies. In other words, one can deal effectively with phase-locked components, which can be recognized via bi-spectrum analysis. When “memory” is involved–which capacitive systems always manifest–it’s an entirely different ball game. And with such systems subjected to stochastic excitation, you get closer to how the climate system appears to operate. Even if I had the free time (and the typing skills) to produce a mini-essay on that subject, by contractual terms I would not be permitted to do so here. Thus I must leave this discussion on a note that satisfies neither one of us. Your civilized discourse nevertheless is much appreciated.

  263. Willis Eschenbach says: Still waiting …
    September 20, 2013 at 5:04 am

    Sorry for the delayed response – my day job’s been pressing in of late and I’ve been working on SSA.

    Your plot demonstrates the lack of spectral resolution in a two point record. The two points can be reconstructed without distortion from their discrete fourier transform (because Nyquist is violated) but the two signals cannot be spectrally separated because there are only two spectral bins in the discrete frequency domain for a two point record. Thus the energy is split into a dc component and a component at the Nyquist frequency. In the sample domain, time only exists at the sample points. The discrete frequency domain corollary is that only frequencies which are harmonics Fs/(2*N), where Fs is the sample frequency and N is the record length can be contained within a single bin. All others will be split across two bins, which is why we use HD when appropriate, it greatly enhances spectral resolution as shown below.

    Clearly a two-point record though was not in mind when I was trying (inartfully) to explain Nyquist sampling to someone not well versed in the subject using a simple geometric analogy. I apologize for my error. I should have stated that two points define a sine wave to within the frequency accuracy determined by the record length.

    All of this has nothing at all to do with the question at hand, since we are not dealing with two point records, and I am not using fourier analysis above, except as a starting point for the fit routine. Which brings us to this ” he claims he can determine a cycle of a most precise length (170.7 years) from 110 years of data,”

    Well let us see with a little experiment using my HD algorithm:

    (* Create a 113 point record of a sine wave of period 170.705678 years sample at yearly intervals*)
    sin = Table[Sin[n*2.*Pi/170.705678], {n, 0, 113}];

    (*extract the frequency of the highest peak as bound for the fit routine *)
    psd = PowerSpectralDensity[sin, w];
    wmax = ArgMax[{psd, w > 0}, w];

    (* do the curve fit, bounding the frequency between wmax found above +/- one bin *)
    fit = FindFit[
    sin, {a*Sin[wo*t + phi], wmax-1/113 < wo < wmax + 1/113, -Pi < phi NMinimize]

    (* express the frequency to 10 decimal digits precision for Willis*)
    N[(2*Pi/wo) /. fit, 10]

    {a -> 1., wo -> 0.0368071, phi -> -0.0368054}
    170.706

    So the good folks at Wolfram extracted the correct frequency not to just to .1 Years but to .001 years.
    Note it got the amplitude (a) exactly, to the decimal precision of MMa and the phase was off by just .037 radians (2 degrees).

    That was fun, let’s try it again, adding in some bandwidth-limited noise of variance .03, the value extracted from the AR model fit.

    sin = Table[Sin[n*2.*Pi/170.705678], {n, 0, 112}] +
    RandomVariate[NormalDistribution[0, .03], 113];
    psd = PowerSpectralDensity[sin, w];
    wmax = ArgMax[{psd, w > 0}, w];
    fit = FindFit[
    sin, {a*Sin[wo*t + phi],
    wmax – 1/113 < wo < wmax + 1/113, -Pi < phi 0}, {a, wo,
    phi}, t, Method -> NMinimize]
    N[(2*Pi/wo) /. fit, 10]

    {a -> 1.00917, wo -> 0.0368329, phi -> -0.0459094}
    170.586

    So we’re off by 170.705678-170.586 = .12 years, sue me.

    Note that in the algorithm above, the only use of the FFT was in the PSD routine whose only purpose is to locate the frequency within a bin. HD has much better spectral resolution (as shown here) which is why we use it.

    “His response to my pointing out that the climate is chaotic was also most puzzling …”

    “”Even Chaos can have an underlying harmonic structure (for example, fractals, Strange attractors, etc.)””
    Perhaps you are puzzled more than you admit. That wasn’t my quote but Steven Groeneveld’s September 12, 2013 at 11:32 am

    As far as the other aspersions on my character, I’ll just chalk those up to Willis being Willis.

    All the best
    JP

  264. 1sky1 says:
    September 16, 2013 at 3:12 pm
    If you ignore physics completely and concentrate solely on DSP algorithms applied to woefully short records, you can find all sorts of “cycles” that have no embodiment in reality.

    I agree, but separating those that embody reality from those that do not is the key. The goodness-of-fit obtained above with a simple harmonic fit is certainly not definitive given the 1/12 probability of it just being a statistical fluke, but neither do I think it is as easily dismissed as you imply. As for being an artifact of the Hadcrut3 record, it is hard to see how a 60 year cycle could be introduced accidentally. The fit algorithm is highly insensitive to noise and single events as it minimizes LSE across the entire record. I would also note, SSA analysis of the SST record shows a similar periodicity (see http://montpeliermonologs.wordpress.com/2013/09/21/periodicity-in-the-sst-record/)

    Regards,
    Jeff

  265. 1sky1 says:
    September 16, 2013 at 3:12 pm And you’ll find that that the results of SSA depend quite critically upon some rather abitrarily chosen analysis paramters

    Actually the analysis I’m doing is quite insensitive to choice of L, the widow parameter, which only controls the fineness of the separation between the two buckets. The motivation for this approach is simple. We are trying to detect a small signal in noisy data. Information theory tells us that information bearing signals must be non-ergodic. SSA allows us to separate the ergodic mode from the non-ergodic modes without loss of information (the sum of the two buckets always equally the original data). Since the ergodic (noise) bucket uses all modes not in the non-ergodic (information bearing) bucket, the analysis depends only on L which in turn only affects the spectral bin width. Using the maximum available L gives us the finest split between the two buckets so that’s what I use, but the results are nearly [identical] until L gets below 20 or so.

    [The "link above" (in the words analysis I'm doing") fails. Mod]

    [Link repaired. -w.]

  266. @1sky1 4:13 pm
    When “memory” is involved–which capacitive systems always manifest–it’s an entirely different ball game. And with such systems subjected to stochastic excitation, you get closer to how the climate system appears to operate.
    The climate, especially with life included, is a capacitive system, no doubt.

    Your civilized discourse nevertheless is much appreciated.
    Likewise. I look forward to reading from 1sky1 in future threads.

  267. @Jeff Patterson 9:42 am
    fit = FindFit[
    sin, {a*Sin[wo*t + phi], wmax-1/113 < wo < wmax + 1/113, -Pi < phi NMinimize]

    sin is your manufactured 113 sample signal with a 170 sample cycle. (BTW, using “sin” as a name for a vector variable for a Sin() function is a sin. Vsin, V1sin, would have been better.)

    Wmax is the maximum frequency in the power spectrum, which will be the first, lowest frequency, 1/113 Hz (are the units of PowerSpectralDensity in Hz or radians per time sample ?)

    What your FindFit() amounts to is:
    Find the best (a=amplitude), (wo=frequency) and (phi=phase) for a single sine wave such that [(sin(t) - a*Sin(wo*t + phi)), sum over t ={1,113}] is minimized,
    Search wo across the narrow range 0 to 2/113 Phi greater than -pi.

    The FindFit you used assumes there is only one sin wave to fit, that you were interested in fewer than 2 cycles over the analysis, and that there is no time-linear bias component. i.e the linear components + b*t + c, are set to zero by their omission. I think your solution was a little over specified for a general problem.

  268. Stephen Rasey says:
    September 21, 2013 at 2:29 pm
    the linear components + b*t + c, are set to zero by their omission. I think your solution was a little over specified for a general problem.

    I don’t think so. Wmax is the maximum frequency found in the psd, not the first. We know it is accurate to within the spectral resolution of the the DFT used in the powers spectral density routine, hence we search over that bandwidth (+/- one bin). The data was generated de-trended so I didn’t include any bias terms. In the actual HD algorithm, the data is de-trended and the mean removed (actually we use a process called periodization which is slightly different and gives better results) but we also include a dc term and sometimes a slope term.

  269. “Wmax is the maximum frequency found in the psd, not the first. ” It is also the first, as in first bin. Which means your search is including the zero point. I’m just going by your Mathmatica code above. If the wo=0.0368 (radians/sample) , the search range had to include values below 1/113 Hz = 2*Pi/113 radians/sample

    Your Mathmatica code did not detrend. So it is not proof that you can get the same answer if you detrend as you say in the ‘HD’ case. You will not. As I showed in My Sept 12 9:44 am, “detrending” is a fiction. Applying a detrend to a non-integer number of cycles will contaminate the analysis with a false trend and alter the frequency.

  270. Stephen Rasey says:
    September 21, 2013 at 10:02 pm
    As I showed in My Sept 12 9:44 am, “detrending” is a fiction.

    You didn’t apply the compensation correctly (as in not at all). Periodization is a two step process.
    1) Detrend abpout the line connecting the endpoints.
    2) DFT the the data and the line
    3) Extract the complex spectral infomation of interest
    4) Correct with the corresponding complex bins from the line

    However, in the HD referred to in the article it isn’t necessary as I have explained many times. The slight shift one gets from using the uncorrected results for the psd just effects the starting point of the fit routine (which BTW is using a global gradient search algorithm which avoids getting trapped in locall minima’s).

  271. @Jeff Patterson at 7:40 am
    My problem is that the detrend line between end points has little to do with any real trend in the data. it is an accident of the sample period. Whether there is a real trend in the data or not is something that connot be determined uniquely. Detrending changes the wavelengths of sub-cycle components.

    Now, if you want to say that your method searches for a maximum match to the signal using one fundamental frequence (3 parameters) on one linear trend (2 parameters), then I agree, a global search will find a solution to a signal with at least 5 sample points. 5 equations, 5 unknowns It doesn’t mean it is the right solution, but it will find one. It presupposes a great deal about the solution.

    What you describe seems to be similar to the Periodisity Transform described by Willis in Cycles without the Mania Certainly we should not let a tool constrain our thinking about the correct components of the solution.

    • Stephen Rasey says:
      My problem is that the detrend line between end points has little to do with any real trend in the data.

      Exactly right but irrelevant. It is the deleterious effect of the trend on the DFT that we are trying to eliminate so that we can measure the spectral energy in the lower bins that would otherwise be masked by the record-edge discontinuity the trend give rise to.It exploits the fact that the Fourier transform is a linear operation to decompose the data into two DFT friendly components.
      .
      “Detrending changes the wavelengths of sub-cycle components”
      True, but the effect is tractable and can be undone on the back end..

      “What you describe seems to be similar to the Periodisity Transform”
      No, it is a front-end technique that can be used with any form of spectral analysis. Think of SSA where the N-length data is decomposed into k N-length components via eigensystem analysis of the data’s covariance matrix. Clearly I can subtract any N point line (or any other N-point function for that matter) from the data and simply add it back in, point for point, to the lowest order SSA reconstruction. I haven’t changed the result one iota so what have I gained? Simply this. By periodizing the record prior to SSA I can get a much better spectral view of the reconstruction’s signal-to-noise ratio which can be used to optimize the choice of the window parameter and reconstruction eigen set.

      Periodization is not quite that transparent with a DFT due to the Gibbs Effect, but these distortions are at high frequencies. In this case, periodization is tool to all us to trade better low frequency dynamic range for reduced high frequency DR.

      “Certainly we should not let a tool constrain our thinking about the correct components of the solution.”

      Amen brother.

  272. Stephen Rasey says:
    September 22, 2013 at 11:22 pm
    What you describe seems to be similar to the Periodisity Transform described by Willis in Cycles without the Mania

    Willis and I are in complete agreement with regard to Scafetta; given enough cycles I can reconstruct any curve (After all that is exactly what a cosine transform does). What struck me as surprising in my result was that so few cycles could give such a good fit, that the 3 main periodic signals were all harmonically related, and the phase stability of the 60 year cycle which still seems pretty astonishing – here it is again in the SST dataset this time.

    The SSA, analyses however has tilted me toward’s Willis’ hypothesis that these are some sort of emergent, long-lived phenomena and not celestial forcing. The period of the mode in the SST (64 years) dataset is far enough removed from that found above in Hadcrut3 (58 years) to be explainable by uncertainty.

  273. @Jeff Patterson 7:58am
    Your 4.2 year lag comes from a cross-corr plot with a broad maximum, where any value from 2 to 6 is above 0.90. But I think you are at least one step removed from the data. So I would not get hung up on the fractional year part of the question. It is not clear from the paragraph above, but your link indicates that global surface temp lags sea surface temp by 4 +/- 4 years.

  274. @Jeff Patterson 10:50 am
    does that make any physical sense?

    And that’s a question the Ocean Heat Content folks are not asking very loudly. Otherwise, they wouldn’t get so worked up over 0.04 deg C of deep water warming.

    A two year lag via Arctic Ocean Ice Cap minimums is easier to believe than a six year lag. Ten year half-life of an exponential decay function, with zero lag is easy to accept. But I find it difficult to visualize a four year impulse lag.

    Maybe a x-year-lag is the wrong question. Perhaps it is a phase-lag. In much the way that the damping force of a pendulum seems out of phase with the pendulum’s position, but in phase with its velocity. Temp-Dot (dTemp/dt) is the Forcing, which can be governed by cloud formation. If we look at cloud formation over time, do we see an x-year lag between ocean and land?

  275. Jeff Patterson:

    Your latest calculations and speculations don’t begin to address the objections I raised earlier. Instead of performing SSA on the three 54-year segments of Hadcrut3, which would show the ABSENCE of seamlessly splicable “trends” and “harmonic periodicities,” you arbitrarily continue to
    use the same stretch of that manufactured index, varying only the number of modes.You then point to a quaint “cross-spectral” comparison between the a low-order mode of SST component and the entire index, as if the handwaving agreement resolves all objections. And finally you claim that ergodicity is a property of noise, not of signal.

    All of this merely makes apparent your inexperience in analyzing geophysical signals, which tend to be gaussian processes to first order. You might acquaint yourself with the work of S. O. Rice of Bell Labs to gain analytic comprehension of such signal processes. FYI, homogenous random wave fields are indeed ergodic. And the cross-spectral density as defined by Wiener-Kintchine is complex-valued, consisting of the coherence and phase, as well as the pair of power densities, instead of simply the magnitude, which you estimate without any concept of the degrees of freedom You ca’t rely upon your experiece in analyzing man-made signals for guidance here. And I can’t take any more time time to explicate well-established fundamentals whose essential grasp is persistently avoided

    • 1sky1 says:
      September 23, 2013 at 5:24 pm
      Your latest calculations and speculations don’t begin to address the objections I raised earlier.

      That;s because I haven’t tried to address the concerns you raised earlier :>) Patience. I did however cut the data in half and got the same result – a 60 cycle quasi periodic mode which when removed yields a constant .05 deg/dec regression slope to the residual.

      “You then point to a quaint “cross-spectral” comparison between the a low-order mode of SST component and the entire index, ”
      Not so. I calculated the cross-correlation between the low order reconstructions of the Hadcrut3 and SSTNH data, the standard way you look for similarity in two signals.

      I think we have a confusion of terms here. The power spectral density of the cross correlation function is a measure of coherence so I’m not sure of the point you are trying to make. I drew no conclusions (because I haven;t done a cross-specrtral analysis) except to note that the resulting line width shows no sign of cross-modulation, indication a straight lag relationship as apposed to a multiplicative one.

      “homogenous random wave fields are indeed ergodic”.- Which is why than can be safely assumed not to carry any information of interest. This is an exercise in small signal detection by process of elimination. We’re looking for a almost linearly increasing AGW that has supposedly been a persistent contribution to temperature for over fifty years. If we remove those ergodic modes that produce a non-persistent,contribution, and the oscillatory modes which are clearly not anthropogenic, we’re left with those components that must contain the signal of interest if it is to be found. I don’t see the flaw in this method. You evidently do but I can’t parse it from your remarks.

      Thanks for taking the time and for the references.

  276. Jeff Patterson:

    The issue is not whether there is a ~60yr “quasi-periodic” component to be found in global or hemispheric SST temperature indices. The issue is that such narrow-band oscillations found in nature are essenially NON-deterministic. They cannot be realistically modeled by any finite set of sinusoids, irrespective of how well such a set may match a short stretch of record. Nor do they constitute “noise” in any physical sense of the word; they are an integral part of the temperature signal. They carry no less information than any longer-term components that are mistakenly taken for “non-ergodic” trends!

    I will leave for you to discover what the (not-only-record-length-imposed) limitations of SSA are. Such decompositions are incabable of seperating signal from noise. As for your notions of what the cross-spectrum is analytically and what it reveals about the relationship between to signals, they are not only quaint, but also entirely misguided. You are the first analyst I’ve encountered in many decades of experience who believes that cross-spectrum analyzing SSA modes is “standard practice.”

  277. In the forecast section you state:

    “If for example, the harmonics are being generated by a stable non-linear climatic response to some celestial cycle, we would expect the model to have skill in forecasting future climate trends. On the other hand, if the periodicities are internally generated by the climate itself (e.g. feedback involving transport delays), we would expect both the fundamental frequency and importantly, the phase of the harmonics to evolve with time making accurate forecasts impossible.”

    Followed by:

    “We assume the periodicity is externally forced and the climate response remains constant. ”

    Is the available data back to 1850 sufficient to identify whether or not the fundamental frequency and phase of the harmonics have evolved w/time? I.e. is it possible to rule in or rule out internal climate factors as dominant? This would not impact judgments wrt external forcings much, but it would be an indicative result, and would lend additional credence to the applied assumption that periodicity is externally forced.

    • Duane Oldsen says:
      September 25, 2013 at 3:00 am
      Is the available data back to 1850 sufficient to identify whether or not the fundamental frequency and phase of the harmonics have evolved w/time?

      No, that’s why the forecast should be taken with a grain of salt. It is only as good as the assumptions stated.
      Best
      Jeff

Comments are closed.