# Crowdsourcing A Full Kernel Cascaded Triple Running Mean Low Pass Filter, No Seriously…

Image Credit: Climate Data Blog

By Richard Linsley Hood  – Edited by Just The Facts

The goal of this crowdsourcing thread is to present a 12 month/365 day Cascaded Triple Running Mean (CTRM) filter, inform readers of its basis and value, and gather your input on how I can improve and develop it. A 12 month/365 day CTRM filter completely removes the annual ‘cycle’, as the CTRM is a near Gaussian low pass filter. In fact it is slightly better than Gaussian in that it completely removes the 12 month ‘cycle’ whereas true Gaussian leaves a small residual of that still in the data. This new tool is an attempt to produce a more accurate treatment of climate data and see what new perspectives, if any, it uncovers. This tool builds on the good work by Greg Goodman, with Vaughan Pratt’s valuable input, on this thread on Climate Etc.

Before we get too far into this, let me explain some of the terminology that will be used in this article:

—————-

Filter:
“In signal processing, a filter is a device or process that removes from a signal some unwanted component or feature. Filtering is a class of signal processing, the defining feature of filters being the complete or partial suppression of some aspect of the signal. Most often, this means removing some frequencies and not others in order to suppress interfering signals and reduce background noise.” Wikipedia.

Gaussian Filter:
A Gaussian Filter is probably the ideal filter in time domain terms. That is, if you consider the graphs you are looking at are like the ones displayed on an oscilloscope, then a Gaussian filter is the one that adds the least amount of distortions to the signal.

Full Kernel Filter:
Indicates that the output of the filter will not change when new data is added (except to extend the existing plot). It does not extend up to the ends of the data available, because the output is in the centre of the input range. This is its biggest limitation.

Low Pass Filter:
A low pass filter is one which removes the high frequency components in a signal. One of its most common usages is in anti-aliasing filters for conditioning signals prior to analog-to-digital conversion. Daily, Monthly and Annual averages are low pass filters also.

A cascade is where you feed the output of the first stage into the input of the next stage and so on. In a spreadsheet implementation of a CTRM you can produce a single average column in the normal way and then use that column as an input to create the next output column and so on. The value of the inter-stage multiplier/divider is very important. It should be set to 1.2067. This is the precise value that makes the CTRM into a near Gaussian filter. It gives values of 12, 10 and 8 months for the three stages in an Annual filter for example.

Triple Running Mean:
The simplest method to remove high frequencies or smooth data is to use moving averages, also referred to as running means. A running mean filter is the standard ‘average’ that is most commonly used in Climate work. On its own it is a very bad form of filter and produces a lot of arithmetic artefacts. Adding three of those ‘back to back’ in a cascade, however, allows for a much higher quality filter that is also very easy to implement. It just needs two more stages than are normally used.

—————

With all of this in mind, a CTRM filter, used either at 365 days (if we have that resolution of data available) or 12 months in length with the most common data sets, will completely remove the Annual cycle while retaining the underlying monthly sampling frequency in the output. In fact it is even better than that, as it does not matter if the data used has been normalised already or not. A CTRM filter will produce the same output on either raw or normalised data, with only a small offset in order to address whatever the ‘Normal’ period chosen by the data provider. There are no added distortions of any sort from the filter.

Let’s take a look at at what this generates in practice.The following are UAH Anomalies from 1979 to Present with an Annual CTRM applied:

Fig 1: UAH data with an Annual CTRM filter

Note that I have just plotted the data points. The CTRM filter has removed the ‘visual noise’ that a month to month variability causes. This is very similar to the 12 or 13 month single running mean that is often used, however it is more accurate as the mathematical errors produced by those simple running means are removed. Additionally, the higher frequencies are completely removed while all the lower frequencies are left completely intact.

The following are HadCRUT4 Anomalies from 1850 to Present with an Annual CTRM applied:

Fig 2: HadCRUT4 data with an Annual CTRM filter

Note again that all the higher frequencies have been removed and the lower frequencies are all displayed without distortions or noise.

There is a small issue with these CTRM filters in that CTRMs are ‘full kernel’ filters as mentioned above, meaning their outputs will not change when new data is added (except to extend the existing plot). However, because the output is in the middle of the input data, they do not extend up to the ends of the data available as can be seen above. In order to overcome this issue, some additional work will be required.

The basic principles of filters work over all timescales, thus we do not need to constrain ourselves to an Annual filter. We are, after all, trying to determine how this complex load that is the Earth reacts to the constantly varying surface input and surface reflection/absorption with very long timescale storage and release systems including phase change, mass transport and the like. If this were some giant mechanical structure slowly vibrating away we would run low pass filters with much longer time constants to see what was down in the sub-harmonics. So let’s do just that for Climate.

When I applied a standard time/energy low pass filter sweep against the data I noticed that there is a sweet spot around 12-20 years where the output changes very little. This looks like it may well be a good stop/pass band binary chop point. So I choose 15 years as the roll off point to see what happens. Remember this is a standard low pass/band-pass filter, similar to the one that splits telephone from broadband to connect to the Internet. Using this approach, all frequencies of any period above 15 years are fully preserved in the output and all frequencies below that point are completely removed.

The following are HadCRUT4 Anomalies from 1850 to Present with a 15 CTRM and a 75 year single mean applied:

Fig 3: HadCRUT4 with additional greater than 15 year low pass. Greater than 75 year low pass filter included to remove the red trace discovered by the first pass.

Now, when reviewing the plot above some have claimed that this is a curve fitting or a ‘cycle mania’ exercise. However, the data hasn’t been fit to anything, I just applied a filter. Then out pops some wriggle in that plot which the data draws all on its own at around ~60 years. It’s the data what done it – not me! If you see any ‘cycle’ in graph, then that’s your perception. What you can’t do is say the wriggle is not there. That’s what the DATA says is there.

Note that the extra ‘greater than 75 years’ single running mean is included to remove the discovered ~60 year line, as one would normally do to get whatever residual is left. Only a single stage running mean can be used as the data available is too short for a full triple cascaded set. The UAH and RSS data series are too short to run a full greater than 15 year triple cascade pass on them, but it is possible to do a greater than 7.5 year which I’ll leave for a future exercise.

And that Full Kernel problem? We can add a Savitzky-Golay filter to the set,  which is the Engineering equivalent of LOWESS in Statistics, so should not meet too much resistance from statisticians (want to bet?).

Fig 4: HadCRUT4 with additional S-G projections to observe near term future trends

We can verify that the parameters chosen are correct because the line closely follows the full kernel filter if that is used as a training/verification guide. The latest part of the line should not be considered an absolute guide to the future. Like LOWESS, S-G will ‘whip’ around on new data like a caterpillar searching for a new leaf. However, it tends to follow a similar trajectory, at least until it runs into a tree. While this only a basic predictive tool, which estimates that the future will be like the recent past, the tool estimates that we are over a local peak and headed downwards…

And there we have it. A simple data treatment for the various temperature data sets, a high quality filter that removes the noise and helps us to see the bigger picture. Something to test the various claims made as to how the climate system works. Want to compare it against CO2. Go for it. Want to check SO2. Again fine. Volcanoes? Be my guest. Here is a spreadsheet containing UAH and a Annual CTRM and R code for a simple RSS graph. Please just don’t complain if the results from the data don’t meet your expectations. This is just data and summaries of the data. Occam’s Razor for a temperature series. Very simple, but it should be very revealing.

Now the question is how I can improve it. Do you see any flaws in the methodology or tool I’ve developed? Do you know how I can make it more accurate, more effective or more accessible? What other data sets do you think might be good candidates for a CTRM filter? Are there any particular combinations of data sets that you would like to see? You may have noted the 15 year CTRM combining UAH, RSS, HadCRUT and GISS at the head of this article. I have been developing various options at my new Climate Data Blog and based upon your input on this thread, I am planning a follow up article that will delve into some combinations of data sets, some of their similarities and some of their differences.

About the Author: Richard Linsley Hood holds an MSc in System Design and has been working as a ‘Practicing Logician’ (aka Computer Geek) to look at signals, images and the modelling of things in general inside computers for over 40 years now. This is his first venture into Climate Science and temperature analysis.

## 355 thoughts on “Crowdsourcing A Full Kernel Cascaded Triple Running Mean Low Pass Filter, No Seriously…”

1. Kirk c says:

That is so cool!

2. Lance Wallace says:

The CO2 curve (seasonally detrended monthly) can be fit remarkably closely by a quadratic or exponential curve, each with <1% error for 650 or so consecutive months:
https://wattsupwiththat.com/2012/06/02/what-can-we-learn-from-the-mauna-loa-co2-curve-2/
The exponential has a time constant (e-folding time) on the order of 60 years (i.e., doubling time for the anthropogenic additions from the preindustrial level of 260 ppm of about 60*0.69 = 42 years).
Can the Full Kernel Triple thingamajig provide any further insight into the characteristics of the curve? For example, due to various efforts by governments to reduce CO2 emissions, can we see any effect on the curve to date? I tried fitting the exponential curve only up to the year 2005, then 2010, finally to September 2013) and there was a very small movement toward lengthened e-folding time (60.26, 60.94, 61.59 years). But also, theoretically, there is some relation between atmospheric CO2 and CO2 emissions, but one has to assume something about the lag time between CO2 emissions and CO2 concentrations. Can the Triple whatsis somehow compare the two curves and derive either a lag time or an estimate of how much of the CO2 emissions makes it into the observed atmospheric concentrations? Simply assuming that all the CO2 emissions make it into the atmosphere in the following year gives an R^2 on the order of 40% (IIRC).

3. geran says:

Since this is your first foray into “climate science”, Richard, let me help with the basics you will need to know.

Climate modeling is about as close to science as are video games. (Except in the better video games, there are sound effects.) Climate modeling works like this: Enter input, run program, no catastrophic results, enter new input and re-start. Continue until you get catastrophic results. Then, publish results and request more funding.

If your program never achieves a catastrophic result, adjust input data until proper results are achieved. (Ever heard of a hockey stick?)

If someone questions your knowledge of science, produce numerous graphics, in color, of your results.

It’s a veritable gold mine!

4. Arno Arrak says:

Why bother with HadCrut and GISS? They are worthless. Use UAH and RSS, but get rid of that annual version and use monthly versions. And draw the trend with a semi-transparent magic marker. See Figure 15 in my book “What Warming?”

5. RichardLH says:

Lance Wallace says:
March 16, 2014 at 1:42 pm

“Can the Triple whatsis somehow compare the two curves and derive either a lag time or an estimate of how much of the CO2 emissions makes it into the observed atmospheric concentrations?”

Not really. Low pass filters are only going to show periodicity in the data and the CO2 figure is a continuously(?) rising curve.

It is useful to compare how the CO2 curve matches to the residual after you have removed the ~60 ‘cycle’ and there it does match quite well but with one big problem, you need to find something else before 1850 to make it all work out right.

Volcanos are the current favourite but I think that getting just the right number and size of volcanos needed is stretching co-incidence a little too far. Still possible though.

6. Eliza says:

Arno 100% correct GISS, HADCRUT are trash data I don’t understand why any posting using that data can be taken seriously *”adjustments”, “UHI ect…”. This is just feeding the warmist trolls.

7. RichardLH says:

Arno Arrak says:
March 16, 2014 at 1:45 pm

“Why bother with HadCrut and GISS? They are worthless. Use UAH and RSS, but get rid of that annual version and use monthly versions.”

They are two of the only series that stretch back to 1850 (which unfortunately the satellite data does not) and with Global coverage.

They do match together quite well though with some odd differences.

What good would a monthly view do when trying to assess climate? Assuming you accept that climate is a long term thing, i.e. longer than 15 years. Monthly is down in the Weather range.

8. RichardLH says:

Eliza says:
March 16, 2014 at 2:00 pm

“Arno 100% correct GISS, HADCRUT are trash data”

You might like to consider the fact that any long term adjustments will show up in the residual, greater than 75 years curve, and, if they have indeed occurred, would only serve to flatten that part of the output.

The ~60 year wriggle will still be there and needs explaining.

9. Steven Mosher says:

Giss and hadcrut are not the only series.
Ncdc. Berkeley. Cowtan and way.

Im using ctrm in some work on gcrs and cloud cover. Thanks for the code richard.

Raise your objections now to the method…folks.

10. Very nice elucidation of the 60 year cycle in the temperature data, Now do the same for the past 2000 years using a suitable filter. A review of candidate proxy data reconstructions and the historical record of climate during the last 2000 years suggests that at this time the most useful reconstruction for identifying temperature trends in the latest important millennial cycle is that of Christiansen and Ljungqvist 2012 (Fig 5)
http://www.clim-past.net/8/765/2012/cp-8-765-2012.pdf
For a forecast of the coming cooling based on the 60 and 1000 year quasi periodicities in the temperatures and the neutron count as a proxy for solar “activity ”
see http://climatesense-norpag.blogspot.com
this also has the Christiansen plot see Fig3. and the 1000 year cycle from ice cores Fig4
The biggest uncertainty in these forecasts is the uncertainty in the timing of the 1000 year cycle peak.
In the figure in Richards post it looks like it is about 2009. From the SST data it looks like about 2003. See Fig 7 in the link and NOAA data at
ftp://ftp.ncdc.noaa.gov/pub/data/anomalies/annual.ocean.90S.90N.df_1901-2000mean.dat
It is time to abandon forecasting from models and for discussion and forecasting purposes use the pattern recognition method seen at the link
http://climatesense-norpag.blogspot.com .

11. J Martin says:

What is the point of all this, what is the end result, projection / prediction ?

12. Arno Arrak says:

Lance Wallace says on March 16, 2014 at 1:42 pm:

“The CO2 curve (seasonally detrended monthly) can be fit remarkably closely by a quadratic or exponential curve, each with <1% error for 650 or so consecutive months…"

So what. CO2 is not the cause of any global warming and is not worth that expenditure of useless arithmetic. The only thing important about it is that it is completely smooth (except for its seasonal wiggle) during the last two centuries. It follows from this that it is physically impossible to start any greenhouse warming during these two centuries. We already know that there has been no warming for the last 17 years despite the fact that there is more carbon dioxide in the air now than ever before. That makes the twenty-first century greenhouse free. Since this constant addition of CO2 is not causing any warming it follows that the theory of enhanced greenhouse warming is defective. It does not work and should be discarded. The only theory that correctly describes this behavior is the Miskolczi theory that ignorant global warming activists hate. But the twentieth century did have warming. It came in two spurts. The first one started in 1910, raised global temperature by half a degree Celsius, and stopped in 1940. The second one started in 1999, raised global temperature by a third of a degree in only three years, and then stopped. Here is where the smoothness of the CO2 curve comes in. Radiation laws of physics require that in order to start an enhanced greenhouse warming you must simultaneously add carbon dioxide to the atmosphere. That is because the absorbency of CO2 for infrared radiation is a fixed property of its molecules that cannot be changed. To get more warming, get more molecules. This did not happen in 1910 or in 1999 as shown by the Keeling curve and its extension. Hence, all warming within the twentieth century was natural warming, not enhanced greenhouse warming. Cobsequently we now have the twentieth and the twenty-first centuries both entirely greenhouse free. Hence that anthropogenic global warming that is the life blood of IPCC simply does not exist. To put it in other words: AGW has proven to be nothing more than a pseudo-scientific fantasy, result of a false belief that Hansen discovered greenhouse warming in 1988.

13. Hlaford says:

The N-path filters make the best notch filters in sampled domains. They fell under the radar for the other kinds of filtering, but here you’d have 12 paths and a high pass at each of them for the monthly data.
I’ve seen a paper where a guy explains how he removed the noise by vuvuzele trumpets using N-path filtering.

14. RichardLH says:

Steven Mosher says:
March 16, 2014 at 2:21 pm

“Giss and hadcrut are not the only series. Ncdc. Berkeley. Cowtan and way.

Im using ctrm in some work on gcrs and cloud cover. Thanks for the code richard.”

No problem – it’s what Greg and Vaughan thrashed out translated into R (for the CTRM).

I know there are other series but I wanted to start with the most commonly used ones first. I have treatments for some of the others as well. At present I am working on a set of Global, Land, Ocean triples as that has thrown up some interesting observations.

It is a great pity that the temperatures series are not available in very in easy to read into R form but require, sometimes, a lot of code just to turn them into data.frames. Makes it a lot more difficult to post the code to the ‘net in an easy to understand form.

The ~60 years ‘cycle’ shows up strongly in all of them as far. :-)

15. jai mitchell says:

Since this constant addition of CO2 is not causing any warming it follows that the theory of enhanced greenhouse warming is defective. It does not work and should be discarded.

. . .hilarious. . .

16. RichardLH says:

J Martin says:
March 16, 2014 at 2:31 pm

“What is the point of all this, what is the end result, projection / prediction ?”

It allows you to look at how the system responded to the inputs over the last 150+ years with less mathematical errors hiding the details. You can use the S-G trend as a loose guide to the immediate future trend if you like.

17. Gary says:

How much useful information is being lost by filtering out the high frequency “noise” ? In other words, how do you judge what the most effective bandwidth of the filter is?

18. RichardLH says:

Hlaford says:
March 16, 2014 at 2:37 pm

“The N-path filters make the best notch filters in sampled domains.”

There are many good notch filters out there, but that is not what this is. In fact it is the exact opposite. A broadband stop/pass filter that will allow ALL of the frequencies above 15 years in length to be present in the output. No need to choose what to look for, anything that is there, will be there.

19. RichardLH says:

Gary says:
March 16, 2014 at 2:42 pm

“How much useful information is being lost by filtering out the high frequency “noise” ? In other words, how do you judge what the most effective bandwidth of the filter is?”

Well what you lose is, Daily, Weather, Monthly, Yearly and Decadal. What you keep is multi-decadal and longer. Which do you consider to be relevant to Climate?

If you are interested in the other stuff you can run a high pass version instead and look at that only if you wish. Just subtract this output signal from the input signal and away you go.

No data is truly lost as such, it is just in the other pass band. The high and low pass added together will, by definition, always be the input signal.

20. David L. Hagen says:

Thanks Richard for an insightful way of exposing the 60 year cycle.

1) Has an algebraic factor been found behind the 1.2067 factor or is this still empirical per von Pratt?
2) Does using the more accurate year length of 365.26 days make any difference?
3) Suggest showing the derivative of the CTRM curve.
That would show more clearly that the rate of warming is declining. If that derivative goes though zero, that would give evidence that we are now entering the next “cooling” period vs just flattening in warming.
PS suggest amending “with a 15 CTRM and” to “with a 15 CTRM year and” to read more clearly.
Interesting interaction you had with TallBloke.

21. Mike McMillan says:

Well, it certainly gets rid of the 1998 peak problem. GISS would be appreciative.

22. Bernie Hutchins says:

Is it possible to humor some of us old-timer digital filter designers by specifically saying what the (Finite) Impulse Response of the CTRM is; in basic terms such as what simpler elements are being cascaded (convolved for the IR response, multiplied for the frequency response, etc.)? This is the key to understanding filtering – old or new. Thanks.

23. RichardLH says:

Dr Norman Page says:
March 16, 2014 at 2:30 pm

“Very nice elucidation of the 60 year cycle in the temperature data, Now do the same for the past 2000 years using a suitable filter.”

Bit of a challenge finding a thermometer series going back that far. ;-)

Proxy series all come with their own problems. They are rarely in a resolution that will allow the ~60 year signal to be seen.

I do have one which is the Shen PDO reconstruction from rainfall which does it quite well back to the 1400s.

Shen, C., W.-C. Wang, W. Gong, and Z. Hao. 2006.
from proxy data of summer rainfall over eastern China.
Geophysical Research Letters, vol. 33, L03702, February 2006.

ftp://ftp.ncdc.noaa.gov/pub/data/paleo/historical/pacific/pdo-shen2006.txt

Looks like the ~60 year is present a long way back then. As to any longer cycles, there the problem is resolution and data length. In most case the ‘noise’ is so great you can conclude almost anything and not be proved wrong by the data unfortunately.

24. Graeme W says:

I have a couple of questions regarding the initial filter.

1. A 365 day filter has an ongoing problem due to leap-years. There are, roughly, 25 leap days in a century, which pushes a 365 day filter almost a month out of alignment. How is this catered for? It’s not a problem with a 12 month filter, but then you hit the problem that each month is not equal in length.

2. You’re doing filtering on anomalies to remove the seasonal component. Aren’t the anomalies supposed to do that themselves, since they’re anomalies from the average for each part of the season. That is, the anomalies themselves are trying to remove the seasonal component, and the filter is also trying to remove the seasonal component. How do ensure that any result we get isn’t an artifact of these two processes interacting?

Please forgive me if these questions show my ignorance, but they’ve been a concern of mine for awhile.

25. Steve Taylor says:

Richard, are you related to the late, great John Linsley Hood, by any chance ?

26. Clay Marley says:

I am not too fond of the triple running mean filter, mainly because it is not easy to implement generically in Excel. By that I mean if I wanted to change the number of points in one stage, it is a fair amount of work. Also there are three smoothing parameters to tinker with rather than a single number of regression points, and I have not seen any good theory on how to pick the three ranges to average over.

I prefer either a LOESS or Gaussian filter for this kind of data. Both can be implemented easily in Excel with macros. Both have the advantage of being able to handle points near either end of the data set. LOESS also works when the points are not equally spaced. By selecting the proper number of regression points, either one will be able to produce results virtually indistinguishable from the triple running mean filter. Also with LOESS it is possible to estimate the derivative at any point.

I have not experimented much with the Savitzky-Golay filter yet because it requires recalculating the coefficients with each change in the order or number of regression points. Can be done of course, just makes the macros more complicated. And I am not sure it is worth the effort. One of these days I’ll probably get around to it.

Any smoothing like this is mainly just a data visualization tool. How it looks is largely subjective. Rarely do I see a smoothed signal being used in later analysis, such as correlating it with something; and when I do it raises a red flag.

27. Steven Mosher says:

RichardLH

“They are two of the only series that stretch back to 1850 (which unfortunately the satellite data does not) and with Global coverage.”

like I said. wrong

28. RichardLH says:

David L. Hagen says:
March 16, 2014 at 3:01 pm

“Thanks Richard for an insightful way of exposing the 60 year cycle.”

Lots of people claim it is not there. This just shows that you cannot really ignore it that easily.

“1) Has an algebraic factor been found behind the 1.2067 factor or is this still empirical per von Pratt?”

Vaughan has a precise set of reasoning which he gave on Greg’s thread over at JCs. It is not empirical but pure mathematical.

” 2) Does using the more accurate year length of 365.26 days make any difference?”

I have found that there may be a 4 year signal in some data which would suggest that it may well be. I haven’t looked in detail at the high pass portion of the signal yet.

“3) Suggest showing the derivative of the CTRM curve.”

I hate Linear Trends and all their cousins. They only hold true over the range they are drawn from and hide rather than reveal as far as I am concerned. Use the residual, greater than 75 years as a better alternative for any long term values. You can run an S-G at that length but please be aware of S-G tendencies to be strongly influenced by the ‘end’ data..

“That would show more clearly that the rate of warming is declining. If that derivative goes though zero, that would give evidence that we are now entering the next “cooling” period vs just flattening in warming.”

I think that the S-G 15 year shows that quite well in any case.

29. RichardLH says:

Steve Taylor says:
March 16, 2014 at 3:17 pm

“Richard, are you related to the late, great John Linsley Hood, by any chance ?”

My Uncle. He got me interested in Audio (and Science) a long time back and was the originator of the 1.3371 value I was originally using in my first CTRM digital work.

30. RLH The Christiansen data I linked to are I think archived. Read the paper and see why I think they do provide a temperature record which is good enough to be useful for analysis. It should replace the Mann hockey stick in the public perception of the NH climate over the last 1000 years.
Again check Figs 3 and 4 at http://climatesense-norpag.blogspot.com.
Forecasting is fairly straightforward if you know where you are on the 1000 year quasi-periodicity.
If you don’t take it into consideration you are lost in a maze of short term noise.

31. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 3:14 pm

“Is it possible to humor some of us old-timer digital filter designers by specifically saying what the (Finite) Impulse Response of the CTRM is”

Basically the triple cascade allows for a near to Gaussian response curve. This can be seen in greater detail on Greg’s thread where the various curves can be compared. CTRM is just slightly better than Gaussian in that it completely removes the corner frequency whereas a true Gaussian does leave a small portion still in the output.

32. RichardLH says:

Mike McMillan says:
March 16, 2014 at 3:10 pm

“Well, it certainly gets rid of the 1998 peak problem. GISS would be appreciative.”

Does throw up another question though. Why is it that GISS and HadCRUT are so far apart in the middle? They are close together at the start and the finish, why so different in the middle? I am not sure GISS (or HadCRUT) will thank me that much.

33. You use the term signal but you haven’t provided a definition for “signal”. Is this data? Is “signal” an entity of some kind? Is there a signal in the probabilities of rolling a dice? I just haven’t come across the word “signal” very often in any of the studies over all the decades of my personal studies and work, perhaps because other terminology is used. My studies have included electronics, but it didn’t involve signal processing. I also spent 10 years full time work in drawing electrical schematic diagrams, about 3 decades of computer science, and I somewhat recently spent several years studying calculus every single day, so now I call myself a mathematician.
If we are using computers to work with a signal, that means it is probably converted to discretized data, consisting of finite numbers, though you can use semantic computation, though I don’t think that’s what I see happening.
Sorry for being somewhat pedantic, but I’m just not conformable with calling the temperature of the global surface a signal, so I’m asking for a better description of “signal”.

34. david moon says:

It’s hard to tell what this is from the description in words. Can you show a block diagram or provide a spreadsheet? Also just showing the time response to a step input, or the frequency response would be helpful.

It sounds like it’s an FIR filter with no feedback of the output. If so, you should be able to multiply out the nested means and the coefficient and get 12 coefficients, one for each tap of an FIR.

One quibble: 75 years is period, not a frequency. It would be more correct to say “…frequencies with periods greater than 75 years…”

35. RichardLH says:

Graeme W says:
March 16, 2014 at 3:16 pm

“I have a couple of questions regarding the initial filter.

1. A 365 day filter has an ongoing problem due to leap-years. There are, roughly, 25 leap days in a century, which pushes a 365 day filter almost a month out of alignment. How is this catered for? It’s not a problem with a 12 month filter, but then you hit the problem that each month is not equal in length.”

True. The advantage of a continuous CTRM filter is that the full sampling frequency is preserved in the output so any error is very tiny. It you were to sub-sample to Annual then the problem you see would be present. This does not have that problem.

“2. You’re doing filtering on anomalies to remove the seasonal component. Aren’t the anomalies supposed to do that themselves, since they’re anomalies from the average for each part of the season. That is, the anomalies themselves are trying to remove the seasonal component, and the filter is also trying to remove the seasonal component. How do ensure that any result we get isn’t an artifact of these two processes interacting?”

Fortunately it doesn’t matter if you do this on Temperature or Anomalies, you end up with the same curve, just a different label to the left. The problem with ‘normals’ is that there are only 30 additions to make the normal. That will leave some error term in that which will then be present in all of the Anomaly outputs. Using a CTRM removes that residual as well rather nicely.

“Please forgive me if these questions show my ignorance, but they’ve been a concern of mine for awhile.”

Mine too which is why I use CTRM now as it is much mathematically purer.

36. RichardLH says:

Steven Mosher says:
March 16, 2014 at 3:21 pm

“RichardLH

“They are two of the only series that stretch back to 1850 (which unfortunately the satellite data does not) and with Global coverage.”

like I said. wrong”

OK. To be precise, there is only one set of thermometers from which the various treatments are drawn.

Some included more thermometers than the others, some less. I have comparisons between most of them and none come out particularly favourably. I have BEST, C&W, NCDC and others. All show surprising disparities with each other with no easy explanations.

I’ll get round to them in time but the next set is Land, Ocean and General triples for the sets I do have to hand right now.

37. RichardLH says:

Dr Norman Page says:
March 16, 2014 at 3:27 pm

“RLH The Christiansen data I linked to are I think archived.”

Thanks, I will. DO you know if the data is on the ‘net and the url? Or is it just the paper?

38. Bernie Hutchins says:

RichardLH says: March 16, 2014 at 3:30 pm

“Bernie Hutchins says:
March 16, 2014 at 3:14 pm

“Is it possible to humor some of us old-timer digital filter designers by specifically saying what the (Finite) Impulse Response of the CTRM is”

Basically the triple cascade allows for a near to Gaussian response curve. This can be seen in greater detail on Greg’s thread where the various curves can be compared. CTRM is just slightly better than Gaussian in that it completely removes the corner frequency whereas a true Gaussian does leave a small portion still in the output.”

Fine – thanks. But we need a “formula” or design procedure that leads to h(n) = …. so that we can calculate the frequency response in a known way (sum of frequency domain cosines in this case). It seems to me that SG with its optimal flatness around DC couldn’t be beaten. Because the data is noisy, spurious resonances (if any) matter. Hence the advantage of SG.

39. RichardLH says:

garymount says:
March 16, 2014 at 3:34 pm

“You use the term signal but you haven’t provided a definition for “signal”. Is this data?”

You are quite right. I do tend to use the term loosely. As far as I am concerned a ‘signal’ is any variation in the temperature data, long term. I tend to class Weather, Monthly, Annual as ‘noise’.

Just my background. Sorry if it confuses.

40. david moon says:

Assuming monthly data points: 1, 2, etc. Is this how it works?

M1 = mean(1,2,3,4)
M2 = mean(5,6,7,8)
M3 = mean(9,10,11,12)

M4 = 1.2067 x mean(M!,M2)
M5 = 1.2067 x mean(M2,M3)

OUTPUT = 1.2067 x mean(M4,M5)

Shift data one place and continue…

41. RichardLH says: March 16, 2014 at 3:51 pm
– – –
Thanks Richard, now I feel better :-)

42. RichardLH says:

david moon says:
March 16, 2014 at 3:36 pm

“It’s hard to tell what this is from the description in words. Can you show a block diagram or provide a spreadsheet?”

…Here is a spreadsheet (https://wattsupwiththat.files.wordpress.com/2010/07/uah-with-annual-ctrm.xlsx) containing UAH and a Annual CTRM.. from above

“Also just showing the time response to a step input, or the frequency response would be helpful.”

If you want freqency response graphs and the like then Greg’s thread on JCs is a better place. I left out a lot of the technical as the post got way too long elsewise.

Remove bits
…all this is at Judith Curry’s site.

http://judithcurry.com/2013/11/22/data-corruption-by-running-mean-smoothers/

Data corruption by running mean ‘smoothers’
Posted on November 22, 2013 by Greg Goodman

or visit Greg’s own site for the same article.

“One quibble: 75 years is period, not a frequency. It would be more correct to say “…frequencies with periods greater than 75 years…””

Again, sorry for the sloppy terminology. I tend to use the two without considering the exact wording required. I think in one and have to write in the other!

43. RichardLH says:

david moon says:
March 16, 2014 at 3:54 pm

“Assuming monthly data points: 1, 2, etc. Is this how it works?”

https://wattsupwiththat.files.wordpress.com/2010/07/uah-with-annual-ctrm.xlsx

Basically you run a single mean over the data. then you use that column as the input to the next stage and so on. Hence the cascaded bit.

44. A Crooks of Adelaide says:

Hi,
Just a note on my own observations with respect to the top graph

What I see is a sixty year sine curve
What I also see is a pattern of two peaks followed by a deep trough – two peaks followed by a deep trough etc. The troughs repeat every 7.5 years next one due in 2016 the peaks are every 3.75 years

I use http://www.climate4you.com/images/AllCompared%20GlobalMonthlyTempSince1979.gif
as my satelite source.

If you plot El Chichon and Pinatubo on the graph you will see how much volcanic events disrupt the this pattern ( http://www.esrl.noaa.gov/gmd/grad/mloapt.html) knocking the shoulders of the plateaus of the running average

Sure there is an oddity around 2005 where the peaks ahve their own little troughs too bad

The graphs I use are:

A = 0.18*SIN(((YEAR-1993)/60)*2*3.14159)+0.2

B = 0.1*COS(((YEAR-1982)/7.5)*2*3.14159

C = 0.25*COS(((YEAR -1980)/3.75)*2*3.14159

See you in 2016 to see how close I get!

45. Arno Arrak says:

RichardLH on March 16, 2014 at 2:03 pm

Richard, you are hung up on useless arithmetic and are defending falsified temperature curves. Both HadCRUT and GISS showed presence of a non-existent warming in the eighties and nineties. My nose tells me that Hansen is behind this. I pointed it out in my book and two years later both sources changed their data retroactively to make it conform to satellites which do not show this warming. They did this secretly and no explanation was given. To coordinate it there had to be a trans-Atlantic communication link between them. This group also includes NCDC which I did not mention before.

Another thing you talk about is volcanos: ” Volcanos are the current favourite but I think that getting just the right number and size of volcanos needed is stretching co-incidence a little too far.” Forget about volcanoes, there is no volcanic cooling. The hot gases from an eruption ascend to the stratosphere where they warm it at first. In about two years time it turns to cooling but this stays up there too and never descends into the troposphere. The alleged “volcanic cooling” is caused by an accidental coincidence of expected cooling with the location of a La Nina cooling that is part of ENSO. The entire global temperature curve is a concatenation of El Nino peaks with La Nina valleys in between. It has been that way ever since the Panamanian Seaway closed. Many so-called “experts” don’t know this and would like nothing better than to detrend it out of existence. But by so doing they are destroying information and information about the exact location of El Nino peaks is required to evaluate the effect of volcanoes. In general, if an eruption is coincident with an El Nino peak it will be followed by a La Nina valley which promptly gets appropriated for a “volcanic cooling” incident. This happened with Pinatubo. But ENSO and volcanism are not in phase and it can happen that an eruption coincides with a La Nina valley. It will then be followed by an El Nino peak exactly where the volcanic cooling is supposed to happen and volcanologists are still scratching their heads about it. This is what happened with El Chichon in Mexico that was equally as strong as Pinatubo. This lack of understanding leads to more stupidity with models. Looking at a CMIP5 feather duster I noticed that all of the threads that completely disagree with one another about the future dip down together at the locations of El Chichon and Pinatubo volcanoes, no doubt because they have volcanic cooling coded into their software. This story is actually no secret because it has been in my book all along but these “climate” scientists just don’t do their homework and are still ignorant about it.

46. A Crooks of Adelaide says:

I guess I needed to add the important bit on how I combine the formulae Sorry about that:

The overarching trend is a sixty year cycle: = A

The moving 20-month average adds a 7.5 year cycle attenuated by the truncation of
the positive peaks of the 7.5 year cycle : = A + (IF B>0, 0, ELSE = B)

The monthly average combines a 7.5 year cycle with a 3.75 year cycle (i.e. twice the
7.5 year cycle) to capture the pattern where every second trough in the 3.75 year COS
function is significantly deeper the others : = A + (3/4) * B + C

47. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 3:51 pm

“Fine – thanks. But we need a “formula” or design procedure that leads to h(n) = …. so that we can calculate the frequency response in a known way (sum of frequency domain cosines in this case).”

Well for Gaussian then Greg refers to this for the full mathematical derivation.
http://www.cwu.edu/~andonie/MyPapers/Gaussian%20Smoothing_96.pdf

For frequency responses then

and

given the various filter responses graphically.

48. RichardLH says:

March 16, 2014 at 4:02 pm

The point of this is that is NOT a curve fit of any form. It is a simple low pass treatment of the data. Any ‘cycle’ you see, you need to explain. This just shows what is there.

49. RichardLH says:

Arno Arrak says:
March 16, 2014 at 4:07 pm

“Richard, you are hung up on useless arithmetic and are defending falsified temperature curves. Both HadCRUT and GISS showed presence of a non-existent warming in the eighties and nineties.”

Low pass filters are not useless arithmetic. They are the way that you can deal with the data you have all the time. You use them all the time in Daily, Monthly and Yearly form. This is just an extension of that.

If you believe that the data has been fiddled, then this should be an ideal tool for you to demonstrate that.

50. Bernie Hutchins says:

RichardLH says:
March 16, 2014 at 4:09 pm

Thanks Richard – that’s what I thought.

But when you cascade Moving Averages (running means, rectangles, “boxcars”) you get all the nulls of the original rectangles in the cascade. When you cascade Gaussians, you get essentially the smallest bandwidth of the set (longest average). With SG, you get an attractive wide low-pass bandwidth (optimally flat) at the expense of less high-pass rejection (never a free lunch).

SG is easy to do. By coincidence I posted an application note on it just a month ago. Note my Fig. 6b for example compared to the two frequency responses you posted for me.
http://electronotes.netfirms.com/AN404.pdf

Without due regard for the frequency response, you COULD BE seeing “colored noise” due to a spurious resonance. You are certainly NOT seeing something totally spurious, because for one thing, we can “see” the 60 year periodicity already in the original data. Less noise – but no new information?

51. AlexS says:

One more instance where people want to produce omelets without eggs.

“They are two of the only series that stretch back to 1850 (which unfortunately the satellite data does not) and with Global coverage.”

52. It would be helpful to know the following:

1. What does this method tell us that a linear trend does not tell us?
2. How clear is it from the method that there is a ~60-year cyclicity in the data?
3. What tests have you performed to see whether this ~60-year cycle is synchronous with the ~60-year cycles of the great ocean oscillations? Or with the ~60-year planetary beat?
4. What are the main reasons why this method is better than other methods now in use?
5. Given that orbital characteristics have already removed any seasonal influence from the data, what need is there to remove it again?
6. Are you advocating this method as a replacement for the linear trends now used by the IPCC etc.?
7. Do you propose to get a paper on the applicability of this method to climate measurements into the learned journals, and to persuade the IPCC to adopt it?

Many thanks

53. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 4:39 pm

“Thanks Richard – that’s what I thought.

But when you cascade Moving Averages (running means, rectangles, “boxcars”) you get all the nulls of the original rectangles in the cascade.”

But that is the reason for the 1.2067 inter-stage value. It places the nulls into the centre of the previous errors and thus the cascade flattens the whole thing to Gaussian, or very nearly so.

Vaughn was kind enough to agree with me when I pointed out that digitisation and range errors dominate over any residual errors with just the three stages.

S-G is all very well, but it does have a tendency to be quite ‘whippy’ at the ends. Full kernel filters never changed when new data s added, just extend. Thus certainty comes from their output, not uncertainty.

So I use CTRM for the majority of the data with S-G only for the ends and verified one against each other for the parameters.

54. RichardLH says:

AlexS says:
March 16, 2014 at 4:42 pm

“One more instance where people want to produce omelets without eggs.

“They are two of the only series that stretch back to 1850 (which unfortunately the satellite data does not) and with Global coverage.””

Show me a Global data set that extends further back and I will use it.

55. RichardLH says:

Monckton of Brenchley says:
March 16, 2014 at 4:47 pm

“It would be helpful to know the following:

1. What does this method tell us that a linear trend does not tell us?”

Linear trends are to my mind almost the most useless of statistics. They are not really valid outside of the range from which they are drawn. A continuous function, such as a filter, is a much better guide as to what is actually happening in the available data. CTRM filters are the most accurate, simple, continuous function you can use.

“2. How clear is it from the method that there is a ~60-year cyclicity in the data?”

If there IS a ~60 year cycle in the data, then it will be present and demonstrable, by measuring peak to peak or mid point to mid point. With only two such cycles available in the data it is right at the edge of any such decision though. Nearly in ‘toss a coin’ land but signal decoders work like this all the time.

” 3. What tests have you performed to see whether this ~60-year cycle is synchronous with the ~60-year cycles of the great ocean oscillations? Or with the ~60-year planetary beat?”

I have quite a few graphs of PDO, AMO/NAO on http://climatedatablog.wordpress.com/. I am in the early days of drawing them all together and that is hopefully going to be part of my next article.

“4. What are the main reasons why this method is better than other methods now in use?”

Mathematically purer than single running means and not much more difficult to create/use.

“5. Given that orbital characteristics have already removed any seasonal influence from the data, what need is there to remove it again?”

Normal/Anomaly only has 30 additions so leaves a surprisingly large error term in that sort of work. This removes all of the quite cleanly. Can be run on either normalised or raw data and will produce the same high quality output for both.

” 6. Are you advocating this method as a replacement for the linear trends now used by the IPCC etc.?”

Absolutely.

Linear Trend = Tangent to the Curve = Flat Earth.

” 7. Do you propose to get a paper on the applicability of this method to climate measurements into the learned journals, and to persuade the IPCC to adopt it?”

My academic days are well past now. It may well be worth while trying to get a paper published though.

56. davidmhoffer says:

Since temperature doesn’t vary linearly with w/m2, this method is as useless as calculating an average temperature is in the first place. Convert all the temperature data to w/m2 and then look at all the trends with all the filters you want. Since averaging temperature data is absurd in the first place, all this accomplishes is a sophisticated analysis of an absurd data set.

The fascination which so many seem to have with trends of averaged temperature data is beyond me. It tells us precisely nothing about the energy balance of the earth, no matter how you filter it.

57. Bernie Hutchins says:

RichardLH says:
March 16, 2014 at 4:49 pm

Thanks again Richard

True enough you may be approximating Gaussian. Convolving many rectangles tends Gaussian of course. So? You are then using low-pass region that is rapidly falling from the start instead of one (SG) that could be optimally flat. What is your reason for choosing one over the other?

And I don’t understand your “S-G is all very well, but it does have a tendency to be quite ‘whippy’ at the ends.” What is “whippy”? Are you saying there are end effects on the ends – that is no surprise. Nobody really knows how to handle ends – just to caution end interpretation. Then you say: “So I use CTRM for the majority of the data with S-G only for the ends…” So you want the whippy behavior?

(Someone around here at times says smoothed data is not THE data. )

58. RichardLH says:

davidmhoffer says:
March 16, 2014 at 5:11 pm

“Since temperature doesn’t vary linearly with w/m2, this method is as useless as calculating an average temperature is in the first place.”

Since matter integrates the incoming power over its volume/density/composition and alters its temperature to suit I disagree.

59. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 5:13 pm

“True enough you may be approximating Gaussian. Convolving many rectangles tends Gaussian of course. So? You are then using low-pass region that is rapidly falling from the start instead of one (SG) that could be optimally flat. What is your reason for choosing one over the other?”

S-G alters over the whole of its length with new data. A CTRM will never alter, only extend. CTRM is very simple to do, S-G is a lot more complex.

Mathematically CTRM is much, much purer than a single running mean and only requires two extra stages, with reducing windows to get near Gaussian. With just three stages digitisation/rounding errors are larger than then error terms left when compared to a true Gaussian response.

I try to avoid placing too much hard decision making on filters other than full kernel ones and use S-G for outline guidance only.

60. RichardLH says:

Dr Norman Page says:
March 16, 2014 at 5:20 pm

Thanks.

61. davidmhoffer says:

RichardLH;
Since matter integrates the incoming power over its volume/density/composition and alters its temperature to suit I disagree.
>>>>>>>>>>>>>.

If the temperature of the earth were uniform, you’d be right. But it isn’t. I suggest you read Robert G Brown’s various articles on the absurdity of calculating an average temperature in the first place. It can easily be demonstrated that the earth can be cooling while exhibiting a warming average temperature, and vice versa.

62. Another way to filter or factor out a strong annual signal is to do a running annual difference (jan 1978-jan 1977 etc). This has the advantage of preserving a signal-to-noise ratio and reveals the relative strengths of longer cycles. If you have hourly data, you can do day to day differences. The limitation is the time resolution of the data.

63. Bernie Hutchins says:

RichardLH says:
March 16, 2014 at 5:40 pm

Thanks Richard –

(1) Somehow I don’t think you and I may not be talking about the same thing. My idea of a FIR filter (which is standard) is that it is fixed length, and convolves with the signal. A moving average and a SG do this. for example. New data is what enters the window one step at a time. The filter is not “redesigned” with each step. Isn’t this what you do? Is your system LTI? Your spreadsheet doesn’t tell me much. Diagrams and equations are often useful in signal processing of course – not just output graphs.

(2) SG is NOT “a lot more complex.” You didn’t say why the SG flat passband was not something to admire. Neither did you tell me what “whippy” meant. I have never heard that term used.

(3) Since you are smoothing (averaging) you are destroying data. If you tell me you see a periodicity of 60 years, I say – yes – I see it in the data itself. If I can’t see it already, but it comes out of the filter, I would want you to show that it emerges in excess of what would be expected just by resonance. So the larger issue is perhaps that if you want to use smoothing and therefore DESTROY information, are we wrong to ask you do define in detail what your smoothing involves, and to demonstrate why it is better than something better understood?

64. bones says:

Lance Wallace says:
March 16, 2014 at 1:42 pm

The CO2 curve (seasonally detrended monthly) can be fit remarkably closely by a quadratic or exponential curve, each with <1% error for 650 or so consecutive months:
https://wattsupwiththat.com/2012/06/02/what-can-we-learn-from-the-mauna-loa-co2-curve-2/
The exponential has a time constant (e-folding time) on the order of 60 years (i.e., doubling time for the anthropogenic additions from the preindustrial level of 260 ppm of about 60*0.69 = 42 years).
——————————————————
Last time I looked, the atmospheric CO2 concentration was increasing at about 5% per decade, which gives it a doubling time of about 140 years.

65. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 6:11 pm

“Thanks Richard –

(1) Somehow I don’t think you and I may not be talking about the same thing. My idea of a FIR filter (which is standard) is that it is fixed length, and convolves with the signal. A moving average and a SG do this. for example. New data is what enters the window one step at a time. ”

A CTRM is just an extension of the standard moving average. Nothing more. The same rules apply. New data enters, old data leaves. The window is a stacked set which has a nearly Gaussian distribution to its weighting values if you work them all out.

“(2) SG is NOT “a lot more complex.” You didn’t say why the SG flat passband was not something to admire. Neither did you tell me what “whippy” meant. I have never heard that term used.”

If you compare one set with another as time evolves then you will see how the S-G has a tendency for its outer end to move around in an almost animal like manner with new data. That Wiki link has a nice animation which shows the underlying function as its passes up a data set which displays it quite well.Too many animations of images in the work I have done elsewhere I suppose, it just looks like a caterpillar to me :-) Sorry.

(3) Since you are smoothing (averaging) you are destroying data.

Wrong. You are assigning stuff to a pass band or a stop band. In fact if you take the output and subtract if from the original data (as in 1-x) you end up with the high pass filter version. Data is never destroyed. High Pass output plus Low Pass output always equals the original data set.

66. RichardLH says:

davidmhoffer says:
March 16, 2014 at 5:47 pm

Pulsed input is also integrated by matter as well so I will still differ.

67. RichardLH says:

fhhaynie says:
March 16, 2014 at 5:53 pm

“Another way to filter or factor out a strong annual signal is to do a running annual difference (jan 1978-jan 1977 etc). This has the advantage of preserving a signal-to-noise ratio and reveals the relative strengths of longer cycles. If you have hourly data, you can do day to day differences. The limitation is the time resolution of the data.”

A single stage filter, either running average or difference, has horrible mathematical errors in its out output.

One look at the frequency response will tell you that.

68. A Crooks of Adelaide says:

RichardLH says:
March 16, 2014 at 4:11 pm
March 16, 2014 at 4:02 pm

The point of this is that is NOT a curve fit of any form. It is a simple low pass treatment of the data. Any ‘cycle’ you see, you need to explain. This just shows what is there.

I’m more interested in the whats left after you take the low pass out. Its the short term cycles that determine if the monthly “Arghhh its getting hotter!” or “Arrgh its getting colder!” is significant or not.

And I dont go along with this … “if you cant explain the cycle, it doesnt exist” story They were predicting eclipses in the Bronze Age and very useful it proved too. You have to find the cycles first – then you start to think what causes them

69. RichardLH says:

Dr Norman Page says:
March 16, 2014 at 5:20 pm

” To look at the longer wavelengths you really need FFT and wavelet analysis of the Holocene data ”

The problem with FFTs and Wavelets is two fold.

1) Large amounts of noise make longer wavelengths very difficult to pick out. You end up with very broad peaks which all COULD be signal.

2) They are both very bad if the signal is made up of ‘half wave’ mixtures. Such as a mixture of 2, 3 and 4 years half waves in some random combination. Or, say, 55, 65, 75 year half waves mixed up to make a 65 year average. Nature, when not operating as a tuned string which is most of the time, has a habit of falling into just such combinations.

The noise is the problem really. In both value and phase. There are no nice clean signals which we can work with.

70. Off the subject:

Most of the time, just read here go elsewhere on the net and seek out those who need to join in and link back here. The first word of this post “Crowdsourceing” struck me.
What we are missing is a “Crowdsourceing” person from the warming side to focus more voters to this site and others like this site and or to pull people out of their daily lives and into the struggle.
Back in say 2006 to 2008 old Al Gore was all about and never ever shut his yap.
Seems we need to find a way to get someone as polarizing Al Gore together with others like him to once more and get them all back into the fray and go all bombastic and over the top with wild claims. Think up a way to gig his or some other ones of their over sized egos and get them in the press and then use that to pull in more people of reason to the battle ground.
Not that these types will ever go away but we need to get this lie based fraud redistribution of wealth pulled way down to earth.

71. RichardLH says:

March 16, 2014 at 6:45 pm

“You have to find the cycles first – then you start to think what causes them”

Well I suspect that we have the Daily, Monthly and Yearly cycles pinned down quite well now. Its the natural cycles longer than 15 years that I am interested in. Nothing much except something big at ~60 years between 15 and 75 years as far as I can tell.

72. Bart says:

RichardLH says:
March 16, 2014 at 1:58 pm

“It is useful to compare how the CO2 curve matches to the residual after you have removed the ~60 ‘cycle’ and there it does match quite well but with one big problem, you need to find something else before 1850 to make it all work out right.”

There’s a bigger problem. It doesn’t really match well at all. The match is between the integral of temperature and CO2.

And, that’s GISS. The match with the satellite record is even better.

CO2 is not driving temperatures to any level of significance at all. It is, instead, itself accumulating due to a natural process which is modulated by temperatures. Once you remove the trend and the ~60 year periodicity, which have been around since at least 1880, well before CO2 is believed to have increased significantly, there is very little left of temperature to be driven.

73. Steven Mosher says:

Richard. They all use different stations

74. RichardLH says:

Steven Mosher says:
March 16, 2014 at 7:03 pm

“Richard. They all use different stations”

As I said, they are all drawn from the same set of thermometers. The selection differs, set to set. When looking at the oldest ones at greater than 150 years or so, the choices available are fairly restricted though.

I have comparisons between the various offerings. They do differ and in some surprising ways. Lots of work still to do :-)

75. RichardLH says:

Bart says:
March 16, 2014 at 7:00 pm

I was pointing out that whilst there may be a superficial match, it fails when comparing to earlier data so therefore may be suspect on those grounds alone.

In any case, unless the temperature starts rising quite quickly fairly soon, the CO2 case gets weaker by the month.

76. Bernie Hutchins says:

RichardLH said in part March 16, 2014 at 6:38 pm
“Too many animations of images in the work I have done elsewhere I suppose, it just looks like a caterpillar to me :-) Sorry.
http://en.wikipedia.org/wiki/File:Lissage_sg3_anim.gif

Richard – Thanks yet again. That helps. Glad we are LTI.

But you are completely wrong (misled by that animation I think) about whipping. None of those yellow polynomials in the animation you posted whipping about are ever even computed. Instead the SG fit is a FIXED FIR filter. Now, to be honest, the first time I learned this, the instructor’s words were exactly: “Astoundingly, this is an FIR filter.” I WAS astounded – still am. No whipping.

[ Aside: In general, polynomials as models of signals are going to be unsuited. Signals are mainly horizontal, while polynomials are vertically in a hurry to get to + or – infinity. Your whipping. ]

All you need do is populate a non-square matrix with integers (like -5 to +5 for length 11), raise these integers to successive powers for the rows below, take the least-squares (pseudo) inverse for the non-square matrix, and use the bottom row as the impulse response. (It is basically “sinc like”.) That’s the nuts and bolts – no polynomial fitting, and always the same answer for a given length and order. See my previous app note link. Yea – magic!

So it is no more complicated (once rather easily computed) than Gaussian. And it’s a much better filter.

77. Matthew R Marler says:

Does anybody know what the result will look like when the “correct” smoothing method is used?

Is this better than piecewise polynomial regression or smoothing by projection on b-splines with knots estimated from the data (Bayesian adaptive regression splines as used by R. E. Kass with neuronal data analysis)?

It might be a useful pedagogical exercise to have a booklet of graphs, with each of 4 or more temperature series smoothed by many different methods, not excluding polynomials and trig polynomials (with fixed and estimated periods) of up to order 10..

78. davidmhoffer says:

RichardLH
Pulsed input is also integrated by matter as well so I will still differ.
>>>>>>>>>>>>>>>.

It most certainly is not. The tropics are net absorbers of energy, the arctic regions are net emitters. Massive amounts of energy are moved from tropics to arctic regions by means of air and ocean currents, and these are not integrated by matter as they would be if the sole mechanism was conductance. You’ve got an interesting way of looking at the data, just apply it to data that is meaningful.

79. A beautiful example of frequency content that I expect to be found in millennial scale uncut temperature records is found in Lui-2011 Fig. 2. In China there are no hocky sticks The grey area on the left of the Fig. 2 chart is the area of low frequency, the climate signal. In the Lui study, a lot of the power is in that grey area.

Lui-2011 Fig.2 Power Spectrum shows large power density at frequencies below 0.05/yr (i.e. cycle times >= 20 years). They note significant power spikes at 1324, 800, 199, 110 Years (and by visual estimate, ~70 and 27 yrs, too).

Copied from a post at ClimateAudit Oct. 31, 2011 Best Menne Slices

Lui sees a minor 60 (or 70) year cycle. But it seems less significant that the others.

80. Bernie Hutchins says:

Matthew R Marler said in part: March 16, 2014 at 7:23 pm
“Does anybody know what the result will look like when the “correct” smoothing method is used?”

No. Nobody knows! Unless you have a physical argument of other evidence, curve fitting is near useless. And the physics – let’s call it “incomplete”. Again, data fit to a curve is not THE data. Curve fitting destroys information. You can always get the proposed curve (again), but you can’t go back.

Many years ago I did a physics lab experiment and plotted data on every possible type of graph paper I could find in the campus store. My professor (Herbert Mahr) was kind enough to compliment my artistic efforts, but then assured me that none of it necessarily MEANT ANYTHING. Indeed.

81. pat says:

meanwhile, the MSM/CAGW crowd are sourcing this today:

16 March: Phys.org: Greenland implicated further in sea-level rise
An international team of scientists has discovered that the last remaining stable portion of the Greenland ice sheet is stable no more.
The finding, which will likely boost estimates of expected global sea level rise in the future, appears in the March 16 issue of the journal Nature Climate Change…
“Northeast Greenland is very cold. It used to be considered the last stable part of the Greenland ice sheet,” explained GNET lead investigator Michael Bevis of The Ohio State University. “This study shows that ice loss in the northeast is now accelerating. So, now it seems that all of the margins of the Greenland ice sheet are unstable.”
Historically, Zachariae drained slowly, since it had to fight its way through a bay choked with floating ice debris. Now that the ice is retreating, the ice barrier in the bay is reduced, allowing the glacier to speed up—and draw down the ice mass from the entire basin…
http://phys.org/news/2014-03-greenland-implicated-sea-level.html

82. John West says:

Sorry guys but this is absolutely worthless. There’s no data-set worth an iota pre-satellite era and post-satellite era lacks enough data to be useful. Even if there were a data-set that was accurate back to circa 1850 that’s still an incredibly short snippet of time wrt Milankovitch Cycle lengths and even the supposed Bond Cycle lengths.

83. John West says:
March 16, 2014 at 8:17 pm
Sorry guys but this is absolutely worthless.

In that case, how would you prove that the warming that is occurring now is not catastrophic?
NOAA and Santer suggested straight lines should be used. Perhaps curves should be used, but it may be harder to define when models are useless in terms of sine waves.

84. Truthseeker says:

jai mitchell says:
March 16, 2014 at 2:39 pm
. . .hilarious. . .
—————————————————-
You quote skepticalscience, and you laugh at other people?
Now that is hilarious.

85. DocMartyn says:

Richard, does the fact that months are of unequal length screw the plot.

86. Lance Wallace says:

bones says:
March 16, 2014 at 6:12 pm

“Last time I looked, the atmospheric CO2 concentration was increasing at about 5% per decade, which gives it a doubling time of about 140 years.”

Bones, recall we are looking at the doubling time of the anthropogenic contribution. One must subtract the constant value of preindustrial times, as I mentioned. The best-fit exponential says that value is about 256.7 ppm. It has been more than 42 years since the first Mauna Loa measurements in March of 1958 so we should see the doubling occurring around the year 2000. In fact that first observation in 1958 was about 314.4 ppm–subtracting 256.7 we get 57.7 ppm added to the preindustrial. When that hits 115.4 we have our doubling. That happened in December 2001, when the observed value was 372.2 ppm. (subtracting 115.4 gets us back to the preindustrial level.) Ergo, observed doubling time is 43 years, pretty close to the calculated 42.

If you want to know the time until the preindustrial concentration is doubled, extending this present best-ft exponential into the future shows it crossing a value of 512 ppm CO2 in 2050. Some say the preindustrial concentration was 280 ppm, so the doubling to 560 ppm occurs in 2061. Note this is not a prediction, just a report on what the present best-fit 3-parameter exponential through Sept 2013 shows. On the other hand,considering the track record of this fit, (<1% error for some 650 consecutive months), I wouldn't bet against it.

87. Richard says:

Hello Richard,

I have had an opportunity to look at your spreadsheet, btw, thanks for sharing it. I have some observations and questions about the methods used.
1. You use three non causal cascaded FIR filters, with 8, 10, and 12 taps respectively.
2. These are just plain averaged filters without any weighting.
3. While doing this will provide you a pseudo Gaussian response, why not use some weighting parameters for a Gaussian response?
4. By using even number taps you are weighting the averages to one side or the other in the time series, and this will affect your phase relationship.
5. Were any windowing corrections considered?
6. Were there any Fourier analysis performed on the data before and after filtering?

I use filters on data all of the time in my work, and am always concerned when someone does filtering and fails to mention Fco, Order, and Filtering Function (i.e. Bessel, Butterworth, Gaussian, etc).

Thanks again,

Richard

88. FrankK says:

jai mitchell says:
March 16, 2014 at 2:39 pm
“Since this constant addition of CO2 is not causing any warming it follows that the theory of enhanced greenhouse warming is defective. It does not work and should be discarded.”

. . .hilarious. . .
——————————————————————————————————–
Even more hilarious is if you do the calcs for the volume of ocean down to the depth quoted then the graph you quote represents only a few hundredths of 1 degree Centigrade rise in temperature. i.e. 2/5ths of bugger-all. And there is no evidence that the CO2 bogey is responsible.

89. Bernie Hutchins says:

RichardLH

Regarding our Savitzky-Golay Discussion: Here is a less cryptic but just single page outline of the SG.m program (first 20 trivial lines at most!) , offered with the Matlab pinv function, and without it, making it possible with any math program that will invert an ordinary square matrix. The example is order three (order 3 polynomial), length seven (7 points moving through smoother). The URL of the app note is on the jpg and in the thread above.

Bernie

90. In examining share (stock) prices I use a multiple moving average tool (in the Metastock program). I have given the short term moving averages (less than 30 days) a different color to the long term moving averages (30 to 60 days). When the long term and short term cross to is a signal of a change which can be used as a buy or sell (with other signals). Have you considered other time periods for your filter and putting them on the same graph. maybe that will point to the unwarranted adjustments made to data or a significant natural shift which many say occurred around 1975 (at least in the Pacific & nothing to do with CO2)
Talking of CO2, there were measurements going back to the early 1800’s see here http://www.biomind.de/realCO2/realCO2-1.htm . There many be questions about accuracy but results show variations and throw doubt on the ASSUMED constant CO2 levels prior to 1966.

91. Jeef says:

On the Y axis: CO2 concentration

On the X axis: temperature over time

X and Y are not necessarily connected. Unless funding depends on a shallow statement in your research paper conclusion to allow this.

92. Bernie Hutchins says:

wbrozek says in part March 16, 2014 at 8:33 pm
“In that case, how would you prove that the warming that is occurring now is not catastrophic?”

You ask a good question and since no one else is apparently still up, I will give it a shot.

In physics, particularly with regard to chaotic non-linear systems (like the climate), but still constrained, there is really no such thing as “proof”. There is instead evidence. The various so-call “global temperature” series, although surely imperfect, are evidence (not proof) that the catastrophic “hockey stick” warming is likely wrong.

The closest thing to establishing the truth is almost certainly the 2nd Law of Thermodynamics. Essentially heat moves from where there is more to where there is less. If the path of flow is not obviously in place, Nature is OBLIGED to provide it. In consequence, certain negative feedback thermostatting mechanisms are mandatory. No engineer or HVAC technician required. The great physicist (is there any other kind of physicist) Eddington famously and amusingly said, often quoted here I believe:

“If someone points out to you that your pet theory of the universe is in disagreement with Maxwell’s equations—then so much the worse for Maxwell’s equations. If it is found to be contradicted by observation—well these experimentalists do bungle things sometimes. But if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.”

Any folks we know!

93. Willis Eschenbach says:

First, Richard, thanks for your work. Also, kudos for the R code, helped immensely.

My first question regarding the filter is … why a new filter? What defect in the existing filters are you trying to solve?

Now, be aware I have no problem with seeking better methods. For example, I use my own method for dealing with the end points problem, as I discussed here.

However, you say:

A 12 month/365 day CTRM filter completely removes the annual ‘cycle’, as the CTRM is a near Gaussian low pass filter. In fact it is slightly better than Gaussian in that it completely removes the 12 month ‘cycle’ whereas true Gaussian leaves a small residual of that still in the data.

Mmmm … if that’s the only advantage, I’d be hesitant. I haven’t run the numbers but it sounds like for all practical purposes they would be about identical if you choose the width of the gaussian filter to match … hang on … OK, here’s a look at your filter versus a gaussian filter:

As you can see, the two are so similar that you cannot even see your filter underneath the gaussian filter … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?

Next, you go on to show the following graphic and comment:

Fig 3: HadCRUT4 with additional greater than 15 year low pass. Greater than 75 year low pass filter included to remove the red trace discovered by the first pass.

Now, when reviewing the plot above some have claimed that this is a curve fitting or a ‘cycle mania’ exercise. However, the data hasn’t been fit to anything, I just applied a filter. Then out pops some wriggle in that plot which the data draws all on its own at around ~60 years. It’s the data what done it – not me! If you see any ‘cycle’ in graph, then that’s your perception. What you can’t do is say the wriggle is not there. That’s what the DATA says is there.

There is indeed a “wiggle” in the data, which incidentally is a great word to describe the curve. It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase. Let me show you why, using your data:

The blue dashed vertical lines show the troughs of the wiggles. The red dashed vertical lines show the peaks of the wiggles.

As tempting as it may be to read a “cycle” into it, there is no “~ 60 year cycle”. It’s just a wiggle. Look at the variation in the lengths of the rising parts of the wiggle—18 years, 40 years, and 41 years. The same is true of the falling parts of the wiggle. They are 29 years in one case and 19 years in the other. Nothing even resembling regular.

The problem with nature is that you’ll have what looks like a regular cycle … but then at some point, it fades out and is replaced by some other cycle.

To sum up, you are correct that “what you can’t do is say the wriggle is not there”. It is there.

However it is not a cycle. It is a wiggle, from which we can conclude … well … nothing, particularly about the future.

Best regards,

w.

94. Greg Goodman says:

That is very interesting.The short term (decadal) correlation of d/dt(CO2) to SST has been discussed quite at bit and is irrefutable. The ice core records seem to show direct CO2 vs temp correlation. So the question is where in between does it change? There will be a sliding mix as the response changes from one extreme to the other.

If earlier data diverges it may be the first indication of the centennial relationship or equally likely an indication of spurious data adjustments. That would be worth investigating.

Do you have a similar graph that goes back to say 1850?

95. Greg Goodman says:

Willis: “As you can see, the two are so similar that you cannot even see your filter underneath the gaussian filter … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?”

It is not indistinguishable , as Richard correctly says it totally removes an annual signal. You attempt to show this is “indistiguishable” by using them to filter temperature “anomaly” data that has already had most of the annual signal removed.

Now do the same with actual temps and you will see the difference. The advantage of having a filter than can fully remove the huge annual cycle is that you don’t need to mess about with “anomaly” data which themselves leak , possibly inverted 12mo signal as soon as the annual cycle changes from that of the reference period.

The one defect I see with triple RM filters is that they are a three pole filter and thus start to attenuate the signal you want to keep ( gaussian being similar has the same defect).

SG has a nice flat pass band, as does the lanczos filter that I also provided code and graphs for in my article.

Perhaps Richard could convert the lanczos into R code too ;)

96. Greg Goodman says:

Willis: “As tempting as it may be to read a “cycle” into it, there is no “~ 60 year cycle”. It’s just a wiggle. ”

There is a long term change in there too Willis. Cooling at end of 19th c , warming since beginning of 20th. When you add a slope to a pure cosine you will find that it shifts the peaks and troughs, when you have differing slopes behind it they will get moved back and forth. That is the cause of the irregularity of the intervals you have noted.

Once again, you display your lack of understanding and over-confidence in your own abilities, then start telling others the way it is. You, like our host, seem to have adopted “the science is settled” attitude to cycles. Just remember that science is never settled and keep an open mind.

It is true that there is no guarantee that this will continue but so for it seems to fit quite well to the “pause” now becoming a down turn since 2005.

97. Greg Goodman says:

Nice article Richard.
Once criticism is the 75y RM . Having roundly accepted how bad and distorting these damn things are you still put one in there. And what is shows is not that much use even if we could believe it.

In general, when the kernel length appears to be a problem you will get a “smoother”, non distorted result with 3RM of half the period. That will not help remove a 60 periodicity.

BTW, I think this ‘cycle’ is closer to a full-wave rectified cosine : abs(cos(x)) than a fully harmonic oscillation. Heavy filtering tends to round things off a bit two much and everything ends up looking like cosines.

This seems to fit arctic ice, though it’s way too soon to see how this up tick will go.
http://climategrog.wordpress.com/?attachment_id=783
Similar abs_cos form in AMO and cyclone ACE:
http://climategrog.wordpress.com/?attachment_id=215

My money is on circa 128y abs_cos rather than 60y full cosine. Fourier techniques will tend to mask this too unless the analyst looks carefully at the presence of harmonics.

Great article though. The traffic on WUWT should be helpful in getting greater awareness of the defects and distortions of runny means.

98. Greg Goodman says:

two much = too much ;)

99. Kelvin vaughan says:

Filter it enough and you will end up with a straight line showing a slow rise in temperature.

100. george e. smith says:

Filters throw away information.

The raw data, is the most information you can ever have.

Filters simply delude one into believing that something else is happening; other that what the measured sampled data values tell.

101. Greg Goodman says:

“Filters throw away information.”
No, filters separate information. What you “throw” away is your choice.

“Filters simply delude one into believing that…”

Delusion is in the eye of the beholder. That some may draw wrong conclusions or make inappropriate analyses is not a reason never to analyse data.

This whole “the data IS the data” meme is totally ignorant. You are not going to learn much by staring at noisy dataset with a massive annual signal except that the data is noisy and has a strong annual cycle. If that is all you feel competent to do them by all means stay where you feel comfortable. That does not means that no one ever got any useful information and that centuries of expertise in data processing are nothing but delusional.

Get real.

102. Greg Goodman says:

Bernie Hutchins. “… the instructor’s words were exactly: “Astoundingly, this is an FIR filter.” I WAS astounded ”

So am I.

I’ve always been dubious of LOESS filters because they have a frequency response that changes with the data content. I’ll have to have a closer look at SG.

It does have a rather lumpy stop band though. I tend to prefer Lanczos for a filter with a flat pass band ( the major defect of gaussian and the triple RM types ).

103. Orson says:

The back-and-forth comments from our more mathematically learned friends is quite excellent for us noobs. All too often, technical analysis goes under the radar of ordinary minds. Thanks, Anthony, for giving us a chance to see some ‘worksheets’ being used…up?

104. I am very grateful to RichardLH for his courteous answers to my questions.

But I should like more detail on why the particular function he has chosen is any better at telling us what is happening in the data than the linear trends used by the IPCC etc. No trend on stochastic data has any predictive value. And we already know from the linear trend on global temperature that the rate of warming is very considerably below what was predicted – indeed, that there has been no global warming to speak of for about a quarter of a century. So, what does your curve tell us about global mean surface temperature that we do not already know?

105. Willis Eschenbach says:

Greg Goodman says:
March 17, 2014 at 12:48 am

Willis:

“As you can see, the two are so similar that you cannot even see your filter underneath the gaussian filter … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?”

It is not indistinguishable , as Richard correctly says it totally removes an annual signal. You attempt to show this is “indistiguishable” by using them to filter temperature “anomaly” data that has already had most of the annual signal removed.

Now do the same with actual temps and you will see the difference. The advantage of having a filter than can fully remove the huge annual cycle is that you don’t need to mess about with “anomaly” data which themselves leak , possibly inverted 12mo signal as soon as the annual cycle changes from that of the reference period.

Not true in the slightest, Greg. You really should try some examples before uncapping your electronic pen.

Once again, the CTRM filter underneath the gaussian filter is scarcely visible … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?

w.

106. cd says:

Richard or others

Nice concise post. I have three questions – sorry.

1) Are the cascade filters based on recursive operations using some basis functions? If yes, then is this not akin to curve fitting? I’ve probably misunderstood but I think you’ve taken issue with this before.

2) In the field I work in we tend to use the Butterworth filter extensively; but in light of the recent post on “signal stationarity” in WUWT, does such an approach seem inappropriate if the composed data series is the sum of many non-stationary signals. I suspect that there will be major issues with significant localised phase shifts (something I understand to be a problem with the Butterworth filter even with “perfect” data sets).

3) Finally, for the purposes of quant operations, is the filtered data any use beyond signal analysis? For example, even if one uses the best filtering method, does the resulting processed series not take with it significant bias: when one tries to statistically quantity something like correlation between two filtered data sets. This seems a little open ended I know, but you see this type of analysis all the time.

107. Willis Eschenbach says:

Greg Goodman says:
March 17, 2014 at 1:25 am

Willis: “As tempting as it may be to read a “cycle” into it, there is no “~ 60 year cycle”. It’s just a wiggle. ”

There is a long term change in there too Willis. Cooling at end of 19th c , warming since beginning of 20th. When you add a slope to a pure cosine you will find that it shifts the peaks and troughs, when you have differing slopes behind it they will get moved back and forth. That is the cause of the irregularity of the intervals you have noted.

And you know that this “is the cause of the irregularity” exactly how?

Once again, you display your lack of understanding and over-confidence in your own abilities, then start telling others the way it is.

Me? I’m the one saying it’s premature to claim cycles with periods in the temperature record. You’re the one trying to tell us the way it is. You’re the one claiming that there are regular cycles in the temperature data. I’m the one saying we don’t know the way it is, and until we do it’s foolish to ascribe it to some mythical “approximately sixty year” cycles …

w.

PS—A fully worked out example, showing the “differing slopes” and how they change a pure cosine wave into the shape shown by the HadCRUT4 data would advance your cause immensely … remember, according to you, you can do it using only a single pure cosine wave plus “differing slopes behind it”. Note that you need to have upswings of 18 years, 40 years, and 41 years. You also need to have downswings of 29 years in one case and 19 years in the other.

I say you can’t do that with only the things you described, a single pure cosine wave plus differing slopes. I say you’re just waving your hands and making things up in an unsuccessful attempt to show that I’m wrong.

But hey, I’m willing to be convinced … break out the pure cosine wave and the differing slopes and show us how it’s done!

108. steveta_uk says:

Richard, can you design a filter to remove comments from the people who don’t believe the data in the first place?

Why they bother commenting is a mystery to me ;(

109. RichardLH says:

davidmhoffer says:
March 16, 2014 at 7:34 pm

“RichardLH
Pulsed input is also integrated by matter as well so I will still differ.
>>>>>>>>>>>>>>>.

It most certainly is not.”

It most certainly is. Both inputs and outputs are. That was you original point you made and I still dispute your claim.

“The tropics are net absorbers of energy, the arctic regions are net emitters. Massive amounts of energy are moved from tropics to arctic regions by means of air and ocean currents, and these are not integrated by matter as they would be if the sole mechanism was conductance.”

Mass transport, phase change and the like are different beasts. They have their own rules but none of them alter what I said.

“You’ve got an interesting way of looking at the data, just apply it to data that is meaningful.”

Thank you, I do.

110. D. Cohen says:

What happens when you apply the same filters to the sunspot data over the last 150 or so years?

111. RichardLH says:

Stephen Rasey says:
March 16, 2014 at 7:45 pm

“A beautiful example of frequency content that I expect to be found in millennial scale uncut temperature records is found in Lui-2011 Fig. 2.”

Ah – the power spectrum graphs.

“Lui sees a minor 60 (or 70) year cycle. But it seems less significant that the others.”

The problem with all these sort of studies is the same as with FFTs and Wavelets (see above). Great if you have noise free data. The more the noise you have, the less the usefulness.

Also that point about ‘half wave’ mixing applies. Nature rarely does things with full sine waves (unless it is a tuned string or the like). Most stuff is a lot more complex and very, very noisy/erratic.

Stuff comes and goes into than noise and makes it very difficult to see reliably. Proxy data is usually worse as each one adds its own variety of noise to pollute the answer.

So with a long proxy record there are some with the resolution required to see a ~60 year record. Shen is one ftp://ftp.ncdc.noaa.gov/pub/data/paleo/historical/pacific/pdo-shen2006.txt which is a rainfall re-construction of the PDO. http://climatedatablog.wordpress.com/pdo/

I am always looking for other with the required resolution. That is one of the points of this thread.

112. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 7:46 pm

“Unless you have a physical argument of other evidence, curve fitting is near useless. And the physics – let’s call it “incomplete”. Again, data fit to a curve is not THE data. ”

But this is most definitely NOT a curve fitting exercise! Indeed this is the exact opposite. If the curve the data draws by low pass filtering does not match your theory then your theory is wrong. This helps (I hope) in that endeavour.

113. RichardLH says:

John West says:
March 16, 2014 at 8:17 pm

“Sorry guys but this is absolutely worthless. ”

IYHO presumably. This does allow quite a detailed look at the temperature series that are available. It allows for meaningful comparisons between those series.

114. RichardLH says:

DocMartyn says:
March 16, 2014 at 8:50 pm

“Richard, does the fact that months are of unequal length screw the plot.”

True I adjust them all to a 1/12 spacing (these are really tiny bar graphs with a width of the sample rate) so there is some jitter associated with that. I don’t think it will affect the outcome in any detectable way.

115. RichardLH says:

Richard says:
March 16, 2014 at 9:18 pm

Thank you.

“I have some observations and questions about the methods used.
1. You use three non causal cascaded FIR filters, with 8, 10, and 12 taps respectively.”

Correct

” 2. These are just plain averaged filters without any weighting.”

Correct

” 3. While doing this will provide you a pseudo Gaussian response, why not use some weighting parameters for a Gaussian response?”

Because creating a function that provides a true Gaussian Kernel and then iterating that kernel over the input doesn’t provide any more accuracy. Occam’s Razor really.

“4. By using even number taps you are weighting the averages to one side or the other in the time series, and this will affect your phase relationship.”

True but for 12 months there is no choice. The spreadsheet does take account of this as the middle column is one row higher to help reduce this. In the end it is only a small phase shift which you can correct for if you believe it is required.

” 5. Were any windowing corrections considered?”

The 1.2067 inter-stage multiplier/divider corrects all/most of the square wave sampling errors if that is what you are asking.

” 6. Were there any Fourier analysis performed on the data before and after filtering?”

No.

“I use filters on data all of the time in my work, and am always concerned when someone does filtering and fails to mention Fco, Order, and Filtering Function (i.e. Bessel, Butterworth, Gaussian, etc).”

As I said, a CTRM is a near (very, very near) Gaussian function.

For frequency responses see Greg’s work

116. RichardLH says:

Bernie Hutchins says:
March 16, 2014 at 9:56 pm

“Regarding our Savitzky-Golay Discussion: Here is a less cryptic but just single page outline of the SG.m program (first 20 trivial lines at most!) ”

Well R already contains a Savitzky-Golay which I use in a mutli-pass form

#”I ran a 5 pass-multipass with second order polynomials on 15 year data windows as per the Savitzky-Golay method.” Nate Drake PhD
SavitzkyGolay <- function(data, period=12)
{
f1 = period * 2 + 1
SavitzkyGolay = signal::sgolayfilt(data,n=f1)
SavitzkyGolay = signal::sgolayfilt(SavitzkyGolay,n=f1)
SavitzkyGolay = signal::sgolayfilt(SavitzkyGolay,n=f1)
SavitzkyGolay = signal::sgolayfilt(SavitzkyGolay,n=f1)
SavitzkyGolay = signal::sgolayfilt(SavitzkyGolay,n=f1)
}

117. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 12:22 am

“First, Richard, thanks for your work. Also, kudos for the R code, helped immensely.”

Thank you.

“My first question regarding the filter is … why a new filter? What defect in the existing filters are you trying to solve?”

Simplicity and accuracy.

“Mmmm … if that’s the only advantage, I’d be hesitant. I haven’t run the numbers but it sounds like for all practical purposes they would be about identical if you choose the width of the gaussian filter to match … hang on … OK, here’s a look at your filter versus a gaussian filter:…As you can see, the two are so similar that you cannot even see your filter underneath the gaussian filter … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?”

Actually it is just slightly better than a Gaussian. It completely removes the 12 month cycle rather than leaving a small sliver of that still in the output.

“There is indeed a “wiggle” in the data, which incidentally is a great word to describe the curve. It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase.”

The choice of words was because I know I can’t prove a ‘cycle’ with what we have. Doesn’t mean you cannot observe what is there though.

“Let me show you why, using your data: The blue dashed vertical lines show the troughs of the wiggles. The red dashed vertical lines show the peaks of the wiggles. As tempting as it may be to read a “cycle” into it, there is no “~ 60 year cycle”. It’s just a wiggle. Look at the variation in the lengths of the rising parts of the wiggle—18 years, 40 years, and 41 years. The same is true of the falling parts of the wiggle. They are 29 years in one case and 19 years in the other. Nothing even resembling regular.”

Hmmm. I would question your choice of inflexion points. To do it properly it would probably be best to de-trend the curve first with the greater than 75 years line (not a straight line!) to get the central crossing points and then do any measurements. Peak to Peak is always subject to outliers so is usually regarded as less diagnostic. But as there are only two cycles this is all probably moot anyway. If there is anything else mixed in with this other than pure sine waves then all bets are off for both period, phase and wave shape.

I just display what is there and see where it goes.

“The problem with nature is that you’ll have what looks like a regular cycle … but then at some point, it fades out and is replaced by some other cycle. ”

The interesting thing is when you do comparisons to some proxy data with the required resolution.

Then out pops some possible correlation that does need addressing.

“To sum up, you are correct that “what you can’t do is say the wriggle is not there”. It is there. However it is not a cycle. It is a wiggle, from which we can conclude … well … nothing, particularly about the future. ”

Well the 15 year S-G trend says the immediate future is downwards. You could conclude that, if the ~60 year ‘cycle’ repeats, then the downwards section is going to be 30 years long. Time alone will tell.

I have used Fourier analysis successfully to detect periodic and pseudo-periodic effects on data. When viewed in the frequency space, the peaks in the Fourier Transform identify such effects, while shallows, regions in the transform space where peaks are notably absent, identify ‘sweet spots’ where the cutoffs for filters can be tuned.
When applied to sunspot numbers. for example, there is a group of peaks corresponding to 11-13 year periods and a secondary group at about a 100 year period, but relatively little amplitude in between.
That is about the limit of applicability of the Fourier transform, however. The periodicity inherent in the mathematics makes it useless for extrapolations or for dealing with secular trends.This will allow optimization of tuned filters, however.

119. davidmhoffer says:

RichardLH;
It most certainly is.
>>>>>>>>>>>>>>>>>>>>>

How sad that both warmists and skeptics have become so obsessed with the measurement of average temperature and the analysis of it that we’ve completely forgotten what the original theory was that we were trying to prove or disprove. The theory is that doubling of CO2 increases downward LW by 3.7 w/m2. So, to determine if the theory is correct or not, we run right out and analyze a metric that has no linear relationship to w/m2 at all, and rationalize that somehow this is OK because the earth integrates it out for us.

One of the first things one ought to learn about understanding a problem is to use data that is as closely related to the problem as possible. Temperature is one order of data removed from the actual problem. Supposing that it is “integrated” is just such a supposition. It cannot be verified except by looking at the root data first and see if that theory holds. If we designed planes and bridges using second order data, there’s be a lot of engineers with their @ss on the line trying to explain why they did such an incredibly stupid thing and wound up responsible for the deaths of so many people.

But this is climate science so wild @ss guesses based on 2nd order data are OK for both advancing and refuting the theory.

pfffffft.

120. RichardLH says:

Greg Goodman says:
March 17, 2014 at 1:48 am

“Nice article Richard.
Once criticism is the 75y RM . Having roundly accepted how bad and distorting these damn things are you still put one in there. And what is shows is not that much use even if we could believe it. ”

Thanks. You and Vaughan got me into using the correct 1.2067 value in the first place.

The single mean for the 75 year is Hobson’s choice really. The data is just not long enough for a full triple pass. Most of the error terms fall outside the pass band hopefully. I could use a 75 year S-G and, if I were to do any de-trending work, that is what I would probably use.

What it does show quite clearly that all the ‘cycle’ is gone at 75 years which was the main point. Whatever is there, it lies between 15 and 75 years and looks to be ~60 years long.

“Great article though. The traffic on WUWT should be helpful in getting greater awareness of the defects and distortions of runny means.”

Thanks again. Single means are just SO bad.

121. RichardLH says:

Kelvin vaughan says:
March 17, 2014 at 1:59 am

“Filter it enough and you will end up with a straight line showing a slow rise in temperature.”

Actually you don’t. You end up with the greater than 75 year curve which is at the limit of what the data shows.

122. RichardLH says:

george e. smith says:
March 17, 2014 at 2:05 am

“Filters throw away information. The raw data, is the most information you can ever have. Filters simply delude one into believing that something else is happening; other that what the measured sampled data values tell.”

Not true. Low pass/High pass filters/splitters always add together to produce the whole of the input signal. No data is ever ‘lost’. You just get to look at a sub-section of it.

If you want the High pass values do a 1-x (i.e. take the output from the 15 year low pass and subtract it from the input signal making sure you get the phase right – which will be a challenge for the 12 month) and you will get that data instead.

123. RichardLH says:

Monckton of Brenchley says:
March 17, 2014 at 2:45 am

“I am very grateful to RichardLH for his courteous answers to my questions. ”

No problem.

“But I should like more detail on why the particular function he has chosen is any better at telling us what is happening in the data than the linear trends used by the IPCC etc.”

A linear trend is the ultimate in removing useful information.

Decide for yourself if this treatment of the UAH data for Land, Ocean and Combined provides a better insight into what happened in the last 34 years or a set of OLS straight lines.

“So, what does your curve tell us about global mean surface temperature that we do not already know?”

That things are much more naturally cyclic than the IPCC would like to concede. The biggest thing that the IPPC try to avoid is “how much of all of this is natural?”

124. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 2:52 am

“Once again, the CTRM filter underneath the gaussian filter is scarcely visible … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?”

Try that again without the full data points to make the difference almost two flat lines and you will see.

125. RichardLH says:

cd says:
March 17, 2014 at 3:04 am

“Nice concise post. I have three questions – sorry.”

Thanks and no problem.

“1) Are the cascade filters based on recursive operations using some basis functions? If yes, then is this not akin to curve fitting? I’ve probably misunderstood but I think you’ve taken issue with this before.”

No – this is most definitely NOT curve fitting. It is just a high quality low pass filter that is simple to implement. Much purer mathematically than the single running means often used.

“2) In the field I work in we tend to use the Butterworth filter extensively; but in light of the recent post on “signal stationarity” in WUWT, does such an approach seem inappropriate if the composed data series is the sum of many non-stationary signals. I suspect that there will be major issues with significant localised phase shifts (something I understand to be a problem with the Butterworth filter even with “perfect” data sets).”

This is pure Gaussian – or even slightly better than Gaussian in that it completely removes the 12 month ‘cycle’ without any additional distortions.

“3) Finally, for the purposes of quant operations, is the filtered data any use beyond signal analysis? For example, even if one uses the best filtering method, does the resulting processed series not take with it significant bias: when one tries to statistically quantity something like correlation between two filtered data sets. This seems a little open ended I know, but you see this type of analysis all the time.”

Well the option exists to do straight comparisons with other similarly filtered signals to see what differences, co-incidence exists without moving to statistics. You know what they say about statisticians :-)

126. RichardLH says:

steveta_uk says:
March 17, 2014 at 3:12 am

“Richard, can you design a filter to remove comments from the people who don’t believe the data in the first place? Why they bother commenting is a mystery to me ;(”

Beyond my capabilities I’m afraid.

127. RichardLH says:

D. Cohen says:
March 17, 2014 at 3:33 am

“What happens when you apply the same filters to the sunspot data over the last 150 or so years?”

I haven’t looked at the sunspot data yet.

128. RichardLH says:

March 17, 2014 at 4:18 am

“I have used Fourier analysis successfully to detect periodic and pseudo-periodic effects on data.”

FFTs suffer great problems with noisy data. Especially at the longer wavelengths as you approach the data series length. Given the noise in the temperature data series I would be very cautious with anything that has less than about 5 (or better still 10) cycles in it when using FFTs..

129. RichardLH says:

davidmhoffer says:
March 17, 2014 at 4:19 am

“How sad that both warmists and skeptics have become so obsessed with the measurement of average temperature and the analysis of it that we’ve completely forgotten what the original theory was that we were trying to prove or disprove. The theory is that doubling of CO2 increases downward LW by 3.7 w/m2. So, to determine if the theory is correct or not, we run right out and analyze a metric that has no linear relationship to w/m2 at all, and rationalize that somehow this is OK because the earth integrates it out for us.”

I notice that you kinda skipped the part where I refuted your claims. Matter integrates incoming and outgoing power, continuous and pulsed, and shows the result as temperature. Or are you claiming different?

“pfffffft.”

Indeed.

130. davidmhoffer says:

RichardLH;
I notice that you kinda skipped the part where I refuted your claims. Matter integrates incoming and outgoing power, continuous and pulsed, and shows the result as temperature. Or are you claiming different?
>>>>>>>>>>>>>>>>>>

On a long enough time scale maybe. On the time scale you are using here, absolutely not.
A perturbation to the system which changes the transport of energy from tropics to poles will result in a smaller change in temperature to the tropics and a larger one in the arctic regions. Average the temperatures out on a time scale too short to integrate them out and you get a result that is over represented by changes to cold temperatures and under represented by changes to warm temperatures.

We’re talking about a perturbation to the system that is decades to centuries long by the most conservative estimates. So no, you cannot rely on matter to integrate the result for you on time scales that you are using.

131. RichardLH says:

davidmhoffer says:
March 17, 2014 at 4:52 am

“On a long enough time scale maybe. On the time scale you are using here, absolutely not.”

On any time scale from milliseconds (or below) on upwards actually. Individual chunks of matter do this all the time. Which was the point I was refuting and you keep dodging.

“A perturbation to the system which changes the transport of energy from tropics to poles will result in a smaller change in temperature to the tropics and a larger one in the arctic regions. Average the temperatures out on a time scale too short to integrate them out and you get a result that is over represented by changes to cold temperatures and under represented by changes to warm temperatures.”

I agree that to get to a steady state through mass transport with the varying inputs we have will take a very long time and probably be cyclic. ~60 years would fall into that category. The greater than 75 years would be another.

“We’re talking about a perturbation to the system that is decades to centuries long by the most conservative estimates. So no, you cannot rely on matter to integrate the result for you on time scales that you are using.”

Well as that is how it does it I am not sure you are supporting your case.

132. Steve Taylor says:

Richard,
John Linsley Hood was one of the greatest analogue electronic design engineers the UK has ever produced. I wish I’d known him. I spent a lot of my time at university back in the 80’s building his designs when I should have been using my project budget on, well, my project.

I am sad he’s gone, but I don’t think he’ll be forgotten.

133. cd says:

RichardLH

With regard to your answer to my question 2. I appreciate that the Butterworth filter (frequency domain) can be seen as an approximation of a Gaussian convolution given complementary parameters. And I’m not talking about the relative merits of either. I guess what I’m really asking is: Is there a problem in designing a filter based on spectral profile (“hard” approach ), as opposed to eye-balling the parameter in the host domain (“soft” approach), if the composed signal (the raw data) is the sum of many non-stationary signals?

In my field of study, the Butterworth is effectively a passband filter where the higher frequencies are typically removed, according to the filter parameters which are determine from the series’ spectral profile. But the recent discussion, with respect to signal stationarity, would deem such an approach foolhardy rather than proper, and yet it’s used extensively.

I was interested in hearing your views on this if you have time.

I hope it doesn’t seem off topic.

134. davidmhoffer says:

RichardLH;
Well as that is how it does it I am not sure you are supporting your case.
>>>>>>>>>>>>

Doubling of CO2 changes the effective black body temperature of earth by precisely zero. There is your integration. Yet it increases downward LW flux. The latter is not integrated on the time scales that you are using, and you can’t measure it as an average by measuring and averaging temperature instead because temperature varies with the 4th root of P.

135. RichardLH says:

Steve Taylor says:
March 17, 2014 at 5:15 am

“John Linsley Hood was one of the greatest analogue electronic design engineers the UK has ever produced. I wish I’d known him. I spent a lot of my time at university back in the 80′s building his designs when I should have been using my project budget on, well, my project.

I am sad he’s gone, but I don’t think he’ll be forgotten.”

Between them he and my father (still alive) brought me up to be interested in engineering and science from a very early age. I doubt he will be forgotten. He used two very simple tools in his work, a precise notch [filter] and a very pure sine wave generator. Allowed him to look in great detail at the work he did with the results you saw.

136. RichardLH says:

EDIT; a precise notch filter

137. Edim says:

One is tempted to conclude,

“It’s the Sun stupid!”

138. Matthew R Marler says:

RichardLH: But this is most definitely NOT a curve fitting exercise! Indeed this is the exact opposite.

Using a word from Willis Eschenbach’s post (the word, I am not attributing an idea to him, though I agree with his main point), what you have done is indistinguishable from curve fitting. “Filtering” is nothing more than than fitting data by a method that uses a set of basis functions, and then separating the results into two components (as said by Greg Goodman), one fit by the method and the basis functions, the other not.

If you know the noise, you can design a filter to reveal the signal. If you know the signal, you can design a filter to reveal the noise. When, as with modern temperature series, you have no claim to know either, you can fit dozens, maybe hundreds (Donoho calls the method “basis pursuit”) of filters equally well to the same data set, each claiming some superiority in some cases. With some filters, “the wiggle is there”, as you wrote; with some filters the wiggle is not there. Whether the “wiggle” is generated by an underlying process that persists and can be described by physical laws can not be determined by looking at the filtering results on the extant data..

139. RichardLH says:

cd says:
March 17, 2014 at 5:20 am

“I guess what I’m really asking is: Is there a problem in designing a filter based on spectral profile (“hard” approach ), as opposed to eye-balling the parameter in the host domain (“soft” approach), if the composed signal (the raw data) is the sum of many non-stationary signals?”

I guess what I am really saying is that by only making a simple 15 years low pass corner frequency choice I am not making any (or very few) assumptions about all of this and what is or is not present.

If you run a standard ‘sweep’ up the available bandwidth, then there is a ‘sweet spot’ between 12-20 years where the output doesn’t change much.

Typically that is always going to be a good choice for a binary chop point, hence the 15 year corner.

The rest is what then comes out.

140. David L. Hagen says:

Richard LH
For a future endeavor, you might find it interesting to explore the Global Warming Prediction Project. and how their automatic model results compare with your filtered data.

141. RichardLH says:

davidmhoffer says:
March 17, 2014 at 5:21 am

“Doubling of CO2 changes the effective black body temperature of earth by precisely zero. There is your integration. Yet it increases downward LW flux. The latter is not integrated on the time scales that you are using, and you can’t measure it as an average by measuring and averaging temperature instead because temperature varies with the 4th root of P.”

I get you don’t think that measuring temperatures or seeing how they evolve over time is a useful exercise. But you logic is faulty as to the methodology of how energy is absorbed/emitted from an object and the temperature of that object over time. IMHO.

142. RichardLH says:

Matthew R Marler says:
March 17, 2014 at 6:12 am

“Using a word from Willis Eschenbach’s post (the word, I am not attributing an idea to him, though I agree with his main point), what you have done is indistinguishable from curve fitting.”

It is not. Curve fitting is deciding a function and then applying it to the data. No functions were harmed in the making of these graphs. :-)

” “Filtering” is nothing more than than fitting data by a method that uses a set of basis functions, and then separating the results into two components (as said by Greg Goodman), one fit by the method and the basis functions, the other not.”

Bandpass splitting, which is what low/high pass filters do, is more a binary choice type of exercise. No decisions are made or needed about what is or isn’t three. Only that it should be longer or shorter than the corner frequency.

“If you know the noise, you can design a filter to reveal the signal. If you know the signal, you can design a filter to reveal the noise.”

Signal = Greater than 15 years
Noise = Less than 15 years
Filter = CTRM or Gaussian

“When, as with modern temperature series, you have no claim to know either, you can fit dozens, maybe hundreds (Donoho calls the method “basis pursuit”) of filters equally well to the same data set, each claiming some superiority in some cases.”

Individual frequencies can be targeted or even groups of frequencies. That is not what is being done here. A binary chop, nothing more, nothing less.

“With some filters, “the wiggle is there”, as you wrote; with some filters the wiggle is not there. Whether the “wiggle” is generated by an underlying process that persists and can be described by physical laws can not be determined by looking at the filtering results on the extant data.”

Well whatever your logic there is something with some power in the 15 to 75 year bracket that the data says is present and needs explaining.

143. RichardLH says:

Edim says:
March 17, 2014 at 5:53 am

“One is tempted to conclude, “It’s the Sun stupid!””

One would need a physical method to connect one to the other in a cyclic fashion. So far nothing has made it past the required scrutiny. Those that fit the cycle seem to lack the power and vice versa.

144. RichardLH says:

David L. Hagen says:
March 17, 2014 at 6:22 am

“For a future endeavor, you might find it interesting to explore the Global Warming Prediction Project. and how their automatic model results compare with your filtered data.”

As soon as I see Linear Trends you have lost me.

Linear Trend = Tangent to the curve = Flat Earth.

It is just that sort of narrow minded thinking that leads one down blind alleyways and into blind corners.

Nature never does things in straight lines, it is always curves and cycles and avalanches. Almost never in sine waves either unless we are talking about orbits and the like. Ordered/Constrained Chaos is at the heart of most things.

145. Henry Clark says:

Now the question is how I can improve it. Do you see any flaws in the methodology or tool I’ve developed?

Because HADCRUT4 data by CRU is greatly incorrect (such as within parts of the 1930-1980 period), as is that from Hansen’s GISS, unfortunately any analysis based on HADCRUT4 is also greatly incorrect. While such has the global cooling scare of the 1960s-1970s occur without substantial cooling beforehand in the global or Northern Hemisphere average, as if it just happened with little basis, I would challenge you or anyone (if inclined to defend it, though you might not be) to find any publication made prior to the CAGW movement of the 1980s onward which shows the 1960s-1970s without showing far more cooling relative to the warmer period of the 1930s-1950s.

For instance, a 1976 National Geographic’s publication of the temperature record of scientists of the time, in the following link, is one I have literally seen in paper form in a library, unlike those repeatedly rewritten-later electronic versions which by strange coincidence (not!) happen to be respectively by a group infamous in Climategroup and by a department which has been under the direction of someone so much an activist as to have been arrested repeatedly in protests:

See that and other illustrations of original temperature data around 40% of the way down in http://tinyurl.com/nbnh7hq

Are there any particular combinations of data sets that you would like to see?

While it would be a significant amount of work, digging up original non-rewritten temperature data for history up through the 1970s (prior to the political era, back when there wasn’t motivation to fudge it), digitalizing it, and then carefully joining it onto later temperature data (from more questionable sources but when no alternative) would be a better starting point. Joining together two datasets like that isn’t in theory ideal but the best option in practice; several years of overlap could be used to help check the method. The Northern Hemisphere average, the Southern Hemisphere average, and the global average could best be each generated, as there are reasons, for instance, Antarctic temperatures follow different patterns than the arctic (as the prior link implies).

Of course, like me, you might not have time to do so in the near future even if you desired. But that is something needed yet utterly lacking, as browsing enough skeptic websites indirectly illustrates. The ideal for convenience of other analysis would be both producing plots and producing a list of the values by year (e.g. a data spreadsheet).

146. davidmhoffer says:

RichardLH;
But you logic is faulty as to the methodology of how energy is absorbed/emitted from an object and the temperature of that object over time. IMHO.
>>>>>>>>>>>>>>>>>>

By your logic, there is no need to average temperature across the earth for analysis at all. The matter having done the integration, all that is required according to you is a single weather station which will over time be representative of the earth’s energy balance. Good luck with that.

147. Henry Clark says:

I would challenge you or anyone (if inclined to defend it, though you might not be) to find any publication made prior to the CAGW movement of the 1980s onward which shows the 1960s-1970s without showing far more cooling relative to the warmer period of the 1930s-1950s.

The Berkeley BEST set made in the modern day by someone who pretended to be a skeptic without pro-CAGW-movement bias but was an environmentalist (as found out in looking at some of his writing beforehand IIRC) does not even remotely meet that challenge. Only original paper publications (or a clear scan not looking rewritten in any way) made prior to the existence of the global warming movement would count.

148. RichardLH says:

Henry Clark says:
March 17, 2014 at 6:44 am

“Because HADCRUT4 data by CRU is greatly incorrect (such as within parts of the 1930-1980 period), as is that from Hansen’s GISS, unfortunately any analysis based on HADCRUT4 is also greatly incorrect. ”

Well they all draw from the same set of physical thermometers. You can take a look at the BEST database, which draws from a wide range of providers, to get a wider picture if you like.

This is more intended to analyse what is there in those series and compare and contrast them. The head figure is one where I have aligned the 4 major series over the 1979 onwards era and shown how they agree and disagree.

A later set from just UAH (CRU is pending) which compares the Land, Ocean and Combined is quite revealing.

149. RichardLH says:

davidmhoffer says:
March 17, 2014 at 6:45 am

“By your logic, there is no need to average temperature across the earth for analysis at all. The matter having done the integration, all that is required according to you is a single weather station which will over time be representative of the earth’s energy balance. Good luck with that.”

150. davidmhoffer says:

RichardLH.
>>>>>>>>>>>>>

No, you answer it. Does matter integrate energy inputs and outputs as you have argued? If so, why is more than a single weather station required for analysis?

151. RichardLH says:

davidmhoffer says:
March 17, 2014 at 6:59 am

“No, you answer it. Does matter integrate energy inputs and outputs as you have argued? If so, why is more than a single weather station required for analysis?”

Bully or what?

Well if you are going to get all technical on that then you would require a statistically representative sample of the various sub-environments that are present on the Earths’ surface.

I’ll start with these area based graphs/sampling sets as a reasonable first pass.

http://climatedatablog.wordpress.com/uah/

Probably needs to augmented by some point sampling values as well (coming up with a CRU combined analysis).

Then you might be able to get close to the true picture of how fast/slow the integration methodology in the various materials are across the whole input surface, day to day and month to month.

152. Greg Goodman says:

Henry Clark: “See that and other illustrations of original temperature data around 40% of the way down in” http://tinyurl.com/nbnh7hq

Very interesting ! In that graph early 60’s is very much the same as early 20th c. Late 19th even cooler rather than warmer as now shown. That does not tell us either is correct.

since Hansen did his little air-con con trick in 1988 , I have no reason to think that he would not be equally misleading with his constant warming adjustments.

It would also not fit a 60 year cycle.

But what we can see is that the long term changes we are seeking to explain are primarily what the various adjustments have stuck in there rather that what the measurements actually were, Whether that is for better or for worse.

153. Greg Goodman says:

MR Marler. “Filtering” is nothing more than than fitting data by a method that uses a set of basis functions, and then separating the results into two components (as said by Greg Goodman),

I said nothing of the sort. Don’t use my name to back up your ignorant claims.

154. davidmhoffer says:

RichardLH;
Probably needs to augmented by some point sampling values as well (coming up with a CRU combined analysis).
>>>>>>>>>>>>>

Probably? In other words, you don’t know.

Willis ran an article some time back on the average temperature of the moon not matching the Stefan-Boltzmann black body temperature. Does this mean SB Law is wrong? No. It means that averaging temperature in such a manner as to accurately represent the energy balance of the moon is near impossible. That’s for a body that is airless and waterless. Doing the same for the earth is orders of magnitude more complex.

I suggest you read Willis’ article as well as the various musings of Robert G Brown on these matters.

155. Henry Clark says:

RichardLH says:
March 17, 2014 at 6:55 am
Well they all draw from the same set of physical thermometers.

So did the data published prior to the 1980s, but that can be seen to be drastically different. When depicting average changes of a tiny fraction of 1% in absolute temperature, of tenths of degree, they depend utterly on the interpolation between widely spaced apart stations, the choice of specific stations used, and, when applicable, hidden adjustments. The practical, convenient way to be certain of no bias in favor of the CAGW movement, without spending thousands of hours personally examining everything, is just to use data published prior to its existence.

In addition to the examples in my prior link, there are others, such as the Northern Hemisphere average history of the National Academy of Sciences, illustrated at http://stevengoddard.files.wordpress.com/2014/03/screenhunter_637-mar-15-11-33.gif in http://stevengoddard.wordpress.com/2014/03/15/yet-another-smoking-gun-of-data-fraud-at-nasa/ , which is utterly different from the CRU/BEST rewritten versions of NH average as well as global average history.

“The head figure is one where I have aligned the 4 major series over the 1979 onwards era and shown how they agree and disagree.”

Obviously, and similar has been seen before, like the undercover CAGW movement supporters on Wikipedia publish a similar plot. However, assuming that RSS or UAH are relatively accurate for the sake of argument, correspondence with them 1979-onwards has absolutely jack to do with disproving rewriting of the pre-1979 section easier to get away with.

If you wish to argue this, then try to meet my challenge:

Find (and link or upload) any publication made prior to the CAGW movement of the 1980s onward which shows the 1960s-1970s without showing far more cooling relative to the warmer period of the 1930s-1950s.

Since you’re arguing this is merely a matter of them all using the same thermometers, that should be easy rather than impossible. Again, what is shown must have been published back then, not a BEST publication of decades later for instance.

156. Greg Goodman says:

cd. “In my field of study, the Butterworth is effectively a passband filter where…”

Many of these filters, commonly used in electroincs are not really applicable to relatively short time series. They can be programmed but usually by recursive formulae. That means they need a long ‘spin-up’ period before they converge and give a reasonably stable result that is close to intended characteristics. The spin up in practice is usually nearly as long as the data for climate!

Also they mostly have really lumpy stop-band leakage.

In electronic applications it may take a few hundreths of a second to settling then be in continuous use. Not really the same thing as climate data. Hence they are not generally much help.

That is why FIR ( finite impulse response ) filters are usually applied here.

My favourite for this use is Lanczos, unless you need to get really close to the end of the data.

157. RichardLH says:

davidmhoffer says:
March 17, 2014 at 7:17 am

“Probably? In other words, you don’t know.”

Don’t put words incorrectly into my mouth.

I was pointing out that area (actually volume) sampling instruments are all very well but can properly be supplemented by point sampling instruments. They both have different time sampling methodologies so that integration alone is non trivial.

Again we are wandering off the original point where you claimed that measuring temperature was useless and I pointed out that integration by matter into temperature over time made your claim invalid.

158. Henry Clark says:

My reply is stuck in moderation at the moment, perhaps from some word in it, probably going to appear later.

159. davidmhoffer says:

RichardLH;
Again we are wandering off the original point where you claimed that measuring temperature was useless and I pointed out that integration by matter into temperature over time made your claim invalid.
>>>>>>>>>>>>>>>

No, we’re not wandering at all. I’m explaining why and you’re coming up with excuses that you can’t defend with anything other than explanations that begin with “probably” and muse about the integration being “non trivial” but never actually answering directly the points I’ve raised.

I suggest again that you read the information that I’ve pointed you to.

160. Greg Goodman says:

Richard: “This is pure Gaussian – or even slightly better than Gaussian in that it completely removes the 12 month ‘cycle’ without any additional distortions.”

It does not have the ugly distortions of the inverted lobes in the stop band that make such a mess with simple running mean but to be accurate you should recognise that progressively attenuating all frequencies right from zero is a distortion (other than the intended “distortion” of removing higher frequencies).

Nothings perfect , it’s always a compromise. That is why at least knowing what the filter does is so important when choosing one. and why this article is so helpful. Just be realistic about how good gaussian or triple running means are.

161. Bart says:

Greg Goodman says:
March 17, 2014 at 12:34 am

“Do you have a similar graph that goes back to say 1850?”

I only really trust the CO2 data to 1958. Proxies are… proxies, i.e., not direct measurements. There are no means truly to verify them.

But, this was a hangup with Ferdinand Englebeen. He was keen to point out that, if you accept the ice core measurements as accurate, the relationship should predict too low a CO2 level before 1958, as here.

Besides the fact that I do not trust the ice core data to deliver a faithful reproduction of CO2 levels, I pointed out it was moot anyway, because knowing what happened since 1958 is enough to discount the influence of human inputs during the era of most significant modern rise.

But, if one really must insist on matching a dodgy set of data farther back, there is no reason that the relationship

dCO2/dt = k*(T – To)

must be steady. The fact that it has been since 1958 is actually quite remarkable. But, there could easily have been a regime change in, say, about 1945 which altered the parameters k and To. I showed the effect of such a regime change to To could make the records consistent here. What could that signify? Possibly an alteration in the CO2 content of upwelling waters at about that time. Maybe Godzilla emerging from hibernation stirred up a cache of latent CO2 at the bottom of the ocean. Who knows?

But, in any case, it is moot. Knowing what happened since 1958 is enough to discount the influence of human inputs during the era of most significant modern rise.

162. Bart says:

RichardLH: To the point of your post, I would say that you need a filter with near unity gain in the passband a little above 1/60 years^-1, which falls off rapidly thereafter. The Parks-McClellan algorithm was all the rage back when I was an undergraduate, and I think the Remez exchange algorithm still forms the kernel of many modern filter design algorithms. Free code for the algorithm may be readily found on the web, though it is generally in FORTRAN.

163. Greg Goodman says:

Thanks Bart, That was not a challenge to what you said, you just mentioned something about it varying earlier and I presumed you had a graph that went further back that may be been interesting. I agree pre-1958 is a different story in CO2 records.

Those who wish to infer something from ice-cores showing stable millennial scale relationships and modern climate are comparing apples to oranges. The magnitude of the long term , in-phase relationship does not tell us anything about the magnitude of the short term orthogonal relationship.

Ferdi did suggest some ice core data for circa 1500 with 50 year resolution, that may be relevant but I won’t digress too far into that discussion here.

Your graph was very interesting. I may have another look that, Thanks.

164. RichardLH says:

davidmhoffer says:
March 17, 2014 at 7:40 am

“No, we’re not wandering at all. I’m explaining why and you’re coming up with excuses that you can’t defend with anything other than explanations that begin with “probably” and muse about the integration being “non trivial” but never actually answering directly the points I’ve raised.

I suggest again that you read the information that I’ve pointed you to.”

Hmmm. I point out that matter integrates energy over time for both inward and outward flows and you never address that point.

Then you raise a straw man about how many sampling points are needed above one. I answer than in a reasonable way and you bluster on.

165. RichardLH says:

Greg Goodman says:
March 17, 2014 at 7:41 am

“It does not have the ugly distortions of the inverted lobes in the stop band that make such a mess with simple running mean but to be accurate you should recognise that progressively attenuating all frequencies right from zero is a distortion (other than the intended “distortion” of removing higher frequencies).

Nothings perfect , it’s always a compromise. That is why at least knowing what the filter does is so important when choosing one. and why this article is so helpful. Just be realistic about how good gaussian or triple running means are.”

Well as you can always get the ‘other half’ out by doing 1-x to get the high pass filter it is pretty good for a few lines of code :-)

True there will always be some blurring of frequencies around the corner value and those will show up in some measure in both pass and stop outputs instead of just one but as you say, nothing is perfect.

166. RichardLH says:

Bart says:
March 17, 2014 at 7:54 am

“RichardLH: To the point of your post, I would say that you need a filter with near unity gain in the passband a little above 1/60 years^-1, which falls off rapidly thereafter. The Parks-McClellan algorithm was all the rage back when I was an undergraduate,”

Thanks for the suggestion but the frequency response curve is way to ‘ringy’ for me.

That is the main problem with most of the higher order filters, there are none that respond well to square wave input. Gaussian is probably the best as far as that goes, hence a near Gaussian with only a few lines of code seemed best.

If you want to use true Gaussian then it will do nearly as well. Just stick with S-G rather than switch disciplines to LOWES in that case.

167. rgbatduke says:

Does throw up another question though. Why is it that GISS and HadCRUT are so far apart in the middle? They are close together at the start and the finish, why so different in the middle? I am not sure GISS (or HadCRUT) will thank me that much.

GISS has a (broken, often backwards) UHI correction. HADCRUT does not. Probably other reasons as well. The UHI correction used by GISS maximally affects the ends.

You cannot really say that they are close at the start and finish and different in the middle, because they are anomalies with separately computed absolute bases. That is, there is no guarantee that the mean global temperature computed from the corrected GISS data will correspond to that computed from the uncorrected HADCRUT4 data. Consequently, you can shift the curves up or down by as much as 1C. So perhaps they should be adjusted to be close in the middle, and maximally different at the start and finish.

There are multiple data adjustments in the two series, and while there is substantial overlap in data sources, the data sources are not identical. And then, as Steve M. notes, there are other long running series with their OWN adjustments and (still overlapping) data sources. It is amusing to see how different they are when plotted on the same axes via e.g. W4T, and then to imagine how their similarities and differences would change if one moved them up or down as anomalies around their distinctly computed global average temperatures on an absolute scale. It is even more amusing to consider how they would change if one actually used e.g. personal weather station data that is now readily available on a fairly impressive if somewhat random grid at least in the United States to compute an actual topographical UHI correction with something vaguely approximating a valid statistical/numerical basis for all of the series. It is more amusing still to consider what the series would look like with error bars, but that, at least, we will never see because if it were ever honestly plotted, the game would be over.

This is a partial entre towards not exactly an explicit correction to your approach, but rather to a suggestion for future consideration.

If one does look at your figure 2 above (HADCRUT4 vs smoothed), you will note that the removed noise scales from the right (most recent) to the left (oldest). This is, of course, a reflection of the increasing underlying uncertainty in the data as one goes into the more remote past. In 1850 we knew diddly about the temperature of the world’s oceans (which cover 70% of the “surface” in the global average surface temperature) and there were whole continental-sized tracts of the planet that were virtually unexplored, let alone reliably sampled for their temperature.

This leaves us with several chicken-and-egg problems that your filter may not be able to compensate/correct for. The most important one is systematic bias like the aforementioned UHI, or systematic bias because places like the Arctic and Antarctic and central Africa and central Australia and Siberia and most of the Americas or the world’s oceans were pitifully poorly represented in the data, and some of those surface areas are make dominant contributions to the perceived “anomaly” today (as maps that plot relative anomaly clearly show). Your smoothed curve may smooth the noisy data to reveal the underlying “simple” structure more clearly, but it obviously cannot fix systematic biases, only statistically neutral noise. I know you make no such claim, but it is important to maintain this disclaimer as the differences between different global average temperature anomalies are in part direct measures of these biases.

The second is that your smoothed curve still comes with an implicit error. Some fraction of the removed/filtered noise isn’t just statistical noise, it is actual measurement error, method error, statistical error that may or may not be zero sum. It might be worthwhile to do some sort of secondary computation on the removed noise — perhaps the simplest one, create a smoothed mean-square of the deviation of the data from the smoothed curve — and use it to add some sort of quasi-gaussian error bar around the smoothed curve. Indeed, plotting the absolute and signed difference between the smoothed curve and the data would itself be rather revealing, I think.

rgb

168. GlynnMhor says:

A better result can be obtained using an Ormsby zero phase filter:
http://www.xsgeo.com/course/filt.htm

Industrial scale treatment of seismic data uses Ormsby filters, applied in the Fourier domain, almost exclusively to frequency limit the data.

The frequency content and filter slopes can be established in the design of the filter operator, and the operator is always run from end to end of the data without truncation of the output.

169. RichardLH says:

rgbatduke says:
March 17, 2014 at 8:15 am

“GISS has a (broken, often backwards) UHI correction. HADCRUT does not. Probably other reasons as well. The UHI correction used by GISS maximally affects the ends.

You cannot really say that they are close at the start and finish and different in the middle, because they are anomalies with separately computed absolute bases.”

Well for that alignment I did a simple OLS for the whole 1979-today period for each source and then adjusted the offsets in the various series to suit. UAH, RSS, GISS and HadCRUT.

http://climatedatablog.wordpress.com/combined/

has the whole story and that was going to be the heart of my next posting (hopefully).

Added some well known proxies in the mixture to extend the data set a few more years backwards into the past as well :-)

170. RichardLH says:

GlynnMhor says:
March 17, 2014 at 8:22 am

“A better result can be obtained using an Ormsby zero phase filter:
http://www.xsgeo.com/course/filt.htm

It would be interesting to see how the output from that differed (if at all) from the CTRM. I suspect that given the data length and the time periods required the output will be nearly identical.

171. Bart says:

RichardLH says:
March 17, 2014 at 8:08 am

“Thanks for the suggestion but the frequency response curve is way to ‘ringy’ for me.”

That’s just an heuristic diagram. You can design for the amount of ripple, and can make it arbitrarily small for arbitrary length of the filter.

That is much better than a filter design which starts attenuating at any frequency above zero, like the Gaussian filter.

172. RichardLH says:

Bart says:
March 17, 2014 at 8:31 am

“That’s just an heuristic diagram. You can design for the amount of ripple, and can make it arbitrarily small for arbitrary length of the filter.”

But as far as I know that requires extra input data series length, which is the one thing we do not have.

You actually want full attenuation of zero to corner frequency, i.e. a brick wall filter, but that is precisely what a single running mean is and look what it does in the frequency domain.

Gaussian seems the best compromise all round, and CTRM is a very simple way to get that.

173. Clay Marley says:

The OP talks about using this triple mean filter to remove the seasonal component of data, but then showed an example using anomaly data, which already has the seasonal component removed.

The easiest and best way to remove seasonal components from seasonal data is to do what everyone else does, use anomalies. Then smooth that if desired with LOESS or Gaussian, which as a few have pointed out, can be made virtually indistinguishable from the triple mean filter.

I’d be more interested to see how filters work on not-quite-so-seasonal data. Sunspots for example, have varying periods so anomalies do not work. Willis has effectively criticized several papers that do a very poor job of smoothing sunspots.

Also, data on total land ice and snow, while seasonal, do not work well with anomalies. The reason is the peak snow coverage is very sharp and varies over a range of 5-6 weeks each year, leaving periodic spikes in the anomaly data.

174. RichardLH says:

Clay Marley says:
March 17, 2014 at 8:47 am

“The OP talks about using this triple mean filter to remove the seasonal component of data, but then showed an example using anomaly data, which already has the seasonal component removed. The easiest and best way to remove seasonal components from seasonal data is to do what everyone else does, use anomalies. ”

The problem with that is that the ‘normal’ is only constructed with, say, 30 samples, often with built-in errors in those samples as they are sub-sampled single running means themselves. This then gets translated into errors in the ‘normal which then get stamped into all the ‘Anomaly’ sets produce using it from then on.

An Annual filter removes those included errors and produces a mathematically more accurate result at monthly sampling rate.

If you want a look at this applied directly to temperature rather than anomalies then try

http://climatedatablog.wordpress.com/cet/

which shows the CET with just such a treatment. You could turn that into an Anomaly of self set by subtracting the overall average if you like.

175. Bernie Hutchins says:

A “zero-phase” filter is not possible with time-series data as it would generally be anti-causal. So when a signal processing engineer says zero-phase he/she usually means a FIR filter with impulse response that is even-symmetric with respect to the center. This is properly called “linear phase”, or constant time delay. The frequency response is a Fourier Series in the frequency variable – just a weighted sum of cosines in this case. A moving average is linear phase, for example, as is Guassian or SG.

Butterworth responses are traditionally IIR and not FIR, and are not linear phase. It is however possible to design a “Butterworth” linear phase by taking a suitable Butterworth characteristic, sampling the magnitude response, substituting in (imposing) a linear phase, and solving for the impulse response. The impulse response can be just the inverse Discrete-Time Fourier Transform (DTFT is slow form of FFT), which is called “frequency sampling” in the textbooks, but this often still has passband ripples (again because it is the sum of cosines after all). So a generalized form of frequency sampling is used where a very large (over-determined) set of magnitude samples is used and the inversion is done by least squares (again the “pseudo-inverse”) down to a reasonable length FIR filter. Works great.

Parks-McClellan is another good option with well defined passband and stopband ripples (Chebyshev error criterion). It is inherently FIR and linear phase. It also has no closed from derivation, hence the PM Algorithm. The book Tom Parks wrote with Sid Burrus “Digital Filter Design” is classic. Filter design is well established and readily available.

176. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 9:16 am

There are many good filter designs out there with their own particular characteristics. Few offer the benign characteristics of Gaussian though.

http://en.wikipedia.org/wiki/Gaussian_filter

…”It is considered the ideal time domain filter”.,.

Being able to implement a filter of that type with only a few lines of code (or addition steps in the case of a spreadsheet) is very difficult. CTRM manages to do so.

Also we have the requirement of a desperately short data series. Most work in other disciplines that use filters of different types have long ‘run up’ times as Greg mentions above.

So a combination of factors which means (pun) that a CTRM is a simple, Occam’s Razor, filter choice :-)

177. cd says:

Greg Goodman

They can be programmed but usually by recursive formulae.

I’m guessing this is what’s need in the time/spatial domain? I’ve only ever used the Butterworth filter in the frequency domain where as I’m sure you know things are a lot “easier”.

That means they need a long ‘spin-up’ period before they converge and give a reasonably stable result that is close to intended characteristics.

Again, my use of them is obviously far more presumptuous than yours. I simply use them as they were intended as passband filters; you obviously have greater experience here and understand the nuances better, but in the frequency domain there is no recursion and no convergence, at least not how I use them ;).

I guess, and I’d be interested in hearing your view. It is often suggested, and for argument’s sake we assume an almost exhaustive data set, that carrying out processing in the frequency domain is the correct approach to applying filters; where the data series is possible/likely the composite of other periodic series. But as the recent post relating to signal stationarity (on WUWT, where you commented) suggested, doing anything according to the “global” spectrum of a signal is fraught with danger.Therefore, applying any filter in the frequency domain is in itself creating local artifacts if the underlying periodic components are non-stationary (e.g. data series composed from several “frequency-modulated” components, I use the term loosely here for brevity).

I’m not pretending to be an expert here, but that recent post seemed somewhat contrived and a little light on the implications.

178. brians356 says:
179. Willis Eschenbach says:

RichardLH says:
March 17, 2014 at 4:14 am

Willis Eschenbach says:
March 17, 2014 at 12:22 am

“First, Richard, thanks for your work. Also, kudos for the R code, helped immensely.”

Thank you.

“My first question regarding the filter is … why a new filter? What defect in the existing filters are you trying to solve?”

Simplicity and accuracy.

You’ll have to define “accuracy” for me in terms of a smoother … how are you measuring that?

The same is true of “simplicity”. Simplicity of what? Implementation? In what computer language?

“Mmmm … if that’s the only advantage, I’d be hesitant. I haven’t run the numbers but it sounds like for all practical purposes they would be about identical if you choose the width of the gaussian filter to match … hang on … OK, here’s a look at your filter versus a gaussian filter:…As you can see, the two are so similar that you cannot even see your filter underneath the gaussian filter … so I repeat my question. Why do we need a new filter that is indistiguishable from a gaussian filter?”

Actually it is just slightly better than a Gaussian. It completely removes the 12 month cycle rather than leaving a small sliver of that still in the output.

http://climatedatablog.files.wordpress.com/2014/02/fig-2-low-pass-gaussian-ctrm-compare.png

As I showed above, your claim about the 12 month cycle is simply not true. Let me repeat it here.

There is no visible difference between your smoother and a standard gaussian smoother. Nor is there any 12 month cycle in the residuals between them (yours minus gaussian). Your claim is falsified.

And as your own link clearly shows, there is only a tiny difference between yours and gaussian, and it’s way out on the edge of the rolloff … color me unimpressed. When a difference makes a difference that is less than a line’s width on the screen, I don’t care.

“There is indeed a “wiggle” in the data, which incidentally is a great word to describe the curve. It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase.”

The choice of words was because I know I can’t prove a ‘cycle’ with what we have. Doesn’t mean you cannot observe what is there though.

I do observe what’s there. What I don’t do is call it a cycle.

“Let me show you why, using your data: The blue dashed vertical lines show the troughs of the wiggles. The red dashed vertical lines show the peaks of the wiggles. As tempting as it may be to read a “cycle” into it, there is no “~ 60 year cycle”. It’s just a wiggle. Look at the variation in the lengths of the rising parts of the wiggle—18 years, 40 years, and 41 years. The same is true of the falling parts of the wiggle. They are 29 years in one case and 19 years in the other. Nothing even resembling regular.”

Hmmm. I would question your choice of inflexion points. To do it properly it would probably be best to de-trend the curve first with the greater than 75 years line (not a straight line!) to get the central crossing points and then do any measurements. Peak to Peak is always subject to outliers so is usually regarded as less diagnostic. But as there are only two cycles this is all probably moot anyway. If there is anything else mixed in with this other than pure sine waves then all bets are off for both period, phase and wave shape.

So you’re taking up the Greg Goodman style of debate, where you just make some off-the-wall claim and then neglect to actually go see if it is true?

Look, Richard. I did the work to produce my estimate of the inflection points, which clearly shows there is not a regular cycle of any kind.

Now if you think that it is not correct because I didn’t “detrend the curve first with the greater than 75 years line (not a straight line!)”, then put your money where your mouth is and do the legwork to show us that your claim is true. I did it for my claim … your turn.

I don’t even know what kind of a “75 years line (not a straight line!)” you’d recommend we use to detrend it. So do the work and come back and show us just how much your (unknown) procedure changes the cycle lengths … my prediction is, not much …

I just display what is there and see where it goes.

No, you display what is there and then extrapolate it unjustifiably.

“The problem with nature is that you’ll have what looks like a regular cycle … but then at some point, it fades out and is replaced by some other cycle. ”

The interesting thing is when you do comparisons to some proxy data with the required resolution.

Then out pops some possible correlation that does need addressing.

That’s your evidence? A graph of the smoothed swings of the PDO? Look, there’s no regular cycle there either. I don’t have your data or code, but by eye, the Chen data (red line) has cycles of lengths 90, 90, 110, 60, and 60 years … is that a “possible correlation that does need addressing”? Is that another of the famous “approximately sixty year” cycles? Or is it an “approximately ninety years” cycle?

In any case, I don’t know if it’s a “possible correlation” that needs addressing, because you haven’t said what it was supposed to be correlated with. Is there another wiggle with cycles of 90, 90, 110, 60, and 60 years that I don’t know about?

“To sum up, you are correct that “what you can’t do is say the wriggle is not there”. It is there. However it is not a cycle. It is a wiggle, from which we can conclude … well … nothing, particularly about the future. ”

Well the 15 year S-G trend says the immediate future is downwards.

Yes … but your method (and a gaussian smooth and a loess smooth), on the other hand says it’s just gone level. Who you gonna believe?

You could conclude that, if the ~60 year ‘cycle’ repeats, then the downwards section is going to be 30 years long. Time alone will tell.

Look, Richard. Either a cycle is 60 years, or it is not. Whenever anyone starts waving their hands and saying that there is an “approximately 60 year cycle”, I tune out because there’s no such thing.

Let me go over the bidding again. If we measure from peak to peak, there are two cycles in the HadCRUT4 data, of length 48 years and 60 years. If we measure trough to trough, there are two cycles in the data, of length 70 years and 61 years.

Now if you wish to call a wiggle with “cycle lengths” of 61, 48, 70, and 60 years an “approximately 60 year cycle”, and then project that putative “cycle” out for 30 years, I can’t stop you.

I will, however, point out that such behavior is a sign of incipient cyclomania, and in the event it persists longer than four hours you should seek immediate medical assistance …

w.

180. John West says:

RichardLH says:
”This does allow quite a detailed look at the temperature series that are available. It allows for meaningful comparisons between those series.”

A detailed look at manipulated junk is still manipulated junk (IMHO). I’ll concede it does allow for comparing one manipulated junk data-set to another manipulated junk data-set and thereby revealing they’re different manipulated junk datasets. (JMHO)

Just look at your figure 4 between 1945 and 1970. Does that look like something that would cause an ice age scare?

Look, I really do appreciate your effort but I’m afraid those that have betrayed us and objectivity have ruined any attempt at data analysis for many more years decades to come.

181. John West says:

wbrozek says:
”In that case, how would you prove that the warming that is occurring now is not catastrophic?”

That’s the problem, we can’t prove it’s not and they can’t prove it is.But we have the satellites now for overwatch so if it does something dramatic like pause (…. LOL …), no, it’d have to be more dramatic than that (like drop significantly) then it’d be extremely difficult for them to explain.

182. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 10:40 am

You’ll have to define “accuracy” for me in terms of a smoother … how are you measuring that?
The same is true of “simplicity”. Simplicity of what? Implementation? In what computer language?”

Mathematically more accurate than a single mean, mathematically simpler than a Gaussian.

“As I showed above, your claim about the 12 month cycle is simply not true. Let me repeat it here. There is no visible difference between your smoother and a standard gaussian smoother. Nor is there any 12 month cycle in the residuals between them (yours minus gaussian). Your claim is falsified.”

Then these two lines would lie one over the other and they don’t. Small difference only, true, but there none the less. Claim substantiated.

“And as your own link clearly shows, there is only a tiny difference between yours and gaussian, and it’s way out on the edge of the rolloff … color me unimpressed. When a difference makes a difference that is less than a line’s width on the screen, I don’t care. ”

So you try and get a Gaussian response curve in a spreadsheet with only two extra columns and no macros. Or only a few lines of R to do the same thing. Look, if you don’t like CTRM – use Gaussian instead. No skin off my back. Move on to where it is really important, the 15 year corner frequency and why that displays a ~60 year signal. All available frequencies above 15 years drop out to something in that bracket and a final residual at longer then 75 years.

“So you’re taking up the Greg Goodman style of debate, where you just make some off-the-wall claim and then neglect to actually go see if it is true?”

No. Please don’t put words in my mouth. Ask and I will respond.

“Look, Richard. I did the work to produce my estimate of the inflection points, which clearly shows there is not a regular cycle of any kind. Now if you think that it is not correct because I didn’t “detrend the curve first with the greater than 75 years line (not a straight line!)”, then put your money where your mouth is and do the legwork to show us that your claim is true. I did it for my claim … your turn.”

Hmmm. You were the one proposing to measure two ‘cycles’ of data and expecting to see a precise answer, not me. I understand all too well what is possible and/or not possible here. I consistently say ~60 years because that is all that can reasonably be stated. In fact if you do subtract the 75 year line you get a non-symmetrical wave shape with a shorter ‘top half’ to a longer ‘bottom half’ but with just two samples I would not stake my life on it. Too many other possibilities including wave shape or other longer ‘cycles’ could also be the reason.

“I don’t even know what kind of a “75 years line (not a straight line!)” you’d recommend we use to detrend it. So do the work and come back and show us just how much your (unknown) procedure changes the cycle lengths … my prediction is, not much … ”

Well I would have thought the blue line on the graph was a clue! The main problem is that is only a single running mean and has loads of distortions in it. I could run a S-G at 75 years which would be much more likely to produce something reasonable.

I can produce the band pass splitter circuit you obviously want by subtracting one output from the other but I thought that was a step too far as yet. I’ve only just brought this subject up!

Now you are demanding it and sort of hinting I don’t know what I am doing. Not very respectful.

“No, you display what is there and then extrapolate it unjustifiably.”

An S-G curve is very, very well respected. The nearest equivalent in statistics is LOWES (which some would claim was based on it in any case). If you wish to discount LOWES then do so for S-G. Otherwise the projection stands.

“That’s your evidence? A graph of the smoothed swings of the PDO? Look, there’s no regular cycle there either. I don’t have your data or code, but by eye, the Chen data (red line) has cycles of lengths 90, 90, 110, 60, and 60 years … is that a “possible correlation that does need addressing”? Is that another of the famous “approximately sixty year” cycles? Or is it an “approximately ninety years” cycle?”

If you would like the code, just ask. You could probably figure it out given that I have a simple example with RSS as a link in the article (see above). Just replace the source data with the Shen data and jaiso data and use a de-trended HadCRUT and you will be there.

Pretty good match with the available thermometer data in their overlap periods.

“In any case, I don’t know if it’s a “possible correlation” that needs addressing, because you haven’t said what it was supposed to be correlated with. Is there another wiggle with cycles of 90, 90, 110, 60, and 60 years that I don’t know about?”

As the thermometer data only goes back to the 1800s that is all the co-relation that is possible.

“Yes … but your method (and a gaussian smooth and a loess smooth), on the other hand says it’s just gone level. Who you gonna believe?”

You are obviously looking at different graphs to me then. It is not a LOWES, it is an S-G. Verified against the Gaussian for parameter choice ‘proof’.

“Look, Richard. Either a cycle is 60 years, or it is not. Whenever anyone starts waving their hands and saying that there is an “approximately 60 year cycle”, I tune out because there’s no such thing.”

Nature has a habit of never being clockwork precise – other than for orbits and then only for a simple two body solution.

“Let me go over the bidding again. If we measure from peak to peak, there are two cycles in the HadCRUT4 data, of length 48 years and 60 years. If we measure trough to trough, there are two cycles in the data, of length 70 years and 61 years.”

As I noted above, it is quite possible that the half cycles are of different periods. That is the way nature often does this stuff. Way too early to tell for sure of course, but this is just a first step.

“Now if you wish to call a wiggle with “cycle lengths” of 61, 48, 70, and 60 years an “approximately 60 year cycle”, and then project that putative “cycle” out for 30 years, I can’t stop you. I will, however, point out that such behavior is a sign of incipient cyclomania, and in the event it persists longer than four hours you should seek immediate medical assistance …”

Thank you for your concern. However that does not make the observations go away. The data has a wriggle in it that needs explaining at around ~60 years and, so far, none of the proposed physics does that.

183. Bart says:

Willis Eschenbach says:
March 17, 2014 at 10:40 am

“Either a cycle is 60 years, or it is not.”

There are no precise cycles in nature. Even our best electronic oscillators wander in frequency over time, and those are meticulously designed with conscious intent. But, they still stay within the general neighborhood of a central frequency, and the oscillations can be extrapolated forward in time on that basis, with the prediction becoming gradually more uncertain the longer the extrapolation interval.

A lot of us saw the ~60 year cycle in the data more than a decade ago. People with their heads in the sand insisted it was “cyclomania”. The rest of us predicted it would show a turnaround mid-decade. Guess what? It did. Denial of its existence is no longer tenable.

184. RichardLH says:

John West says:
March 17, 2014 at 10:42 am

Well we have to work with what is there, not what we would like to be there. Just because you don’t like what there is, there is no reason to discard it completely.

You do realise that if a natural ~60 ‘wriggle’ is there in the data then AGW is a lot less than is currently calculated don’t you?

185. Bernie Hutchins says:

Greg Goodman said March 17, 2014 at 2:24 am in part:

“I’ve always been dubious of LOESS filters because they have a frequency response that changes with the data content. I’ll have to have a closer look at SG. It does have a rather lumpy stop band though. I tend to prefer Lanczos for a filter with a flat pass band ( the major defect of gaussian and the triple RM types ).”

I am not familiar enough with LOESS to comment. If the frequency response changes with the data, it is not a Linear Time-Invariant (LTI) filter. SG is LTI. True enough the trade-off for a flat passband is a bumpier stop-band. Always is – No free lunches.

As for Lanczos, if I recall correctly it is an impulse response that is nothing more than a truncated sinc. As such it is subject to Gibbs Phenomenon transition band ripples – perhaps 23% or so – hardly a flat passband.

These points are well-considered in the signal processing literature. There is a danger to using a “built-in” without understanding it in detail. Hope I’m not being unfair – don’t intend to be.

186. RichardLH says:

Bart says:
March 17, 2014 at 11:26 am

“A lot of us saw the ~60 year cycle in the data more than a decade ago. People with their heads in the sand insisted it was “cyclomania”. The rest of us predicted it would show a turnaround mid-decade. Guess what? It did. Denial of its existence is no longer tenable.”

Well you do have to stack up: volcanos, SO2 and CO2 in just the right order and magnitude with no lag and you CAN get there! if you try really hard :-)

187. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 10:40 am

P.S. Care to re-do your plot without the green line so that a true comparison can be made? If I plotted temperature data with the full yearly cycle still in it then any deviations of the Anomaly would be hard to spot as well.

Play fair at least.

188. Bernie Hutchins says:

Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely. If you argue that you can recover the original from the smoothed alone, in the general case, you are wrong. Inversion is often at least ill-conditioned and usually impossible due to singularities. Most useful filters have what are intended to be stopbands. As such, they generally have nulls (unit-circle zeros). The inverse filter has unit-circle poles and is useless.

A moving average has nulls. SG has nulls. Continuous infinite-duration Gaussian, I believe, has no nulls, but I’m not sure about a truncated Gaussian which is convolved with a sinc? Successive passes (cascading) through moving-average rectangles have nulls – they accumulate.

Smoothing may be curve fitting (such as intentional interpolation), but it need not be. But if you claim insight from a smoothed curve, all you can really claim is that the data provides some evidence for what you feel is a proper model of the data, and the rest you assume is noise, worthy of being discarded. This is certainly circular reasoning at least in part. Such a claim requires considerable caution and full analysis and disclosure.

189. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 12:04 pm

“Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely. If you argue that you can recover the original from the smoothed alone, in the general case, you are wrong. ”

Not true in the case of a digital low pass/high pass band pass splitter filter such as a CTRM. You can do a 1-x by subtracting one band from the input source and get the other.

Has to be the case because here is no-where else for the data to go! In digital terms only of course, analogue could have some losses :-)

(Rounding errors and the like excepted of course but we can assume they are not likely to be even close to a dominant term – quantisation errors come much,much higher than that).

190. Bernie Hutchins says:

Richard –

“Not true in the case of a digital low pass/high pass band pass splitter filter such as a CTRM. You can do a 1-x by subtracting one band from the input source and get the other.”

I said:

“Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely.”

You say you haven’t thrown data away BECAUSE you kept a copy of the input. Didn’t I say that?

191. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 12:28 pm

I did. People are always suggesting that a simple low pass filter throws information away. That is not the case here (and rarely is in climate work). The input digitisation/rounding/truncation errors are by far and away the largest term. Anything to do with the calculations in the filter are in the dust (assuming reasonable floating point operations anyway). There are only a few adds and three divisions to get to any single output value. If you do the calculations of error propagation you will see.

It becomes a sort of mantra that people roll out to say why this is not a valid treatment without considering if that is truly the case.

Sorry you just pushed one of my buttons and I responded. probably in an ill considered way. Apologies.

If we were talking about 16 or 24 bit audio or video streams I would agree that this is not completely lossless, but with the data streams we have in climate, filter errors are the least of our worries.

192. RichardLH says:
March 16, 2014 at 1:58 pm

Lance Wallace says:“Can the Triple whatsis somehow compare the two curves and derive either a lag time or an estimate of how much of the CO2 emissions makes it into the observed atmospheric concentrations?”

RichardLH :Not really. Low pass filters are only going to show periodicity in the data and the CO2 figure is a continuously(?) rising curve.

A LPF by definition, passes frequencies below it’s cut-off frequency. Since D.C. (i.e. frequency = 0) is always below the cut-off, the above statement by RLH is incorrect. A ramp-like input into a LPF will cause a lagged ramp-like output, with the lag determined by the filters group delay characteristic.

Note that it is incorrect I think to speak of an LTI filter (such as the one described here) as causing “distortion”, which as a term of signal processing art refers exclusively to artifacts caused by system non-linearity. If you can sum the output of your filter with the residual (i.e. the rejected signal) and reconstruct the input, no distortion has occurred.

Another way of saying this is that an LTI filter can not create energy at a given frequency that is not present in the input signal, even if the filter response peaks badly at that frequency. Such peaking/sidelobe/pb-ripple may extenuate frequencies you wish to extinguish (an so in a sense “distort” the variance present at that frequency relative to some other frequency band of interest), but this is not distortion as the term is normally used.

193. Bart You are right about there being no precise cycles in nature .The 60 year cycle is very obvious as a modulation of the approximate 1000 year quasi periodicity which comes and goes as it beats with other cycles. See Fig 4 at
http://climatesense.norpag.blogspot.com
It looks good at 10000.9000 8000. 5000? 2000 1000 and 0 ( the present warming)
The same link provides an estimate of the possible coming cooling based on the 60 and 1000 year periodicities and the neutron count as the best proxy for solar “activity”

194. Bernie Hutchins says:

Richard –

Except as you can construct a proper inverse filter, smoothing (filtering, reduction of bandwidth) will remove information, and here you can’t construct this inverse. You say that you have not lost information because you kept the input separately. That doesn’t count!

Nobody is talking about numerical issues of the processing. Measurement issues (systematic) are a problem as is often discussed on this blog. It is very difficult to see how smoothing helps these.

And we all already agree we can see the apparent 60 year periodicity. What does your method offer that is better, let alone new?

If you are instead suggesting that you are not discarding any important information, then make that case, both in showing that you have chosen the correct parameters for a correct model, and that you understand the errors involved.

195. Willis Eschenbach says:

Bart says:
March 17, 2014 at 11:26 am

Willis Eschenbach says:
March 17, 2014 at 10:40 am

“Either a cycle is 60 years, or it is not.”

… A lot of us saw the ~60 year cycle in the data more than a decade ago. People with their heads in the sand insisted it was “cyclomania”. The rest of us predicted it would show a turnaround mid-decade. Guess what? It did. Denial of its existence is no longer tenable.

That’s a marvelous anecdote, Bart, but kinda short on links. The high point of the detrended HadCRUT4 data you are using occurred in 2003, which means that “predictions” after that point are hindcasts. So to examine your claim that “a lot” of people predicted what happened “more than a decade ago”, what we need are links to contemporaneous accounts of say three of your and other folks’ predictions made pre-2003 that global temperatures would peak by “mid-decade”.

I mean if “a lot” of you were making such predictions as you claim, surely linking to two or three of them isn’t a big ask.

w.

PS—Here is what I think about cycles:

Does the temperature go up and down? Yes, it does, on all timescales. They are best described as wiggles.

Do these wiggles imply the existence of cycles stable enough to have predictive value? I’ve never found one that could survive the simplest out-of-sample testing, and believe me, I’ve looked very hard.

Y’all seem to think I’m refusing to look at cycles. I’m not

I’m refusing to look at cycles AGAIN, because over the decades I’ve looked at, tested, and tried to extrapolate dozens and dozens of putative cycles, and investigated reports of others’ attempts as well. I wrote an Excel spreadsheet to figure out the barycentric orbit of the sun, for heaven’s sake, you don’t get more into cycles than that. I was as serious a cyclomaniac as the next man, with one huge, significant exception … I believe in out-of-sample testing …

Net result?

Nothing useful for predictions. After testing literally hundreds of natural timeseries datasets for persistent cycles, I found nothing that worked reliably out-of-sample, and I found no credible reports of anyone else finding anything that performed well out-of-sample. Not for lack of trying or lack of desire to find it, however. I’d love to be shown something like that the Jupiter-Saturn synodic period was visible in climate data, and I’ve looked hard for that very signal … but I never encountered anything like it, or any other cycles that worked for that matter.

So, as an honest scientist, I had to say that if the cycles are there, nobody’s found them yet, and turn to more productive arenas.

w.

• brians356 says:

Willis,

I’m still waiting [for] someone to produce a hunk of hide or bone from a Sasquatch, as well.

“Where has all the rigor gone? Long time pa-a-ssing …”

196. brians356 says:

” … for someone …”

197. Willis Eschenbach says:

RichardLH says:
March 17, 2014 at 12:56 pm

People are always suggesting that a simple low pass filter throws information away. That is not the case here (and rarely is in climate work).

That doesn’t seem right to me. Any filter which is not uniquely invertible loses information.

For example, while we can get from the numbers {1,2,3,4,5} to their mathematical average of 3, there is no way to be given the average of “3” and work backwards to {1,2,3,4,5} as a unique answer. So even ordinary averaging is not uniquely invertible, and thus it loses information. The same is true for the gaussian average, or a boxcar average, or most filters.

w.

198. Willis I repeat my comment to Bart from above
“Bart You are right about there being no precise cycles in nature .The 60 year cycle is very obvious as a modulation of the approximate 1000 year quasi periodicity which comes and goes as it beats with other cycles. See Fig 4 at
http://climatesense.norpag.blogspot.com
It looks good at 10000.9000 8000. 5000? 2000 1000 and 0 ( the present warming)
The same link provides an estimate of the possible coming cooling based on the 60 and 1000 year periodicities and the neutron count as the best proxy for solar “activity”
These quasi periodicities in the temperature data suggest what periodicities in the driver data are worth investigating. You do not need to understand the processes involved in order to make reasonable forecasts – see the link above.
The chief uncertainty in my forecasts is in knowing the timing of the peak in the 1000 year quasi periodicity .As Bart says- there no precise cycles – this one seems to oscillate from say 950 – 1050
. Looking at the state of the sun I think we are just past a synchronous peak in both the 60 and 1000 year periodicities. I’d appreciate your view on this .

199. Greg Goodman says:

Bernie H: “As for Lanczos, if I recall correctly it is an impulse response that is nothing more than a truncated sinc. As such it is subject to Gibbs Phenomenon transition band ripples – perhaps 23% or so – hardly a flat passband. ”

I went into some detail about all this on hte CE thread that Richard linked at the top . I included links to freq spectra of all discussed filters including Lanczos, and provided example code implementation in the update, if anyone wants to try it.

Here is a comparison of the freq response of the symmetric triple RM that Richard is providing, a Lanczos and a slightly different asymmetric triple RM that minimises the negative lobe that inverts part of the data. (It’s a fine difference, but while we’re designing a filter, we may as well minimise it’s defects).
http://climategrog.wordpress.com/2013/11/28/lanczos-filter-script/

Close-up of the stop-band ripple:
http://climategrog.wordpress.com/?attachment_id=660

There’s link in there that provides a close up of the detail of the lobes for those interested.

Yes there is a little bump in the pass band but it’s not the gregarious ripple that is seen in SG Butterworth, Chebychev, etc., and yes it’s a symmetric kernel so linear phase response.

I used a three lobe lanczos because it does not have too much Gibbs effect. You can obtain narrower transition band at the expense of accepting more ripple and over-shoot. Horses for courses as always in filter design.

The basic idea with Lanczos is to start with the ideal step function frequency response: the ‘ideal’ filter. But this requires an infinite length *sinc function’ as the kernel to achieve it. As soon as you truncate the kernel you multiply the nice, clear step your started with by another sinc fn.

Lanczos minimises the latter defect by fading out the truncation of the kernel rather than cutting it off. How quickly you cut it off, or fade it out determines the balance between the ideal step and the ringing. Lanczos was quite genius at maths and calculated this as the optimum solution. So it is somewhat more subtle than “nothing more than a truncated sinc”.

You can’t do it in three columns in a spreadsheet but it can be defined fairly easily as a macro function and it’s not that hard to code. It’s still a simple kernel based convolution. In fact it is no more complicated to do than a gaussian.

Gaussian or 3RM are useful, well-behaved low-pass filters but they make a pretty poor basis for high-pass or band-pass since they start to attenuate right from zero. If you want to make compound filters something with at least some portion of flat passband is much better. That is why I provided a high-pass Lanczos in my article too.

200. Greg Goodman says:

@Willis:

Comparison of freq resp of gaussian and triple RM:
http://climategrog.wordpress.com/?attachment_id=424

It can be seen gauss leaks about 4% of the nominal cut-off that the 3RM blocks completely. This can be a useful difference in the presence of a strong annual cycle. If you use a thick enough pencil, and scale your graph to see the cycle that was so big you wanted to get rid of it, it may not be that noticeable.

When the 4% does not matter, there’s nothing wrong with gaussian, it’s a good filter.

201. steverichards1984 says:

“RichardLH:

Excellent post. a good description of an easy to use filter to help expose underlying behaviour.

I find it sad that some complain about cycles when you said: “If you see any ‘cycle’ in graph, then that’s your perception. What you can’t do is say the wriggle is not there”

I hope the crowd sourcing aspect will enable the multiplier (1.2067) to be refined more quickly with more R users on to it.

I find it sad that people say “where is your code and where is the data you used” when it is on your linked website.

I hope you continue to develop these ideas.

If it is significantly better than fft’s as you suggest, then I can see its usage moving into other technical areas.

202. Greg Goodman says:

Willis: “So even ordinary averaging is not uniquely invertible, and thus it loses information. The same is true for the gaussian average, or a boxcar average, or most filters.”

That is true for straight average and RM since all the weights are equal. It is not the case for a gaussian or other kernel were the weights are unique.

The reverse process is called deconvolution. It’s basically a division in the frequency domain in the same way a convolution is multiplication in the freq domain. For example you can ‘soften’ a digital image with a gaussian ‘blur’ ( a 2-D gaussian low-pass filter ). You can then ‘unblur’ with a deconvolution. There are some artefacts due to calculation errors but in essence it is reversible.

This can be taken further to remove linear movement blur, or to refocus an out of focus photo. Yeah, I’d spent years telling people once you had taken a blurred shot there was NOTHING you could do with photoshop or anything else to fix it because the information just wasn’t there to recover. Wrong!

It is even possible to remove quite complex hand shake blurs, if you can find a spot in the photo that gives you a trace of the movement. This is also the way they initially corrected the blurring caused by the incorrect radius on the Hubble space telescope mirror. They found a dark bit of sky, with just one star in the field of view. They took a (blurred) image of it and then deconvolved all further shots using the FT of the blurred star image. That got them almost back to design spec and tied them through until they could get someone up there to install corrective optics.

203. Greg Goodman says:

One way to get a better look at how regular the supposed 60 year cycle is, would be to ‘detrend’ by fitting, say a 2nd order polynomial to the unfiltered data, subtract it and then do the filter. This should produce a levelled out wiggle which can be compared to a pure cosine or peaks and troughs picked out to see how stable the cycle length and amplitude is.

Just in case anyone is inclined, I have something else I’d rather work on now.

204. Bernie Hutchins says:

Greg Goodman says:
March 17, 2014 at 2:47 pm

Greg – Regarding Lanczos: thanks for that – I learned something. It is not a truncated sinc but the truncated product of two sincs.

The two sincs have different widths, so their produce in the time domain corresponds to the convolution of two unequal width rectangles – which is a trapezoid. This is then in turn truncated (by a rectangular product of the sync product in time) so the trapezoid in frequency is smeared by a frequency domain sinc. [ We use this, (it’s an old friend), but don’t give it a special name – it’s a “linear transition region”.]

Because of the gradual transition, the truncation is not the full 23% Gibbs but rather a much smaller bump. The result would look very much like a trapezoid with small bumps both sides of the linear slope. In fact it would look like something like – EXACTLY what you plotted! [Good homework problem. In practice we would probably truncate with a Hamming window rather than with a rectangle – with virtually all the Gibbs gone. Or best of all, you minimize the frequency domain squared error and make that transition region a “don’t care” band – like with Parks-McClellan.]

I like that filter pretty well. No significant passband ripple, and a Hamming window would fix even that. Nice control over roll-off if that is important. Good candidate along with SG. Thanks for the reply.

I am not sure, however why you suggest “gregarious ripple that is seen in SG Butterworth, Chebychev, etc”. Certainly SG and BW have no passband ripple. Chebychev certainly does, but note that SG looks a lot like inverse Chebyshev.

Bernie

205. RichardLH says:

Jeff Patterson says:
March 17, 2014 at 1:46 pm

“A LPF by definition, passes frequencies below it’s cut-off frequency. Since D.C. (i.e. frequency = 0) is always below the cut-off, the above statement by RLH is incorrect. A ramp-like input into a LPF will cause a lagged ramp-like output, with the lag determined by the filters group delay characteristic.”

That is true but not what was asked which was.

Lance Wallace says:“Can the Triple whatsis somehow compare the two curves and derive either a lag time or an estimate of how much of the CO2 emissions makes it into the observed atmospheric concentrations?”

I gave an answer that I intended to indicate that a low pass filter is incapable of such a connection.

I apologise it the words I used failed to convey that meaning.

P.S. The lag in this case would be one month, i.e. the sample frequency. AFAIK.

206. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 2:00 pm

“And we all already agree we can see the apparent 60 year periodicity.”

If I thought that the statement was true and universally accepted then I would probably not have constructed the post.

“What does your method offer that is better, let alone new?”

Only a very simple way to construct a Gaussian filter. Two extra columns in a spreadsheet and a few lines of R.

The reasoning about using high quality filters to examine the temperature traces rather than a statistical analysis hold regardless of the particular filter employed.

I believe that Gaussian is the best. I believe that a 15 year low pass uncovers details that others are missing/ignoring.

207. Willis Eschenbach says:

Greg Goodman says:
March 17, 2014 at 3:06 pm
@Willis:

Comparison of freq resp of gaussian and triple RM:
http://climategrog.wordpress.com/?attachment_id=424

It can be seen gauss leaks about 4% of the nominal cut-off that the 3RM blocks completely. This can be a useful difference in the presence of a strong annual cycle. If you use a thick enough pencil, and scale your graph to see the cycle that was so big you wanted to get rid of it, it may not be that noticeable.

When the 4% does not matter, there’s nothing wrong with gaussian, it’s a good filter.

Thanks for that, Greg. However, as you might imagine (and as I mentioned above) I also looked at the residuals from the process. There is no annual signal in the residual (CTRM minus gaussian).

As a result, once again I reject the claim of both yourself and Richard that the Gaussian doesn’t remove the annual signal. Both of them remove it.

Finally, although the difference at a given point might be 4%, the difference between the filters is the difference in the integrals … and that looks by eye to be much less than 4%

w.

208. ThinkingScientist says:

RichardLH

Nice article, I like the simplicity of getting a high quality filter/splitter from just three spreadsheet columns, easy to understand and easy to apply. And simple to get the low and high pass parts without much loss. Neat.

Concerning the quasi-60 year periodicity, I was reminded to go and dig out a spreadsheet I made where I was looking at the periodicity in the temperature data too. My method was based on sine wave cross-correlation, looking for the peak cross-correlation as I swept the frequency up. Bit laborious in a spreadsheet, but I get a period of 66 years using that method. Takes more than 3 columns though…:-)

The other interesting data set to look at which is a long reconstruction (but no tree rings!) and only using other published proxies is Loehle (2007). On that data set the peak cross-correlation period (with the swept frequency cross-correlation I used) is 1560 years.

209. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 2:02 pm

“So, as an honest scientist, I had to say that if the cycles are there, nobody’s found them yet, and turn to more productive arenas.”

So as an honest scientists I observe that there is a wriggle in the current data which needs explaining and cannot just be swept under the carpet.

Something caused what we see. Any offers as to what? Co-incidence?

210. Bernie Hutchins says:

Greg Goodman replied to Willis March 17, 2014 at 3:28 pm in part:
“ Willis: “So even ordinary averaging is not uniquely invertible, and thus it loses information. The same is true for the gaussian average, or a boxcar average, or most filters.” Greg replied: “That is true for straight average and RM since all the weights are equal. It is not the case for a gaussian or other kernel were the weights are unique.” “

I believe the real issue is whether or not the filter has notches. Students are often surprised that so many digital filters have unit circle zeros, and ask why: Because so many useful filters (nearly all?) have stopbands and we go through zero to be reasonably close to zero. But inverting unit circle zeros does not work. If you notch out something, it’s gone and you can’t get it back. That is, you have poles on the unit circle (infinite response at the corresponding, former, notch frequencies).

Further it has nothing to do with unequal weights. RM has unit circle zeros, as does SG, CTRM, Lanctos, and most other filters except Gaussian. Occasionally filters don’t have stopbands (such as audio equalizers – like tone controls) and avoid unit-circle zeros. It is always possible to specify such a filter – it just may not have useful applications.

211. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 2:29 pm

“That doesn’t seem right to me. Any filter which is not uniquely invertible loses information. ”

Your intuition is wrong however. There is no data loss, only removal from this plot. The other high pass plot can be constructed that shows the removed information by subtracting this trace from the input trace.. Together they always add to the original input.

It is possible that some frequencies around the corner value will be distributed into both high and low in various mixtures, but when added back together the answer will still be correct.

212. cd says:

Greg

It is not the case for a gaussian or other kernel were the weights are unique.

Sorry as someone who has to program professional applications that is just wrong. Sometimes I wonder whether you read something online only to spout some more “smoothed” and rather “convoluted” version here…

How on earth with overlapping kernel does one untangle the interference, how would one do it EVEN IN A DISCRETE VERSION WITH THE A GAUSSIAN FILTER KERNEL!!!!!!!!!!!!!!!!!!!!!

Image processing is a very easy and an accessible way to test this:

You cannot apply a Gaussian filter to an image, save the image then pass it to someone without the original image, then given just the kernel design reconfigure reproduce the original image. There are many “deconvolution” methods but they never recreate the original image, that would need to assume continuity (and determinism) in a discretised series where the best one can hope for is semi-continuity in a stochastic series.

You sir are full of BS! And I am starting to wonder whether I’ve given Richard to much kudos his answers to my question – hardly technically demanding – have produced answers which are neither here nor there.

213. Personally I don’t see cycles, I see damping. In my days of training for my airplane pilot license I was taught about the stability of the controlling surfaces of the airplane, ailerons, rudder, the pitch, yaw and roll axis. What happens when you apply power, pitch the nose up, or down, and fatefully when you drop like a rock when you put yourself into a stall (I pursued other interests after that exercise). And one thing that I remember is the stability of flight from the damping of the flight path due to how airplanes are designed.
Years latter while studying calculus, once again I came across the damping physics but this time from a deep mathematical perspective and analysis. The graph produced looks like damping to me. I’ll have to revisit the math for damping before I can do any further analysis. Others have compared to feedbacks in electronics circuitry.
– – –
On the other hand, the filtered graph looks like the path of a zombie chase with a large lag following a drunk walker, or a Lizzy looking at the flowers.

214. Willis Eschenbach says:

RichardLH says:
March 17, 2014 at 4:40 pm

Willis Eschenbach says:
March 17, 2014 at 2:29 pm

“That doesn’t seem right to me. Any filter which is not uniquely invertible loses information. ”

Your intuition is wrong however. There is no data loss, only removal from this plot. The other high pass plot can be constructed that shows the removed information by subtracting this trace from the input trace.

So you are telling us that you can flawlessly reconstruct the original input trace, given only the result of a gaussian smoothing … but only if you have access to the original input trace??

Dang, that’s some fine detective work …

w.

215. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 4:34 pm

“As a result, once again I reject the claim of both yourself and Richard that the Gaussian doesn’t remove the annual signal. Both of them remove it.”

Are you saying that this plot of the various frequency responses is wrong?

Or that you were able to find that difference (admittedly tiny, 4% Greg puts it at) when you did your work?

As I observed it might depend on which source data set you use as digitisation/sampling errors dominate in most cases. UAH has a range of less than 200 possible sample input values and HadCRUT has less than 2000. 4% of those can be easy to miss.

216. Willis Eschenbach says:

RichardLH says:
March 17, 2014 at 4:36 pm

Willis Eschenbach says:
March 17, 2014 at 2:02 pm

“So, as an honest scientist, I had to say that if the cycles are there, nobody’s found them yet, and turn to more productive arenas.”

So as an honest scientists I observe that there is a wriggle in the current data which needs explaining and cannot just be swept under the carpet.

Something caused what we see. Any offers as to what? Co-incidence?

Richard, that is the underlying question in climate science—what causes the gradual swings in temperature. So indeed we study that.

The idea that we can squint at a smoothed temperature record and say what has caused the swings, however, is a bridge far too far.

w.

217. cd says:

RichardLH

I think you are playing fast with the rules here and being quite “obfuscatious” with your interpretation of Willis’ comments.

Any filter will result in loss of information – that is without doubt; are you using the notion that all filters preserve data (modified as such) but not information (which is correct). I have always assumed you to be honest and decent but I cannot reconcile that notion with your response to Willis’ point. You either know exactly what he meant yet decided to dismiss it with pedantic obfuscation or you didn’t understand it.

218. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 4:45 pm

“So you are telling us that you can flawlessly reconstruct the original input trace, given only the result of a gaussian smoothing … but only if you have access to the original input trace??”

I did not say that if you re-read my post. What I said was that the signal is split accurately into two parts, only one of which is plotted above. Please look carefully before you comment.

219. Greg says:

“In practice we would probably truncate with a Hamming window rather than with a rectangle”

The idea is to use the central lobe of the second sinc in the way you would use a Hamming or other windowing function, to ensure it fades to zero at the end of the kernel. This is times to be at zero crossing of the of the first sinc so as to have the smoothest fade out possible. In practice the means you usually have 3,5 or 7 etc lobes. Outside that it’s zero so the windowing sinc never goes negative. That is , in effect, a further rectangular truncation.

If you think replacing the rectangular truncation by a second bell shaped function would be better I’d like to see it, though I suspect it would do more damage to the nice trapezoidal form than it would help by reducing the ripple. For a start it would mess up the nice flat section of the pass band.

I’m always ready to be proved wrong and if you can improve on Lanczos I’d be happy to be wrong and probably adopt your filter.

BTW I usually use lanczos if I need to remove annual before doing FFT, since it does less damage to the bit I want to keep. I can remove 12mo and lower yet still have 24mo and above pretty much intact.

220. RichardLH says:

cd says:
March 17, 2014 at 4:40 pm

“You cannot apply a Gaussian filter to an image, save the image then pass it to someone without the original image, then given just the kernel design reconfigure reproduce the original image. ”

No-one – certainly not me anyway – is claiming any such thing.

What you can say is that the low pass part is an accurate part of the signal below 15 years in period. The high part of the signal can be obtained by subtracting that output from the original input (possible 1 month phase delays being correctly accounted for).

Do you know how a band pass splitter circuit work? In most cases they are cascaded low pass filters with each stage subtracted from the previous stage to reduce component count and so as not have to construct full band pass, two sided, filters for each stage.

This is the digital equivalent of such a first stage splitter.

221. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 4:50 pm

“The idea that we can squint at a smoothed temperature record and say what has caused the swings, however, is a bridge far too far.”

Well you have a fairly good centre frequency with a fairly tight bandwidth around it to look for potential clients in :-)

There has to be something doing it and I don’t buy convenient volcanos, SO2 and CO2 as being the cause.

222. Greg says:

cd “Any filter will result in loss of information ”

No, nothing is “lost”. If you have box fruit and put all the apples in one bowl and all the oranges in another, you have not “lost” the apples. You still have them. What you have is two bowls or fruit. This may help you determine the weight of apples that you have bought,. for example. You would not be able to do this with all the fruit in one box.

Once you have analysed the weight of the apples, the oranges and assorted ‘noise’ of fruit remaining in the box you can either put them all back to together, sell the oranges or eat the apples.

Nothing is “lost”

223. RichardLH says:

cd says:
March 17, 2014 at 4:51 pm

“I think you are playing fast with the rules here and being quite “obfuscatious” with your interpretation of Willis’ comments.

Any filter will result in loss of information”

No I was being accurate. There is no loss, the ‘missing’ data is in the other ‘stop’ band as it must be.

The two parts added together MUST be the input. There is a possibility that some frequencies around the corner value will ‘leak’ into both but the sum of the parts….

224. Bernie Hutchins says:

cd says:
March 17, 2014 at 4:51 pm

CD – exactly so, and Richard said the same thing to me at least twice above. Your guess!

It would be like filing your income tax by submitting half the data and then argue that nothing was actually missing because you yourself still have a full copy! That should work!

225. Greg says:

“There has to be something doing it and I don’t buy convenient volcanos, SO2 and CO2 as being the cause.”

I agree about what it’s not but you need to be careful about seeing it as a definite cycle which “must” have a cause, when there are barely two full cycles and a lot of what is there is “adjusted” to buggery.

3RM is close to gaussian and it is a trap of gaussian filters to make everything look like nice smooth sine waves. Be careful not to jump to too firm conclusions.

226. cd says:

Richard

I am disappointed to say the least, I find myself on Willis’ side whose dealing with you has been far from fair – or at least I thought.

What you can say is that the low pass part is an accurate part of the signal below 15 years in period.

Right, that is a mantra, you’ve been repeating that throughout even when it is irrelevant to the question being asked. I agree with your work but not to you answer to Willis’ rather proper question.

Do you know how a band pass splitter circuit work?

Sorry you didn’t understand me this obfuscation, it is what people do when they tie themselves in knots. Do you understand sample density prevalence in discrete series and on and on we go. I think Willis’ statement was clear your answer far from it.

In most cases they are cascaded low pass filters…

You see a cascade filter IS akin to a Basis led method such as B-Spline. Yet when asked you said it wasn’t, I really doubt that! It certainly sound like one.

227. RichardLH says:

cd says:
March 17, 2014 at 5:12 pm

“I am disappointed to say the least, I find myself on Willis’ side whose dealing with you has been far from fair – or at least I thought.”

I am sorry if I am not explaining myself better then. My fault.

“Right, that is a mantra, you’ve been repeating that throughout even when it is irrelevant to the question being asked. I agree with your work but not to you answer to Willis’ rather proper question.”

Which is what? That filters MSUT lose information? Pray tell, where does that loss go? It must go somewhere. Where do you think it is and what is its magnitude?

“Sorry you didn’t understand me this obfuscation, it is what people do when they tie themselves in knots. Do you understand sample density prevalence in discrete series and on and on we go. I think Willis’ statement was clear your answer far from it. ”

I am sorry, I’ll try again.

“You see a cascade filter IS akin to a Basis led method such as B-Spline. Yet when asked you said it wasn’t, I really doubt that! It certainly sound like one.”

A cascaded filter set as described allows for appropriate correction of the previous stage until you get to the point where any errors that are left are below the input digitisation ones.

228. RichardLH says:

Greg says:
March 17, 2014 at 5:11 pm

“3RM is close to gaussian and it is a trap of gaussian filters to make everything look like nice smooth sine waves. Be careful not to jump to too firm conclusions.”

The only conclusion I have jumped to is that there is a strong ~60 wriggle to the data that needs a good, physics based, explanation. Chance and convenience are the last choices items on my list.

229. cd says:

Greg

Once you have analysed the weight of the apples, the oranges and assorted ‘noise’ of fruit remaining in the box you can either put them all back to together, sell the oranges or eat the apples.

One should judge the weight of someones understanding by the appropriateness and clarity of their analogy. This however, is rather pathetic. And if you wonder why I am so terse and to the point, it is because you’re just as dismissive with others. If you demand such high standards with others, can I suggest you either lower your standards or stop pretending coming from such high ground to begin with.

Nothing is “lost”

So give us a workflow using a freeware image processing package (e.g. gimp). If you can do it then you’ve just got the Noble prize for physics cause I know a whole populous of astronomers (including astrophysicist) who need your brilliance right now.

230. cd says:

Sorry Nobel…

You seem the type of chap to make a big deal of this Greg.

231. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 5:10 pm

“It would be like filing your income tax by submitting half the data and then argue that nothing was actually missing because you yourself still have a full copy! That should work!”

It would be more like taking some money out of your account and still having the full value at your disposal (provided you don’t spend any of it anyway). It is just in two places now, not one. The total will be the same.

232. 1sky1 says:

No matter what low-pass filter is used to reveal to the eye the low-frequency signal components, it doesn’t change the fact that no sound basis for predicting their future is provided thereby. Unlike the tides, temperature variations are NOT simply the superposition of discrete sinusoids (line spectra); they are stochastic processes whose spectrum may contain fairly narrow-band components, but encompasses a wide band CONTINUUM of frequencies nevertheless. Even in the multidecadal range, GISP2 data show significant peaks at periods considerably different than ~60yrs. Only amateurs would contend that the precise nature of the LP filter makes a critical difference in the practical problem. Ironically enough, the great side-lobe leakage of SG filters is blithely ignored here, while arguing the “superiority” of cascaded boxcar or gaussian filters.

233. RichardLH says:

cd: What is it that I am not explaining well?

A simple low pass filter (Gaussian or whatever) is a binary chop into two bands, high and low pass. Together they sum to the original series. There is no loss (assuming that we do not get rounding errors in the calculations anyway which in this case is not even going to be close to being a factor).

This ‘filters must lose information’ is a myth. Please tell me where you think the loss is going?

234. RichardLH says:

1sky1 says:
March 17, 2014 at 5:27 pm

“Ironically enough, the great side-lobe leakage of SG filters is blithely ignored here, while arguing the “superiority” of cascaded boxcar or gaussian filters.”

Well as the output of the S-G can be directly compared to the CTRM where they overlap then any such error should be easy to see. :-)

235. Greg says:

cd, Bernie et al.

Information is only lost if you throw it away. Filtering does not throw anything away, it divides it. Richard has separated out the long term components to examine that part. Like separating out the apples to weight them. I don’t see any suggestion from him that this should now be the new surface temperature record and we should throw away the rest.

The way I’d look at this would be to try to isolate the supposed 60 y periodicity and subtract it and see what’s left. Then do a similar thing again. If there’s something around 34 years ;) subtract that. By this time the data should be long enough to do a reliable FFT on what remains.

Now that would not be possible if the information was “lost.”

Enough of this rather pointless argument, please.

236. cd says:

RichardLH

Pray tell, where does that loss go? It must go somewhere.

Now if ever there was red flag that it is it. It is transferred, depending to the coefficients (and even if dynamic) information is transferred to one part of the kernel window (data point) to another (typically THE kernel cell).

Do you know how convolution work? No honestly!

A cascaded filter set as described allows for appropriate correction of the previous stage until you get to the point where any errors that are left are below the input digitisation ones.

That is the same principle of a B-Spline, and is based in the prescribed level of the spline.

237. cd says:

Greg

I appreciate your response: but information is lost?! All you’re doing is squirming, you now seem to be moving toward a transform argument; that we move toward a kernel with a series of operations with widths 1 to n-1 then we can reconstruct the signal. I don’t think this is even right, perhaps I misunderstand but remember if I got to implement it in code then waffle wont do. Give me an algorithm in pseudocode!

I actually like Richard’s argument, but your response to Willis’s tight statement (and yes in my rather modest understanding (but I’m no novice either), shows that you guys are not so sure of your own turf but rather than coming clean you obfuscate – the ultimate condescension.

238. Greg says:

cd “One should judge the weight of someones understanding by the appropriateness and clarity of their analogy. This however, is rather pathetic.”

Well you say it’s pathetic and start getting all philosophical about the worth of a man from his analogies etc. but you don’t have any counter to the ‘pathetic’ analogy.

Nothing is lost by filtering.

Quite what you think I should do with gimp or what bearing this has on whether something is lost when using a low pass filter to split a dataset into h.f and l.f bands, is not clear.

239. Greg says:

Enough of this rather pointless argument. I’m out;.

240. RichardLH says:

cd says:
March 17, 2014 at 5:32 pm

“Now if ever there was red flag that it is it. It is transferred, depending to the coefficients (and even if dynamic) information is transferred to one part of the kernel window (data point) to another (typically THE kernel cell).”

As I pointed out above and more than once. There will be some frequencies around the corner value where some portion will end up in the wrong ‘bin’. Not all, just some. And only for a small range of frequencies.

As I deliberately chose a portion of the spectrum where there is little ‘power’ (i.e. did the sweep needed to check the output sensitivity to corner values) then any such values are going to be very small in any case.

In any case Gaussian is the least likely to cause any in-band re-assignments. It is considered to be the ‘ideal’ time domain filter which is what is required.

So I ask again, what are the losses and where are they going? For a 15 year corner Gaussian (or near Gaussian) filter specifically.

241. Bernie Hutchins says:

Greg says:
March 17, 2014 at 5:31 pm
cd, Bernie et al.

“Information is only lost if you throw it away. Filtering does not throw anything away, it divides it. Richard has separated out the long term components to examine that part. Like separating out the apples to weight them. I don’t see any suggestion from him that this should now be the new surface temperature record and we should throw away the rest.

The way I’d look at this would be to try to isolate the supposed 60 y periodicity and subtract it and see what’s left. Then do a similar thing again. If there’s something around 34 years ;) subtract that. By this time the data should be long enough to do a reliable FFT on what remains.

Now that would not be possible if the information was “lost.”

Enough of this rather pointless argument, please.”

Yes – let’s stop being silly. It’s not lost because it WAS lost but you bring it back when you subtract.
Recall all the students back to class. Tell them we have been lying for decades. Always a good idea to keep them off balance by changing the rules.

242. cd says:

Greg

Nothing is lost by filtering.

OK. And again I ask you:

1) Apply filter to image.
2) With only the filtered image and the filter kernel design.
3) Give me the original image.

I use Gimp cause others can then try your suggestions (without expense), after all no information is lost so it should be simple to do. Please tell us how to get go through steps 1 to 3 without any knowledge of other steps other than the filter design?

243. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 5:52 pm

“Yes – let’s stop being silly. It’s not lost because it WAS lost but you bring it back when you subtract.”

If you subtract the output (low pass) from the input stream (this is digital after all so that CAN be done with no phase delay as would be the case if this was analogue) then you end up with the high pass output instead. Simple maths.

The sum of the parts will always be the whole (rounding errors, etc. excepted).

244. RichardLH says:

cd says:
March 17, 2014 at 5:56 pm

“OK. And again I ask you:

1) Apply filter to image.”

Stop right there. Take that filtered output image and subtract it from the original. You get the ‘negative’ filter to what you just applied.

In this case we have, low pass – high pass. No losses.

245. cd says:

RichardLH

Again you answer another question. For all the huff and puff from Greg and yourself, Willis’ pertinent point (though poorly worded), has not been answered by either of you because for all your waffle (and while I think it may even be immaterial to Richards original point) you have backed yourselves into a corner on a sideline issue.

As I say, I feel I have been taken in by your self-asserted expertise.

246. RichardLH says:

cd says:
March 17, 2014 at 6:03 pm

“As I say, I feel I have been taken in by your self-asserted expertise.”

I am sorry if I have disappointed. Please just tell me where you think the losses are going in the 15 year corner Gaussian and I will do my best to explain further.

247. Bernie Hutchins says:

RichardLH says:
March 17, 2014 at 5:57 pm

Perhaps – to be constructive here – why not consider presenting your viewpoint in terms of “Perfect Reconstruction Filters” (related to “Sub-band Coders” and wavelets), and not up against the traditional jargon of “filtering”. That would solve everything.

248. cd says:

RichardLH

Take that filtered output image and subtract it from the original. You get the ‘negative’ filter to what you just applied.

That is not what Willis asked you.

Again you answer a different question one could assign a best fit plane say to a grey scale image (something you hate) to give you a colour gradient image, and yes by adding the residual between the original image to the gradient image you can recreate the original image. But that does not prove that the gradient image has not lost information, rather the opposite in that you need the original image to get back to the original image. And yes the same is true for processed image using a convolution filter.

249. RichardLH says:

cd says:
March 17, 2014 at 6:10 pm

“That is not what Willis asked you. ”

Remind me again just exactly what it was you think Willis asked that I have not answered to your satisfaction.

I have been playing with images, filters, compressions and the like for way too long I suspect. :-)

250. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 6:05 pm

“Perhaps – to be constructive here – why not consider presenting your viewpoint in terms of “Perfect Reconstruction Filters” (related to “Sub-band Coders” and wavelets), and not up against the traditional jargon of “filtering”. That would solve everything.”

!!! You think that would be simpler?

I just don’t get how people can claim that filters lose information then, when asked how and where the loses go if they do that, fail to come up with any explanation as to where and why. Just a bald statement of ‘fact’.

An assertion, nothing more.

251. cd says:

RichardLH

Look I actually like your points on climate variability. But Willis’ point regarding loss of “data” (I think he meant information) after convolution stands, and a sharp one too.

In my opinion you have got involved in the wrong argument here and you’re wrong. Side tracked but that is in the nature of blogs. Anyways, goodnight.

252. RichardLH says:

cd says:
March 17, 2014 at 6:21 pm

“Look I actually like your points on climate variability. But Willis’ point regarding loss of “data” (I think he meant information) after convolution stands, and a sharp one too.”

Thanks. Actually he was wrong. For a simple band pass splitter circuit in digital there are no losses as such, just possible incorrect band assignment.

It is a binary chop. Either low or high. Nowhere else to go. The parts always sum to the whole. You can choose which part to look at (or both if you require).

It is all always there.

Night.

253. Bart says:

Willis Eschenbach says:
March 17, 2014 at 4:50 pm

“The idea that we can squint at a smoothed temperature record and say what has caused the swings, however, is a bridge far too far.”

If you need to squint to see the ~60 year signal, it’s probably time to get a new lens prescription.

It also isn’t as important to say what caused it as to say what assuredly did not. Rising CO2 did not. It started before the rise, and it looks nothing like it.

254. cd: “That is the same principle of a B-Spline, and is based in the prescribed level of the spline.”

No. There is no curve fitting or basis functions involved in a discrete filter. Convolution in the time domain is mathematically equivalent to multiplication in the frequency domain. The filter’s frequency response, (ideally for a LP, the response = 1 in the pass-band and 0 for frequencies above) is multiplied by the signal’s spectrum, retaining the spectrum in the pass-band and rejection that part of the signal’s spectrum in the stop-band. A B-spline, on the other hand merges polynomial fits at the segment boundary points (the so-called knots) and thus unlike a filter, has no definable, data-independent frequency response. That’s the key point, a filter’s frequency response is invariant and independent of the data and the technique is unrelated to curve fitting algorithms.

All that being said, the point about information loss assumes the spectrum of the information of interest is confined to the pass-band of the filter. If it is not, information is surely lost. In Richard’s plot’s for instance, all information about annual variation is lost. If I’m listening to a violin concerto on my stereo and turn down the treble knob, I can no longer hear the high notes as well and again, information is lost.

Assuming the signal is spectrally localizable, it is up to the user to decide what defines the “signal” of interest and to design a filter appropriate to the task, It should be pointed out that residual (the part of the input signal rejected by the LPF) by definition attenuates long term-trend information and thus all of this information, which is what we are after, is contained in the output. That is, I think, what Greg and Riochard mean by “nothing is lost”, although that’s not how I would phrase it.

Regards,
Jeff

255. Bernie Hutchins says:

RichardLH said March 17, 2014 at 6:18 pm in part:
” !!! You think that would be simpler? ”

Not if you and/or Greg do not understand EXACTLY what I am suggesting and that this already pretty much what you are using to defend your point of view: putting all the information in two or more bands that can then be perfectly reconstructed. You seem to be trying to reinvent this notion. It’s been done, and it’s easy in concept. Even simpler! The simplest PR filters are in fact, a length-2 moving average and a length-2 differencer. Imperfect filters, but no information lost – as you want. No one will argue information loss if this is your angle. Because no information IS lost.

256. ThinkingScientist says:

To RichardLH

Just to let you let you know, I think you have been extraordinarily patient with some of the critics here. As a geophysicist myself, with more than a passing understanding of low and high pass filtering (and as an audophile and user of PA equipment including subwoofer design) I have no problem in understanding the utility of the simple (but clever) filter you have presented or the significance of the low pass/high pass threshold being a non-ciritcal choice over a wide range of frequencies. Makes sense to me.

Some of the criticisms on this web page from people with apparently high IQ’s are quite strange. Its as though they want to pick a fight with what seems to me a simple and pragmatic way of implementing a low pass/high pass splitter filter with no significant side effects. In a spreadhsheet. With jut 3 columns.

Simples.

257. cd says:

Jeff Patterson

No. There is no curve fitting or basis functions involved in a discrete filter.

Perhaps I’m missing it, but I don’t think he is using a discrete filter in the sense of the sample window.

Convolution in the time domain is mathematically equivalent to multiplication in the frequency domain.

Thanks for that, I’d assumed we all knew that. Although binary filters such as mode/median kernel operations have no equivalent. And please don’t involve wavelet transforms at this stage!

All that being said, the point about information loss assumes the spectrum of the information of interest is confined to the pass-band of the filter.

Well there’s the money statement!

I think we’re all in agreement and WIllis is right on the point of loss of “data”. Richard and Greg knew exactly what he meant (loss of information) but thought by admission on being wrong on one point meant all was wrong. Bit like Richard Dawkins, who would have anyone who challenges established Darwinian theory as being some sort of creationist (akin to denier). On the side of science, APPARENTLY, he proclaims there shall be no debate! How enlightened these experts are.

258. cd says:

259. cd says:

RichardLH

For a simple band pass splitter circuit in digital there are no losses as such, just possible incorrect band assignment.

Again, without all the gobbledigook (obfuscation) can you reconstruct the original signal from the post-processed series + the “inverse” convolution (what’s that you say, there is no inverse!).

So you can’t. OK then information is lost! And Willis was right!

260. Willis Eschenbach says:

Greg Goodman says:
March 17, 2014 at 3:28 pm

Willis:

“So even ordinary averaging is not uniquely invertible, and thus it loses information. The same is true for the gaussian average, or a boxcar average, or most filters.”

That is true for straight average and RM since all the weights are equal. It is not the case for a gaussian or other kernel were the weights are unique.

The reverse process is called deconvolution. It’s basically a division in the frequency domain in the same way a convolution is multiplication in the freq domain. For example you can ‘soften’ a digital image with a gaussian ‘blur’ ( a 2-D gaussian low-pass filter ). You can then ‘unblur’ with a deconvolution. There are some artefacts due to calculation errors but in essence it is reversible.

Well, I’ll be the third person to strongly question this claim. To start with, whether the weights are unique is immaterial to this question, because the weights in a gaussian average are symmetrical and thus decidedly not unique …

The problem, however, is not the uniqueness of the weights. It is that at the end of the day we have more variables than equations. It is inherent in the convolution process. For each step, we add one value to the front end of the filter, and remove one from the other end.

As a result, we can definitely improve the resolution as you point out regarding the Hubble telescope. But (as someone pointed out above) if you are handed a photo which has a gaussian blur, you can never reconstruct the original image exactly.

You can test this mentally by considering a gaussian blur of a black and white image at successively larger and larger scales. The image gets more and more blurry as you increase the size of the gaussian filter, until eventually it is just fluctuations in some middle gray.

IF your theory were correct, then knowing the convolution weights you could take that sheet of fluctuations in gray and reconstruct the original photo … and even you must admit that’s not possible whether or not you know the weights of the convolution.

In other words, you sound very persuasive and knowledgeable, but you are very wrong. Here’s a simple example:

The green columns and the purple columns are the two datasets, Data A and Data B. Each dataset has three points (N=3), which are quite different.

In each case I used the identical filter, a 13-point = ± 3 standard deviations gaussian filter. The red line is the filtered version of Data B. The blue line is the filtered version of Data A. What blue line, you say? The one you can’t see because it’s hidden by the red line …

Now, can you deconvolve that? Sure … but as you can see, gaussian averages that are indistinguishable from each other lead to widely differing answers as to the underlying data.

Now that doesn’t mean all is lost. It’s not. While there are a variety of datasets that would have nearly indistinguishable gaussian averages, it is a finite number, and a finite number of results. And those results (possible data configurations) have a distribution. That means you can take an educated guess at the underlying data, and you’ll do better than flipping coins.

It doesn’t mean that knowing the weights you can reconstruct the original data. Not possible.

w.

261. The issue of filter inversion (not that it’s germane) need not be argued. Every filter is uniquely defined by it’s Z-transform. In the z-domain, Y[z] = C[z]*X[z] where x[n] , y[n] are the input and output signals respectively and c[n] the impulse response of the filter. Since X[z] = Y[z] C^-1[z], the input signal can be recovered exactly from y[n] by the inverse filter C^-1[z] iff C^-1[z] is causal and stable. Some filters are invertible, some aren’t. To be invertible, the ROC of C[z] and C^-1[z] must overlap which is not always the case.

262. cd says:
Perhaps I’m missing it, but I don’t think he is using a discrete filter in the sense of the sample window.

I see nothing in the description that would lead me to believe that the filter is not a simple FIR. What leads you to the counter-conclusion?

263. Bart says:
March 17, 2014 at 6:31 pm
Willis Eschenbach says:
March 17, 2014 at 4:50 pm

“The idea that we can squint at a smoothed temperature record and say what has caused the swings, however, is a bridge far too far.”

Bart: It also isn’t as important to say what caused it as to say what assuredly did not. Rising CO2 did not. It started before the rise, and it looks nothing like it.

This is the point that invariably gets lost in these my IQ-is-higher-than-your-IQ slug fests that break out anytime someone posts on the subject of quasi-periodicity the observational data. Whether the “65 year cycle” is a near line spectrum of astronomical origin, a resonance in the climate dynamics, a strange attractor of an emergent, chaotic phenomena or a statistical anomaly, what it isn’t is man-made. It is however, responsible for the global warming scare. If it had not increased the rate of change of temperature during its maximum slope period of 1980’s and 90s, the temperature record would be unremarkable (see e.g. the cycle that changed history and we wouldn’t be here arguing the banalities of digital filter design.

264. Bart says:

Jeff Patterson says:
March 17, 2014 at 8:57 pm

Bingo!

265. Bart says:

Though, I take issue with your link where it says: “natural, emergent cycles simply do not exhibit this type of stable behavior”. They do all the time. It is simply a matter of having energy storage with a high “Q” response.

266. Bart says:

Your critic was amusing. Seems to think that once someone conjectures a potential cause for something, however thinly supported, he is then entitled to assume it is the cause, and the question yields to his begging it.

267. Matthew R Marler says:

Richard LH: Curve fitting is deciding a function and then applying it to the data.

That may be your definition, but it is certainly idiosyncratic. Most curve fitting entails choosing lots of functions that may be fit to all or part of the data, as with Bayesian adaptive regression splines. The result is then called the “fitted curve” , or the “fitted curves” when many are compared.

268. Matthew R Marler says:

Greg Goodman: No, filters separate information. What you “throw” away is your choice.

Marler: “Filtering” is nothing more than than fitting data by a method that uses a set of basis functions, and then separating the results into two components (as said by Greg Goodman),

Greg Goodman: I said nothing of the sort. Don’t use my name to back up your ignorant claims.

269. Matthew R Marler says:

I forgot to add. I cited Greg Goodman as the source of the idea of “separating information”, not as an authority, because he mentioned it before I did.

270. Matthew R Marler says:

Richard: Well whatever your logic there is something with some power in the 15 to 75 year bracket that the data says is present and needs explaining.

For a hypothesized signal you can estimate the noise. For a hypothesized noise, you can estimate a signal. Quoting Greg Goodman again: No, filters separate information. What you “throw” away is your choice. To paraphrase, which you call the “signal” and which you call the “noise” depends on other information, such as future data. When doing a decomposition into frequencies, there is no “automatic” or theory-free method to decide whether the “noise” resides in the high or low frequencies. You stated your hypotheses, and got the corresponding estimates. Which frequencies persist into the future will be discovered in the future. That you could get a decomposition to fit your hypotheses was guaranteed ( p = 1.0, so to speak, no matter what the true nature of the finite data set.)

271. Splice says:
272. RichardLH says:

Jeff Patterson says:
March 17, 2014 at 6:36 pm

“Assuming the signal is spectrally localizable, it is up to the user to decide what defines the “signal” of interest and to design a filter appropriate to the task, It should be pointed out that residual (the part of the input signal rejected by the LPF) by definition attenuates long term-trend information and thus all of this information, which is what we are after, is contained in the output. That is, I think, what Greg and Riochard mean by “nothing is lost”, although that’s not how I would phrase it.”

Indeed that is precisely it.

You can have a function

lowpass(signal) -> L

you can also either have a function

highpass(signal) -> H

or use

signal – L = H

in either case no information is ‘lost’. It is always in either L or H.

True with real, practical functions, then some part of ‘signal’ part could be in both L and H by mistake but, provided you ensure the corner frequency is chosen to stay away from points of interest, then no real problems occur.

Sure if you drop L or H on the floor and scrub some dirt over It with your foot then stuff is most definitely ‘lost’ but I do try not to do that :-)

273. RichardLH says:

Bernie Hutchins says:
March 17, 2014 at 6:41 pm

“No one will argue information loss if this is your angle. Because no information IS lost.”

I know – but it doesn’t stop the mantra being rolled out.

274. RichardLH says:

ThinkingScientist says:
March 17, 2014 at 6:59 pm

“Just to let you let you know, I think you have been extraordinarily patient with some of the critics here.”

Thank you

“As a geophysicist myself, with more than a passing understanding of low and high pass filtering (and as an audophile and user of PA equipment including subwoofer design) I have no problem in understanding the utility of the simple (but clever) filter you have presented or the significance of the low pass/high pass threshold being a non-ciritcal choice over a wide range of frequencies. Makes sense to me.”

Me too. A very useful tool to explore with. In reality it doesn’t much matter if you use CTRM or a true Gaussian. The results will be nearly indistinguishable. The CTRM is just easier to construct and use by almost everybody. If you have the knowledge to use higher order stuff then that will produce similar results but it does require more knowledge and understanding and brings its own complications.

275. RichardLH says:

cd says:
March 17, 2014 at 7:05 pm

“Perhaps I’m missing it, but I don’t think he is using a discrete filter in the sense of the sample window.”

A moving average uses a sliding sample window, and this is just an extension of that base concept.

276. RichardLH says:

cd says:
March 17, 2014 at 7:13 pm

“Again, without all the gobbledigook (obfuscation) can you reconstruct the original signal from the post-processed series + the “inverse” convolution (what’s that you say, there is no inverse!).

So you can’t. OK then information is lost! And Willis was right!”

Straw man.

Lowpass(signal) -> L
Highpass(signal -> H
or
signal – L = H

no information is ‘lost’. Ever.

277. RichardLH says:

Willis Eschenbach says:
March 17, 2014 at 7:32 pm

“As a result, we can definitely improve the resolution as you point out regarding the Hubble telescope. But (as someone pointed out above) if you are handed a photo which has a gaussian blur, you can never reconstruct the original image exactly.”

Straw man.

No one other than you, certainly not me, has ever claimed such.

I repeat what I said above to cd

Lowpass(signal) -> L
Highpass(signal) -> H
or
signal – L = H

no information is ‘lost’. Ever.

278. RichardLH says:

Jeff Patterson says:
March 17, 2014 at 8:57 pm

“Whether the “65 year cycle” is a near line spectrum of astronomical origin, a resonance in the climate dynamics, a strange attractor of an emergent, chaotic phenomena or a statistical anomaly, what it isn’t is man-made. It is however, responsible for the global warming scare. If it had not increased the rate of change of temperature during its maximum slope period of 1980′s and 90s, the temperature record would be unremarkable (see e.g. the cycle that changed history and we wouldn’t be here arguing the banalities of digital filter design.”

+10

279. RichardLH says:

Matthew R Marler says:
March 18, 2014 at 12:32 am

“Richard LH: Curve fitting is deciding a function and then applying it to the data.”

Curve fitting is deciding a set of functions and then applying them to the data.

Sorry.

280. RichardLH says:

Matthew R Marler says:
March 18, 2014 at 12:52 am

“To paraphrase, which you call the “signal” and which you call the “noise” depends on other information, such as future data. When doing a decomposition into frequencies, there is no “automatic” or theory-free method to decide whether the “noise” resides in the high or low frequencies. ”

Signal = anything longer than 15 years in period
Noise = anything shorter than 15 years in period

281. RichardLH says:

Splice says:
March 18, 2014 at 1:26 am

“I have a better and simplier forecast:”

As I have said before on many occasions (and I DO mean it)

Linear trend = Tangent to the curve = Flat Earth

The way you get to the conclusion that the Earth is flat is to concentrate on a local area and then extrapolate that observation to areas outside of the range the observations are drawn on.

Exactly the same is done if you apply a straight line (in a general OLS statistical fashion) to a set of temperature observations and then draw any conclusion about what it means (pun) outside of that range, future or past.

If you use a continuous function, such as a filter, then you are much closer to the ‘truth’ about was is, was and maybe about to happen.

282. cd says:

RichardLH

no information is ‘lost’.

You’re shifting the goal posts. If all you have is the smoothed data then you’ve lost information. The original point stated this.

While not being an expert I am not a novice either, and what you’ve done here with regard to the above point is arm wave and obfuscate.

Again, I have no problem with your work.

283. RichardLH says:

cd says:
March 18, 2014 at 3:54 am

“You’re shifting the goal posts. If all you have is the smoothed data then you’ve lost information. The original point stated this.”

I am not. I have chosen to display only part of the information – that is true. That is not the same thing as losing information. The ‘other part’ is still there , just not displayed. If you like I can do the

signal-LowPass = HighPass

284. cd says:

RichardLH

Willis original point.

Any filter which is not uniquely invertible loses information.

signal-LowPass = HighPass

Is an answer to a different question. Note that what Willis states that:

LowPass

Nothing else! It then follows that HighPass is the lost information and he is correct. As I said, it is a side issue, one that you could have dealt with a simple admission. It doesn’t even challenge your work, but it is annoying when you deal with a point with obfuscation.

285. RichardLH says:

cd says:
March 18, 2014 at 4:41 am

“RichardLH
Willis original point.
Any filter which is not uniquely invertible loses information.

signal-LowPass = HighPass
Is an answer to a different question. Note that what Willis states that:

LowPass
Nothing else! It then follows that HighPass is the lost information and he is correct. As I said, it is a side issue, one that you could have dealt with a simple admission. It doesn’t even challenge your work, but it is annoying when you deal with a point with obfuscation.”

It is not obfuscation, it is fact.

True if you were to confine yourself to looking only at the LowPass, then it is only part of the information, so in that limited sense only the other part is ‘lost’. I have never claimed otherwise.

Fortunately we can look outside of that very,very narrow viewpoint and observe that the ‘lost’ information is still there, over in the HighPass bit as I have always done.

What part of those observations and conclusions do you disagree with?

286. RichardLH says:

cd:

Here is a plot of the High Pass for UAH data just so that you can be sure that nothing is ‘lost’

287. RichardLH says:

cd: And the distribution of the same

288. RichardLH says:

indpndnt says:
March 18, 2014 at 1:29 pm

I have replicated this work in Python, for those who prefer it or might find it useful. It can be found on GitHub at:
https://github.com/indpndnt/ipython/blob/master/ClimateSmoothing.py

Useful addition for those who wish to work in Python rather that R.

Thanks to ‘indpndnt’.

289. Willis Eschenbach says:
March 17, 2014 at 12:22 am
“It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase. Let me show you why, using your data:”

Sorry Willis but your analysis is bogus. The LPF doesn’t remove the trend. For the sake of argument assume the trend is linear and the filter output is simply a*t + B*Sin[wo*t + phi]. The derivative is a Cosine plus a constant and is thus not zero mean. The timing of zero crossings of the derivative (the points where you draw the vertical lines to show the supposed timing inconsistency) depends on the constant (slope of the trend) and will not be consistent unless the slope is zero. This effect becomes even more pronounces if there are other sub-harmonics present because these low frequency signals make the slope of the trend time dependent.

290. Matthew R Marler says:
March 18, 2014 at 12:52 am
Richard: Well whatever your logic there is something with some power in the 15 to 75 year bracket that the data says is present and needs explaining.

Matthw: To paraphrase, which you call the “signal” and which you call the “noise” depends on other information, such as future data. When doing a decomposition into frequencies, there is no “automatic” or theory-free method to decide whether the “noise” resides in the high or low frequencies.

Your first sentence is indecipherable (the signal definition depends on future data??). The second sentence is wrong. It is the AWG hypothesis that defines the signal of interest, namely a long-term temperature trend imposed on the natural variation of the climate which itself may contain a natural trend. Spectral decomposition, regardless of the technique used, throws away the high frequency data to improve the signal to noise ratio where each is defined as above. The main point is that by definition, the discarded “noise” is detrended and zero-mean and thus cannot contain any information about anthropogenic effects. Likewise the wiggles which remain, regardless of cause, are clearly of natural origin, Remove them as well and the temperature record shows a remarkably consistent ~.5 degC/century slope since at least 1850 with no discernible dependency on CO2 concentration.

“Which frequencies persist into the future will be discovered in the future. That you could get a decomposition to fit your hypotheses was guaranteed ( p = 1.0, so to speak, no matter what the true nature of the finite data set.)”

You still misunderstand the difference between filtering and curve fitting but no matter. Prediction of the climate is a fool’s errand. Since there is no detectable effect of CO2 in the temperature record the motive for forecasting is obviated. The climate will do what it has always done- meander about.

291. Jeff – there are obviously many motive sfor forecasting future climate without regard to CO2. If for example we have reason to believe that we might expect a significant cooling -then this would suggest various actions in mitigation or adaptation.It is very obvious- simply by eye balling the last 150 years of temperature data that there is a 60 year natural quasi periodicity at work. Sophisticated statistical analysis actually doesn’t add much to eyeballing the time series. The underlying trend can easily be attributed to the 1000 year quasi periodicity. See Figs 3 and 4 at
http://climatesense-norpag.blogspot.com
The 1000 year period looks pretty good at 10000,9000,8000,7000, 2000.1000. and 0
This would look good I’m sure on a wavelet analysis with the peak fading out from 7000- 3000.
The same link also provides an estimate of the timing and extentof possible future cooling using the recent peak as a synchronous peak in both the 60 and 1000 year cycles and the neutron count as supporting evidence of a coming cooling trend as it appears the best proxy for solar “activity” while remaining agnostic as to processes involved.

292. Some suggestions for RichardLH

1) Plot the frequency response in dB. This allows much better dynamic range in observing how far down the sidelobes and passband ripples are.

2) Run randomly generated data through the filter and look at the power spectral density of the output. If the PSD is flat in the passband, the filter is exonerated as the source of the wiggles.

3) Show the step response of the filter, (not the theoretical) by running a step through the actual filter and plotting the result to prove there is no error in the implementing code and to show there is no substantial overshoot and/or ringing. This step is necessary because the temperature data is not WGN and contains large short-term excursions which will ring the filter if it is underdamped.

4) I think I saw somewhere that Greg had a high-pass version of the filter. Set the cut-off of the HP version as high as possible while still passing the 65 year cycle unattenuated. Subtract the HP filter output from the input data after lagging the input data by the (N-1)/2 samples (the filter delay) where N is the number of taps (coefficients) in the filter (N should be odd). The result will be the trend, which I suspect will show a remarkably low variance about a ~.5 degC/century regression.

293. Dr Norman Page says:
March 18, 2014 at 7:56 am
Jeff – there are obviously many motive sfor forecasting future climate without regard to CO2. If for example we have reason to believe that we might expect a significant cooling -then this would suggest various actions in mitigation or adaptation.

Perhaps but I doubt that the uncertainties will be low enough for actionable predictions until we have amassed another couple of hundred years worth of reliable data. Until then, my primary concern is derailing the destructive agenda of the warming hysterics. After that threat is removed, knock yourselves out.

294. Bernie Hutchins says:

RichardLH at March 18, 2014 at 2:16 am quotes me SELECTIVELY as saying:
“No one will argue information loss if this is your angle. Because no information IS lost.”

Note I said “If this is your angle” but you never get around to saying it is. With FILTERING (processing of a signal) information IS lost. That’s the purpose of filtering. With multi-band ANALYSIS (like with an FFT), information is retained in different outputs. The purpose is insight.

Although analysis is apparently your viewpoint (you at 5:28PM yesterday, and Greg at 5:31PM yesterday), I’m not sure you understand this is what you are contending, or the important difference (in concept) from filtering.

It is somewhat disconcerting to have to keep mentioning this here.

295. RichardLH says:

Willis:

So that you can get an idea of what the HadCRUT 15 year CTRM de-trended with a 75 year S-G looks like

A very nice 65 year (well one ‘cycle’ of anyway) wriggle. :-)

And, no, I do not expect that underlying trend of the 75 year S-G to continue in the manner suggested, an inflexion in that is due fairly soon I suspect, but will take a few more years to be visible (before you ask)

296. RichardLH says:

Jeff Patterson says:
March 18, 2014 at 7:57 am

“Some suggestions for RichardLH
1) Plot the frequency response in dB. This allows much better dynamic range in observing how far down the sidelobes and passband ripples are.”

That would emphasise what is happening but would make the plots SO much different to other existing plots out there. I’ll stick with a straight linear scale for now.

“2) Run randomly generated data through the filter and look at the power spectral density of the output. If the PSD is flat in the passband, the filter is exonerated as the source of the wiggles.”

It is (very near to) Gaussian. It will have a Gaussian response. No work required.

“3) Show the step response of the filter, (not the theoretical) by running a step through the actual filter and plotting the result to prove there is no error in the implementing code and to show there is no substantial overshoot and/or ringing. This step is necessary because the temperature data is not WGN and contains large short-term excursions which will ring the filter if it is underdamped. ”

It is (very near to) Gaussian. It will have a Gaussian response. No work required.

“4) I think I saw somewhere that Greg had a high-pass version of the filter. Set the cut-off of the HP version as high as possible while still passing the 65 year cycle unattenuated. Subtract the HP filter output from the input data after lagging the input data by the (N-1)/2 samples (the filter delay) where N is the number of taps (coefficients) in the filter (N should be odd). The result will be the trend, which I suspect will show a remarkably low variance about a ~.5 degC/century regression.”

Did the high pass version as a quick piece of work and posted the results above. Only from the clipboard but that is the fastest way.

297. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 8:13 am

“Note I said “If this is your angle” but you never get around to saying it is. With FILTERING (processing of a signal) information IS lost. That’s the purpose of filtering. ”

Wrong. Filtering is about selecting, not losing, data. The ‘other’ data is still there, just in a different plot as I showed above.

298. Jeff You are too pessimistic – there will be straws in the wind by 2020 and by 2040 predictions until 2100 will probably be actionable .It is already clear that CO2 is of minor significance and that inaction in that regard is the best course.

299. Matthew R Marler says:

Jeff Patterson: Your first sentence is indecipherable (the signal definition depends on future data??).

I subverted my point by being too succinct. It isn’t the “definition” of the “signal” that depends (among other things) on future data. Rather, it is an evaluation and decision about what persists (hence might be called “signal” and what is transient (hence might be called “noise”.) If the 60 year oscillation persists, it might be called “signal”; if it does not persist, it is simply another instance of the fact that a finite time series can be well-modeled by a variety of methods: piecewise polynomials, polynomials, trig polynomials, smoothing splines, regression splines, and so forth.

You still misunderstand the difference between filtering and curve fitting but no matter.

What I wrote, and repeat, is that they are “indistinguishable” in practice. All “filtering” entails “curve-fitting”; then identification of some of the “fitting” results as “signal”, and other results as “noise”. “Band-pass filtering”, for example, is approximately equivalent to subtracting out the part of the data that is well fitted by the functions thought to represent the noise, which have been derived from the assumption about the noise (i.e., the coefficients are calculated to give the hoped for “appropriate” transfer function); there is no universal a priori reason for calling either the “low” frequency components or the “high” frequency components the “signal”, but depends on context.

Prediction of the climate is a fool’s errand. Since there is no detectable effect of CO2 in the temperature record the motive for forecasting is obviated. The climate will do what it has always done- meander about.

If it were possible, and it may become so, to predict climate (the distribution of the weather over space and time) that would be valuable. At the present time, the GCMs have a record of over-predicting the mean; they are not ready to be relied upon, but failure up til now does not imply failure in perpetuity (paraphrase of quotes by Thomas Edison and many others.) Perhaps (this is what I expect) in 20 years they’ll be accurate enough at predicting the seasonal distribution of the weather in reasonably sized regions (it would have been nice, 6 months ago, to have predicted the US combination of Western warmth and Eastern cold and snow.) Whether CO2 concentration will have to be included in order to make accurate enough forecasts is an open research question.

The assertion that there is a hard and fixed distinction between “curve fitting” and “filtering” I think results from a narrow view of all the ways and contexts in which these related computational techniques are used.

300. Matthew R Marler says:

Signal = anything longer than 15 years in period
Noise = anything shorter than 15 years in period

With that definition set, your post illustrates what I wrote: for any definition, using a finite data set, you can find representations of the noise and signal. Deciding or learning what represents a persistent process requires more than that.

301. Bernie Hutchins says:

Richard –

You say your SG filter is length 75, but you did not specify the order of the fit.

And is this a cascade or just one pass?

Nice result (good picture), but we need specific details to verify and continue.

302. RichardLH says:

Matthew R Marler says:
March 18, 2014 at 8:32 am

“With that definition set, your post illustrates what I wrote: for any definition, using a finite data set, you can find representations of the noise and signal. Deciding or learning what represents a persistent process requires more than that.”

True. Except for the fact that a 15 year corner exposed just a ~60 year wriggle. That in itself is interesting.

There is nothing of any real power above 15 years and below 75 years except that rather cyclic ~60 year wriggle.

303. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 8:34 am

“You say your SG filter is length 75, but you did not specify the order of the fit.”

Second order

“And is this a cascade or just one pass?”

5 pass, multi pass (See the R code and comments on my blog). The reference to Nate Drake PhD is because he put his PhD on the table to bet that a CTRM was not an FIR filter :-). See the Nature Missing Heat thread for all the gory details.

“Nice result (good picture), but we need specific details to verify and continue.”

Thanks :-)

304. Matthew R Marler says:

Jeff Patterson: “Whether the “65 year cycle” is a near line spectrum of astronomical origin, a resonance in the climate dynamics, a strange attractor of an emergent, chaotic phenomena or a statistical anomaly, what it isn’t is man-made. It is however, responsible for the global warming scare. If it had not increased the rate of change of temperature during its maximum slope period of 1980′s and 90s, the temperature record would be unremarkable (see e.g. the cycle that changed history and we wouldn’t be here arguing the banalities of digital filter design.”

Only in part. Scientists in the 19th century warned of the potential of increasing CO2 to cause increasing climate warmth. Numerous textbooks (e.g. “Principles of Planetary Climate” by Raymond T. Pierrehumbert) have calculated the amount by which the “equilibrium” temperature of the Earth should increase in response to a doubling of CO2, and some other calculations based on a hypothetical H2O feedback have raised the estimate above that. All these calculations have their problems, but the overall increase in temp since the mid 1800s, despite the oscillation in the rise, are what was predicted. All the data analyses to date do not lead to a very clear-cut result; accumulating CO2 might or might not be warming the Earth climate, and the possible effect might or might not continue into the future (that is, even if CO2 has increased temps up til now, some negative feed back of the sort presented by Willis Eschenbach might prevent warming in the future.)

After reviewing this evidence for years, I am persistently dismayed by people who confidently believe that CO2 does or does not have an effect of a particular size.

305. RichardLH says:

Matthew R Marler says:
March 18, 2014 at 8:46 am

“After reviewing this evidence for years, I am persistently dismayed by people who confidently believe that CO2 does or does not have an effect of a particular size.”

I am cautious about ascribing cause without taking properly into account what looks to be very significant natural variations, possibly of a quite cyclic nature.

As far as I am concern, the data is not yet strong enough to be able to properly decide what proportions are assigned correctly to what.

Mostly it is just speculation and models. I have spent my life in computing trying to understand what assumptions people have made and what turns out to be wrong in those assumptions.

306. Matthew R Marler says:

Richard LH: True. Except for the fact that a 15 year corner exposed just a ~60 year wriggle. That in itself is interesting.

There is nothing of any real power above 15 years and below 75 years except that rather cyclic ~60 year wriggle.

That reminds me: did I thank you for your work and for a good, stimulating post? Thank you for your work, and thank you for your post.

You understand, I hope, that a new method to reveal a 60 year “oscillation” that had already been revealed by many others makes your result look post-hoc. Scafetta, Pratt and others have found something similar by different methods. This constant reworking of a well-studied finite data set doesn’t reveal what we want (well, what I want) to know: what mathematical result is a reasonably accurate representation of a process that will persist approximately the same for the next few decades? That’s the part that requires “out of sample data”.

307. Jeff Patterson says:
March 18, 2014 at 7:57 am

“Some suggestions for RichardLH
1) Plot the frequency response in dB. This allows much better dynamic range in observing how far down the sidelobes and passband ripples are.”

>That would emphasise what is happening but would make the plots SO much different to other >existing plots out there. I’ll stick with a straight linear scale for now.
Plotting in dB is the standard method in DSP circles. “Showing what is happening” would seem most important in light of your primary claim that it is near Gaussian.

“2) Run randomly generated data through the filter and look at the power spectral density of the output. If the PSD is flat in the passband, the filter is exonerated as the source of the wiggles.”

>It is (very near to) Gaussian. It will have a Gaussian response. No work required.
So you say but without the log plots it is impossible to say how close and whether it is coded correctly.

“3) Show the step response of the filter, (not the theoretical) by running a step through the actual filter and plotting the result to prove there is no error in the implementing code and to show there is no substantial overshoot and/or ringing. This step is necessary because the temperature data is not WGN and contains large short-term excursions which will ring the filter if it is underdamped. ”
>It is (very near to) Gaussian. It will have a Gaussian response. No work required.

Gaussian filters have no overshoot. Close-to-guassian will have some. How much?

>Did the high pass version as a quick piece of work and posted the results above. Only from the clipboard but that is the fastest way.

Did you subtract it from the signal? If so I couldn’t find the plot you referenced.

308. RichardLH says:

Matthew R Marler says:
March 18, 2014 at 8:56 am

“That reminds me: did I thank you for your work and for a good, stimulating post? Thank you for your work, and thank you for your post.”

Thank you. Discussing what is observed is thanks enough.

“You understand, I hope, that a new method to reveal a 60 year “oscillation” that had already been revealed by many others makes your result look post-hoc.”

I am all to well aware that what I am showing has been done elsewhere by other previously and probably better. I just hope that by using well know data sources and an irrefutable methodology I have got the observation of some form of ‘natural cycle’ to stick.

” Scafetta, Pratt and others have found something similar by different methods.”

Nicola was nice enough to say that I had confirmed some of his work. That is only partially true. I confirmed the presence to the ~60 year cycle. I have made no real offering as to why it is there yet. More observations needed before we go there :-)

“This constant reworking of a well-studied finite data set doesn’t reveal what we want (well, what I want) to know: what mathematical result is a reasonably accurate representation of a process that will persist approximately the same for the next few decades? That’s the part that requires “out of sample data”.”

All we need to do is wait. Hard in this Internet age but it WILL answer all the questions :-)

309. RichardLH says:

Jeff Patterson says:
March 18, 2014 at 9:00 am

“Plotting in dB is the standard method in DSP circles. “Showing what is happening” would seem most important in light of your primary claim that it is near Gaussian. ”

It is near Gaussian. Vaughn Pratt is the author of that statement and I’m not going to fight with him!

“So you say but without the log plots it is impossible to say how close and whether it is coded correctly.”

Again, I am only repeating work already done and discussed at much length by Greg and Vaughan. They (and me) are confident that it is Gaussian. If you believe that statement is wrong I would love to see some plots.

“Gaussian filters have no overshoot. Close-to-guassian will have some. How much?”

I suspect that you will find it hard to distinguish the two (as Willis did for the frequency plots).

“Did you subtract it from the signal? If so I couldn’t find the plot you referenced.”

No I just showed that it was possible to create the Annual High Pass output. There is a lot of stuff in there which, being sub-Annual, is probably mostly reflected values from the ‘normal’ and other very short term variations.

I did do the rather more interesting 15-75 to get the ‘cycle’ clearly though :-)

310. Bernie Hutchins says:

Richard –

A second-order fit, length 75 SG has impulse response and frequency response as shown here:

The red line shows 60 years, and note that it is somewhat down the slope at about 1/2. Five passes would take it to about 3%. Not good. I think we are not on the same page with respect to sampling rate (I assumed yearly) while you have a denser sampling. So – how long is your impulse response in terms of samples? Is it 75 x 12 or something like that?

If my impulse response is correct (?) then exactly what does YOUR impulse response look like? It is, as you say, still FIR. Is there a math description of what you do or just a spread sheet?

Thanks for any specific information.

311. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 9:18 am

“A second-order fit, length 75 SG has impulse response and frequency response as shown here:
http://electronotes.netfirms.com/SG2-75.jpg

Thanks for that plot.

“The red line shows 60 years, and note that it is somewhat down the slope at about 1/2. Five passes would take it to about 3%. Not good. I think we are not on the same page with respect to sampling rate (I assumed yearly) while you have a denser sampling. So – how long is your impulse response in terms of samples? Is it 75 x 12 or something like that? ”

The sampling always stays at the underlying sample rate, in this case Monthly. In fact in order to match to the kernel size of the Annual CTRM which is 12 + 10 + 8 = 30 wide I used a factor of 2 to get an appropriate Annual S-G, i.e. 12 * 2 + 1. The 15 (and 75) year wide CTRM and S-G are similarly scaled. The results were verified by observing the closeness of the resultant 15 year curves during the whole of their overlap period. I suspect that it would be possible to get an even better fit by slightly modifying the *2 but the results were good enough ‘as is’.

“If my impulse response is correct (?) then exactly what does YOUR impulse response look like? It is, as you say, still FIR. Is there a math description of what you do or just a spread sheet? ”

For CTRM it is as close to Gaussian as you could possibly wish with a much simpler implementation. In any case, digitisation/sampling/rounding/truncation errors dominate over just about all the filter maths anyway as Vaughn was kind enough to agree when I pointed it out. There are only some 200 possible input values for UAH and some 2000 for HadCRUT because of the number of decimal places reported. That means that in most cases even a quite truncated Gaussian will have weighting factors that are below those.

“Thanks for any specific information.”

NP

312. Bart says:

Dr Norman Page says:
March 18, 2014 at 7:56 am

“Sophisticated statistical analysis actually doesn’t add much to eyeballing the time series.”

I would go further, and say statistical analysis should never, ever, be applied without eyeballing the time series to provide what is commonly referred to as a sanity check.

The utility of statistics is mostly in data compression, in distilling large amounts of information into a few representative numbers. Canned statistical routines, like computer simulations, are very susceptible to GIGO. They are only as good as the assumptions built into them.

The flip side of the old caveat of seeing patterns where they do not exist is having an analysis tool which misses patterns which are obviously there. The analysis tool is just a dumb algorithm. You have a brain. Use it. (not you, specifically, Dr. Page – you need no admonition).

313. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 9:18 am

I should point out that the correct way to get the various inter-stage values for the CTRM is next=round(prev*1.2067) which is the value that Vaughn provided. Both Greg and I had used 1.3371 previously, though from completely different sources.

314. Bart says:

You really don’t need a filter at all. The ~60 year periodic behavior is readily apparent in the raw data. Filtering it out for better resolution is really just gilding the lily.

315. RichardLH says:

Bart says:
March 18, 2014 at 9:37 am

“You really don’t need a filter at all. The ~60 year periodic behavior is readily apparent in the raw data. Filtering it out for better resolution is really just gilding the lily.”

You would think so. But I am told it is all down to co-incidental Volcanos, SO2 and CO2 in just the right mixture and timing :-)

316. Bart Thanks for your very pertinent comment – the capacity of the establishment IPCC contributing modelers to avoid the blindingly obvious natural periodicities in the temperature record is truly mind blowing.

317. Bernie Hutchins says:

Richard –

I guess I still don’t have much of a clue what you are doing!

What you do is really quite (unnecessarily?) complicated, more complicated that SG or Gaussian, etc., but still FIR and you should be able to provide the impulse responses of the individual stages you are cascading. That would be complete, concise, well defined, and unambiguous. Is that available?

318. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 10:21 am

“I guess I still don’t have much of a clue what you are doing!”

Utilising work done by others to create a high quality filter analysis of the available climate data.

“What you do is really quite (unnecessarily?) complicated, more complicated that SG or Gaussian, etc., but still FIR and you should be able to provide the impulse responses of the individual stages you are cascading. That would be complete, concise, well defined, and unambiguous. Is that available?”

Not really complicated. Just cascading the single stages in an approved fashion to get better responses. The CTRM came from Greg and Vaughn, the S-G from standard utilisation of that to match to the CTRM.

CTRM is just so easy and is so near to Gaussian that the differences hardly matter. Gaussian is the ideal filter to work with on almost all signal work. Usually has too slow a roll-off but in climate no-one is looking for 1/3 octave filters or better!

I am fairly sure that you don’t need the full 5 stage cascade for the S-G. You do need 3 to get to a reasonable match but the extra 2 are probably not really required. I left them in mainly because Nate was SO determined to prove me wrong about the CTRM and they do no real harm – and I do like to credit him with the original as you may see in the R at my blog:-).

The underlying reasoning is to use high quality filters to do the analysis. The particular values came out of seeing what worked and looking at what was there.

The 15 year corner came from a standard low pass sweep up the available bandwidth as you would do in any other signal work and looking for the cut point.

If you have any suggestions of how to improve/simplify the various filters or their responses then I would welcome the advice.

319. Bart says:

RichardLH says:
March 18, 2014 at 9:38 am

You would think so. But I am told it is all down to co-incidental Volcanos, SO2 and CO2 in just the right mixture and timing :-)

Yes, our old friend, the epicycles. Plus ca change…

320. RichardLH says:

Bart says:
March 18, 2014 at 10:54 am

“Yes, our old friend, the epicycles. Plus ca change…”

There is a LONG way to go to get to there. It does rather require a physical mechanism with sufficient capability which currently is not there, as far as I am concerned. Lots of suggestions, but nothing that really seems valid in detail.

The response does look very cyclic and that would suggest orbital but HOW is then the problem.

321. Bernie Hutchins says:

Richard –

It is evident that we are on quite different wavelengths! I do not understand what the limitations are on your resources such that you see an advantage. You seem to conclude that your method is almost as good as Gaussian (converges on Gaussian?). So – Why not use Gaussian?

Indeed, why not use any of the long-available superior alternatives: classic windowing, frequency domain least squares, Parks-McClellan, generalized Frequency Sampling, SG, and probably a half dozen others? Further, these methods already push the performance into the corners (trade-offs) – limited by such things as the uncertainty relationship for Fourier-transform-related signals.

My only suggestions would be to try these standard things and compare them, FIRST. Then try your method and see if it is really better in a sense you can elaborate.

Moreover – assuming a reliable outcome, what have you discovered that is new? As Bart says above, essentially, “We already see the periodicity”. No mathematical method that detects periodicity (filtering, spectral analysis) speaks even indirectly to the origins of that periodicity.

And – you forgot to mention if you have the impulse responses I asked about. If not, just please say so.

322. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 11:07 am

“It is evident that we are on quite different wavelengths! I do not understand what the limitations are on your resources such that you see an advantage. You seem to conclude that your method is almost as good as Gaussian (converges on Gaussian?). So – Why not use Gaussian?”

Because a CTRM is a good enough match not to need that rather more complex routine? The answer will be the same. Differences between the various filter types are not going to be sufficient to alter any of the outcomes in any case (other than in a worse direction).

“Indeed, why not use any of the long-available superior alternatives: classic windowing, frequency domain least squares, Parks-McClellan, generalized Frequency Sampling, SG, and probably a half dozen others? Further, these methods already push the performance into the corners (trade-offs) – limited by such things as the uncertainty relationship for Fourier-transform-related signals. ”

There is no need to get a ‘better’ response than Gaussian and I am not sure that one is available in any case. Please, if you feel that there IS a better alternative, then the data is freely available so run one of your choice and see how it differs. My betting is not enough to notice.

“My only suggestions would be to try these standard things and compare them, FIRST. Then try your method and see if it is really better in a sense you can elaborate. ”

Other than true Gaussian (which I have run) the ones I am currently using seem to be sufficient to the task. If you feel there is a need to do a complete set of alternatives then go for it. I have enough on my plate trying to get through the various data sets
with the two I have right now.

“Moreover – assuming a reliable outcome, what have you discovered that is new? As Bart says above, essentially, “We already see the periodicity”. No mathematical method that detects periodicity (filtering, spectral analysis) speaks even indirectly to the origins of that periodicity. ”

In that case then the ~60 year natural ‘cycle’ is established and all the other stuff needs more detailed looking at. Things such as the various parts of the UAH signal.

Quite intriguing and a source of investigation/work to come.

“And – you forgot to mention if you have the impulse responses I asked about. If not, just please say so.”

No. But as the response curve is so nearly Gaussian I would be incredibly surprised in there are any significant or worthwhile differences between the two of them.

323. Bart says:

I am always perplexed by the insistence that “we don’t have an identified mechanism”. You don’t need to know how a diesel engine works to know you better get out of the way of the train.

We have a situation of two scientists. Scientist #1 has a hypothesis, but no evidence to back it up. Scientist #2 has no hypothesis, but hard data which demonstrate a phenomenon contradicting #1’s hypothesis. Which to choose, oh, which to choose…

Cyclic phenomena abound in nature. They’re everywhere. Find some piece of metal near you and tap it. Does it not ring? Pour some water in a glass. Does it rise uniformly?

Climate scientists would do better spending their time looking for the mechanism than denying that it is lying in plain sight before them.

324. Bart says:

…that its signature is lying in plain sight…

325. Bernie Hutchins says:

Richard –

But you have nothing.

Just my thanks.

Bernie

326. RichardLH says:

Bart says:
March 18, 2014 at 11:39 am

“I am always perplexed by the insistence that “we don’t have an identified mechanism”. You don’t need to know how a diesel engine works to know you better get out of the way of the train.”

But knowing the science that unpins it allows you to run in the right direction!

327. RichardLH says:

Bernie Hutchins says:
March 18, 2014 at 11:43 am

Thanks.

“But you have nothing.”

Really? A nice way of showing what is actually there with sufficient precision not to be refuted. Nothing? A pity you think so.

“Just my thanks.”

Thank you in return.

328. 1sky1 says:

Bernie Hutchins:

On March 18, 2014 at 11:07 am, you list a panoply of well-established alternatives to the home-brewed filters touted here and follow up with: “My only suggestions would be to try these standard things and compare them, FIRST. Then try your method and see if it is really better in a sense you can elaborate. Moreover – assuming a reliable outcome, what have you discovered that is new?”

While all your points are spot on the mark and would be acknowleged as such in a professional technical dicussion, that is scarcely the purpose or tenor of a dog and pony show intended to establish “street cred” in blogland. Only the most naive, however, will buy the self-promoting notion that Pratt, Goodman or Hood have produced anything new or useful.

329. RichardLH says:

1sky1 says:
March 18, 2014 at 3:00 pm

“Only the most naive, however, will buy the self-promoting notion that Pratt, Goodman or Hood have produced anything new or useful.”

Hmmm. That was just establishing the validity of the methods and demonstrating quite clearly that there is something in the ~60 year bracket that HAS happened and needs explaining.

Next (assuming that Anthony will allow) we have a comparison/alignment of the 4 major temperature sets to allow a combined satellite/thermometer set stretching back to the 1800s with the ability to directly compare/contrast the various sources and some possible lines of enquiry from that.

Then, against assuming permission, an analysis of UAH and HadCRUT across the globe with an area by area breakdown and direct comparison between them. Raises some interesting questions about possible UHI contamination which will be hard to dismiss.

Want to see?

330. RichardLH says:

EDIT: …Then, again assuming permission, …

and

..who Vaughan Pratt is …

combination of fingers and auto complete.

331. Willis Eschenbach says:

Jeff Patterson says:
March 18, 2014 at 6:56 am

Willis Eschenbach says:
March 17, 2014 at 12:22 am
“It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase. Let me show you why, using your data:”

Sorry Willis but your analysis is bogus. The LPF doesn’t remove the trend. For the sake of argument assume the trend is linear and the filter output is simply a*t + B*Sin[wo*t + phi]. The derivative is a Cosine plus a constant and is thus not zero mean. The timing of zero crossings of the derivative (the points where you draw the vertical lines to show the supposed timing inconsistency) depends on the constant (slope of the trend) and will not be consistent unless the slope is zero. This effect becomes even more pronounces if there are other sub-harmonics present because these low frequency signals make the slope of the trend time dependent.

I do love all the theory … Greg talks about “differing slopes” and “pure cosines”. I invite him to demonstrate it. He hasn’t replied.

You say my analysis is “bogus”, for the same reason Greg posited … but you don’t give us a worked out example using the actual data either.

I go about it the other way. I do the exercise myself, run the numbers, and then I think about the theory. Depending on how you choose to count (peaks or valleys) using the detrended HadCRUT4 data (detrended as Richard recommended with a 60-year S-G filter) we end up with cycles of 51, 61, 64, and 67 years. That’s not a cycle of any kind, not approximately sixty years, not approximately anything.

So I’m sorry, Jeff, but as much as I generally respect your science-fu, a range from 51 to 67 years is a huge difference, and so I will say it again. There is nothing in the data to indicate that there is any kind of regular cycle in the data. Or as I put it above, “It is a grave mistake, however, to assume or assert that said wiggle has a frequency or a cycle length or a phase.”

NOTE THAT I AM NOT SAYING THAT THERE IS NO REGULAR CYCLE IN THE TEMPERATURE.There may be such a cycle, although to my knowledge no one has ever shown it with any solid evidence. But as always lack of evidence is not evidence of a lack.

I am saying it is a mistake to ASSUME that there is such a regular cycle and proceed with your analysis on that basis.

w.

332. Willis Eschenbach says:
March 18, 2014 at 6:25 pm
we end up with cycles of 51, 61, 64, and 67 years. That’s not a cycle of any kind, not approximately sixty years, not approximately anything.

Is it possible that there really is a 60 year cycle out there which just happens to be masked and pushed or pulled in one direction or another due to other cycles? For example, one could plot the times when all ice is gone from Lake Superior and while we know about the 365.24 day cycle when the sun comes around to the same point, we may find that sometimes melt times may be 10 months apart and at other times 14 months apart for example.

333. Edim says:

I think nobody assumes that the cycles are regular. The cycles are quasiperiodic – oscillations that appear to follow a regular pattern but which do not have a fixed period. Just like the solar cycle, which is ~11 year in average, but actually ~9 – 13 years long.

334. RichardLH says:

Willis Eschenbach says:
March 18, 2014 at 6:25 pm

“I go about it the other way. I do the exercise myself, run the numbers, and then I think about the theory. Depending on how you choose to count (peaks or valleys) using the detrended HadCRUT4 data (detrended as Richard recommended with a 60-year S-G filter) we end up with cycles of 51, 61, 64, and 67 years. That’s not a cycle of any kind, not approximately sixty years, not approximately anything.”

I think you may have missed to bit where I did the work you had requested (the 15 year CTRM de-trended with a 75 year S-G) and posted the result above. (This was just a quick exercise and posted straight from my clipboard so apologies for the lack of style)

This shows a very ‘clean’ cycle at either 65 years (if you take the peaks) or 70 years (if you take the zero crossings).

Given that there is only one cycle truly possible to measure I’ll still call that ~60 years (I could make it ~67.5 if you want to be very technical).

I will stand by what that shows and note that future data may allow for a more precise estimate.

335. RichardLH says:

Edim says:
March 19, 2014 at 2:31 am

“I think nobody assumes that the cycles are regular. The cycles are quasiperiodic – oscillations that appear to follow a regular pattern but which do not have a fixed period. Just like the solar cycle, which is ~11 year in average, but actually ~9 – 13 years long.”

Indeed. I would be VERY surprised if this was a single cycle at all. Given the data my best guess is that this is most probably a combination of 55, 65 and 75 year cycles mixed in some semi-random combination of you forced me to decide right now.

That is right out there with just about a thousand other possibilities so should probably be given an appropriate weighting for likely accuracy.

336. RichardLH says:

wbrozek says:
March 18, 2014 at 7:43 pm

“For example, one could plot the times when all ice is gone from Lake Superior and while we know about the 365.24 day cycle when the sun comes around to the same point, we may find that sometimes melt times may be 10 months apart and at other times 14 months apart for example.”

If ther is indeed an ~60 year cycle to the data then the averages for a 30 year portion would be slightly longer than 365.25 and the rest slightly shorter. If that cycle its was modulated by others then the averages would change based on them also. It can get really, really messy and, given the semi random variations that will always exist, it can take a LONG time to be sure/definite about anything.

337. RichardLH says:

Willis Eschenbach says:
March 18, 2014 at 6:25 pm

And in case you were wondering how that other plot is derived from the HadCRUT series

As I also said before, I do not expect the underlying 75 year S-G to continue upwards indefinitely. I would expect an inflexion soon based on a likely projection of the current data to also make it into a ‘cycle’.

I do not accept that CO2 is the main driver to that longer trend. It may well be some portion of it however. The ratios are still up for grabs in my world.

338. Willis Eschenbach says:
March 18, 2014 at 6:25 pm

“You say my analysis is “bogus”, for the same reason Greg posited … but you don’t give us a worked out example using the actual data either.”

I didn’t think an example necessary since clearly a cosine wave (the derivative of a sine wave) oscillating about a non-zero mean (derivative of the slope) has zero-crossings (peaks of of the undifferentiated waveform) that are not equally spaced in time. Your method is simply not a valid measure of period. That’s why I said your analysis (not necessarily your conclusion) is bogus.

If you measured, instead of the half cycle periods you used, the peak-to-peak or trough to trough intervals you’d get a better estimate but as I said before, even this approach assumes the trend is constant across the measurement interval which is certainly not the case. For a worked out example that assumes the underlying trend is itself a low frequency sinusoid, see figure 5 of my first WUWT post. The red curve is a three-sine fit to the data (I know you object to the method, I’m just using it as an illustration). Note the lack of consistency in the timing of the peaks/troughs even though it is comprised of purely sinusoidal functions.

Warm regards (and my “science-fu” can’t hold a candle to yours but thanks)
JP

339. RichardLH says:

Jeff Patterson says:
March 19, 2014 at 12:55 pm

“If you measured, instead of the half cycle periods you used, the peak-to-peak or trough to trough intervals you’d get a better estimate but as I said before, even this approach assumes the trend is constant across the measurement interval which is certainly not the case”

That may be my fault I’m afraid. I prefer ‘zero crossing’ as a detection point because peaks are much more subject to outliers, etc.

As you can see I prefer to de-trend with a longer residual rather than eyeball though if I can (though I can do that as well if required).

I suspect it is just that Willis is not as good or experienced at doing it as I am. Not his fault really.

As you can see from the work I posted above, I’m am pretty good at doing that eyeball stuff as well :-).

340. You can of course use zero crossings as long as you use positive slope crossing to positive slope crossing (or neg-to-neg). But again, if the trend is non-constant over the measurement interval all bets are off. That’s why God gave us the periodogram :>)

341. DocMartyn says:

RichardLH, do you have an email so I could drop you a note?

342. RichardLH says:

Jeff Patterson says:
March 19, 2014 at 3:59 pm

“You can of course use zero crossings as long as you use positive slope crossing to positive slope crossing (or neg-to-neg).”

No – if you have any sense you make no assumptions but de-trend with the next stage in the band-pass splitter. Done in about all other disciplines :-). You don’t have to make an assumption about what the later curve/line/wriggle looks like then.

343. RichardLH says:

DocMartyn says:
March 19, 2014 at 5:42 pm

“RichardLH, do you have an email so I could drop you a note?”

http://climatedatablog.wordpress.com/

344. GlynnMhor says:

I keep seeing this repeated: “Gaussian is the ideal filter…”, yet the Gaussian filter only passes 100% of the signal in a narrow zone immediately around its peak, and attenuates both above and below.

An ideal filter would have as minimal an effect as possible over the whole frequency range that one wants to retain, while strongly suppressing frequencies outside that range.

The only filter that has a nearly flat frequency response of that sort is an Ormsby.

http://www.xsgeo.com/course/filt.htm

345. RichardLH says:

GlynnMhor says:
March 20, 2014 at 11:36 am

“I keep seeing this repeated: “Gaussian is the ideal filter…”, ”

From the Wiki

http://en.wikipedia.org/wiki/Gaussian_filter
…[Gaussian} is considered the ideal time domain filter, just as the sinc is the ideal frequency domain filter…

As we are mainly interested in keeping the time domain low pass portion with the least distortions then that is why it is considered ‘best’ IMHO.