Image Credit: Climate Data Blog
By Richard Linsley Hood – Edited by Just The Facts
The goal of this crowdsourcing thread is to present a 12 month/365 day Cascaded Triple Running Mean (CTRM) filter, inform readers of its basis and value, and gather your input on how I can improve and develop it. A 12 month/365 day CTRM filter completely removes the annual ‘cycle’, as the CTRM is a near Gaussian low pass filter. In fact it is slightly better than Gaussian in that it completely removes the 12 month ‘cycle’ whereas true Gaussian leaves a small residual of that still in the data. This new tool is an attempt to produce a more accurate treatment of climate data and see what new perspectives, if any, it uncovers. This tool builds on the good work by Greg Goodman, with Vaughan Pratt’s valuable input, on this thread on Climate Etc.
Before we get too far into this, let me explain some of the terminology that will be used in this article:
—————-
Filter:
“In signal processing, a filter is a device or process that removes from a signal some unwanted component or feature. Filtering is a class of signal processing, the defining feature of filters being the complete or partial suppression of some aspect of the signal. Most often, this means removing some frequencies and not others in order to suppress interfering signals and reduce background noise.” Wikipedia.
Gaussian Filter:
A Gaussian Filter is probably the ideal filter in time domain terms. That is, if you consider the graphs you are looking at are like the ones displayed on an oscilloscope, then a Gaussian filter is the one that adds the least amount of distortions to the signal.
Full Kernel Filter:
Indicates that the output of the filter will not change when new data is added (except to extend the existing plot). It does not extend up to the ends of the data available, because the output is in the centre of the input range. This is its biggest limitation.
Low Pass Filter:
A low pass filter is one which removes the high frequency components in a signal. One of its most common usages is in anti-aliasing filters for conditioning signals prior to analog-to-digital conversion. Daily, Monthly and Annual averages are low pass filters also.
Cascaded:
A cascade is where you feed the output of the first stage into the input of the next stage and so on. In a spreadsheet implementation of a CTRM you can produce a single average column in the normal way and then use that column as an input to create the next output column and so on. The value of the inter-stage multiplier/divider is very important. It should be set to 1.2067. This is the precise value that makes the CTRM into a near Gaussian filter. It gives values of 12, 10 and 8 months for the three stages in an Annual filter for example.
Triple Running Mean:
The simplest method to remove high frequencies or smooth data is to use moving averages, also referred to as running means. A running mean filter is the standard ‘average’ that is most commonly used in Climate work. On its own it is a very bad form of filter and produces a lot of arithmetic artefacts. Adding three of those ‘back to back’ in a cascade, however, allows for a much higher quality filter that is also very easy to implement. It just needs two more stages than are normally used.
—————
With all of this in mind, a CTRM filter, used either at 365 days (if we have that resolution of data available) or 12 months in length with the most common data sets, will completely remove the Annual cycle while retaining the underlying monthly sampling frequency in the output. In fact it is even better than that, as it does not matter if the data used has been normalised already or not. A CTRM filter will produce the same output on either raw or normalised data, with only a small offset in order to address whatever the ‘Normal’ period chosen by the data provider. There are no added distortions of any sort from the filter.
Let’s take a look at at what this generates in practice.The following are UAH Anomalies from 1979 to Present with an Annual CTRM applied:
Fig 1: UAH data with an Annual CTRM filter
Note that I have just plotted the data points. The CTRM filter has removed the ‘visual noise’ that a month to month variability causes. This is very similar to the 12 or 13 month single running mean that is often used, however it is more accurate as the mathematical errors produced by those simple running means are removed. Additionally, the higher frequencies are completely removed while all the lower frequencies are left completely intact.
The following are HadCRUT4 Anomalies from 1850 to Present with an Annual CTRM applied:
Fig 2: HadCRUT4 data with an Annual CTRM filter
Note again that all the higher frequencies have been removed and the lower frequencies are all displayed without distortions or noise.
There is a small issue with these CTRM filters in that CTRMs are ‘full kernel’ filters as mentioned above, meaning their outputs will not change when new data is added (except to extend the existing plot). However, because the output is in the middle of the input data, they do not extend up to the ends of the data available as can be seen above. In order to overcome this issue, some additional work will be required.
The basic principles of filters work over all timescales, thus we do not need to constrain ourselves to an Annual filter. We are, after all, trying to determine how this complex load that is the Earth reacts to the constantly varying surface input and surface reflection/absorption with very long timescale storage and release systems including phase change, mass transport and the like. If this were some giant mechanical structure slowly vibrating away we would run low pass filters with much longer time constants to see what was down in the sub-harmonics. So let’s do just that for Climate.
When I applied a standard time/energy low pass filter sweep against the data I noticed that there is a sweet spot around 12-20 years where the output changes very little. This looks like it may well be a good stop/pass band binary chop point. So I choose 15 years as the roll off point to see what happens. Remember this is a standard low pass/band-pass filter, similar to the one that splits telephone from broadband to connect to the Internet. Using this approach, all frequencies of any period above 15 years are fully preserved in the output and all frequencies below that point are completely removed.
The following are HadCRUT4 Anomalies from 1850 to Present with a 15 CTRM and a 75 year single mean applied:
Fig 3: HadCRUT4 with additional greater than 15 year low pass. Greater than 75 year low pass filter included to remove the red trace discovered by the first pass.
Now, when reviewing the plot above some have claimed that this is a curve fitting or a ‘cycle mania’ exercise. However, the data hasn’t been fit to anything, I just applied a filter. Then out pops some wriggle in that plot which the data draws all on its own at around ~60 years. It’s the data what done it – not me! If you see any ‘cycle’ in graph, then that’s your perception. What you can’t do is say the wriggle is not there. That’s what the DATA says is there.
Note that the extra ‘greater than 75 years’ single running mean is included to remove the discovered ~60 year line, as one would normally do to get whatever residual is left. Only a single stage running mean can be used as the data available is too short for a full triple cascaded set. The UAH and RSS data series are too short to run a full greater than 15 year triple cascade pass on them, but it is possible to do a greater than 7.5 year which I’ll leave for a future exercise.
And that Full Kernel problem? We can add a Savitzky-Golay filter to the set, which is the Engineering equivalent of LOWESS in Statistics, so should not meet too much resistance from statisticians (want to bet?).
Fig 4: HadCRUT4 with additional S-G projections to observe near term future trends
We can verify that the parameters chosen are correct because the line closely follows the full kernel filter if that is used as a training/verification guide. The latest part of the line should not be considered an absolute guide to the future. Like LOWESS, S-G will ‘whip’ around on new data like a caterpillar searching for a new leaf. However, it tends to follow a similar trajectory, at least until it runs into a tree. While this only a basic predictive tool, which estimates that the future will be like the recent past, the tool estimates that we are over a local peak and headed downwards…
And there we have it. A simple data treatment for the various temperature data sets, a high quality filter that removes the noise and helps us to see the bigger picture. Something to test the various claims made as to how the climate system works. Want to compare it against CO2. Go for it. Want to check SO2. Again fine. Volcanoes? Be my guest. Here is a spreadsheet containing UAH and a Annual CTRM and R code for a simple RSS graph. Please just don’t complain if the results from the data don’t meet your expectations. This is just data and summaries of the data. Occam’s Razor for a temperature series. Very simple, but it should be very revealing.
Now the question is how I can improve it. Do you see any flaws in the methodology or tool I’ve developed? Do you know how I can make it more accurate, more effective or more accessible? What other data sets do you think might be good candidates for a CTRM filter? Are there any particular combinations of data sets that you would like to see? You may have noted the 15 year CTRM combining UAH, RSS, HadCRUT and GISS at the head of this article. I have been developing various options at my new Climate Data Blog and based upon your input on this thread, I am planning a follow up article that will delve into some combinations of data sets, some of their similarities and some of their differences.
About the Author: Richard Linsley Hood holds an MSc in System Design and has been working as a ‘Practicing Logician’ (aka Computer Geek) to look at signals, images and the modelling of things in general inside computers for over 40 years now. This is his first venture into Climate Science and temperature analysis.
A better result can be obtained using an Ormsby zero phase filter:
http://www.xsgeo.com/course/filt.htm
Industrial scale treatment of seismic data uses Ormsby filters, applied in the Fourier domain, almost exclusively to frequency limit the data.
The frequency content and filter slopes can be established in the design of the filter operator, and the operator is always run from end to end of the data without truncation of the output.
rgbatduke says:
March 17, 2014 at 8:15 am
“GISS has a (broken, often backwards) UHI correction. HADCRUT does not. Probably other reasons as well. The UHI correction used by GISS maximally affects the ends.
You cannot really say that they are close at the start and finish and different in the middle, because they are anomalies with separately computed absolute bases.”
Well for that alignment I did a simple OLS for the whole 1979-today period for each source and then adjusted the offsets in the various series to suit. UAH, RSS, GISS and HadCRUT.
http://climatedatablog.wordpress.com/combined/
has the whole story and that was going to be the heart of my next posting (hopefully).
Added some well known proxies in the mixture to extend the data set a few more years backwards into the past as well 🙂
GlynnMhor says:
March 17, 2014 at 8:22 am
“A better result can be obtained using an Ormsby zero phase filter:
http://www.xsgeo.com/course/filt.htm”
It would be interesting to see how the output from that differed (if at all) from the CTRM. I suspect that given the data length and the time periods required the output will be nearly identical.
RichardLH says:
March 17, 2014 at 8:08 am
“Thanks for the suggestion but the frequency response curve is way to ‘ringy’ for me.”
That’s just an heuristic diagram. You can design for the amount of ripple, and can make it arbitrarily small for arbitrary length of the filter.
That is much better than a filter design which starts attenuating at any frequency above zero, like the Gaussian filter.
Bart says:
March 17, 2014 at 8:31 am
“That’s just an heuristic diagram. You can design for the amount of ripple, and can make it arbitrarily small for arbitrary length of the filter.”
But as far as I know that requires extra input data series length, which is the one thing we do not have.
You actually want full attenuation of zero to corner frequency, i.e. a brick wall filter, but that is precisely what a single running mean is and look what it does in the frequency domain.
http://climatedatablog.files.wordpress.com/2014/02/fig-1-gaussian-simple-mean-frequency-plots.png
Gaussian seems the best compromise all round, and CTRM is a very simple way to get that.
The OP talks about using this triple mean filter to remove the seasonal component of data, but then showed an example using anomaly data, which already has the seasonal component removed.
The easiest and best way to remove seasonal components from seasonal data is to do what everyone else does, use anomalies. Then smooth that if desired with LOESS or Gaussian, which as a few have pointed out, can be made virtually indistinguishable from the triple mean filter.
I’d be more interested to see how filters work on not-quite-so-seasonal data. Sunspots for example, have varying periods so anomalies do not work. Willis has effectively criticized several papers that do a very poor job of smoothing sunspots.
Also, data on total land ice and snow, while seasonal, do not work well with anomalies. The reason is the peak snow coverage is very sharp and varies over a range of 5-6 weeks each year, leaving periodic spikes in the anomaly data.
Clay Marley says:
March 17, 2014 at 8:47 am
“The OP talks about using this triple mean filter to remove the seasonal component of data, but then showed an example using anomaly data, which already has the seasonal component removed. The easiest and best way to remove seasonal components from seasonal data is to do what everyone else does, use anomalies. ”
The problem with that is that the ‘normal’ is only constructed with, say, 30 samples, often with built-in errors in those samples as they are sub-sampled single running means themselves. This then gets translated into errors in the ‘normal which then get stamped into all the ‘Anomaly’ sets produce using it from then on.
An Annual filter removes those included errors and produces a mathematically more accurate result at monthly sampling rate.
If you want a look at this applied directly to temperature rather than anomalies then try
http://climatedatablog.wordpress.com/cet/
which shows the CET with just such a treatment. You could turn that into an Anomaly of self set by subtracting the overall average if you like.
A “zero-phase” filter is not possible with time-series data as it would generally be anti-causal. So when a signal processing engineer says zero-phase he/she usually means a FIR filter with impulse response that is even-symmetric with respect to the center. This is properly called “linear phase”, or constant time delay. The frequency response is a Fourier Series in the frequency variable – just a weighted sum of cosines in this case. A moving average is linear phase, for example, as is Guassian or SG.
Butterworth responses are traditionally IIR and not FIR, and are not linear phase. It is however possible to design a “Butterworth” linear phase by taking a suitable Butterworth characteristic, sampling the magnitude response, substituting in (imposing) a linear phase, and solving for the impulse response. The impulse response can be just the inverse Discrete-Time Fourier Transform (DTFT is slow form of FFT), which is called “frequency sampling” in the textbooks, but this often still has passband ripples (again because it is the sum of cosines after all). So a generalized form of frequency sampling is used where a very large (over-determined) set of magnitude samples is used and the inversion is done by least squares (again the “pseudo-inverse”) down to a reasonable length FIR filter. Works great.
Parks-McClellan is another good option with well defined passband and stopband ripples (Chebyshev error criterion). It is inherently FIR and linear phase. It also has no closed from derivation, hence the PM Algorithm. The book Tom Parks wrote with Sid Burrus “Digital Filter Design” is classic. Filter design is well established and readily available.
Bernie Hutchins says:
March 17, 2014 at 9:16 am
There are many good filter designs out there with their own particular characteristics. Few offer the benign characteristics of Gaussian though.
http://en.wikipedia.org/wiki/Gaussian_filter
…”It is considered the ideal time domain filter”.,.
Being able to implement a filter of that type with only a few lines of code (or addition steps in the case of a spreadsheet) is very difficult. CTRM manages to do so.
Also we have the requirement of a desperately short data series. Most work in other disciplines that use filters of different types have long ‘run up’ times as Greg mentions above.
So a combination of factors which means (pun) that a CTRM is a simple, Occam’s Razor, filter choice 🙂
Greg Goodman
They can be programmed but usually by recursive formulae.
I’m guessing this is what’s need in the time/spatial domain? I’ve only ever used the Butterworth filter in the frequency domain where as I’m sure you know things are a lot “easier”.
That means they need a long ‘spin-up’ period before they converge and give a reasonably stable result that is close to intended characteristics.
Again, my use of them is obviously far more presumptuous than yours. I simply use them as they were intended as passband filters; you obviously have greater experience here and understand the nuances better, but in the frequency domain there is no recursion and no convergence, at least not how I use them ;).
I guess, and I’d be interested in hearing your view. It is often suggested, and for argument’s sake we assume an almost exhaustive data set, that carrying out processing in the frequency domain is the correct approach to applying filters; where the data series is possible/likely the composite of other periodic series. But as the recent post relating to signal stationarity (on WUWT, where you commented) suggested, doing anything according to the “global” spectrum of a signal is fraught with danger.Therefore, applying any filter in the frequency domain is in itself creating local artifacts if the underlying periodic components are non-stationary (e.g. data series composed from several “frequency-modulated” components, I use the term loosely here for brevity).
I’m not pretending to be an expert here, but that recent post seemed somewhat contrived and a little light on the implications.
Thanks for your reply, Richard. Comments follow.
RichardLH says:
March 17, 2014 at 4:14 am
You’ll have to define “accuracy” for me in terms of a smoother … how are you measuring that?
The same is true of “simplicity”. Simplicity of what? Implementation? In what computer language?
As I showed above, your claim about the 12 month cycle is simply not true. Let me repeat it here.
There is no visible difference between your smoother and a standard gaussian smoother. Nor is there any 12 month cycle in the residuals between them (yours minus gaussian). Your claim is falsified.
And as your own link clearly shows, there is only a tiny difference between yours and gaussian, and it’s way out on the edge of the rolloff … color me unimpressed. When a difference makes a difference that is less than a line’s width on the screen, I don’t care.
I do observe what’s there. What I don’t do is call it a cycle.
So you’re taking up the Greg Goodman style of debate, where you just make some off-the-wall claim and then neglect to actually go see if it is true?
Look, Richard. I did the work to produce my estimate of the inflection points, which clearly shows there is not a regular cycle of any kind.
Now if you think that it is not correct because I didn’t “detrend the curve first with the greater than 75 years line (not a straight line!)”, then put your money where your mouth is and do the legwork to show us that your claim is true. I did it for my claim … your turn.
I don’t even know what kind of a “75 years line (not a straight line!)” you’d recommend we use to detrend it. So do the work and come back and show us just how much your (unknown) procedure changes the cycle lengths … my prediction is, not much …
No, you display what is there and then extrapolate it unjustifiably.
That’s your evidence? A graph of the smoothed swings of the PDO? Look, there’s no regular cycle there either. I don’t have your data or code, but by eye, the Chen data (red line) has cycles of lengths 90, 90, 110, 60, and 60 years … is that a “possible correlation that does need addressing”? Is that another of the famous “approximately sixty year” cycles? Or is it an “approximately ninety years” cycle?
In any case, I don’t know if it’s a “possible correlation” that needs addressing, because you haven’t said what it was supposed to be correlated with. Is there another wiggle with cycles of 90, 90, 110, 60, and 60 years that I don’t know about?
Yes … but your method (and a gaussian smooth and a loess smooth), on the other hand says it’s just gone level. Who you gonna believe?
Look, Richard. Either a cycle is 60 years, or it is not. Whenever anyone starts waving their hands and saying that there is an “approximately 60 year cycle”, I tune out because there’s no such thing.
Let me go over the bidding again. If we measure from peak to peak, there are two cycles in the HadCRUT4 data, of length 48 years and 60 years. If we measure trough to trough, there are two cycles in the data, of length 70 years and 61 years.
Now if you wish to call a wiggle with “cycle lengths” of 61, 48, 70, and 60 years an “approximately 60 year cycle”, and then project that putative “cycle” out for 30 years, I can’t stop you.
I will, however, point out that such behavior is a sign of incipient cyclomania, and in the event it persists longer than four hours you should seek immediate medical assistance …
w.
RichardLH says:
”This does allow quite a detailed look at the temperature series that are available. It allows for meaningful comparisons between those series.”
A detailed look at manipulated junk is still manipulated junk (IMHO). I’ll concede it does allow for comparing one manipulated junk data-set to another manipulated junk data-set and thereby revealing they’re different manipulated junk datasets. (JMHO)
Just look at your figure 4 between 1945 and 1970. Does that look like something that would cause an ice age scare?
Look, I really do appreciate your effort but I’m afraid those that have betrayed us and objectivity have ruined any attempt at data analysis for many more
yearsdecades to come.wbrozek says:
”In that case, how would you prove that the warming that is occurring now is not catastrophic?”
That’s the problem, we can’t prove it’s not and they can’t prove it is.But we have the satellites now for overwatch so if it does something dramatic like pause (…. LOL …), no, it’d have to be more dramatic than that (like drop significantly) then it’d be extremely difficult for them to explain.
Willis Eschenbach says:
March 17, 2014 at 10:40 am
“Thanks for your reply, Richard. Comments follow.
You’ll have to define “accuracy” for me in terms of a smoother … how are you measuring that?
The same is true of “simplicity”. Simplicity of what? Implementation? In what computer language?”
Mathematically more accurate than a single mean, mathematically simpler than a Gaussian.
“As I showed above, your claim about the 12 month cycle is simply not true. Let me repeat it here. There is no visible difference between your smoother and a standard gaussian smoother. Nor is there any 12 month cycle in the residuals between them (yours minus gaussian). Your claim is falsified.”
http://climatedatablog.files.wordpress.com/2014/02/fig-2-low-pass-gaussian-ctrm-compare.png
Then these two lines would lie one over the other and they don’t. Small difference only, true, but there none the less. Claim substantiated.
“And as your own link clearly shows, there is only a tiny difference between yours and gaussian, and it’s way out on the edge of the rolloff … color me unimpressed. When a difference makes a difference that is less than a line’s width on the screen, I don’t care. ”
So you try and get a Gaussian response curve in a spreadsheet with only two extra columns and no macros. Or only a few lines of R to do the same thing. Look, if you don’t like CTRM – use Gaussian instead. No skin off my back. Move on to where it is really important, the 15 year corner frequency and why that displays a ~60 year signal. All available frequencies above 15 years drop out to something in that bracket and a final residual at longer then 75 years.
“So you’re taking up the Greg Goodman style of debate, where you just make some off-the-wall claim and then neglect to actually go see if it is true?”
No. Please don’t put words in my mouth. Ask and I will respond.
“Look, Richard. I did the work to produce my estimate of the inflection points, which clearly shows there is not a regular cycle of any kind. Now if you think that it is not correct because I didn’t “detrend the curve first with the greater than 75 years line (not a straight line!)”, then put your money where your mouth is and do the legwork to show us that your claim is true. I did it for my claim … your turn.”
Hmmm. You were the one proposing to measure two ‘cycles’ of data and expecting to see a precise answer, not me. I understand all too well what is possible and/or not possible here. I consistently say ~60 years because that is all that can reasonably be stated. In fact if you do subtract the 75 year line you get a non-symmetrical wave shape with a shorter ‘top half’ to a longer ‘bottom half’ but with just two samples I would not stake my life on it. Too many other possibilities including wave shape or other longer ‘cycles’ could also be the reason.
“I don’t even know what kind of a “75 years line (not a straight line!)” you’d recommend we use to detrend it. So do the work and come back and show us just how much your (unknown) procedure changes the cycle lengths … my prediction is, not much … ”
Well I would have thought the blue line on the graph was a clue! The main problem is that is only a single running mean and has loads of distortions in it. I could run a S-G at 75 years which would be much more likely to produce something reasonable.
I can produce the band pass splitter circuit you obviously want by subtracting one output from the other but I thought that was a step too far as yet. I’ve only just brought this subject up!
Now you are demanding it and sort of hinting I don’t know what I am doing. Not very respectful.
“No, you display what is there and then extrapolate it unjustifiably.”
An S-G curve is very, very well respected. The nearest equivalent in statistics is LOWES (which some would claim was based on it in any case). If you wish to discount LOWES then do so for S-G. Otherwise the projection stands.
“That’s your evidence? A graph of the smoothed swings of the PDO? Look, there’s no regular cycle there either. I don’t have your data or code, but by eye, the Chen data (red line) has cycles of lengths 90, 90, 110, 60, and 60 years … is that a “possible correlation that does need addressing”? Is that another of the famous “approximately sixty year” cycles? Or is it an “approximately ninety years” cycle?”
If you would like the code, just ask. You could probably figure it out given that I have a simple example with RSS as a link in the article (see above). Just replace the source data with the Shen data and jaiso data and use a de-trended HadCRUT and you will be there.
Pretty good match with the available thermometer data in their overlap periods.
“In any case, I don’t know if it’s a “possible correlation” that needs addressing, because you haven’t said what it was supposed to be correlated with. Is there another wiggle with cycles of 90, 90, 110, 60, and 60 years that I don’t know about?”
As the thermometer data only goes back to the 1800s that is all the co-relation that is possible.
“Yes … but your method (and a gaussian smooth and a loess smooth), on the other hand says it’s just gone level. Who you gonna believe?”
You are obviously looking at different graphs to me then. It is not a LOWES, it is an S-G. Verified against the Gaussian for parameter choice ‘proof’.
“Look, Richard. Either a cycle is 60 years, or it is not. Whenever anyone starts waving their hands and saying that there is an “approximately 60 year cycle”, I tune out because there’s no such thing.”
Nature has a habit of never being clockwork precise – other than for orbits and then only for a simple two body solution.
“Let me go over the bidding again. If we measure from peak to peak, there are two cycles in the HadCRUT4 data, of length 48 years and 60 years. If we measure trough to trough, there are two cycles in the data, of length 70 years and 61 years.”
As I noted above, it is quite possible that the half cycles are of different periods. That is the way nature often does this stuff. Way too early to tell for sure of course, but this is just a first step.
“Now if you wish to call a wiggle with “cycle lengths” of 61, 48, 70, and 60 years an “approximately 60 year cycle”, and then project that putative “cycle” out for 30 years, I can’t stop you. I will, however, point out that such behavior is a sign of incipient cyclomania, and in the event it persists longer than four hours you should seek immediate medical assistance …”
Thank you for your concern. However that does not make the observations go away. The data has a wriggle in it that needs explaining at around ~60 years and, so far, none of the proposed physics does that.
Willis Eschenbach says:
March 17, 2014 at 10:40 am
“Either a cycle is 60 years, or it is not.”
There are no precise cycles in nature. Even our best electronic oscillators wander in frequency over time, and those are meticulously designed with conscious intent. But, they still stay within the general neighborhood of a central frequency, and the oscillations can be extrapolated forward in time on that basis, with the prediction becoming gradually more uncertain the longer the extrapolation interval.
A lot of us saw the ~60 year cycle in the data more than a decade ago. People with their heads in the sand insisted it was “cyclomania”. The rest of us predicted it would show a turnaround mid-decade. Guess what? It did. Denial of its existence is no longer tenable.
John West says:
March 17, 2014 at 10:42 am
Well we have to work with what is there, not what we would like to be there. Just because you don’t like what there is, there is no reason to discard it completely.
You do realise that if a natural ~60 ‘wriggle’ is there in the data then AGW is a lot less than is currently calculated don’t you?
Greg Goodman said March 17, 2014 at 2:24 am in part:
“I’ve always been dubious of LOESS filters because they have a frequency response that changes with the data content. I’ll have to have a closer look at SG. It does have a rather lumpy stop band though. I tend to prefer Lanczos for a filter with a flat pass band ( the major defect of gaussian and the triple RM types ).”
I am not familiar enough with LOESS to comment. If the frequency response changes with the data, it is not a Linear Time-Invariant (LTI) filter. SG is LTI. True enough the trade-off for a flat passband is a bumpier stop-band. Always is – No free lunches.
As for Lanczos, if I recall correctly it is an impulse response that is nothing more than a truncated sinc. As such it is subject to Gibbs Phenomenon transition band ripples – perhaps 23% or so – hardly a flat passband.
These points are well-considered in the signal processing literature. There is a danger to using a “built-in” without understanding it in detail. Hope I’m not being unfair – don’t intend to be.
Bart says:
March 17, 2014 at 11:26 am
“A lot of us saw the ~60 year cycle in the data more than a decade ago. People with their heads in the sand insisted it was “cyclomania”. The rest of us predicted it would show a turnaround mid-decade. Guess what? It did. Denial of its existence is no longer tenable.”
Well you do have to stack up: volcanos, SO2 and CO2 in just the right order and magnitude with no lag and you CAN get there! if you try really hard 🙂
Willis Eschenbach says:
March 17, 2014 at 10:40 am
P.S. Care to re-do your plot without the green line so that a true comparison can be made? If I plotted temperature data with the full yearly cycle still in it then any deviations of the Anomaly would be hard to spot as well.
Play fair at least.
Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely. If you argue that you can recover the original from the smoothed alone, in the general case, you are wrong. Inversion is often at least ill-conditioned and usually impossible due to singularities. Most useful filters have what are intended to be stopbands. As such, they generally have nulls (unit-circle zeros). The inverse filter has unit-circle poles and is useless.
A moving average has nulls. SG has nulls. Continuous infinite-duration Gaussian, I believe, has no nulls, but I’m not sure about a truncated Gaussian which is convolved with a sinc? Successive passes (cascading) through moving-average rectangles have nulls – they accumulate.
Smoothing may be curve fitting (such as intentional interpolation), but it need not be. But if you claim insight from a smoothed curve, all you can really claim is that the data provides some evidence for what you feel is a proper model of the data, and the rest you assume is noise, worthy of being discarded. This is certainly circular reasoning at least in part. Such a claim requires considerable caution and full analysis and disclosure.
Bernie Hutchins says:
March 17, 2014 at 12:04 pm
“Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely. If you argue that you can recover the original from the smoothed alone, in the general case, you are wrong. ”
Not true in the case of a digital low pass/high pass band pass splitter filter such as a CTRM. You can do a 1-x by subtracting one band from the input source and get the other.
Has to be the case because here is no-where else for the data to go! In digital terms only of course, analogue could have some losses 🙂
(Rounding errors and the like excepted of course but we can assume they are not likely to be even close to a dominant term – quantisation errors come much,much higher than that).
Richard –
Please read what I wrote, and then think. You said:
“Not true in the case of a digital low pass/high pass band pass splitter filter such as a CTRM. You can do a 1-x by subtracting one band from the input source and get the other.”
I said:
“Smoothing is generally the destruction of at least some of the data. If you argue that you still have the original data stashed away – then true enough (obviously) you haven’t lost anything completely.”
You say you haven’t thrown data away BECAUSE you kept a copy of the input. Didn’t I say that?
Bernie Hutchins says:
March 17, 2014 at 12:28 pm
“Please read what I wrote, and then think.”
I did. People are always suggesting that a simple low pass filter throws information away. That is not the case here (and rarely is in climate work). The input digitisation/rounding/truncation errors are by far and away the largest term. Anything to do with the calculations in the filter are in the dust (assuming reasonable floating point operations anyway). There are only a few adds and three divisions to get to any single output value. If you do the calculations of error propagation you will see.
It becomes a sort of mantra that people roll out to say why this is not a valid treatment without considering if that is truly the case.
Sorry you just pushed one of my buttons and I responded. probably in an ill considered way. Apologies.
If we were talking about 16 or 24 bit audio or video streams I would agree that this is not completely lossless, but with the data streams we have in climate, filter errors are the least of our worries.
RichardLH says:
March 16, 2014 at 1:58 pm
Lance Wallace says:“Can the Triple whatsis somehow compare the two curves and derive either a lag time or an estimate of how much of the CO2 emissions makes it into the observed atmospheric concentrations?”
RichardLH :Not really. Low pass filters are only going to show periodicity in the data and the CO2 figure is a continuously(?) rising curve.
A LPF by definition, passes frequencies below it’s cut-off frequency. Since D.C. (i.e. frequency = 0) is always below the cut-off, the above statement by RLH is incorrect. A ramp-like input into a LPF will cause a lagged ramp-like output, with the lag determined by the filters group delay characteristic.
Note that it is incorrect I think to speak of an LTI filter (such as the one described here) as causing “distortion”, which as a term of signal processing art refers exclusively to artifacts caused by system non-linearity. If you can sum the output of your filter with the residual (i.e. the rejected signal) and reconstruct the input, no distortion has occurred.
Another way of saying this is that an LTI filter can not create energy at a given frequency that is not present in the input signal, even if the filter response peaks badly at that frequency. Such peaking/sidelobe/pb-ripple may extenuate frequencies you wish to extinguish (an so in a sense “distort” the variance present at that frequency relative to some other frequency band of interest), but this is not distortion as the term is normally used.