Guest post by Richard Saumarez
Correlations are often used to relate changes in climate variables in order to establish a potential relationship between them. These variables may have been processed in some way before they are correlated, possibly by filtering them. There has been some debate, which may have shed more heat than light, about the validity of combining these processes and whether they interact to make the conclusions drawn from them invalid. The object of this post is to explain the processes of correlating and filtering signals, the relationship between them and show that the results are predictable provided one takes care.
The importance of the Fourier Transform/Series.
Fourier analysis is of central importance in filtering and correlation because it allows efficient computation and gives theoretical insight into how these processes are related. The Fourier Transform is an analytical operation that allows a function that exists between the limits of – and + infinity to be expressed in terms of (complex) frequency and is a continuous function of frequency. However, a Fourier Transform of a real-World signal, which is sampled over a specific length of time – a record -, is not calculable. It can be approximated by a Fourier series, normally calculated through the discrete Fourier Transform algorithm, in which the signal is represented as the sum of a series of sine/cosine waves whose frequencies are an exact multiple of the fundamental frequency (=1./length of the record). Although this may seem to be splitting hairs, the differences between the Fourier Transform and the series are important. Fortunately, many of the relationships for continuous signals, for example a voltage wave form, are applicable to signals that are samples in time, which is the way that a signal is represented in a computer. The essential idea about the Fourier transform is that it takes a signal that is dependent on time, t, and represents it in a different domain, that of complex frequency, w. An operation performed on the time domain signal has an equivalent operation in the frequency domain. It is often simpler, far more efficient, and more informative to take a signal, convert it into the frequency domain, perform an operation in the frequency domain and then convert the result back into the time domain.
Some of these relationships are shown in figure 1.
Figure 1. The relationship between the input and output of a system in the time and frequency domains and their correlation functions. These are related through their (discrete) Fourier Transforms.
If a signal, x(t) passes through a linear system, typically a filter, the output, y(t) can be calculated from the input and the impulse response of the filter h(t), which is, mathematically, its response to an infinite amplitude spike that lasts for an infinitesimal time. The process by which this is calculated is called a convolution, which is often represented as “*”, so that:
y(t)=x(t)*h(t)
Looking at figure 1, this is shown in the blue upper panel. The symbol t, representing time, has a suffix, k, that indicates that this is a sampled signal at t0, t1, t2 …… Arrows represent the Discrete Fourier Transform (DFT) that convert the signal from the time to the frequency domain the inverse transform (DFT-1) that converts the signal from the frequency to the time domain. In the frequency domain, convolution is equivalent to multiplication. We can take a signal and transform it from x(tk) to X(wn). If we know the structure of the filter, we can calculate the DFT, H(wn), of its impulse response. We can write, using the symbol F as the forward transform and F-1 as the inverse transform, the following relationships to get the filter output:
X(w)=F[x(t)]
H(w)=F[h(t)]
Y(w)=X(w)H(w)
y(t)=F-1[Y(w)]
What we are doing is taking a specific frequency component of the input signal, modifying it by the frequency response of the filter to get the output at that frequency. The importance of the relationships shown above is that we can convert the frequency response of a filter, which is how filters are specified, into its effect on a period of a time domain signal, which is usually what we are interested in. These deceptively simple relationships allow the effects of a system on a signal to be calculated interchangeably in the time and frequency domains.
Looking at the lower panel in Figure 1, there is a relationship between the (discrete) Fourier Transform and the correlation functions of the inputs and outputs. The autocorrelation functions, which are the signals correlated with themselves are obtained by multiplying the transform by a modified form, the complex conjugate, written as X(w)*, (see below), which gives the signal power spectrum and taking the inverse transform. The cross correlation function is obtained by multiplying the DFT of the input by the complex conjugate of the output to obtain the cross-power spectrum, Gxy(w) and taking the inverse transform, i.e.:
Gxy(w)=X(w)Y*(w)= X(w)X*(w)H*(w)
Rxy(t)=F-1[Gxy(w)]
Therefore there is an intimate relationship between time domain signals representing the input and output of a system and the correlation functions of those signals. They are related through their (discrete) Fourier Transforms.
We now have to look in greater detail at the DFT, what we mean by a frequency component and what a cross-correlation function represents.
Figure 2 shows reconstruction of a waveform, shown in the bottom trace in bold by adding increasingly higher frequency components of its Fourier series. The black trace is the sum of the harmonics up to that point and the red trace is the cosine wave at each harmonic. It is clear that as the harmonics are summed, its value approaches the true waveform and when all the harmonics are used, the reconstruction is exact.
Up to this point, I have rather simplistically represented a Fourier component as being a cosine wave. If you compare harmonics 8 and 24 in figure 2, the peak of every third oscillation of harmonic 24 coincides with the peak in harmonic 8. In less contrived signals this does not generally occur.
Figure 2 A wave form, shown in bold, is constructed by summing its Fourier components shown in red. The black traces show the sum the number of Fourier components up to that harmonic.
The Importance of Phase
Each Fourier component has two values at each frequency. A sine and cosine waves are generated by a rotation of a point about an origin (Figure 3). If it starts on the y axis, the projection of the point is a sine wave and its projection on the x axis is a cosine wave. When the Fourier coefficients are calculated, the contribution of both a sine and a cosine wave to the signal at that frequency are determined. This gives two values, the amplitude of the cosine wave and its phase. The red point on the circle is at an arbitrary point and so its projection becomes a cosine wave that is shifted by a phase angle, usually written as j. Therefore the Fourier component at each frequency has two components, amplitude and phase and can be regarded as a vector.
Earlier, I glibly said that convolution is performed by multiplying the transform of the input by the transform of the impulse response (this is true since they are complex). This is equivalent to multiplying their amplitudes and adding their phases. In correlation, rather than multiplying X(w) and Y(w), we use Y(-w), the transform represented in negative frequency, the complex conjugate. This is equivalent to multiplying their amplitudes and subtracting their phases. Understanding the phase relationships between signals is essential in correlation[i].
Figure 3 The Fourier component is calculated as a sine and cosine coefficient, which may be converted to amplitude, A, and phase angle j. The DFT decomposes the time domain signal into amplitude and phases at each frequency component. The complex conjugate at is shown in blue.
Signal shape is critically determined by phase. Figure 4 shows two signals, an impulse and a square wave shown in black. I have taken their DFT, randomised the phases, while keeping their amplitudes the same, and then reconstructed the signals, shown in red.
Figure 4. The effect of phase manipulation. The black traces are the raw signal, and the red trace is the signal with a randomised phase spectrum but an identical amplitude spectrum
This demonstrates that phase has very important role in determining signal shape. There is a classical demonstration, which space doesn’t allow here, of taking broad band noise and imposing either the phase spectrum of a deterministic signal, while keeping the amplitude spectrum of the noise unaltered or doing the reverse: imposing the amplitude spectrum of the deterministic signal and keeping the phase of the noise unaltered. The modified spectrum is then inverse-transformed into the time domain. The phase manipulated signal has a high correlation with the deterministic signal, while the amplitude manipulated signal has a random correlation with deterministic signal, so underlining the importance of phase in determining signal shape.
Figure 5 The phase spectra of delayed impulses.
A very important concept is that phase represents delay in a signal. A pure time delay is a linear change in phase with frequency as shown in figure 5. The amplitude of the signal is unaltered, but in the case of a delay, there is increasing negative phase with frequency. However, any system that changes the shape of the input signal as it is passed through it, as is usually the case, will not have a linear phase spectrum. This is a particularly important concept when related to correlation.
We are now in a position to understand the cross-correlation function. Looking at the formula for correlation shown in figure 1, the CCF is:
,
which can be normalised to give the coefficient: 
, where N is the number of points in the record.
This rather formidable looking equation is actually quite straightforward. If k is zero, this is simply the standard formula for calculating the correlation coefficient and x is simply correlated with y. If k is one, the y signal is shifted by one sample and the process is repeated. We repeat this for a wide range of k. Therefore the function rxy is the correlation between signals two signals at different levels of shift, k and this tells one something about the relationship between the input and output.
We have a signal x(t) which has been passed through a physical system, with specific characteristics, which results in an output y(t) and we are trying to deduce the characteristics of the system, h(t). Since, from Figure1, the DFT, Y(w) of the output is the product of the DFTs of the input X(w) and the impulse response, H(w), could we not simply divide the DFT of the output by the DFT of the input to get the response? In principle, we can, providing the data is exact.
However most real world measurements contain noise, which is added to the inputs and outputs, or even worse other deterministic signals, and this renders the process somewhat error prone and the results of such a calculation are shown below (figure 6), illustrating the problem:
Figure 6. Left: Input (black) and output (red) signals for a system. Right: the calculated impulse response with 2.5% full scale amplitude noise added to the input and output (black) compared with the true impulse response (red). The low pass filtered response is shown in green.
This calculation illustrates another very important concept: linear physical systems store and dissipate energy. For example, a first order system, which could be the voltage output of a resistor/capacitor network or the displacement of a mechanical spring/damper system, absorbs energy transients and then releases the energy slowly, resulting in the negative exponential impulse response shown in figure 6. The variables which fully define the first order system are its gain and the time constant. The phase spectrum of the impulse response in distinctly non-linear. Attempts to measure another variable, for example delay, which implies a linear phase response, is doomed to failure because it doesn’t really mean anything. For example, if one is looking at the relationship between CO2 and temperature, this is likely to be a complex process that is not defined by delay alone and therefore the response of the system should be identified rather than a physically meaningless variable.
Noise and Correlation
Correlation techniques are used to reduce the effects of noise in the signal. They depend on the noise being independent of, and uncorrelated with, the underlying signal. As explained above, correlation is performed by shifting the signal in time, multiplying the signal by itself (auto-correlation) or with another signal (cross-correlation), summing the result, and performing this at every possible value of shift.
Figure 7. Broad band noise ( black) with the autocorrelation function( red) superimposed.
In broad band noise, each point is, by definition uncorrelated with its neighbours. Therefore, in the auto-correlation function, when there is no shift, there will be perfect correlation between it and its non-shifted self. For all other values of shift, the correlation is, ideally, zero, as shown in figure 7.
The auto correlation function of a cosine wave is obtained in the same manner. When it is unshifted, there will be a perfect match and the correlation will be 1. When shifted by ¼ of its period, the correlation will be zero, be -1 when shifted by ½ a period and zero when shifted by ¾ of period.
The ACF of a cosine wave is a cosine wave of the same frequency with an amplitude of the square of the original wave. However if there is noise in the signal, the value of the correlation will be reduced.
Figure 8 shows the ACF of broadband noise with two sine wave embedded in it. This indicates recovery of two deterministic signals that have serial correlation and are not correlated with the noise. This is a basis for spectral identification in the presence of noise.
Figure 8 The ACF of a signal containing two sine waves of different frequencies embedded in noise. The ACF (red) is the sum of two cosine waves with the same frequencies.
A very important feature of the ACF is that if destroys phase information. Referring to Figure 1, the DFT of the ACF is X(w) (or Y(w)) multiplied by its complex conjugate, which has the same amplitude and negative phase. Thus when they are multiplied together, the amplitudes are squared and the phases are added together making the resultant phase zero. This is the “power spectrum” and is the ACF is its inverse DFT. Therefore the ACF is composed entirely of cosine waves and is symmetrical about a shift of zero.
However, the cross-correlation function, which is the inverse DFT of the cross-power spectrum contains phase. By multiplying the complex conjugate of the output by the input in the frequency domain, one is extracting the phase difference and the delays at each frequency between the input and the output and the cross-correlation function reflects this relationship. If the power spectrum, e.g.: the DFT, of rxx(t) is Gxx(w) and the cross-power spectrum of rxy(t) is Gxy(w), then:
H(w)=Gxy(w)/Gxx(w)
Figure 9 shows the same data as used in figure 6 to calculate the impulse response and the error is very much reduced because signal correlation is a procedure that separates the signal from noise.
Figure 9. Estimate of the impulse response using the data in figure 6 via cross-correlation (black) and pre-filtering the data (green).
These calculations are based on a single record of the input and output. When available, one uses multiple records and calculates the estimated response E[H(w)] from the averaged power spectra:
E[H(w)]=< Gxy(w)>/<Gxx(w)>
Where < x> means the average of x. This leads to a better estimate of the impulse response. It is possible to average because correlation changes the variable from the time domain to relative shift between the signals so aligning them.
One simple check that can be performed to check that one is getting reasonable results, assuming that one has enough individual records, is to calculate the coherence spectrum. This is effectively the correlation between the input and output at each frequency component in the spectrum. If this is significantly less than 1.0, it is likely that there is another input, which hasn’t been represented, or the system is non-linear.
One of the major problems in applying signal processing methods to climate data is that there is only one, relatively short, record and therefore averaging cannot be applied to improve estimates.
Improving resolution by record segmentation and filtering.
One can improve estimates of the response if one has a model of what the signal represents. If one is dealing with a short term process, in other words the output varies quickly and transiently in response to a short term change in input, one can estimate the length of record that is required to capture that response and hence the frequency range of interest. This enables one to segment the record into shorter sections. Each segment has the same sampling frequency, therefore the highest frequency is preserved. By shortening the length of each record we have thrown away low frequencies because the lowest frequency is 1/(record length). However, we have created more records containing high frequencies, which can be averaged to obtain a better estimate of the response.
The other strategy is filtering. This, again, involves assumptions about the nature of signal. Figure 10 shows the same data as in figures 7 & 8 after low-pass filtering. The ACF is no longer an impulse but is expanded about t=0. However the ACF of the deterministic signal is recovered with higher accuracy.
This can be done here because the signal in question has a very small, low frequency, bandwidth and is not affected by the filtering (figure 11). The effects of the filter are easily calculable. If it has a frequency response of A(w), the input and output signals become X(w)A(w) and Y(w)A(w). The cross correlation spectrum is therefore simply:
Gxy(w)= A(w)X(w)A*(w)Y*(w)=A2(w)X(w)Y*(w)
Figure 10 The ACFs using the same data as in Figure 6. Note that the ACF of noise is no longer an impulse at t=0 and that there has been a considerable improvement in the ACF of the two sine waves as they now represent a higher fraction of the total power in the signal.
A2(w)is the autocorrelationfunction of the filter, which has no phase shift and will not affect the phase of the cross-power spectrum, provided the same filter is used on the input and output. This is because the phase reversal of the complex conjugate of the filter in the output cancels out that applied to the input, so the timing relationships between the input and output will not be affected. Provided the ACF spectrum is in the pass band of the filter, it will be preserved. In figure 9, the estimated impulse responses are shown using filtered (green) and non-filtered data. If one wishes to characterise the response, by assuming it is a first order system (which this is), one can fit an exponential to the data so getting its gain and time constant. The filtered result gives marginally better estimates but one has to design the filter rather carefully, appreciate that filtering modifies the impulse response and correct the results for this.
Thus, it is possible to filter signals and perform correlations, provided that the frequency band being filtered does not overlap to much the system response, as illustrated in figure 11, and one is careful.
Figure 11. The signal spectrum is composed of the true signal and noise spectra. A good filter response (green) will attenuate (grey) some of the noise but preserve the true signal, while a bad filter will modify the true signal spectrum and hence the ACF.
In practice, however there is likely to be an overlap between the noise and signal spectra. If the system response is filtered, the correlation functions will be filtered and become widened and oscillatory. In this case, the results won’t mean much and almost certainly will not be what you think they are! There are more advanced statistical methods of determining which part of the spectra contain deterministic signal but, in the case of climate data, the short length of the modern record and the poor quality of the historical record makes this very difficult.
Degrees of Freedom.
Suppose we have two signals and we want to determine if they have different means. They both have a normal distribution and the same variance. Can we test the difference in means by performing a “Student’s t” test? This will almost certainly be wrong, because in most simple statistical tests, there is an assumption that each observation is independent. In figure 7, the ACF is an impulse and nominally zero elsewhere, showing that each point is independent of each other. If the signal is filtered, the points are no longer independent because we have convolved the signal with the impulse response of the filter, as shown in figure 10. Looking at figure 1, the time domain convolution is given by:
This is similar to the correlation formula, except that the impulse response is reversed in time. It shows that the output at any point is a weighted sum of the inputs that have preceded it and are therefore no longer independent. Therefore in applying statistical tests to signal data, one has to measure the dependence of each sample on others by using the autocorrelation of the signal to calculate the number of independent samples or “degrees of freedom”.
Conclusion.
Correlation and filtering are highly interdependent through a set of mathematical relationships. The application of these principles is often limited because of signal quality and the “art” of signal processing is to try to get the best understanding of a physical system in the light of these constraints. The examples shown here are very simple, giving well defined results but real world signal processing may be messier, require much more statistical characterisation and give results that may be limited statistically by inadequate data.
One always has to ask what is the goal of processing a signal and does this make any sense physically? For example, as discussed earlier, cross correlation is often used interchangeably with “delay and it is only meaningful if one believes that phase response of the system in question has a linear phase response with frequency. If one is estimating something that is not meaningful, additional signal processing will not be helpful.
Rather, if one has a model of the system, one can then calculate the parameters of the model and, having done this, one should look carefully at the model to see if it accounts for the measured signals. Ideally, this should be tested with fresh data if it is available, or one can segment the raw data and use one half to create the model and test it with the remaining data.
Modern scripting programs such as “R” allow one to perform many signal processing calculations very easily. The use of these programs lies in not applying them blindly to data but in deciding how to use them appropriately. Speaking from bitter experience, it is very easy to make mistakes in signal processing and it is difficult to recognise them. These mistakes fall into three categories, programming errors, procedural errors in handling the signals and not understanding the theory as well as one should. While modern scripting languages are robust, and may largely eliminate straight programming errors, they most certainly do not protect one from making the others!
[i] I have used the electrical engineering convention in this post, i.e.: the basis function of the Fourier Transform is a negative complex exponential.
Related articles
- Fourier series before Fourier (johndcook.com)
- Retrotechtacular: The Fourier Series (hackaday.com)
- Sampling Rate (tamino.wordpress.com)
Richard,
“A sine and cosine waves are generated by a rotation of a point about an origin (Figure 3). If it starts on the y axis, the projection of the point is a sine wave and its projection on the x axis is a cosine wave.”
Thank you for this article. I find it most in lighting, but I’m confused. Above is what you “wrote” and below is what I “heard”.
“If it starts on the y axis” > Then I image a point in the diagram on the little circle at x = 0, y = -1 (or minimum), because the angle measurement seem to be starting from down position.
“the projection of the point [on to the y-axis] is a sine wave” > here I image the point will be projected again to x =0 (zero because it is the starting point) and to y = -1 (or minimum because that appears what the projection would look like to me and what appear in the black trace in the figure), but sin(0) is not -1.
“its projection on the x axis is a cosine wave.” > the projection I believe would be x = 0, y = 0 (rotating the x and y axes), but cosine(0) is not 0.
What am I doing wrong?
Thanks again,
David
Very nice review, but it makes me miss my last job. What I am doing now is rather boring. The most complicated math I use now is some basic graph theory 🙁
BTW, I have been getting interested in wavelet analysis.
Excellently written article, clear and concise.
Jeff L ” Now we need to assume stationarity & linearity (and those assumptions ultimately would need to be tested) but in theory, using the convolutional model, we can calculate h(t) .”
Temperature will not be stationary, however, dT/dt probably will be. That is the first step I usually take when looking at temperature time series.
Thanks, Richard.
Your excellent article transported me back to electronics engineering school.
This is one way to look at climate data and clearly see that we know almost nothing yet.
I hope more knowledge will bring back sanity to the climate debate.
I second the comments of richardscourtney and OldWierdHarold.
My reaction upon seeing the post was the same as that of, I suspect, a not insignificant minority: the post seems unlikely to contain anything I don’t already know (well, knew at one time), and, since its introduction fails to identify a specific question before the house to which the post would apply its basic signal processing, reading it does not appear worth my time. So I haven’t read it.
This is not to say that it may not be a valuable post for many here; it’s just to let you know how it plays to one segment of your audience.
vukcevic says:
A correlation coincidence, or maybe not.
http://www.vukcevic.talktalk.net/Peru-Ice.htm
North and the South hemisphere marching in step, warmth in the north with ice in the south.
===
How about ANTARCTIC melting period (inverse) correlating with ARCTIC Oscillation index?
http://climategrog.wordpress.com/?attachment_id=208
Ironically it appears correlate somewhat better than the Arctic melting period does. That was quite a surprise.
This is still work in progress, more details once I’ve checked it more thoroughly.
Any suggestions on how to get a numerical measurement of correlation between daily AO and once per year melting periods?
Excellent article. I have always wondered why FFT analysis has not been applied to the GW data. My only guess is that they want the scam and not the truth.
It is amazing what a FFT can do. In 1977 I was a nuclear engineer (with a second degree in applied mathematics) responsible for starting up a nuclear power plant. I had spent weeks trying to find the cause of a problem with the flow of one of the pumps. One of the engineers was using the HP 5420A Digital Signal Analyzer (I believe this was the very first FFT analyzer of its kind) on the vibration transducer in an effort to see if there were bearing problems. My math degree kicked in and a light-bulb went off in my head. I borrowed the HP 5420, and hooked it up to the flow transducer and pressure transducer. The signal was coupled through a capacitor so that only the AC signal and not the DC voltage were going to the HP. Within 15 – 20 minutes I had determined that one of the volutes on the pump was causing minor, but enough, cavitation to create the difference in flow from this pump. I later used this to tune the feed-water control system to such a degree that we were able to get about 2% more power out of the plant. Two percent of 1000 megawatts is 20 megawatts more power! I went on to solve several other problems with the analyzer, The HP allowed me to solve problems in a few hours instead of days or weeks. The HP analyzer became my best friend, I used it more than my new fangled pocket calculator. Many of the new “intelligent” flow, level, and temperature control systems have a bare-bones FFT analyzer in their electronics today. If the AGW crowd really wants to know the truth they will start analyzing the data correctly, not massaging to fill their pocket book.
Richard
Firstly, whoa!
That’s a lot to take in, would it not have been easier just to refer to the Wiener–Khinchin theorem – would it not have cut down your article by half. Perhaps I’ve missed the point (or completely off track) but it did seem like hard going until I got to the best bits.
Secondly, do you not think you are taking a sledgehammer to crack a nut. Surely simple regression analysis will do for most bivariate issues. I agree that the t-stat may be a little to primitive but it could be vetted with other tests such as the Durbin Watson stat and simple cross-correlation. Don’t get me wrong, it seems really good to look at every scenario, but everyone seems data processing nuts in this field.
Thirdly, while I’m not questioning anything you say (you’re obviously the expert), you fleetingly mention phase shift as a result of certain types of filtering. In one of the examples above you show what looks like a Butterworth filter; this, as you allude to, will introduce a phase shift, surely this will affect any subsequent analysis. Perhaps a more elaborate discussion of this with result from a single study would make for another good article.
Joe Born
Well bully for you. If the article is below you then no need to comment unless you want a brownie point for being so knowledgeable.
Yes the article is long winded and probably a referral to some good online articles would’ve saved a lot of screen space, but he at least has put something out there and does attempt, I admit unsuccessfully, to communicate some commonly applied techniques in a less technical fashion. In short, he hasn’t condescended the reader…unlike you I hasten to add.
I think a good follow up would be to show why such approaches are better than just using more common approaches or we even need worry about such things in this (or any) field.
You don’t have to please all the people all the time; that’s why it’s good for websites to host all kinds of articles. Just thought you might want to know what other posters think.
This is a good introductory article on data processing and should be a must read and comprehend for scientists interpreting data.
Someone that has the time and energy should write an article on the acquisition of the data. The the data from the point of the measurement: Pt RTD, TC, PH probe etc, to the point where the data is stored on disk in a useful format for data processing.
This is one of the areas I work in. Very often I run across situations where the investigator is not purely measuring what they believe they are measuring.
The article should address the problems of the actual measurement process: linear, non-linear sensors, S&H aperture time, sample interval choice, A/D conversion problems, driver problems, operating system problems (not real time vs real time), choice of floating point or binary, etc, etc.
The way I see it, this the point where the starting materials for the sausages are made.
Great post Richard!
Do you mind if I keep a copy for reference; not for publicizing?
I want to second (third, fourth, ….) Richard’s comment to the effect that access to a good “tool box” of processing algorithms (e.g., the R suite of tools) isn’t sufficient to employ properly used those tools. My example is the man who is an expert in the use of every tool know to man. He is put in him in a room where any tool ever made is available to him and he knows how to properly use every tool. He is asked to build a car. His first question will likely be: “What’s a car?” I, too, have experience with the improper use of signal processing tools–especially digital signal processing tools.
I would like to add that for signal processing that can be implemented to a high degree of fidelity convolving the signal with a finite duration impulse response (e.g., lowpass, bandpass, differentiation, Hilbert Transform, etc.), the Discrete Fourier Transform (DFT) can be used (and often is used because it is computationally efficient) to filter a uniformly sampled time waveform. Although the magnitude of the DFT of a finite duration segment of an input signal provides a measure of the spectral content of that segment of the signal and thus provides a way to visualize the filtering process in the frequency domain, the spectral properties of the DFT are secondary to the filtering process. The DFT (and other transform procedures whose outputs give little if any information about the frequency content of the signal) can be used to efficiently implement finite impulse response convolution, The DFT is one of several digital signal processing algorithms having the property that the circular convolution of two equal length sampled waveforms is, within a scale factor, equal to the inverse DFT of the product of the DFTs of the two waveforms. The fact that the DFT of a finite duration segment of a waveform provides information about the spectral content of the waveform is a side benefit of the process.
William C Rostron, thank you for raising the central point of physical sanity of “world temperature,” measured to a tenth of a degree C. Only someone who has lived their life in books would think that has real world meaning. One might just as well ask how many feathers are there on an average bird 100, 1000, 10,000 years ago, calculated to the nearest feather.
Ooops. In my immediately foregoing comment, replace employ properly used with properly employ.
Thanks for the comments.
This post was in response to an argument about filtering data and then cross correlating them – is it valid?
1) @ur momisugly Joe Born and others. I agree that if you are familiar with DSP, this is a very simple exposition. My goal was to show that filtering and correlation are related and this may not be realised by some readers. If done incorrectly, correlation will be affected by filtering and this has been the subject of a number of posts and disputes – Eschenbach, Mckintyre, Briggs et al.
2) @ur momisuglyOtsar. Thanks, but I would need to write a textbook!
3) @ur momisuglycd. The purpose of the last section was explain, in simple terms, that autocorrelation affects statistical degrees of freedom and so needs to be taken into account when testing hypotheses statistically. As regards filtering, if one is using a physical filter with a phase shift on the input and output, the point of cross-correlation is that it eliminates the phase response of the filter, provided the filters have the same characteristics (and generally one should use Bessel filters that linear phase characteristics). If one uses a digital zero-phase filter for both, you won’t screw up the phase response at least.
4) @ur momisuglyUseless grad student. Yes you are basically correct. One has to careful in defining the sign of phase, otherwise the signal comes out backwards! Also, the magnitude in most DFTs needs care because it represents HALF the spectrum. If you remember that cos(wt)=(exp(jwt)+exp(-jwt))/2, the spectrum is represented in both positive and negative frequencies (with sine, imaginary, component negative) you will calculate amplitude/2 if you use a real transform.
5) @ur momisuglyDavid in Texas. A point rotating in unit radius circle will trace out cos(wt) on the x axis and sin(wt) on the y axis, i.e. they will be 90 degrees out of phase. I had some difficulty in drawing figure 3, and the solution to your problem depends on when you consider time to zero. If I start measuring a cosine wave at t=0, the phase will be zero but if I start 1/4 of a period later, the phase will 90 degrees. Phase is related to delay and this depends in a given signal on when you started and finished recording it. The important point in signals is not the absolute phase but the way phase changes with frequency – the phase spectrum.
A more general point is this. The type of analysis I have described is related to LINEAR systems, that obey 1) the output is proportional to the input, 2) the output of the sum of the two different input functions is equal to the sum of the outputs of the responses of the individual functions (superposition) and 3) the process does not vary with time (stationarity)..
In the brutal real world, as opposed to technological signal processing, systems are rarely truly linear or stationary and outputs may depend on the product of the inputs; signal processing in these systems is much more difficult (tiger country!). If you start with a basic proposition of, say, the oceans absorb heat and release it, they may act as a simple mixer and one can treat their behaviour through linear methods. In practice, mixing between levels, currents etc (I’m no expert) may make this invalid. However, for small fluctautions, over short periods they may be sufficiently linear for this not to matter – but this depends on the problem that you are trying to model and why you are doing so .
The importance of this idea is signal signal processing is not a “dark art” but a tool to gain insight into the behaviour of physical systems, especially when one has noise, or other signals mixed in. Therefore one has to start with an idea of what one is trying to establish rather than using a battery of techniques blindly (and incorrectly).
Jeff L says:
April 9, 2013 at 8:01 am
“Now we need to assume stationarity & linearity (and those assumptions ultimately would need to be tested) but in theory, using the convolutional model, we can calculate h(t) “
Actually, a valid model over the past 55 years is not stationary, but it has stationary increments. The relationship is essentially
dCO2/dt = k*(T – To)
dCO2/dt = rate of change of CO2
k = coupling factor in ppm/degC/unit-of-time
T = global temperature anomaly
To = equilibrium level of temperature
This highly representative model shows that temperature is necessarily driving CO2, and not the reverse. The arrow of causality is necessarily from temperature to CO2, as it would be absurd to argue the converse, that temperatures are being driven by the rate of change of CO2 – once the CO2 reached a plateau, no matter what its level, the temperature would return to the equilibrium value of To.
A cross correlation done between the rate of change of CO2 and temperature will yield a flat spectrum and essentially an impulse for the impulse response.
I know it is hard to believe, but the climate modelers have got even the most basic relationship backwards and wrong.
RCSaumarez says:
April 9, 2013 at 11:04 am
“Therefore one has to start with an idea of what one is trying to establish rather than using a battery of techniques blindly (and incorrectly).”
So true. The climate community should have started by employing identification techniques, such as PSD estimation, to identify how things were actually working, then fit their models to the observations. Instead, they have attempted to work up models, and then fit the data to them. Doyle’s famous detective warned against such methodology over a century ago:
“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” – Sherlock Holmes
Bart says:
April 9, 2013 at 11:10 am
On
dCO2/dt = k*(T – To)
This type of relationship is to be expected in a continuous transport system where CO2 is constantly entering and exiting the surface system, and the differential rate at which it does so is modulated by surface temperature.
It’s a done deal. I’m just waiting for others to recognize it.
Greg Goodman says:
April 9, 2013 at 9:15 am
…..
Hi Greg
It makes sense (the CET temperatures correlate with AO in winter months, the Antarctic’s summer). For correlation I use Excel, but it requires same ‘scale resolution’ for both variables. As a test I suggest use the average of monthly AO’s data ( for the ice melting months) tabulated as annual data. Hope someone else may know of a better method.
There is a serious caveat when it comes to applying this stuff. It works only for linear time-invariant systems. If we understand a system well enough, we can work it so our analysis is LTI (or close enough). In the case of the climate, that probably isn’t the case.
Bart says: “A cross correlation done between the rate of change of CO2 and temperature will yield a flat spectrum and essentially an impulse for the impulse response.”
It will or it could do ?
Call be skeptical, but I’d like to see at least a simple demonstration that is the case. I have no reason to think you are wrong but that is not enough for me to accept such a bold statement without question.
Bob: ” In the case of the climate, that probably isn’t the case.”
Aw, com’on Bob. Climate is CO2 plus stochastic variation. I know cos Tamino told me so.
commieBob says:
April 9, 2013 at 12:31 pm
“In the case of the climate, that probably isn’t the case.”
Actually, it probably is the case. The inapplicability of these methods is more the exception than the rule. And, as I demonstrate above, this system is clearly amenable to such an approach.
OK, you lost me after “Correlations”, but I love the way you fill out an x-y axis, so I trust you. What’s your opinion of the 5 second rule? And, is it safe to use a meat-tenderizing hammer on a ganglion cyst?