John Graham-Cumming has posted an interesting analysis, he could benefit from some reader input at his blog.
See here and below: http://www.jgc.org/blog/
Adjusting for coverage bias and smoothing the Met Office data
As I’ve worked through Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850 to reproduce the work done by the Met Office I’ve come up against something I don’t understand. I’ve written to the Met Office about it, but until I get a reply this blog post is to ask for opinions from any of my dear readers.
In section 6.1 Brohan et al. talk about the problem of coverage bias. If you read this blog post you’ll see that in the 1800s there weren’t many temperature stations operating and so only a small fraction of the Earth’s surface was being observed. There was a very big jump in the number of stations operating in the 1950s.
That means that when using data to estimate the global (or hemispheric) temperature anomaly you need to take into account some error based on how well a small number of stations act as a proxy for the actual temperature over the whole globe. I’m calling this the coverage bias.
To estimate that Brohan et al. use the NCEP/NCAR 40-Year Reanalysis Project data to get an estimate of the error for the groups of stations operating in any year. Using that data it’s possible on a year by year basis to calculate the mean error caused by limited coverage and its standard deviation (assuming a normal distribution).
I’ve now done the same analysis and I have two problems:
1. I get much wider error range for the 1800s than is seen in the paper.
2. I don’t understand why the mean error isn’t taken into account.
Note that in the rest of this entry I am using smoothed data as described by the Met Office here. I am applying the same 21 point filter to the data to smooth it. My data starts at 1860 because the first 10 years are being used to ‘prime’ the filter. I extend the data as described on that page.
First here’s the smooth trend line for the northern hemisphere temperature anomaly derived from the Met Office data as I have done in other blog posts and without taking into account the coverage bias.
And here’s the chart showing the number of stations reporting temperatures by year (again this is smoothed using the same process).
Just looking at that chart you can see that there were very few stations reporting temperature in the mid-1800s and so you’d expect a large error when trying to extrapolate to the entire northern hemisphere.
This chart shows the number of stations by year (as in the previous chart), it’s the green line, and then the mean error because of the coverage bias (red line). For example, in 1860 the coverage bias error is just under 0.4C (meaning that if you use the 1860 stations to get to the northern hemisphere anomaly you’ll be too hot by about 0.4C. You can see that as the number of stations increases and global coverage improves the error drops.
And more interesting still is the coverage bias error with error bars showing one standard deviation. As you might expect the error is much greater when there are fewer stations and settles down as the number increases. With lots of stations you get a mean error near 0 with very little variation: i.e. it’s a good sample.
Now, to put all this together I take the mean coverage bias error for each year and use it to adjust the values from the Met Office data. This causes a small downward change which emphasizes that warming appears to have started around 1900. The adjusted data is the green line.
Now if you plot just the adjusted data but put back in the error bars (and this time the error bars are 1.96 standard deviations since the published literature uses a 95% confidence) you get the following picture:
And now I’m worried because something’s wrong, or at least something’s different.
1. The published paper on HadCRUT3 doesn’t show error bars anything like this for the 1800s. In fact the picture (below) shows almost no difference in the error range (green area) when the coverage is very, very small.
2. The paper doesn’t talk about adjusting using the mean.
So I think there are two possibilities:
A. There’s an error in the paper and I’ve managed to find it. I consider this a remote possibility and I’d be astonished if I’m actually right and the peer reviewed paper is wrong.
B. There’s something wrong in my program in calculating the error range from the sub-sampling data.
If I am right and the paper is wrong there’s a scary conclusion… take a look at the error bars for 1860 and scan your eyes right to the present day. The current temperature is within the error range for 1860 making it difficult to say that we know that it’s hotter today than 150 years ago. The trend is clearly upwards but the limited coverage appears to say that we can’t be sure.
So, dear readers, is there someone else out there who can double check my work? Go do the sub-sampling yourself and see if you can reproduce the published data. Read the paper and tell me the error of my ways.
UPDATE It suddenly occurred to me that the adjustment that they are probably using isn’t the standard deviation but the standard error. I’ll need to rerun the numbers to see what the shape looks like, but it should reduce the error bounds a lot.
WUWT readers please go to http://www.jgc.org/ to discuss and see the latest updates.
I’ve found another interesting analysis. Apols if it has been noted before:
http://strata-sphere.com/blog/index.php/archives/11932
The best bit is the final graph:
http://cdiac.ornl.gov/epubs/ndp/ushcn/ts.ushcn_anom25_diffs_urb-raw_pg.gif
If you look at it, it looks very much like the increase in temps we have been told is going on. But it’s not. It is the changes made to the raw data sets to produce the ‘adjusted’ data sets.
Conspiracy? Where’s my working? Well, this is their graph, not an independent one! It comes from http://cdiac.ornl.gov/ itself.
So this tells us, in no uncertain terms, and completely unambiguously as far as I can see, that the increase of 0.6C is entirely fabricated. True I have not searched for a justification of that fabrication – it may be valid, but it should be reported as such!
“”” Toho (08:26:18) :
I don’t see how the sampling theorem is relevant. We are not interested in exactly recreating daily temperature variations, but estimation of variations over decadal time frames (besides, if you have a priori information that there is a 24-hour signal with harmonics, the sampling theorem doesn’t really apply, does it). That said, I certainly agree that there are large errors due in part to poor sampling methods, and the HadCRUT3 error estimates seem to be off by an order of magnitude or so. “””
Well then you don’t understand the sampling theorem. Standard Sampled Data theory shows that an out of band signal at a frequency B + b sampled at a rate 2B results in an in band error signal at a frequency B-b, which cannot be removed by any filter without also removing valid in band signals at the same frequency. Violation of the Nyquist criterion by a factor of only 2, as in sampling an out of band signal at a frequency of B + B or higher, results in an aliassed signal at B-B which is the average value of the function.
That means that by undersampling the data either temporally or spatially for the duration of the baseline period (30 years or whatever) that is used as the reference value for the “anomaly” calculation; that baseline average over whatever time period is corrupted by zero frequency aliassed noise components. It doesn’t matter that one may only be interested in trends; the trends are illusionary anyway. Anybody can see that plots of temperature anomalies over various time scales exhibit fractal like properties; longer and longer plotting intervals show “trends” of greater and greater extent over longer and longer time frames. I don’t understand your comment that a priori knowledge of a 24 hjour cycle with harmonics somehow dismisses the sampling theorem. Current methodology records a daily min/max temperature. That is a twice daily sampling rate, which only satisfies Nyquist in the event that the dailly cycle is a pure sinusoidal function. If it is not sinusoidal in waveform and is still periodic, there must be at least a second harmonic 12 hour periodic signal present, and 12 hour sampling of that violates Nyquist by a factor of two rendering the average contaminated by aliassing noise. Now as it turns out, min/max recording rather that simple 12 hour sampling eliminates the dgenerate case of exact twice signal frequency sampling, that can identify the signal presence, but not its amplitude, for example, when samples are taken exactly at the zero crossings, so recording zero. The min/max strategy does at least record roughly the correct amplitude, but not the correct time average, due to the presence of the 2F or higher signal components. So no, that foreknowledge does not erase the need for the sampling theorem.
Pachauri’s silly plot that Viscount Monckton called him out on in Copenhagen is a great demonstration of that fact. Statisticians are kidding themselves when they claim to be extracting additional information by applying regressions and filterings like running five year averages. We are talking about a chaotic function that never ever repeats; it only happens once; so what statistical significance is there in anything that only happens once ?
Yes it is true that the common mathematics of statistics can be applied to sets of totally unrelated numbers; and averaages, medians, standard deviations, or any other buzzword of standard statistics can be applied to number sets with no relational significance whatsoever.
That does not mean that the mere mechanics of doing that somehow will reveal “information” which was never in there in the first place.
It is often stated that White Noise contains more “information” than any other signal; it is totally unpredicatble, and no matter how long a string of white noise values you collect and process, you can never learn anything about the very next value to come along. In that sense the incoming signal is 100% information about itself.
The anomaly concept may seem advantageous, and it certainly does have some merits as to incorporating new stations into an existing network. But that is about the only merit.
Consider a solid sphere that is enclosed in a close fitting rubber (latex) skin so the skin is in contact with the sphere everywhere, but is just barely stretched, so it touches the sphere at every point.
Now take a hold of the skin at any point, and pull it away from the sphere by some small distance. The skin can then be stretched in any direction to move that point you have a hold of, over so some other point on the sphere by stretching the skin, and then the skin can be released at that point, so the skin is distorted, and points around the moved point are all moved to new locations.
It can be shown, that no matter how the skin is stretched and moved, there must always be at least two points somewhere on the skin that have not moved at all, and are still in their orighinal locations. No matter what contortions are applied there will always be at least two stationary points. Of course thoes point change for every different application of the stretch operation.
Now that simple problem in topology, is not unlike the mapping of tempearture aroound the globe and noting anomlies at each point. A zero anomaly report (at any point) is like one of the stationary points on the latex skin. The fact that some points did not move; and therfore record zero anomaly, does not grant a licence to assume that neighboring points also remained stationary, and would yield a zero anomaly, in the event that they too were actually sampled.
The complete global temperature continuous function ( at any time epoch) is exactly analagous to the latex skin on the sphere. Adjacent points can be stretched away from some stationary point, and nothing can be known about their locations (in the temperature anomaly realm) without actually taking a sample there.
In the case of temperatures the greater difference between adjacent points is like a greater stretching of the rubber skin. In weather terms, such diferences lead to the development of winds; but nothing can be learned about that unless all of those points are sampled. Anomalies yield no information that can be used to map wind patterns or virtually any other weather phenomenon, and after all, climate is supposed to be the long term average of weather.
On another issue, Bill had raised the concept of MP3 encoding, presumably as an argument counter to my suggestion to sub sample a digital data strem such as a music piece for example. Here Bill is confusing “Data compression” with “data aquisistion”
MP3 is an adaptive endoding of data THAT HAS ALREADY BEEN RECORDED.
The encoding algorithm, reacts to prior knowledge of data that has yet to arrive to be processed. similar concepts were already being applied in the 1960s in the recording of long playing 33 1/3 RPM phonograph (gramophone) records. During low level passages being cut on the master disc, the recording groove spacing was reduced to place the grooves closer together. When a louder passage was about to be recorded, the cutting lathe increased the groove spacing prior to the arrival of the loud signal, and that allowed the recording of a larger dynamic range than earlier constant spacing recording, and it also allowed different frequwncy compensation curves, that gave improeved signal to noise ratio; such as the standard RIAA recording and playback curves.
As Bill pointewd out, MP3 encoding allows data compression by factors of ten or more, which permits storing a whole lot of rather boring music on small players with play back quality, that passes for hi-fi to the unsuspecting purchasers of such music.
Just to be sure that some new technology hadn’t somehow snuck past this old fogey, I contacted a department manager at Creative; Soundblaster to some of us. They know about as much about MP3 and lookalikes as anybody.
He confirmed that there is no such thing as live real time MP3. The adaptive processing relies on prior knowledge of data yet to be processed, so some form of “buffering” is absolutely mandatory. In simple terms, the DATA must already be gathered, BEFORE you can process it, and compress it to store the pertinent information in less storage space.
So it is not a method of acquiring more information with less resources; just imagine running from place to place with a Stevenson screen to be in the right place at just the right time to record an important temperature anomnaly. Somehow basic information theory does not permit gathering more information with less resources. I’t somewhere in that whole signal to noise ratio, data rate, and channel bandwidth relationship. There’s that
Clude Shannon and his theorem in there, anothe bell Telephone Laboratories Product, Like Nyquist. How sad it is that we have lost that great National Treasure.
Signal recovery and signal processing, are somewhat different tasks than efficient data storage.
So sorry Bill, but no cigar.
George E. Smith (23:59:30) :
Really, that hits the nail right on the head. There is no way to zero out the noise and say with any sort of replicable or predictive calculations, “This is the contribution to warming because where x^2/y/(z*x) there is B warming.” And that is the part that I think that should be demonstrable before we drive our societies off of the economic cliff.
2 notes:
1) Made up the calculation above to illustrate, hopefully obviously.
2) If this kind of evidence exists, no one has ever shown it to me when I asked.
Toho.
The sampling theorem is definitely not about recreating a signal exactly.
As you correctly say, sampling loses information. There is generally no prospect of exact recovery of the original signal in practical systems. A well sampled system (more-than-observing the limit of the sampling theorem) might be able to recreate a very good approximation to the original signal by interpolation between samples.
Rather than exactness, it would better to talk about the adequacy of sampling. For a bandlimited signal, the sampling theorem tells us to absolutely avoid aliasing as that will destroy our ability to reconstruct a reasonably faithful representation of the original signal.
Is aliasing a big deal in temperature reconstructions?
Perhaps the point is that we don’t know. We can share opinions on this and compare examples. That’s all good fun and helps to spread knowledge and experiences. But where is the formal analysis in the literature?
I haven’t seen it. So right now, it looks like there could be a potentially serious gap in the whole approach to recreating historic temperature series on a global scale. And that means they could be junk.
I do tend to agree with you that localized atmospheric energy will not stay localized for long. But people who claim that the MWP was “localised” would appear to have a contrary view. To repeat my example above, an El Nino event is relatively localised, and (without comfort on spatial aliasing) I wouldn’t be too ready to accept the 1998 spike was anything more than an artefact of an inadequate network.
Check out Mann’s article in the Washington Post:
http://www.washingtonpost.com/wp-dyn/content/article/2009/12/17/AR2009121703682.html
E-mail furor doesn’t alter evidence for climate change
The really fun part is the comments. 29 pages ripping him apart!
Seeing the comments and rankings on the UK Daily Mail expose of the ‘Trick’ to ‘Hide the Decline’ is almost as much fun too.
The tide is turning, and turning fast…
Sorry to jump tracks …… the following has been posted on a warmist blog. Can somebody please let know how Anthony responded?
“Oh dear, Anthony Watts?. Probably new to you, but that UHI crock has long been debunked: http://www.ncdc.noaa.gov/oa/about/response-v2.pdf
“One analysis was for the full USHCN version 2 data set. The other
used only USHCN version 2 data from the 70 stations that surfacestations.org classified as good or best. We would expect some differences simply due to the different area covered: the 70 stations only covered 43% of the country with no stations in, for example, New Mexico, Kansas, Nebraska, Iowa, Illinois, Ohio, West Virginia, Kentucky, Tennessee or North Carolina. Yet the
two time series, shown below as both annual data and smooth data, are remarkably similar. Clearly there is no indication from this analysis that poor station exposure has imparted a bias in the U.S. temperature trends”
Reply: Covered here. I would also like to point out that you can tell the writers at that blog that the Talking Points memo is not a part of the peer-reviewed climate literature and therefore by their own standards something which should probably be given little weight. ~ charles the moderator
John
Difficult to contact you at your sitel.
please look at climatereason.com and then send me an email from there
thanks
tonyb
I think it would be useful at this juncture to review the difference between standard deviation (SD) and standard error of the mean (SEM).
For example, if you want to know how tall the human race is on average, you can measure the height of N people chosen at random, and then calculate the mean. If you want to know how confident you can be in your mean, that will depend on how variable the population is (standard deviation, or SD) and how many samples you have (N). In the extreme case of a uniform population, it would be sufficient to measure a single person. It turns out that SEM = SD/sqrt(N). It bears repeating that SD is a measure of how variable your measurement is *within* the population, and SEM is how confident can be in the estimate of the true mean across the entire population.
All of this depends on the randomness of the N samples. If Japanese or Norwegians were overrepresented, your estimate would contain an error that is not expressed by SEM. The same is true for temperature. There are separate techniques that attempt to correct for sampling bias, which in the case of global temperature, is expected to be the driving source of uncertainty, and difficult to adjust for.
As an earlier person posted, rather than the number of measurements, it is the geographic distribution of measurements that does more to determine the reliability of the estimate of the mean. Reasonable people may also question the validity or relevance of global mean temperature as a rather meaningless concept, akin to the average human, with one breast and one testicle. Indeed, the last ice age was characterized less as a drop in the global average temperature, and more as a large increase in the difference between the tropics and the northern latitudes. In the terms of this discussion, SD increased more than the mean decreased.
George E. Smith (23:59:30) :
“I don’t understand your comment that a priori knowledge of a 24 hjour cycle with harmonics somehow dismisses the sampling theorem.”
It doesn’t dismiss the sampling theorem (I probably did not communicate what I meant very clearly). The sampling theorem provides a sufficient condition to be able to exactly recreate the signal using samples. But it is only a sufficient condition. With the sampling theorem comes a method of reconstruction. (But it is not the only method of reconstruction conceivable, given an arbitrary set of samples and an arbitrary sampling methodology.)
If you happen to have a-priori knowledge that the signal is a pure sine wave of a certain frequency you only need a single sample to be able to perfectly recreate it to prepetuity (i.e. the only missing piece of information is the amplitude). That is a lot less than what the sampling theorem asks for. If you know that the signal is a 24-hour sine with a certain number of harmonics you are missing some more information, but you can still recreate the signal (to perpetuity) with a finite number of samples.
Besides, knowing the daily max and min temps give you more information than two equally spaced temperature samples.
I still stand by my post above.
In the time domain its been shown that (Tmax+Tmin)/2 is a good estimate. If you like go get CRN 5 minute data and see for your self. . In the spatial dimension the field is coherent over large distances. Think about it. The field can be sparsely sampled and long term trends can still be captured. That’s all you care about. Look at the records of the 4 longest temperature stations and compare them to the global average ( IPCC Ar4 ch06)
There are more important issues. Put your brain power there.
Toho
“If you happen to have a-priori knowledge that the signal is a pure sine wave of a certain frequency you only need a single sample …”
It’s a highly idealised example, and not sure it get us very far with the question of whether there is aliasing in the temperature data.
Looking at a more recent post on WUWT, I see even more evidence to warn us that there is likely to be aliasing in the temperature data.
Before that, pause to imagine a sequence of samples of a signal (in time or in space) as a sequence of point values. If we join up the points, we can turn this into a sequence of trapezia. We’re interested to know whether the resulting stepwise-linear curve is a decent reflection of the original signal. (More sophisticated interpolation using curves doesn’t really add to this).
If a smoothly changing sequence of trapezia is a “good” representation of the original signal, it must follow that the closest neighbouring sample data points are highly autocorrelated.
We can say this because erratic changes in neighbouring sample points would be a good indicator of overlapping bands in the “frequency” domain, and therefore aliasing. (Frequency in quotes becuse this point also holds for spatial data sampling.)
And as discussed, the consequence of aliasing is that we cannot rely on the sampled data to reconstruct the original signal.
Now consider the latest WUWT post about Darwin. It is claimed that a “neighbouring” station (for a hologenising algorithm) was some 1500 miles away. There are closer stations, but they are not sufficiently correlated for a homogenising algorithm. In fact some of the closer stations are negatively correlated to Darwin.
What better hint do we need that spatial aliasing is a potential problem.
Toho (13:11:31) :
You are referencing only one point in the system and Mr. Smith is referencing the entire system. This maybe why you are not seeing things eye to eye?
If you are looking to make a mean temperature for the entire planet, you need to be thinking in terms of the entire system, and not just the particular sampling point in the system. It seems pretty silly (to me anyway, no disrespect) to argue against the idea of taking samples at evenly spaced intervals for reasons other than actual physical difficulty.
Of course, I feel that this is all a moot point and the range of the entire climate system is the important bit of knowledge. If the radiative warming theory is true the whole system should read warmer and the best way to show that, or its absence, is to show the bounds increasing. Of course, that would not prove CO2 involvement, but it would eliminate a lot of data issues.
Hey guys, come on now. Don’t put words in my mouth. I was talking about the sampling theorem specifically, and I was trying to make a point with my simple examples. I don’t really disagree with the sentiment here that the error bounds on instrumental temperature records are probably significantly understated (I would guess by an order of magnitude, i.e. a big deal). But to make that case we need a better argument than the sampling theorem. It is a mathematical theorem which does not say what some people here try to argue it does.
Also, I don’t believe that there are large errors (in multidecadal trend line regressions) caused by aliasing (but I would be happy to be proven wrong about that). However, I do agree there are serious problems with the homogenisation algorithms that seem to be employed.
Jordan:
“If we join up the points, we can turn this into a sequence of trapezia. We’re interested to know whether the resulting stepwise-linear curve is a decent reflection of the original signal.”
No, I think you go wrong here. We are not interested in a decent reconstruction of the daily temperature variations. The thermometer readings are just viewed as a statistic.
“”” Toho (13:11:31) :
George E. Smith (23:59:30) :
“I don’t understand your comment that a priori knowledge of a 24 hjour cycle with harmonics somehow dismisses the sampling theorem.”
If you happen to have a-priori knowledge that the signal is a pure sine wave of a certain frequency you only need a single sample to be able to perfectly recreate it to prepetuity “””
This is worse than pulling teeth. If I have a sinusoidally varying signal with an exactly known frequency, and I take one sample per cycle; I get zero information about whether the amplitude of the cycle is 20 degrees C or 20 millidegrees C. I get exactly the same measurment every sample; and any process for obtaining an average, will naturally give exactly the value of that sample. Now remember I did say that the case of sampling at exactly 2.B (your are talking only 1.B) is a degenerate case with an indeterminate result. That is one of the reasons for random sampling; but that only works for a truly sinusoidal signal, which by definition has exactly the same amplitude each and every cycle. (if the amplitude changes from cycle to cycle it isn’t a sinusoidal function).
Some are saying that the min/max average is good enough. How good is good enough when we are talking hundredths of a degree.
I have looked at a whole bunch of daily “weather” maps; do it every day for the SF bay area, and daily min-max ddifferneces of 30-40 deg F are very common, with tens of degrees differences over distances of a handful of km separation.
When it comes to shorter than daily cyclic temperature changes, (we actually have clouds in California) there is no prior frequency knowledge.
One can argue that “it all comes out in the wash” and the averages are good enough. Well the climate does not depend totally on average temperature.
At the global mean temperatuyre there is no “Weather”, so there can’t be any climate either since that is defined as the average of weather.
Unfortunately, the operating Physics doesn’t pay any attention to averages. The surface emitted thermal radiation that is a major earth cooling process, follows a more 4th power of temperature law, and the spectral peak of the thermal radiation which is what is of interest to GHG capture, varies as the 5th power of the temperature.
So if you take the integral of the 4th or 5th power of any cyclic temperature curve over a complete cycle, the result is always higher than simply integrating the average; so cycles do matter, and in the case of climate temperatures there are notable annual and daily cycles that result in an always positive enhancement of the total radiant emmission from the earth, and hence estimates of whether we are warming or cooling.
It is ironic that all you statisticians think that a tenth of a degree change in the average of a variable that has a 150 degree C possible maximum range, of values on any given day is somehow significant; but you can then dismiss similar errors in your homogenised data, that result from plain and simple experimental errors; because well you aren’t really interested in the data, just the imagined “trends”.
Just don’t call it science if that is what you truly believe.
Take a look in Al Gore’s book “An Inconvenient Truth”; pages 66-67 specifically, where we have Al’s impression of atmospheric CO2 and Temperature from some ice cores, over a long period of time.
So tell us what “TREND” is depicted in those graphs of CO2 and Temperature ? I’m only interested in Climate not weather so just give me one number for the whole 600,000 years. That should be long enough for you to get some good statistical average of the trend, and I would expect a pretty small standard deviation after all that time.
Just try telling your cell phone service provider, that the sampling theorem doesn’t matter, and he should be able to give you pretty good; though “Average” service without paying any attention to the Nyquist theorem.
The CRU supporters complain that we are taking the whistle blown e-mails out of context; in other words; we aren’t getting the full story form just a few e-mail snippets. Funny how that works with e-mails but we can get perfectly good climate data from out of context “sampling”.
“”” Toho (14:15:51) :
Jordan:
I agree with most of what you write. But my point is that it doesn’t follow from the sampling theorem. The sampling theorem is about recreating exactly. Sure, you are going to lose information when sampling, and that will cause errors in the temperature estimates. I certainly agree with that. But it has nothing to do the sampling theorem. I don’t think aliasing is a big deal by the way, because localized atmospheric energy will not stay localized for long. “””
Here’s a Quote From Professor Julius T. Tou’s book on sampled systems.
” Sampling Theorems. Fundamenta Theorem of Sampling. If a signal f(t) has a frequency spectrum extending from zero to B cps, it is completely determined by the values of the signal (i.e. the samples) taken at a series of instants separated by T = 1/2B sec, where T is the sampling period.
This theorem implies that if a signal is sampled instantaneously, at a constant rate equal to twice the highest signal frequency, the samples contain all of the information in the original signal. ”
Now I changed some of his symbols which don’t repicate well here; but otherwise it is verbatim.
And I would point out that the theorem says the signal is COMPLETELY determined by the samples, which contain ALL the information.
That statement, is immediately followed by a mathematical proof of the theorem, based on Fourier Integrals, and the Fourier Transform.
That proof is then followed by a proof that the original signal can be COMPLETELY reconstructed from the samples, and how to do that.
So please don’t try to tell me that a properly sampled signal can’t be correctly reconstructed; the reconstruction is only limited by practical technological questions; not by theoretical mathematical limitations.
Besides that my point is not that the signal needs to be reconstructed. A consequence of the sampling theorem is that THE AVERAGE cannot be recovered, given only a modest violation of the Nyquist Criterion (factor of two), and min/max daily sampling already is at or beyond that limit of error.
And it shouldn’t be necessary for this audience to point out that “frequency ” can be applied to any cyclic variable, not just electrical signals. In the case of global temperature, it applies to two different variable, namely time and space; both of which are subject to the limitations of sampling.
But I’m not here to try and free anybody from their delusion that statistical machinations can produce information out of nothing. You local telephone directory would be a good place to start a new science of statistical manipulations. Or if you are also a greeniie, or a WWF enthusiast; why not start a global project to determine the average number of animals per hectare all over the earth; animal meaning anything in the biological animal kingdom. Don’t pay any attention to what sort of animals; yes they range from smaller than ants to whale size; but that shouldn’t matter any more than the kind of terrain matters to the significance of the temperature (or anomaly) recorded there, matters to global climate. If you’ve seen one thermometer reading, you’ve seen them all.
Toho:
I agree global reconstructions are not expressed at daily reolution, but most are monthly. Do we have the well behaved “thermal surface”? (Note, spatial resolution is the question, not time.)
My doubts remain. We know of significant regional fluctuations which prevail for many months and even years (like ENSO, AMO, PDO). What picture do we get from sparse and irregular spatial samples?
I agree with your comment that thermometer readings are just viewed as a statistic, but aliasing can introdice a nasty systematic distortion. Not necessarily a zero-mean random variable which will “melt away” in the calculation of a mean. If the network gives us a distorted picture of (say) ENSO in 1998, the same network might give a similarly distorted picture of other ENSO events.
Another example to illustrate concerns about loss of information in sampling, and why statistical analysis does not get us “out of jail”:
You’re travelling at constant speed on a bicycle, and compelled to keep your eyes closed. You are permitted to blink your eyes (open) at regular intervals to capture some information about the road ahead. This is all you have to make decisions about direction. You’re fine at (say) 10 blinks per second, but then your are required to reduce your blink rate. There will come a point when the blink rate is so low that you are unable to gather enough information about the road ahead to avoid a crash. Loss of information is too great beyond that point. The lost information is permanently lost, there is no statistical analysis or modelling from the samples to get it back. So when you are at risk of crashing, the only practical option is to increase the blink rate to something that meets your requirements.
To George:
My encounters with the sampling theorem are all founded in discrete control systems design. It is easy to show matehmatically how a stable closed loop system can be driven to instability by extending the sampling interval.
I think your familiarity of the sampling theorem will be greater than mine. However I understand the mathematical analysis usually starts with an absolutely band-limited spectrum (dropping to zero amplitude beyond an assumed maximum frequency). The sample sequence “repeat spectra” can then be completely isolated and removed in theory. Do I take it that’s the reason why you suggest the original signal can be perfectly recovered from the samples?
I asseted that perfect reconstruction is not possible – without explaining that practical systems cannot totally bandlimit a signal in the way of the theory. We can realistically reduce frequency-folding to an immaterial level (for a purpose, such as control), but that also means the samples can only ever produce an imperfect reconstruction of the continuous signal.
(OK – perhaps there is an exception of well cyclic signals, but that doesn’t add much to discussion of sampling the climate system.)
To Steven
You could well be right. Or maybe wrong – I don’t know and feel we’re not really in a position to say either way. As you;ll see from above, I feel inclined to hang onto my scepticism. The issue of aliasing needs to be formally investigated and submitted for review in the usual way. Until then, could we put an asterisk next to the global reconstructions, just to remind us of this loose end?
(Good discussion about an interesting topic.)
“”” Jordan (15:54:27) : “””
Well Jordan bear in mind that what I just related above, extracted from a sampled data control system Text book, is a purely mathematical construction. But it does assert that in the mathematical sense the band limited continuous functioon is in fact perfectly represented by the set of properly spaced discrete samples, and in the mathematical sense it can be completely reconstructed. Now it is true that often we can’t be sure the signal really is band limited; global temperature data for example, we really can’t know the resolution limits of actual spatial temperature variations, or even temporal ones.
In the signal processing realm, this problem is usually dealt with by running the raw signal through an anti-aliassing filter to be sure that it is truly band limited before sampling. Generally that means reducing out of band signal information to below the LSB of the A-D converter.
Aliassing noise is a real problem for example with optical mice. Your basic Optical mouse is a digital camera that may take 2000 frames per second images of whatever surface the mouse is sitting on. It cares not what that surface variation is, just that there is some. Crosscorrelation of incoming images, with the previous stored image, is converted into cursor move information, after deducing the movement between the two successive images.
Now the high frame rate (sample rate) is only possible because the image has only a small number of pixels, and quite large ones. So the camera may only have between 15 x 15 to perhaps 30 x 30 pixels, somehwere in the 30 -60 micorn pixel size range. This is not like your favorite Nikon Digital SLR.
Now I can easily design single element camera lenses (1:! relay) that can resolve way below 30 microns, since we are not constrained to spherical surfaces, since the lenses are molded. As a result the camera lens in proper focus can provide an image on the silicon sensor, that the sensor can not properly sample, without aliassing noise. this problem will be set off my ssurfaces containing repetitive patterns, so the dot patetrn color printing images, and even things like Tatami mats can result in eratic cursor movement, if you try mousing on them.
We have eliminated that problem in at least LED mouses, since I can build a completely Optical anti-aliassing filter, and put it right on the actual camera lens itself. In effect I can design a deliberately fuzzy lens that won’t resolve below the pixel size; with an accurately manufacturable cutoff frequency, of the Modulation Transfer Function. In fact I have several Patents on the method.
It is much harder to implement that in laser mice because of the effects of the beam coherence.
So yes I am up to my elbows every day in Sampling theorem realities.
And yes, in practice it is practical limitations that prevent exact reconstruction; but not mathematical theoretical reasons.
But the whole modern communication technology is intimately tied to sampled data. The sampling theorem says I can completely represent say a 4 KHz band limited voice signal by discrete samples taken at 125 microsecond intervals. Well those samples can themselves take less time than say one microsecond. So I can have 124 microseconds of silence between the adjacent samples of a typical voice message. Or alternatively, I can fill that empty space with another 124 sets of one microsecond samples for a total of 125 total voice messages all happening at the same time. Adn at the receiving end, I can sort those samples into 125 channels, and then reconstruct each of them at its end destination.
Well of course you need some time for synchronisation, and management overhead. Now you aren’t getting something for nothing; because in order to transmit that pulse train of one microsecond pulses, with sharp transition steps between channel samples, you need a transmission channel that can handle one microsecond pulses, instead of a slow 4 khz audio signal; well all of that is dicatated by the Shannon theorem.
So accurate (sufficiently) reconstruction of properly sampled data, is a well developed technology, and the telephone companies spent mucho bucks and time, making sure that the theory behind such transmission methodologies is sound.
Now it is a lot more complex than I have indicated here, because they also go out of their way to compress the data to its minimum intelligible size as Bill alluded to in his posts; and many years of study has gone into the development of digital data encoding and transmission techniques, to improve capacity, and signal to noise ratios while containing total channel bandwidth.
But all of that magic is post processing of already snared information, and the recipient, knows exactly what surgery was carried out on his stuff and how to unbury it, and recover his data.
That luxury is not present in the climate data gathering field. Satellite systems that can scan offer a big improvement, but have their own difficulties, in terms of understanding just what the blazes your remote sensors are really responding to.
I forgot to add above that it is not any need to reconstruct a global temperature map that concenrs me. It is that aliassing noise can swamp even the average, with just modest undersampling, and the base time data sets used to compute temperature anomalies are just such long time averaged data that completely ignores the fact that that base average temperature is itself corrupted by noise; no matter how long the base interval is.
Jordan:
“I agree global reconstructions are not expressed at daily reolution, but most are monthly. Do we have the well behaved “thermal surface”? (Note, spatial resolution is the question, not time.) ”
No, I don’t think we have. I think the thermal surface is a lot more noisy than is what is percieved in AGW circles.
“I agree with your comment that thermometer readings are just viewed as a statistic, but aliasing can introdice a nasty systematic distortion. Not necessarily a zero-mean random variable which will “melt away” in the calculation of a mean. If the network gives us a distorted picture of (say) ENSO in 1998, the same network might give a similarly distorted picture of other ENSO events.”
Maybe, but my gut feeling as a physicist is that aliasing specifically is a non-issue compared to a lot of other potential error sources. The reason for my feeling is that energy will tend to move pretty quickly in the atmosphere. However, this is something that probably could (and should) be statistically tested. I have a feeling that most of the testing that has been done and published in this area relies on homogenized data which will tend to hide systematic errors by making station records more appear more correlated than they are in reality.
George:
“And I would point out that the theorem says the signal is COMPLETELY determined by the samples, which contain ALL the information.”
True, but the theorem does not say the inverse, that you can’t have complete information of the signal without all the samples, or with a smaller number of sample points. All your arguments seem to be based on this inverse of the theorem, which isn’t generally true. In particular it isn’t true when you have additional information about the system dynamics. The theorem provides a sufficient condition for reconstruction, not a necessary condition.
“That proof is then followed by a proof that the original signal can be COMPLETELY reconstructed from the samples, and how to do that.
So please don’t try to tell me that a properly sampled signal can’t be correctly reconstructed; the reconstruction is only limited by practical technological questions; not by theoretical mathematical limitations.”
That’s not what I said. My comment was made in response to something that Jordan wrote above, and I think I was pretty clear that it wasn’t regarding the sampling theorem, but regarding temp records. To recreate exactly as per the sampling theorem you need an infinite number of samples, even if the bandwidth is limited. In real life you don’t have that, so in real life sampling an arbitrary signal will lose information.
“This is worse than pulling teeth. If I have a sinusoidally varying signal with an exactly known frequency, and I take one sample per cycle; I get zero information about whether the amplitude of the cycle is 20 degrees C or 20 millidegrees C. I get exactly the same measurment every sample;”
If you have a sine signal you by definition know the phase. I explicitly said the only missing piece of information was the amplitude. If that’s the case you only need one sample (not one per period) to reconstruct the entire signal. If you are missing phase and frequency as well you need three carefully chosen samples instead (not per period, three total). This is a simple counter-example to the inverse of the sampling theorem if you will. It show that you can get by with a lot less than two samples per period if you have knowledge about the system dynamics and carefully select where you sample your signal.
The example was in response to one of your posts above where you didn’t understand (my probably poorly worded) statement about a-priori information. But the main point is, the inverse of the sampling theorem is not generally true.
Toho (03:32:41) : ” … homogenized data .. will tend to hide systematic errors by making station records more appear more correlated than they are in reality.”
Yes, that’s a good point.
“”” Toho (03:32:41) :
“” If you have a sine signal you by definition know the phase. “””
OK Toho; I give up; you win.
So yes I know it’s a pure sinusoidal signal with absolutely no harmonic content.
I can also tell you that the period is exactly 86,400 seconds.
So I just read my thermometer and got my one sample. The thermometer reads 59 deg F/15 deg C. It is 10:30 AM PST.
Please reconstruct the complete signal and give me the following information;
1/ Minimum temperature
2/ Maximum temperature
3/ Time of either minimum temperature or maximum temperature.
Or if for some reason you are unable to provide any or all of those numbers; please give me instead;
4/ The Average temperature for the Cycle.
So that should be fairly straight forward Toho;
So it’s your move now.
This graph contains raw data from NASA GIStemp – global and US.
http://i629.photobucket.com/albums/uu20/blouis79/USglobal_anomaly_vs_jetfuel-1.png
For some reason, the unsmoothed US data swings wildly compared to unsmoothed global data – the original global data source included month-by-month data too, so I think it can be reasonably trusted.
A statistical analysis of the US data would easily indicate a very wide predictive uncertainty.
Is anyone else interested in the airport heat island effect and the correlation with jetfuel consumption on takeoff?
George
There was no doubt some misunderstandings and dead ends in the above discussion, and the sine wave might be one of them. But I wouldn’t be too dismissive of Toho’s position.
Toho also makes this very good point (for a non-periodic signal):
” the theorem does not say the inverse, that you can’t have complete information of the signal without all the samples, or with a smaller number of sample points. All your arguments seem to be based on this inverse of the theorem, which isn’t generally true.”
Seems to make sense to me.
Toho- you have said the following on a number of ocasions: “The theorem provides a sufficient condition for reconstruction, not a necessary condition.” I don’t follow – could you please expand.
One of the things I take from the above discussion is the apparently conflicting requirements of statistical analysis (sampling error) versus the sampling theorem (errors due to aliasing) when it comes to sample autocorrelation.
If the objective is to measure statistical aggregates, autocorrelation in the samples tends to be a problem – making life generally more difficult and perhaps even obstructing an analysis. If the objective is to recreate a faithful representation of the continuous signal from a finite sample, autocorrelation between the samples would appeat to be an absolute necessity.
Jordan:
“If the objective is to measure statistical aggregates, autocorrelation in the samples tends to be a problem – making life generally more difficult and perhaps even obstructing an analysis. If the objective is to recreate a faithful representation of the continuous signal from a finite sample, autocorrelation between the samples would appear to be an absolute necessity.”
Yes, I think that is a pretty good simple summary.
“Toho- you have said the following on a number of ocasions: “The theorem provides a sufficient condition for reconstruction, not a necessary condition.” I don’t follow – could you please expand.”
That is just an attempt to express my point about the sampling theorem in a different way. I suppose the above is a formulation that would appeal to a mathematician.
The theorem essentially says that IF you have a bandwidth limited signal and sample it often enough and in a specific way, THEN you can recreate the signal (that is the sufficient part). However, it does not say that IF you DON’T sample it that often or IF you DON’T sample it in that specific way, THEN you CAN’T recreate the signal (that would be the necessary part).
I.e. there may be other ways to sample that would require a smaller average sample rate for reconstruction.
Related is the second point I am trying to make, that if you know something about the dynamics of the system, you can use that knowledge along with a much sparser set of samples in order to get much more information out than you would get from the samples alone. I also gave a very simplistic example of such a case. Another more realistic example is meteorologists making short term weather predictions from (samples of) initial value conditions.
George:
I have never claimed that real temps are a pure sine signal. However, there are published statistical relationships between the max/min temp and the cycle average. It seems pretty clear to me that if you are looking at the anomalies at a decadal scale, then such statistics should be pretty good (for local temperatures). If you have a good argument as to why they are incorrect I am more than willing to listen, but it simply does not follow from the sampling theorem. Errors from UHI effects, changes in instrumentation, location changes etc should be larger by orders of magnitude.
And George, you do have a number of good points above that I tend to agree with.