KEVIN KILTY
Introduction
A guest blogger recently1 made an analysis of the twice per day sampling of maximum and minimum temperature and its relationship to Nyquist rate, in an attempt to refute some common thinking. This blogger concluded the following:
(1) Fussing about regular samples of a few per day is theoretical only. Max/Min temperature recording is not sampling of the sort envisaged by Nyquist because it is not periodic, and has a different sort of validity because we do not know at what time the samples were taken.
(2) Errors in under-sampling a temperature signal are an interaction of sub-daily periods with the diurnal temperature cycle.
(3) Max/Min sampling is something else.
The purpose of the present contribution is to show that these first two conclusions are misleading without further qualification; and the third conclusion could use fleshing out to explain Max/Min values being “something else”.
1. Admonitions about sampling abound
In the world of analog to digital conversion admonitions to bandlimit signals before conversion are easy to find. For example, consider this verbatim quotation from the manual for a common microprocessor regarding use of its analog to digital (A/D or ADC) peripheral. The italics are mine.
“…Signal components higher than the Nyquist frequency
(fADC/2) should not be present to avoid distortion from unpredictable signal convolution. The user is advised to remove high frequency components with a low-pass filter before applying the signals as inputs to the ADC.”
![clip_image001[4]](https://i0.wp.com/wattsupwiththat.com/wp-content/uploads/2019/02/clip_image0014.gif?resize=82%2C2&ssl=1 "clip_image001[4]"){kind=small}
Date: February 14, 2019.
1 Nyquist, sampling, anomalies, and all that, Nick Stokes, January 25, 2019
2. Distortion from signal convolution
What does distortion from unpredictable signal convolution mean? Signal convolution is a mathematical operation. It describes how a linear system, like the sample and hold (S/H) capacitor of an A/D, attains a value from its input signal. For a specific instance, consider how a digital value would be obtained from an analog temperature sensor. The S/H circuit of an A/D accumulates charge from the temperature sensor input over a measurement interval, 0 → t, between successive A/D conversions.
(1) 
Equation 1 is a convolution integral. Distortion occurs when the signal (s(t)) contains rapid, short-lived changes in value which are incompatible with the rate of sampling with the S/H circuit. This sampling rate is part of the response function, h(t). For example the S/H circuit of a typical A/D has small capacitance and small input impedance, and thus has very rapid response to signals, or wide bandwidth if you prefer. It looks like an impulse function. The sampling rate, on the other hand, is typically far slower, perhaps every few seconds or minutes, depending on the ultimate use of the data. In this case h(t) is a series of impulse functions separated by the sampling rate. If s(t) is a slowly varying signal, the convolution produces a nearly periodic output. In the frequency domain, the Fourier transform of h(t), the transfer function (H(ω)), also is periodic, but its periods are closely spaced, and if the sample rate is too slow, below the Nyquist rate, spectra of the signal (S(ω)) overlap and add to one another. This is aliasing, which the guest blogger covered in detail.
From what I have just described, several things should be apparent. First, the problem of aliasing cannot be undone after the fact. It is not possible to figure the numbers making up a sum from the sum itself. Second, aliasing potentially applies to signals other than the daily temperature cycle. The problem is one of interaction between the bandwidth of the A/D process and the rate of sampling. It occurs even if the A/D process consists of a person reading analog records, and recording by pencil. Brief transient signals, even if not cyclic, will enter the digital record so long as they are within the passband of the measurement apparatus. This is why good engineering seeks to match the bandwidth of a measuring system to the bandwidth of the signal. A sufficiently narrow bandwidth improves the signal to noise ratio (S/N), and prevents spurious, unpredictable distortion.
One other thing not made obvious in either my discussion, or that of the guest blogger, concerns the diurnal signal. While a diurnal signal is slow enough to be captured without aliasing by a twice per day measurement cycle, it would never be adequately defined by such a sample. One would be relatively ignorant of the phase and true amplitude of the diurnal cycle with twice per day sampling. For this reason most people sample at least as fast as 2 and one-half times the Nyquist rate to obtain usefully accurate phase and amplitude measurements of signals near the Nyquist rate.
3. An example drawn from real data
![clip_image005[4]](https://i0.wp.com/wattsupwiththat.com/wp-content/uploads/2019/02/clip_image0054.jpg?resize=549%2C235&quality=83&ssl=1 "clip_image005[4]")
Figure 1. A portion of AWOS record.
As an example of distortion from unpredictable signal convolution refer to Figure 1. This figure shows a portion of temperature history drawn from an AWOS station. Note that the hourly temperature records from 23:53 to 4:53 show temperatures sampled on schedule which vary from −29◦F to −36◦F, but the 6 hour records show a minimum temperature of −40◦F.
Obviously the A/D system responded to and recorded a brief duration of very cold air which has been missed in the periodic record completely, but which will enter the Max/Min records as Min of the day. One might well wonder what other noisy events have distorted the temperature record. Obviously the Max/Min temperature records here are distorted in a manner just like aliasing– a brief, high frequency, event has made its way into the slow, twice per day Max/Min record. The distortion is about 2◦F difference between Max/Min and the mean of 24 hourly temperatures–a difference completely unanticipated by the relatively high sampling rate of once per hour, if one accepts the blogger’s analysis uncritically. Just as obviously, if such event had occurred coincident with one of the hourly measurement schedules, it would have become a part of the 24 samples per day spectrum, but at a frequency not reflective of its true duration. So, there are two issues here. The first one being the distortion from under-sampling, and the second being that transient signals possibly aren’t represented at all in some samples but are quite prevalent in others.
In summary, while the Max/Min records are not the sort of uniform sampling rate that the Nyquist theorem envisions, they aren’t far from being such. They are like periodic measurements with a bad clock jitter. It is difficult to argue that a distortion from unpredictable convolution does not have an impact on the spectrum resembling aliasing. Certainly sampling at a rate commensurate with the brevity of events like that in Figure 1 would produce a more accurate daily “mean” than does midpoint of the daily range; or, alternatively one could use a filter to condition the signal ahead of the A/D circuit, just as the manual for the microprocessor suggests, and just as anti-aliasing via the Nyquist criterion, or improvement of S/N would demand. Trying to completely fix the impact of aliasing from digital records is impossible after the fact. The impact is not necessarily negligible, nor is it mainly an interaction with the diurnal cycle. This is not just a theoretical problem; especially considering that Max/Min temperatures are expected to detect even brief temperature excursions, there isn’t any way to mitigate the problem in the Max/Min records themselves. This provides a segue into a discussion about the “something otherness” of Max/Min records.
4. Nature of the Midrange
The midpoint of the daily range of temperature is a statistic. It is among a group known as order statistics, as it comes from data ordered from low to high value. It serves as a measure of central tendency of temperature measurements, a sort of average; but is different from the more common mean, median, and mode statistics. To speak of the midpoint range as a daily mean temperature is simply wrong.
If we think of air temperature as a random variable following some sort of probability distribution, possessing a mean along with a variance, then the midpoint of range may serve as an estimator of mean so long as the distribution is symmetric (kurtosis, excess, and higher moments are zero). It might also be an efficient or robust estimator if the distribution is confined between two hard limits, a form known as platykurtic for having little probability in the distribution tails. In such case we could also estimate a monthly mean temperature using a midrange value from the minimum and maximum temperatures of the month or even an annual mean using the highest and lowest temperatures for a year.
In the case of the AWOS of Figure 1 the annual midpoint is some 20◦F below the mean of daily midpoints, and even a monthly midpoint is typically 5◦F below the mean of daily values. The midpoint is obviously not an efficient estimator at this station, although it could work well perhaps at tropical stations where the distribution of temperature is more nearly platykurtic.
The site from which the AWOS data in Figure 1 was taken is continental; and while this particular January had a minimum temperature of −40◦F, it is not unusual to observe days where the maximum January temperature rises into the mid 60s. The weather in January often consists of a sequence of warm days in advance of a front, with a sequence of cold days following. Thus the temperature distribution at this site is possibly multimodal with very broad tails and without symmetry. In this situation the midrange is not an efficient estimator. It is not robust either, because it depends greatly on extreme events. It is also not an unbiased estimator as the temperature probability distribution is probably not symmetric. It is, however, what we are stuck with when seeking long-term surface temperature records.
One final point seems worth making. Averaging many midpoint values together probably will produce a mean midpoint that behaves like a normally distributed quantity, since all elements to satisfy the central limit theorem seem present. However, people too often assume that averaging fixes all sorts of ills–that averaging will automatically reduce variance in a statistic by the factor 1/√n. This is strictly so only when samples are unbiased, independent and identically distributed. The subject of data independence is beyond the scope of this paper, but here I have made a case that the probability distribution of the maximum and minimum values are not necessarily the same as one another, and may vary from place to place and time to time. I think precision estimates for “mean surface temperature” derived from midpoint of range (Max/Min) are too optimistic.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Hey Kevin,
Thanks for your post – another interesting piece around consequences of aliasing in the temperature record.
A guest blogger recently made an analysis of the twice per day sampling of maximum and minimum temperature and its relationship to Nyquist rate, in an attempt to refute some common thinking.
Nick was responding to the earlier article by Mr Ward who argued that the way historical temperatures were captured, namely daily min/max introduces errors (differences between daily midrange value and true arithmetic mean). Those errors due to aliasing are not accounted in subsequent analysis that are based on averages derived from daily midrange values. Nick recognized the problem and had a good attempt trying to show that we can still preserve reasonable good daily mean by reducing number of samples per day and applying later adjustments knowing regularity of daily oscillations. Unfortunately, his adjustments do not apply to most of the historical record where all we have is daily min/max. Nick’ method requires (1) periodic two samples per day (not min/max) and (2) highly sampled reference signal where we can validate adjustments. For most of the historical record we don’t have (1) and for many places we don’t have (2) either. So it is not a magic wand generating valid information out of nothing – it’s basically way of saying that if we have good data we can correct bad data.
In summary, while the Max/Min records are not the sort of uniform sampling rate that the Nyquist theorem envisions, they aren’t far from being such. They are like periodic measurements with a bad clock jitter.
Under previous posts around this subject there were ferocious, almost religious arguments whether Nyquist, or indeed even very concept of ‘sampling’ applies to daily recordings of min/max.
The articles on Nyquist problem by William Ward (?) were misguided. The current article by Kevin Kilty does not help that much to clarify the confusion sowed by Ward. His example of S/H does not apply to temperature measurement. The original problem was about whether Min and Max daily measurement can be used to estimate the long term averages or the gradient of averages. Yes, they can. The issue is what is the error and how to estimate it. This can be done numerically on real data. One you do it you realize there is no point of wasting time on pseudo-theoretical analysis by Ward, Stocks and Kilt.
unka
What reason(s) do I have to reject Ward and Kilty and accept your assertions? You make a claim without providing an explanation or even examples. Do you have any publications that you can direct us to to support your assertions?
Much of the current thread seems concerned with matters that are far removed from issues truly pertinent to actual in situ measurements of temperature by Max/Min thermometers. There simply is no A/D conversion of the type specified by the convolution of Eq. 1 and no attendant signal distortion. The extreme values are simply picked off a continuous signal of a mechanical instrument whose time-constant is typically on the order of tens of seconds. The highly irregular times of occurrence of extrema are not recorded nor are they particularly relevant to the (limited) utility of the data. They differ categorically from the digital samples obtained by multiplying the original signal by a Dirac comb (q.v.), which is the prerequisite for any possibility of frequency aliasing. But so does the very purpose of the data, which is not to record a signal consisting of a huge, asymmetric diurnal cycle (with harmonic line structure) and a much weaker random component, but to suppress the former in order to reveal the climatic variations of the latter.
There’s little presented here that truly clarifies or advances that practical purpose. That the mid-range value is persistently different from the true signal mean is well-understood. What is needed is a scientifically based reconciliation of historical data with far-more-revealing modern records. Instead we’re offered a smorgasbord of practical irrelevancies and analytic misconceptions based on superficial parsing of technical terminology.
If this was a drawing we are discussing pen pressure rather that the picture. For sure the pen pressure produces different grades of blacks at certain points but the overall painting remains the same.
What makes a picture, pressure or pattern? Pattern is the trend, pressure is a daily measurement.
If this was a drawing we are discussing pen pressure rather that the picture. For sure the pen pressure produces different grades of blacks at certain points but the overall painting remains the same.
If all you need is just a sketch which somehow resembles an original is fine. But if you need precise scientific analysis – using for that a rough sketch may be not a great idea. Here trends for Goodwell, OK, 2007-2017. Green line is the trend computed from monthly averages of daily midrange values. Red line is the actual reference trend computed from highly-sampled records (every 5-min). Dotted line represents the bias where monthly trend deviates from the slope of the reference trend. Very little matches. Endpoints differ. Slopes differ.
Congrats there is definitely a bias during those years. Try the averages across a group of stations in this area or same station over a longer period. Trend is gone I can guarantee this 100%. Try it.
Try the averages across a group of stations in this area or same station over a longer period. Trend is gone I can guarantee this 100%. Try it.
We’ve got high quality records for no longer than 12-14 years. But you can synthesize a signal that closely resembles highly sampled signal for longer periods, say 150 years, de-trend it, then apply min/max approach and compare the reference signal with no trend and averaged min/max. And for many runs spurious trend appears.
Try it.
I will!
The best source I know of is USCRN. Is that OK with you? This thread is getting old but I’ll share what I find when me meet next. OK?
That’s what you get by mismatching the dates of the 5-minute-resolution diurnal cycles and the daily mid-range values right from the start; see:
The characteristic mid-range value should be the average of the morning trough and the afternoon crest of the SAME day.
1sky1, until you acknowledge that any representation of an analog signal with discrete values is a sample, governed by Nyquist, your knowledge and analysis will be limited. The “purpose” you mention (understand climatic variations) will also be limited by the error resulting from not complying with Nyquist. You can certainly make a case that the error is insignificant – and perhaps you could do that by defining what is and is not significant. But you can’t access (accurately – to engineering precision) long term trends when the extrema you are using are tainted by aliasing.
Your desire for scientifically based reconciliation of historical data with modern data/methods will only yield disappointment. Once aliased there is no way to remove the error. Even Nick’s noble ideas are going to be cut short by the lack of good data to use for the correction – and furthermore you will never know if your “correction” is more correct – you will just know it is different. There is no advancement to be had. The data we have is inferior due to the methods used. The only advancement would be to statistically determine the range of possible error and increase the stated uncertainty in the data and subsequent calculations.
If I understand all this correctly, it would seem the data to test Mr. Kilty’s criticism of Mr. Stokes’ article is unavailable, due to the fact that most temp sampling is done twice a day? If I do understand this correctly, is there another, non-theoretical way to test Mr. Kilty’s theory?
Kevin,
Thank you for this added clarity.
Some of us who are used to working with numbers dismissed the early historical weather data as unfit for purpose when the purpose is to estimate global warming.
At some stage technology was advanced enough to start to estimate global warming. The date for this depends on error analysis lowering the limits to be acceptable for the purpose.
How would you recommend that both sampling and error analysis now be performed for daily captures, for the result to be likely fit for purpose? At what date was adequate performance achieved? (I put it about year 2000 for Australian data).
Geoff
What is the Nyquist frequency for temperature? I expect there us bi maximun unless you apply an arbitrary bandpass filter.
Otherwise, what prevents temperature from changing on timescales of less than a second when the sun pops out from behind a cloud?
Q: What prevents temperature from changing on timescales of less than a second when the sun pops out from behind a cloud?
..
..
A: Thermal inertia
How much thermal inertia does air have?
Answer: Very little.
The oceans which comprise over 70% of the surface of the planet have a significant impact on the air temperature. So the thermal inertia of the water in the oceans is very high. Ever hear of “El Nino?”
Ferdberple,
Nyquist tells us what the relationship must be between what goes into the ADC and the clock frequency that runs the ADC. The filter is designed to control what goes into the ADC. In the real world there are no bandlimited signals. We attempt to limit them with filters, but imperfectly – however the results can still be very low error. Someone with knowledge of climate needs to decide what is “signal” and what is “noise”. Is that sun popping out a part of climate (signal) or just noise? Climate scientists care. Engineers may or may not care – but they will make sure to design a system that works properly for what the climate scientists decide. The engineer always recommends to take more data in because you can discard it later. You can’t go back and get it later if you decide to throw it away up front.
There is not really a Nyquist frequency for atmospheric temperature, per se. But analysis from the previous discussion suggests 24-sample/day does pretty good, but 288-samples/day does better to cover more scenarios and corner cases. The question is, how much error can you tolerate? Nyquist gets set from this.
William
I remember when I was in the Army and assigned to the Photographic Interpretation Research and Development lab. I went to Greenland for a month. The lab’s photographer advised me to take lots of film and take lots of pictures because the cost of film and processing, although expensive by usual standards, was trivial compared to the costs of travel and housing while there. And, I might never get a second chance — which I haven’t!
Clyde – good story! Good advice from the photographer.
If you ever do go back, bring some ice with you. I hear they are running low.
Kevin, I think most of us already believed that averaging Tmin and Tmax did not accurately represent the average temperature, which itself is used to infer an energy imbalance. The previous posts on the subject and my limited explorations demonstrated this. The question I have is about the recent “high quality” data based on five-minute averages. Do these satisfy the requirements for calculating a meaningful average temperature for a particular location?
Loren
While energy is of interest, the alarmist claims of future cataclysm are based on the effects of temperature. That is, ocean temperatures killing coral and fish, land temperatures causing extinction of plants and animals that are altitude bound, and supposed declines in crop production. Therefore, it really is important to be able to accurately and precisely characterize temperatures, and to be able to predict both high and low temperature changes, not just the temperature between two extremes. After all, there are an infinite number of temperature pairs that can provide the same mid-range value. Relying on any kind of ‘average’ usually results in a loss of information. While climatologists are focused on averages and trends, there might well be things of interest happening with the variance of the highs and lows.
Eq. (7) in Shannon’s is indeed the key to understanding what Kevin wrote, but it is not at all elementary as you think. I’m not sure if Kevin himself understands it completely.
Yes, convolving unit impulses over an arbitrary signal generates that signal, in a trivial way which is not useful because it only returns the sample points. If you convolve continuously, only then does it return the entire range. Not useful because we want to reconstruct the entire function from evenly spaced discrete samples.
You will recognize eq (7) as the convolution of the sampled function with a _normalized sinc_ pulse sin(πx)/πx, which results in the perfect reconstruction of the entire original band-limited signal.
How is this possible?
Shannon did not explain the motivation for the sinc pulse in his paper, but it related to the fact that the Fourier transform of the sinc function is the rectangle function. If we multiply the spectrum of the band-limited signal by a rectangle with height=1 over the support of the spectrum and zero elsewhere, it is clear that result is the same band-limited signal. So, applying the convolution theorem, we know that we can obtain the same result in the time domain by convolving the sample points with the normalized sinc pulse, which has a value of 1 at t=0 and zero elsewhere, i.e. a _real_ unit pulse.
The resulting continuous function is exactly the original signal, not an approximation.
I have met very few engineers who correctly understand this theorem. Most insist it only returns an approximate reconstruction of the original signal. But it is perfect, in exactly the same sense that two points determine a line. Two points are also sufficient to reconstruct the highest frequency in a band-limited signal, iff the Nyquist limit is obeyed
Again my reply to Joe Born February 15, 2019 at 1:36 pm is misplaced. I’m pretty sure I pressed the correct reply button. (But several hours passed while I was composing)
There’s been much talk of time here but not of space.
I’m not sure what the overall goal is, but when you start averaging all the different stations together it smacks of wanting to know the total energy (or potential) in the boundary layer so you can track how it changes over time. If that’s wrong well, then forget the rest of this comment.
If I wanted to get the total energy I would sample the boundary layer temperature everywhere at the same time and then average that. That means sampling some stations during day and some at night, others at dawn or dusk, etc. depending on where they are at the time of sampling.
Taking min/max over the whole day causes things that move over time like warm or cold air masses to all get all blurred into readings in many different stations. Plus vertical mixing in the atmosphere throws another wrench into things. My conclusion is that the min/max data is not all that useful for tracking total energy/potential changes in the atmosphere.
Seems like the min/max data would be more useful to monitor local climate — not for tracking over large regions. For example, is it getting warmer in Greenland where there’s lots of ice (and very few thermometers)?
Just my two cents…and likely worth about what you paid for it.
Observer
I think that if one is looking for a daily, global average, to be used for calculating an annual global average, it does make sense to synchronize all readings to be done simultaneously. With high temporal resolution sampling, that could be done in post processing without any special need for coordination around the world. Although, one problem that I see is that, because the land masses are not uniformly distributed, one might want to do that twice daily, for when the minimum land area is experiencing heating and the second time when the maximum land area is experiencing heating.
Kevin,
The use of this old historic max/min temperature is wrong for a number of reasons.
For example, T max happens at a time when the balance of several competing effects, some negative and some positive over time, reaches a maximum detected according the th thermometer design.
The timing of Tmax and inexorably some of its value is thus governed by some physical effects that have little to no relevance to whether the globe is warming or not.
E.g., the maximum might occur on an overcast day at an uncharacteristic time because there happened to be a break in the cloud long enough to capture the short period of elevated temperature. Maybe this happened, but at the same time some rain had just ceased to pour down and a variable evaporative cooling effect of rain on temperature was present at the particular time the sun shone. Neither sunshine nor rainfall are supposed to be primary determinants of the energy assessed through its temperature proxy. There is no place for elevation of temperatures like this to the power of 4 when dealing with S-B math. (There are more interfering exogenous variables, but I hope I made the point with rain and sunshine).
In Australia, it was not until about year 2000 that thermometry and data acquisition became good enough to measure temperatures as relevant to global energy.
Therefore, discussion of Nyquist frequencies is largely academic, of little practical effect but tremendously interesting. Thank you for your essay. Geoff.
Die Zeit verfliegt im Sauseschritt und wir wir fliegen mit.
Wilhelm Busch.
carpe diem.
Lets look at this Tmin and Tmax a bit more. The assumption behind simply averaging these as the “temperature” is entirely false. Say we have a Tmin which is pretty much constant at 0C and a few seconds of Tmax of 100C. Is the temperature really 50C? Of course not. Tmin and Tmax are perhaps useful for weather, but have nothing useful to contribute if averaged. The correct outcome needs two things, the first a time estimate of each temperature, taken with even sampling at a short enough period to not “miss” the peak values due to the rate of change of the temperature; and the second a time weighted average of the temperature value. This is an approximation to the low pass characteristic I suggested earlier, but capable of reasonable implementation digitally. So the data for 24 hours with my example, 900 samples of 0C and 100 samples of 100C gives a heating value of 10C, not the “average” value of 50C.
All the discussion of convolution above is really not attacking the data problem, the data of simple max and minimum values actually says almost nothing about climate or even perceived temperature. Weather forecasts tend to use maximum temperature or minimum depending on if they think it is cold or hot for the time of year. You choose clothes to suit, but what is that to do with climate?
Realistically all the data used by climate scientists is very unsatisfactory, and is processed in ways which are invalid. However if it gives the result they want, who is going to notice? I was “tutored” on the satellite data above, and you will see that duplicating samples every 18 days, is not going to give much that is accurate. It must contain huge variation due to weather, wind, time of day and season. If it is accurate to 0.01 degrees how can I filter out all the other variables? I cannot, so I must “adjust” the data in invalid ways to get any result at all. Once I start to add in area averaging as well one has very little idea of the error band, except that it is unknown and probably large.
The most interesting feature of climate science is the difficulty of obtaining the original raw data!
One of my pet peeves. Invalid processing. Taking daily temperatures recorded in integer values, averaging them and adding a digit of precision so you now have temps accurate to one tenth of a degree. Then averaging monthly values to annual ones and adding another digit of precision again, out to one one-hundredth of a degree. Or converting from degrees F to some other unit by multiplying by an improper fraction and again adding extra precision. The same exact thing is done to anomalies, basically because they are using a base that has false precision.
I just can’t believe that scientists in other more robust physical specialties haven’t roundly criticized the data practices being used by climate scientists. I would have flunked most of my lab classes if I had done similar data torturing.
Jim
This is a point I made to Nick above after reading his post from a few years back trying to use the theory of Large Numbers as a catch all. Pure theoretical fantasy really.
What Nick fails to understand is that the person making the apparatus will plan the uncertainty in the instrument depending on various factors, including reliability and maintenance. What Nick and others seem to think is that when a reading can be averaged that it means those extra decimals mean anything. He doesn’t see that the apparatus is often designed to be “insensitive” to small variations.
I took his March example and showed that if you assume the 1 degree system and add an offset ot 0.3 degC, you can have drifts or offsets in the data (against a well-known ideal) and it won’t matter as long as it is within the uncertainty of the apparatus. The average is 25.3 +/- 0.5 degC whereas the ideal real (obtained with a more accurate system reads 24.9 +/- 0.05 degC. If all you had is the system with higher uncertainty (which applies to the temperature measurement systems we have) then you wouldn’t know or WOULDN’T CARE for small changes like this.
Because the use of the apparatus (the tools) is related to the variation it is trying to measure. This is also what Kevin is talking about in his essay about Nyquist. You need to develop the tools to acquire data at uncertainties related to the variation you are seeing.
But you don’t assume that just because you can numerically calculate higher precision, that it means anything.
Jim
A voice of sanity in the darkness!
I was amazed how the smartest guy in this thread always stays so calm and answer with facts and examples (as he always does). Now I get it.
When you know and understand something deeply and others don’t it is really fun to read their comments and straw men arguments. Here we have spent millions of characters describing a tree in the middle of a forest. What the forest looks like? -no idea.
Who is he smartest in this thread is your guess. (He knows for sure and he is still smiling).
Kevin — Thanks for a remarkable and thought provoking essay. And thanks to everyone else for an equally remarkable assortment of comments — most of them well worth reading.
Just an idea. Why not integrate a graph of a curve with vertical axis in degrees of temperature in degrees kelvin and horizontal axis in time for 24 hours in one minute intervals and then compare the areas under curves on a year to year basis. What if anything will this tell.
Jim, Micky, Nick,
I’d like to believe this discussion should be settled empirically. What about recipe below?
1. Take some temperature series or generate artificial one – one decimal place. That represents measurements.
2. Each measurement from the point 1 has uncertainty +/- 0.05.
3. To each measurement add a random value drawn from the set [-0.05, 0.05]. The sum represents true, precise temperature.
4. Average the measurements and true temperatures
5 Compare results from the step 4.
Does it sound sensible?
OK, I did it for daily max, Melbourne, monthly means of daily max to 1 decimal (GHCND) for 2012. Adding normal dist error sd 0.05. The error, to 2 decimals was imperceptible. Here were the max as measured:
27.39 26.96 23.65 21.93 16.96 14.44 14.90 15.24 18.56 20.75 23.30 25.70
and, after adding noise
27.39 26.96 23.65 21.93 16.96 14.44 14.90 15.24 18.56 20.75 23.30 25.70
I had to multiply the differences by 100 to show at 2 dp. Then they were:
-0.02 0.17 -0.13 -0.03 -0.05 -0.09 0.01 -0.04 -0.01 -0.03 0.09 0.05
Even adding noise sd 0.5 barely made a difference. The differences multiplied by 100 were
0.63 -0.57 -0.01 -0.44 0.77 0.08 0.46 0.74 1.52 0.85 1.62 0.52
The effect is proportional.
No
You didnt read what I wrote. If you add random samples then they will cancel out with averaging.
You are assuming as many theorists do that the errors are random.
But what about discontinous drift which is common? Or slow non linear skews? And what about when you have small variations that are within the appartus uncertainty you just leave them there.
The assumption being made is the magnitude value obtained from a measurement contains intrinsic information about the quantity being measured. The reality is that it also includes the limitations and decisions made by the person maintaining the equipment. And there is no way of deconvoluting them to achieve a higher precision. That is why the real value is magnitude and error. It is up to the user to decide the context of this.
“You didnt read what I wrote.”
What you wrote was
“All of the differences you calculated need to have the +/- 0.5 degC error added.”
And that is just wrong, as these many examples show. When you quote an uncertainty, it is an error for each reading that could go either way. And there is no basis for adding that to the average. As you occasionally concede, that is reduced (a lot) by averaging.
What you keep coming back to is a claim that the instruments might be out by a fixed, repeated amount. That isn’t uncertainty; it is just wrongly calibrated instruments. It has different consequences and a different remedy. And it isn’t something to which you can sensibly assign a probability.
Nick
What you fail to realise is that they can intentionally be designed to drift but without compromising the overall use.
By your method you have no way of telling drift from a real value.
If an appartus is well maintained and regularly calibrated then this is less of an issue. But if left alone for years as often is the case, or if the whole system phase space is not fully characterised you have no way of telling what fluctuations in the magnitude of the readings are. And it may be designed precisely for this use.
You keep thinking that the magnitude of the readings can be used to elicit greater precision than what the system is designed for.
Micky, I’ve had similar discussions with Nick, but to no avail. In Nick’s world, all instruments are perfectly accurate and have infinite resolution. Nick-world error is always random and always averages away.
Nick either has no concept of systematic error or perhaps just refuses to acknowledge its existence.
Even highly accurate platinum resistance thermometer air temperature sensors suffer from systematic measurement error unless the shield is aspirated. The measurement error arises from uncompensated solar irradiation, including reflected heat, and slow wind speed.
Only the new CRN network uses aspirated shields. Only they provide temperatures reliable to about ±0.1 C (assuming everything remains in repair).
All prior air temperature measurements, and still including worldwide measurements outside the US CRN and perhaps parts of Europe, even the best modern air temperatures are not good to better than ±0.3 C, under ideal conditions of calibration, repair, and siting.
A better global uncertainty estimate is ±0.5 C, and in some places and times, probably rises to ±1-2 C.
Ssystematic error violates the assumption of the Central Limit Theorem, and does not average away. In proper science, it gets reported as the root-mean-square uncertainty in any mean.
But consensus climate science is not proper science.
And in Nick-world, physical science does not exist at all. Every Nick-world process follows the statistical ideal. It makes life there very easy.
I think one issue is that application pf uncertainty in pure theory is very different than in real world practice. Jim touched on this.
When determining systematic (irreducible) uncertainty, or just, error the decision is made to follow a convention dependent on the use of the measuring system.
For an average of measured data points you either root mean square the individual systematic errors or you assume the average belongs to the set of measurements and thus only the systematic error applies. Often it’s a mix and is subjective because in the real world error deterrmination is an art and often conservative.
As I have said on multiple occasions you design the tools for the job. Hence the reliability of the data is related to its use.
Nick –>
Read the documents I provided. The uncertainty arises in the next unstated digit. If the temp recorded is an integer, then the error component is +/- 0.5. If it is recorded to the tenth, then the error is 0.05. It is uncertain because you have no way of knowing what it’s value is.
When you then average numbers, for example, 29 and 32 you don’t know if you are averaging 29-0.5 and 32+0.5, or perhaps 29-0.0 and 32+0.3. That is why it is called uncertainty and why it must carry through. You simply can’t say 29+32=61/2=30.5 and keep the tenths digit. At best and depending on rounding you could say 30+/- 0.5 or 31+/-0.5.
Either way, you can’t then average another reading and say you can add another digit of precision, for example, 30.53. If you could do that, you could end up with an average containing 30, 60, who knows how many, numbers to right of the decimal place. Heck, you could end up with a repeating decimal out to infinity!
I still don’t think you are getting the point. When you did your data run you added in evenly distributed random noise that was artificially meant to cancel out. And it did.
With instruments, you don’t *know* what the “noise” from each instrument actually is. You know what the error “band” is, i.e. +/- some value. But you don’t know where in the band each instrument actually is. You can’t just assume that the errors have a perfect gaussian distribution in the error band and will all cancel out when you average all the values together.
Think of it this way. I am manufacturing thermocouples on a semi-conductor substrate for a digital thermometer. As my equipment ages are those thermocouples going to have the same error band or wider bands? Will the error bands have higher positive biases or negative biases? Or will they be equal? As the temperatures changes will the substrate expand or contract and what will that do the tolerance of the thermocouple? What will that do to the error band of the thermocouple?
If you don’t know all of this for each individual station then any kind of averaging of readings becomes a crap shoot for trying to cancel out errors using significant digits beyond the intrinsic capability of the instrument. In essence the error band of your average reading remains the error band of the instruments used.
If you add random samples then they will cancel out with averaging.
Indeed. Software randomizers usually generate random samples from strictly normal distribution. When I used however randomizer which draws samples from the Rayleigh distribution an error in averages, as expected, was significantly larger.
So, condition sine qua non if the ‘improving by averaging’ technique may work is that the error guys need to be strictly ‘white, Gaussian’? Any deviation from that as systematic errors, drifts, biases etc. would make ‘improving by averaging’ questionable?
That’s an interesting example.
An even simpler one is having a slow varying drift of +/- 0.3 degC over a few years that when using the 1 degC system produces a variation in the average magnitude.
But NOT any significant change in the measurement (magnitude with error)
You may think it’s interesting. It may be a real signal. But you would not be able to tell the difference unless you used a more accurate and higher precision system to check.
If you didn’t have this then attributing the drift to a real signal would be false.
In fact you probably wouldn’t care because you would not be using a 1degC system to determine sub degree changes.
In fact you probably wouldn’t care because you would not be using a 1degC system to determine sub degree changes.
That is one of the key points in such discussions, I reckon. Many historical temperature records were rounded to nearest C/F, as far as I’m aware. Still, climate science wizards claim that tiny variations in the mean are true representation of the reality.
You may think it’s interesting. It may be a real signal. But you would not be able to tell the difference unless you used a more accurate and higher precision system to check.
If I remember correctly Jim said he’s working on a paper that highlights issues around that. Looking forward to see that, hopefully supported by real life examples.
“So, condition sine qua non if the ‘improving by averaging’ technique may work is that the error guys need to be strictly ‘white, Gaussian’?”
No, it isn’t. In fact, that is the point of the central limit theorem. It says not only that the distribution of the mean will tend to gaussian, even if the samples aren’t, but that the sd of that normal mean will be what you would get by combining those of the samples as if they were normal (IOW, it doesn’t matter).
Huh? What is the “distribution of the mean”? And if the errors are not gaussian in distribution then they provide a bias to any calculation based on them since the positive errors cannot cancel the negative errors.
“What is the “distribution of the mean”? “
A sum of random variables is a random variable, and has a distribution. If you divide by N that is just a scaling – it still has a distribution. And it is what the Central Limit Theorem says tends to normal.
“since the positive errors cannot cancel the negative errors”
Positive errors always cancel negative, and this has nothing to do with being gaussian.
Probability doesn’t always work in the real world. If I give you a coin and ask you to flip it one hundred times can you tell me how many heads and how many tails you will get?
You can tell me the probability of each combination coming up but you can’t tell me for any single run of 100 flips exactly what combination you will get. That’s what the error band represents in a physical measurement from a large number of measurement devices. You can *assume* that the error distribution from all those devices will take on a gaussian distribution and that the average from all of them will be the most likely outcome but you can’t *know* that! What you *can* know is that the actual value of the average will lie somewhere in the error band representative of the totality of the instruments.
The central limit theorem only works when the samples themselves are defined with no error bands associated with them. For instance, samples taken from a population distribution. The population distribution may be skewed, e.g. based on income so that you have a large number of individuals on the low income side of the curve and few on the high side of the curve. A distribution far from being gaussian. If you take enough test sample collections from the distribution and calculate their averages you will find a gaussian distribution of the averages from those calculations. But those samples have no error band associated with them! Their income is their income! The minute you create uncertainty about what each individuals income is then you automatically create uncertainty about what the average is for each sample. And it then follows that the distribution for all of the samples put together is uncertain. Any graph of the probability curve should be done with an unsharpened carpenters pencil and not a fine point No. 2 pencil.
And this is what the the “average temperature of the Earth” should have. A huge error band associated with it! And that error band should be at least as large as the error band of the least reliable instrument used to calculate the average!
“And it is what the Central Limit Theorem says tends to normal.”
I should have also pointed out in my comment that with a normal distribution you *still* don’t KNOW, for any specific run, exactly what result you will get. You have a probability curve. But the mean of that curve is not *the* answer, not in the physical world. It’s not the most accurate. It’s only the most probable. But that’s why people bet on horse races, the most probable horse doesn’t always win in the physical world. And when that probability distribution curve has to be drawn with an unsharpened carpenters pencil you can’t even tell what the value of the mean at the center actually *is*. That’s why systemic error bands carry through in any measurement of the physical world. Not every measurement device gives perfectly repeatable measurements, i.e. the same absolute error, time after time over years of operation. Wasps can build nests in vent holes, mold can build up around critical components, even quantum effects can have impacts on semiconductor elements.
Mathematicians and programmers can assume perfect precision out to any number of significant digits, can assume probability distributions always resolve to one single number every single time (i.e. the same horse always wins), and that measurement devices have no systemic error bands but engineers that live in the real world will all tell you differently. So will most physical scientists. I often wonder why climate scientists won’t admit the same.
You can do that but you won’t prove anything because the hundreths digit is uncertain. The distribution might be random, but it might not be. That is the reason that the last unknown digit of precision is always stated as the full range of possible numbers. You don’t know the “true value” so it must be stated as a range.
Another issue with error is using standard deviation. Many places require the use of standard deviation in order to get an idea of the size of the possible values.
I did this for a Topeka, Kansas station in August 2018. Here is what I got when mapping daily averages (rounded to two significant digits).
mean – 79.13
min – 70
max – 89
standard deviation – 5.01
standard error – 0.9
3 sigma’s – 15.03
79.13 + 15.03 = 94.16
79.13 – 15.03 = 64.10
Funny how the ~ 95+% confidence level is outside the total range of recorded values. The standard error is also very large. Makes one wonder how accurate the monthly mean is when using in averaging.
In the (now 4?) recent threads on the subject of two measurements a day (Tmax and Tmin without associated times), a central question is: Is it is even possible to reconstruct an underlying continuous-time signal from just two samples? That is, assuming we are talking about a continuous-time signal that is suitably bandlimited. Any temperature curve driven by the earth’s rotation would seem to be limited to a constant plus a day-length sinusoid (assumption of daily periodicity, just a fundamental, at least for a start). The sampling theorem would seem to suggest that we need a number of samples GREATER THAN 2: perhaps 2.1 or 2.5 or 4. This is the common wisdom, and IT IS WRONG.
With the use of simple FFT interpolation techniques,
http://electronotes.netfirms.com/AN398.pdf
we CAN get perfect recovery with just two samples/cycle. Here N=2 (even) so we generally do have to assume we have energy at k=1 (half the sampling frequency of 2). (If we didn’t, we have only a constant so it would not be interesting.) In this case (see app note) we split that “center term” (k=1 for N=2)
The figure here: http://electronotes.netfirms.com/TwoSamps.jpg
Shows the 2 samples/day case. The top panel show a raised (by an additive 2) cosine sampled twice in one cycle (red stems at +3 and at +1). Here we overplotted the supposed cosine in blue, but this known origin is not input to the program – we intend to get this from the invented samples. The seconds panel is the FFT of the first, for k=0 and k=1. Because this is a length-(N=2) DFT, X(0) is the sum (4) and X(1) the difference (2), purely real.
Now to zero-pad. We suppose that we will learn where the DFT “thinks the two samples came from” if we had perhaps 16 samples rather than just 2. So we interpolate by zero-padding the CENTER of the FFT with 14 zero. If N were odd, we would have zero at the center. But N=2 (even) so we split X(1) in two and thus place 1’s at k=1 and k=15 with 13 zeros in-between to form the FFT of the interpolated signal (panel 3 – follow the green lines). This is REVERSING THE ALIASING. Then taking the inverse FFT (and multiplying by 8) we get back 16 samples of the cosine (last panel). The whole exercise is the four lines of MATLAB/Octave code:
x = [3,1]
X = fft(x)
XI = [X(1), X(2)/2, zeros(1,13), X(2)/2]
xi = 8*ifft(XI)
There was earlier talk about “Signals 101” and missing the first week. This sort of thing here is what you learn with true experience. It is important to understand HERE when simultaneously worrying about aliasing, bandlimiting, and minimal samples/cycle. It’s neat too!
Bernie
Hi Bernie,
That is neat. Thanks.
You said: “The sampling theorem would seem to suggest that we need a number of samples GREATER THAN 2: perhaps 2.1 or 2.5 or 4. This is the common wisdom, and IT IS WRONG.”
The theorem states fs >= 2B, so the fs = 2B case is covered, correct? But what about this scenario?
Sample 1sin(x), where the sample clock falls at exactly x = nπ, where n = 0, 1, 2, 3, …
This is one of the reasons it is usually written fs > 2B.
Every practical application exceeds 2x for a number of good reasons.
William –
My FFT-interpolation demo is merely a more recent answer to the Oldie-but-Goodie philosophical issue of a sinusoid sampled exactly twice/cycle which you have reintroduced here.
The older rebuttal? Well a sinusoid of constant freq, constant amplitude, and constant phase (no FM, AM, PM, pauses) has only one bit of INFORMATION – it’s either there of it isn’t. Further, its (one-sided) bandwidth is zero (B=0). So any sampling rate greater than zero is sufficient to assure that “no information is lost”. Even one sample. The probability that this sample is EXACTLY at a zero-crossing is zero.
Bernie
Bernie,
One thing a lot of us share in common is a curious mind and the love of learning things – and knowing things. But we have to try to keep clear whether we are at the chalkboard or in the real world with our interests. The fun things you brought up with the recent points are interesting but not applicable in the real world for any practical purpose.
You said: “The probability that this sample is EXACTLY at a zero-crossing is zero.”
Yes. But you were discussing theory. The probability of finding a real world application where we need to sample a signal-generator-grade sine wave and try to sample it at exactly 2x the frequency is zero.
The topic is really how sampling affects the accuracy and therefore value of our temperature record. We are dealing with a real world application. What is the point of bringing up academic-only special cases that have no application to the subject? None of those special cases can help us go back and improve the record and none of them would be useful if we were to try to improve the way to do capture data.
William Ward at February 20, 2019 at 10:13 am said in part
“. . . . . . . . We are dealing with a real world application. What is the point of bringing up academic-only special cases that have no application to the subject? . . . . . . . .”
Well – I guess it’s the same point YOU intended in bringing up the SAME special case:
William Ward at February 19, 2019 at 11:18 pm
“. . . . . . . . . . But what about this scenario? Sample 1sin(x), where the sample clock falls at exactly x = nπ, where n = 0, 1, 2, 3, … . . . . . . “
So William employs these sort of academic parlor-games in formulating an original “gotcha” but objects to someone else, using standard basic DSP theory, following up to a FULL understanding of the simplest of examples. Toy examples still have to follow theory – no slight-of-hand is used. There are no “academic-only special cases”. Instead only Nature reminding us (sotto voce) that She can be subtle.
At the same time, he misuses DSP notions of sampling (values, but no corresponding times given), aliasing (what spectral overlaps occur), and jitter (the change of average time between samples that 1sky1 pointed out).
Bernie
Bernie, I’m sorry for offending you. I hope you will accept my apology.
I addressed your technical points in another recent reply.