A condensed version of a paper entitled: “Violating Nyquist: Another Source of Significant Error in the Instrumental Temperature Record”.

By William Ward, 1/01/2019

The 4,900-word paper can be downloaded here: https://wattsupwiththat.com/wp-content/uploads/2019/01/Violating-Nyquist-Instrumental-Record-20190112-1Full.pdf

The 169-year long instrumental temperature record is built upon 2 measurements taken daily at each monitoring station, specifically the maximum temperature (Tmax) and the minimum temperature (Tmin). These daily readings are then averaged to calculate the daily mean temperature as Tmean = (Tmax+Tmin)/2. Tmax and Tmin measurements are also used to calculate monthly and yearly mean temperatures. These mean temperatures are then used to determine warming or cooling trends. This “historical method” of using daily measured Tmax and Tmin values for mean and trend calculations is still used today. However, air temperature is a signal and measurement of signals must comply with the mathematical laws of signal processing. The Nyquist-Shannon Sampling Theorem tells us that we must sample a signal at a rate that is at least 2x the highest frequency component of the signal. This is called the Nyquist Rate. Sampling at a rate less than this introduces aliasing error into our measurement. The slower our sample rate is compared to Nyquist, the greater the error will be in our mean temperature and trend calculations. The Nyquist Sampling Theorem is essential science to every field of technology in use today. Digital audio, digital video, industrial process control, medical instrumentation, flight control systems, digital communications, etc., all rely on the essential math and physics of Nyquist.

NOAA, in their USCRN (US Climate Reference Network) has determined that it is necessary to sample at 4,320-samples/day to practically implement Nyquist. 4,320-samples/day equates to 1-sample every 20 seconds. This is the practical Nyquist sample rate. NOAA averages these 20-second samples to 1-sample every 5 minutes or 288-samples/day. NOAA only publishes the 288-sample/day data (not the 4,320-samples/day data), so to align with NOAA the rate will be referred to as “288-samples/day” (or “5-minute samples”). (Unfortunately, NOAA creates naming confusion with their process of averaging down to a slower rate. It should be understood that the actual rate is 4,320-samples/day.) This rate can only be achieved by automated sampling with electronic instruments. Most of the instrumental record is comprised of readings of mercury max/min thermometers, taken long before automation was an option. Today, despite the availability of automation, the instrumental record still uses Tmax and Tmin (effectively 2-samples/day) instead of a Nyquist compliant sampling. The reason for this is to maintain compatibility with the older historical record. However, with only 2-samples/day the instrumental record is highly aliased. It will be shown in this paper that the historical method introduces significant error to mean temperatures and long-term temperature trends.

NOAA’s USCRN is a small network that was completed in 2008 and it contributes very little to the overall instrumental record. However, the USCRN data provides us a special opportunity to compare a high-quality version of the historical method to a Nyquist compliant method. The Tmax and Tmin values are obtained by finding the highest and lowest values among the 288 samples for the 24-hour period of interest.

 

NOAA USCRN Examples to Illustrate the Effect of Violating Nyquist on Mean Temperature

The following example will be used to illustrate how the amount of error in the mean temperature increases as the sample rate decreases. Figure 1 shows the temperature as measured at Cordova AK on Nov 11, 2017, using the NOAA USCRN 5-minute samples.

clip_image002

Figure 1: NOAA USCRN Data for Cordova, AK Nov 11, 2017

The blue line shows the 288 samples of temperature taken that day. It shows 24-hours of temperature data. The green line shows the correct and accurate daily mean temperature that is calculated by summing the value of each sample and then dividing the sum by the total number of samples. Temperature is not heat energy, but it is used as an approximation of heat energy. To that extent, the mean (green line) and the daily-signal (blue line) deliver the exact same amount of heat energy over the 24-hour period of the day. The correct mean is -3.3 °C. Tmax is represented by the orange line and Tmin by the grey line. These are obtained by finding the highest and lowest values among the 288 samples for the 24-hour period. The mean calculated from (Tmax+Tmin)/2 is shown by the red line. (Tmax+Tmin)/2 yields a mean of -4.7 °C, which is a 1.4 °C error compared to the correct mean.

Using the same signal and data from Figure 1, Figure 2 shows the calculated temperature means obtained from progressively decreased sample rates. These decreased sample rates can be obtained by dividing down the 288-sample/day sample rate by a factor of 4, 8, 12, 24, 48, 72 and 144. Therefore, the sample rates will correspond to: 72, 36, 24, 12, 6, 4 and 2-samples/day respectively. By properly discarding the samples using this method of dividing down, the net effect is the same as having sampled at the reduced rate originally. The corresponding aliasing that results from the lower sample rates, reveals itself as shown in the table in Figure 2.

clip_image004

Figure 2: Table Showing Increasing Mean Error with Decreasing Sample Rate

It is clear from the data in Figure 2, that as the sample rate decreases below Nyquist, the corresponding error introduced from aliasing increases. It is also clear that 2, 4, 6 or 12-samples/day produces a very inaccurate result. 24-samples/day (1-sample/hr) up to 72-samples/day (3-samples/hr) may or may not yield accurate results. It depends upon the spectral content of the signal being sampled. NOAA has decided upon 288-samples/day (4,320-samples/day before averaging) so that will be considered the current benchmark standard. Sampling below a rate of 288-samples/day will be (and should be) considered a violation of Nyquist.

It is interesting to point out that what is listed in the table as 2-samples/day yields 0.7 °C error. But (Tmax+Tmin)/2 is also technically 2-samples/day with an error of 1.4°C as shown in the table. How can this be possible? It is possible because (Tmax+Tmin)/2 is a special case of 2-samples per day because these samples are not spaced evenly in time. The maximum and minimum temperatures happen whenever they happen. When we sample properly, we sample according to a “clock” – where the samples happen regularly at exactly the same time of day. The fact that Tmax and Tmin happen at irregular times during the day causes its own kind of sampling error. It is beyond the scope of this paper to fully explain, but this error is related to what is called “clock jitter”. It is a known problem in the field of signal analysis and data acquisition. 2-samples/day, regularly timed, would likely produce better results than finding the maximum and minimum temperatures from any given day. The instrumental temperature record uses the absolute worst method of sampling possible – resulting in maximum error.

Figure 3 shows the same daily temperature signal as in Figure 1, represented by 288-samples/day (blue line). Also shown is the same daily temperature signal sampled with 12-samples/day (red line) and 4-samples/day (yellow line). From this figure, it is visually obvious that a lot of information from the original signal is lost by using only 12-samples/day, and even more is lost by going to 4-samples/day. This lost information is what causes the resulting mean to be incorrect. This figure graphically illustrates what we see in the corresponding table of Figure 2. Figure 3 explains the sampling error in the time-domain.

clip_image006

Figure 3: NOAA USCRN Data for Cordova, AK Nov 11, 2017: Decreased Detail from 12 and 4-Samples/Day Sample Rate – Time-Domain

Figure 4 shows the daily mean error between the USCRN 288-samples/day method and the historical method, as measured over 365 days at the Boulder CO station in 2017. Each data point is the error for that particular day in the record. We can see from Figure 4 that (Tmax+Tmin)/2 yields daily errors of up to ± 4 °C. Calculating mean temperature with 2-samples/day rarely yields the correct mean.

clip_image008

Figure 4: NOAA USCRN Data for Boulder CO – Daily Mean Error Over 365 Days (2017)

Let’s look at another example, similar to the one presented in Figure 1, but over a longer period of time. Figure 5 shows (in blue) the 288-samples/day signal from Spokane WA, from Jan 13 – Jan 22, 2008. Tmax (avg) and Tmin (avg) are shown in orange and grey respectively. The (Tmax+Tmin)/2 mean is shown in red (-6.9 °C) and the correct mean calculated from the 5-minute sampled data is shown in green (-6.2 °C). The (Tmax+Tmin)/2 mean has an error of 0.7 °C over the 10-day period.

clip_image010

Figure 5: NOAA USCRN Data for Spokane, WA – Jan13-22, 2008

 

The Effect of Violating Nyquist on Temperature Trends

Finally, we need to look at the impact of violating Nyquist on temperature trends. In Figure 6, a comparison is made between the linear temperature trends obtained from the historical and Nyquist compliant methods using NOAA USCRN data for Blackville SC, from Jan 2006 – Dec 2017. We see the trend derived from the historical method (orange line) starts approximately 0.2 °C warmer and has a 0.24 °C/decade warming bias compared to the Nyquist compliant method (blue line). Figure 7 shows the trend bias or error (°C/Decade) for 26 stations in the USCRN over a 7-12 year period. The 5-minute samples data gives us our reference trend. The trend bias is calculated by subtracting the reference from the (Tmaxavg+Tminavg)/2 derived trend. Almost every station exhibits a warming bias, with a few exhibiting a cooling bias. The largest warming bias is 0.24 °C/decade and the largest cooling bias is -0.17 °C/decade, with an average warming bias across all 26 stations of 0.06C. According to Wikipedia, the calculated global average warming trend for the period 1880-2012 is 0.064 ± 0.015 °C per decade. If we look at the more recent period that contains the controversial “Global Warming Pause”, then using data from Wikipedia, we get the following warming trends depending upon which year is selected for the starting point of the “pause”:

1996: 0.14°C/decade

1997: 0.07°C/decade

1998: 0.05°C/decade

While no conclusions can be made by comparing the trends over 7-12 years from 26 stations in the USCRN to the currently accepted long-term or short term global average trends, it can be instructive. It is clear that using the historical method to calculate trends yields a trend error and this error can be of a similar magnitude to the claimed trends. Therefore, it is reasonable to call into question the validity of the trends. There is no way to know for certain, as the bulk of the instrumental record does not have a properly sampled alternate record to compare it to. But it is a mathematical certainty that every mean temperature and derived trend in the record contains significant error if it was calculated with 2-samples/day.

clip_image012

Figure 6: NOAA USCRN Data for Blackville, SC – Jan 2006-Dec 2017 – Monthly Mean Trendlines

clip_image014

Figure 7: Trend Bias (°C/Decade) for 26 Stations in USCRN

Conclusions

1. Air temperature is a signal and therefore, it must be measured by sampling according to the mathematical laws governing signal processing. Sampling must be performed according to The Nyquist Shannon-Sampling Theorem.

2. The Nyquist-Shannon Sampling Theorem has been known for over 80 years and is essential science to every field of technology that involves signal processing. Violating Nyquist guarantees samples will be corrupted with aliasing error and the samples will not represent the signal being sampled. Aliasing cannot be corrected post-sampling.

3. The Nyquist-Shannon Sampling Theorem requires the sample rate to be greater than 2x the highest frequency component of the signal. Using automated electronic equipment and computers, NOAA USCRN samples at a rate of 4,320-samples/day (averaged to 288-samples/day) to practically apply Nyquist and avoid aliasing error.

4. The instrumental temperature record relies on the historical method of obtaining daily Tmax and Tmin values, essentially 2-samples/day. Therefore, the instrumental record violates the Nyquist-Shannon Sampling Theorem.

5. NOAA’s USCRN is a high-quality data acquisition network, capable of properly sampling a temperature signal. The USCRN is a small network that was completed in 2008 and it contributes very little to the overall instrumental record, however, the USCRN data provides us a special opportunity to compare analysis methods. A comparison can be made between temperature means and trends generated with Tmax and Tmin versus a properly sampled signal compliant with Nyquist.

6. Using a limited number of examples from the USCRN, it has been shown that using Tmax and Tmin as the source of data can yield the following error compared to a signal sampled according to Nyquist:

a. Mean error that varies station-to-station and day-to-day within a station.

b. Mean error that varies over time with a mathematical sign that may change (positive/negative).

c. Daily mean error that varies up to +/-4°C.

d. Long term trend error with a warming bias up to 0.24°C/decade and a cooling bias of up to 0.17°C/decade.

7. The full instrumental record does not have a properly sampled alternate record to use for comparison. More work is needed to determine if a theoretical upper limit can be calculated for mean and trend error resulting from use of the historical method.

8. The extent of the error observed with its associated uncertain magnitude and sign, call into question the scientific value of the instrumental record and the practice of using Tmax and Tmin to calculate mean values and long-term trends.

Reference section:

This USCRN data can be found at the following site: https://www.ncdc.noaa.gov/crn/qcdatasets.html

NOAA USCRN data for Figure 1 is obtained here:

https://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2017/CRNS0101-05-2017-AK_Cordova_14_ESE.txt

NOAA USCRN data for Figure 4 is obtained here:

https://www1.ncdc.noaa.gov/pub/data/uscrn/products/daily01/2017/CRND0103-2017-AK_Cordova_14_ESE.txt

NOAA USCRN data for Figure 5 is obtained here:

https://www1.ncdc.noaa.gov/pub/data/uscrn/products/subhourly01/2008/CRNS0101-05-2008-WA_Spokane_17_SSW.txt

NOAA USCRN data for Figure 6 is obtained here:

https://www1.ncdc.noaa.gov/pub/data/uscrn/products/monthly01/CRNM0102-SC_Blackville_3_W.txt

0 0 vote
Article Rating
575 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
January 14, 2019 2:12 pm

“NOAA, in their USCRN (US Climate Reference Network) has determined that it is necessary to sample at 4,320-samples/day to practically implement Nyquist. 4,320-samples/day equates to 1-sample every 20 seconds. This is the practical Nyquist sample rate. ”

How about far simple alternative: put a digital thermometer inside a 5 gallon bucket of water, housed in a Stevenson screen, then read temperature just once a day at 2pm.

Greg F
Reply to  vukcevic
January 14, 2019 2:47 pm

From where I am standing that bucket of water would have been frozen solid a couple of weeks ago. ;o)

William Ward
Reply to  Greg F
January 14, 2019 3:25 pm

Greg F: LOL! Maybe antifreeze needs to be added.

vukcevic: Interesting idea: using a specified thermal mass to integrate the heat energy. I’m not sure if this would be good or not, but might be worthy of exploring. Of course, you would still need to determine what the maximum frequency content was for that situation and sample at the rate that complied with Nyquist. You still have a signal, but likely an even slower one. But what NOAA did with the USCRN is probably the way to go. We would need a much better global coverage with stations of this quality and then we would need to process all of the 288 samples/day for each location. Not mentioned in the paper but is one of the next obvious failings of the way climate science is done, is the concept of “global average temperature”. What is the science – what is the thermodynamics that says averaging temperature signals from multiple locations has any meaning at all? In engineering, we feed signals into characteristic equations. What do we feed the “global average temperature” into? First, we don’t even end up with a signal. We end up with a number. This is a science fail. But what it is fed into is the Climate Alarmism machine and thats about all.

Samuel C Cogar
Reply to  William Ward
January 15, 2019 7:33 am

Quoting from article: William Ward, 1/01/2019

The 169-year long instrumental temperature record is built upon 2 measurements taken daily at each monitoring station, specifically the maximum temperature (Tmax) and the minimum temperature (Tmin). These daily readings are then averaged to calculate the daily mean temperature as …… etc., etc.

The author, William Ward, is assuming things that are not based in fact or reality.

First of all, there is no per se “169-year long instrumental temperature record” that defines or portrays global, continental, national or regional near-surface air temperatures. But now there is, or is claimed to be, 1 or 2 or 3 or so, LOCAL +/- 169-year long instrumental temperature records whose contents covering their 1st 100 years have to be adjudged as highly questionable and of little importance other than use as “reference data/info”.

Secondly, the same as above is true for the per se, United States’ 148-year long instrumental temperature record.

And thirdly, in the early years of recording the US’s “temperature record” there were only 19 stations reporting temperatures, ……. all of them east of the Mississippi River …… and those temperature were recorded twice per day, at specified hours, ……. which had absolutely nothing to do with the daily Tmax or Tmin temperatures.

Now just about everything you wanted to know about the US Temperature Record and/or the NWS, …… and maybe a few things you don’t want to know, …… can be found at/on the following web sites, ….. just as soon as the current “government shutdown” is resolved, 😊 😊 ….. to wit:

The beginning of the National Weather Service
http://www.nws.noaa.gov/pa/history/index.php

National Weather Service Timeline – October 1, 1890:
http://www.nws.noaa.gov/pa/history/timeline.php

History of the NWS – 1870 to present
http://www.nws.noaa.gov/pa/history/evolution.php

Reply to  Samuel C Cogar
January 16, 2019 9:07 am

Thank you, Mr. Cogar, for commenting on the first sentence:
“The 169-year long instrumental temperature record …”

There is no 169-year real-time record
of global temperatures !

The Southern Hemisphere had insufficient
coverage until after World War II — an argument
could be made that pre-World War II
surface temperatures represent only
the Northern Hemisphere !

How can we take this author seriously
when his FIRST sentence is wrong !

In addition, the author seems obsessed
with the sampling rate of surface
thermometers.

There are MANY major problems
with surface temperature “data”
BEFORE we get to the sampling rate !

How about a MAJORITY of the data
being infilled (wild guessed) by government
bureaucrats with science degrees, for grids
with no data, or incomplete data?

How about the CHARACTER of the
people compiling the data — the same
people who had predicted a lot of
global warming, have a huge
conflict of interest because they
also get to compile the “actuals”,
and they certainly want to show
their predictions were right,
so they make “adjustments”
that push the actuals closer
to their computer game predictions ?

How about the fact that weather satellite data,
with far less infilling, has been available
since 1979, but are ignored simply because they
show less warming than the surface numbers?

How about the fact that weather balloon data
confirm that satellite data are right, but the
surface data are “outliers” — yet the weather
balloon data are also ignored ?

The biggest puzzle of all is why one number,
the global average temperature, is the
right number to represent the “climate”
of our planet. After all, not one person
actually lives in the “average temperature” !

My climate science blog:
http://www.elOnionBloggle.Blogspot.com

William Ward
Reply to  Samuel C Cogar
January 16, 2019 1:31 pm

Samuel and Richard,

Thanks for commenting here. I actually agree with some of what you said. However, I think maybe you judged my paper too quickly based upon your critiques about the instrumental record comments. My point here is to show problems with how climate science is conducted with the use of the data in the record. With a limited number of words for this post, I didn’t want to waste them on details that were not relevant to the core message. Please feel free to add details about the record history – that is helpful. And I’m not negating all of the other problems with temperature measurements you mention. In a post to someone else I listed out 12 things wrong with the “record”.

I have cataloged 12 significant “scientific” errors with the instrumental record:
1) Instruments not calibrated
2) Violating Nyquist
3) Reading error (parallax, meniscus, etc) – how many degrees wrong is each reading?
4) Quantization error – what do we call a reading that is between 2 digits?
5) Inflated precision – the addition of significant figures that are not in the original measurements.
6) Data infill – making up data or interpolating readings to get non-reported data.
7) UHI – ever encroaching thermal mass – giving a warming bias to nighttime temps.
8) Thermal corruption – radio power transmitters located in the Stevenson Screen under the thermistor or a station at the end of a runway blasted with jet exhaust.
9) Siting – general siting problems – may be combined with 7 and 8
10) Rural station dropout – loss of well situated stations.
11) Instrument changes – changing instruments that break with units that are not calibrated the same or instruments that are electronic where previous instruments were not. Response times likely increase adding greater likelihood to capture transients.
12) Data manipulation/alteration – special magic algorithms to fix decades old data.

This paper just focuses on 1 of them. Many others have already written about the other 11 here.

Samuel C Cogar
Reply to  Samuel C Cogar
January 17, 2019 4:32 am

William Ward – January 16, 2019 at 1:31 pm

My point here is to show problems with how climate science is conducted with the use of the data in the record. …………… ………………. And I’m not negating all of the other problems with temperature measurements you mention. In a post to someone else I listed out 12 things wrong with the “record”.

It is MLO that, the per se, US Historical Temperature Record (circa 1870-2018), …… that is controlled/maintained by the NWS and/or NOAA, …… is so corrupted that it is utterly useless for conducting any research or investigations of an actual, factual scientific nature.

William Ward, I would like to point out the fact that your …… “list of 12 things wrong with the record” ……. are only a minor part of “the problem”.

In actuality, there is NO per se, actual, factual US Historical Temperature Record prior to 1970’s, or possibly prior to 1980’s. And the reason I say that is, all temperature data recorded at specified locations by government employees and/or volunteers, from the early days in 1870/80’s, up thru the 1960’s/1970’s, was transmitted daily to the NWS where its ONLY use was for generating 3 to 7 day weather forecasts, ….. originally only for areas east of the Mississippi River.

“DUH”, the first 70+ years of the aforesaid “daily temperatures” were only applicable for 7 days max, and then of no value whatsoever, other than garbage reference data.

And that NWS archived near-surface temperature “data base” didn’t become the US Historical Temperature Record until the proponents of AGW/CAGW gave it its current name after adopting/highjacking “it” to prove their “junk science”.

Reply to  Greg F
January 15, 2019 1:29 am

/sarc is in short supply, to be use in moderation.

[Oh yes, we use /sarc all time in our secret moderator forum. Oh wait, you meant something else. Nevermind. -mod]

getitright
Reply to  Greg F
January 15, 2019 11:03 pm

Because the constant lag time of you integrating scheme would produce a variable error as the temperature derivative is never constant.

Samuel C Cogar
Reply to  vukcevic
January 15, 2019 4:07 am

vukcevic – January 14, 2019 at 2:12 pm

How about far simple alternative: put a digital thermometer inside a 5 gallon bucket of water, housed in a Stevenson screen, then read temperature just once a day at 2pm.

“HA”, apparently no one liked that “idea” when I proposed a similar one 3 or 4 years ago ….. even though my proposal suggested using a one (1) gallon enclosed container (aluminum), ….. a -30F water-antifreeze mixture ……. and two (2) temperature sensors, ……. with said sensors connected to a radio transmitter located underneath or on top of the Stevenson screen ….. and which is programmable to transmit temperature data at whatever frequency desired.

The liquid in the above noted container …… NEGATES any further “averaging” of daily temperatures ….. because it is a 100% accurate real-time “average’er”.

Peta of Newark
Reply to  vukcevic
January 15, 2019 5:14 am

Thank you Vuk – nailed it
Is it really beyond possibility that the observed temp changes are due to a decrease in te water comtent of the land/ground/Earth’s land surface?
Might explain an ever so gentle sea-level rise in places?

If you wanna measure Earth Surface Temperature – measure it.
Put a little datalogger running at 4 samples per day max, pop it in a plastic bag with some silica gel, put that inside an old jam-jar with a good fitting/sealing lid..
THEN: bury it under 18″ of dirt.

Return once per month to unload it and check its battery.
If you fancy an average of a wide area of ground, locate the underground stop-cock tap of your home’s water supply pipe and put the data-logger in there.
Saves digging a hole in your garden too.

If you wanna be *really* scientific, run a twin data-logger, in a classic style Stephenson Screen directly above or nearby.
Maybe then, get Really Hairy and put a *third* logger inside a nearby forest or densely wooded area.
A city slicking pent-house dwelling friend might be a handy thing to have for even greater excitement.
😀

Then you might realise that temp is maybe not the driver or the cause of Climate – temp is the symptom or the effect. As Vuk intimates by attempting to measure *energy* rather than temperature.
If anyone is being violated by Climate Science, it is in fact Boltzmann, Wien and Planck.
Nyquist is ‘a squirrel’. A distraction.

Editor
Reply to  Peta of Newark
January 15, 2019 5:15 pm

Peta => Nyquist is another way of demonstrating that the long-term historical surface temperature record is not fit for purpose when applied to create a global anomaly in the range of less than 1 or 2 degrees.

You are quite right that climate models particularly violate a lot of “more important” physical laws by using “false” (simplified) versions of non-computable non-linear equations.

If one considers the temperature record as a “signal” (which is one valid way of looking at it) then Nyquist applies if one expects an accurate and precise result. Since the annual GAST anomaly changes by a fraction of a degree C, the errors and uncertainty resulting from the (necessary for historical records) violation of Nyquist.

January 14, 2019 2:15 pm

Maximum error is what the Warmistas want as it best fits the CAGW theory.

Steve O
January 14, 2019 2:18 pm

What if, instead of taking 2 samples per day we took 730 samples per year?

Gary Pearse
Reply to  Steve O
January 14, 2019 3:48 pm

Randomly of course!

Ferdberple
Reply to  Gary Pearse
January 14, 2019 10:39 pm

730 random samples would actually give a more reliable annual average.

ghl
Reply to  Steve O
January 14, 2019 4:11 pm

Steve
Make it 732, then you would see some real aliasing.

William Ward
Reply to  Steve O
January 14, 2019 4:11 pm

Steve O – LOL!

January 14, 2019 2:22 pm

Nonsense about Nyquist
“The Nyquist-Shannon Sampling Theorem tells us that we must sample a signal at a rate that is at least 2x the highest frequency component of the signal. This is called the Nyquist Rate. Sampling at a rate less than this introduces aliasing error into our measurement.”
The Theorem tells you that you can’t resolve frequencies beyond that limit. But we aren’t trying to resolve high frequencies. We are trying to get a monthly average. It isn’t a communications channel.

As for aliasing, there are no frequencies to alias. The sampling rate is locked to the diurnal frequency.

As for the examples, Fig 1 is just 1 day. 1 period. You can’t demonstrate Nyquist, aliasing or whatever with one period.

Fig 2 shows the effect of sampling at different resolution. And the differences will be due to the phase (unspecified) at which sampling was done. But the min/max is not part of that sequence. It occurs at times of day depending on the signal itself.

There is a well-known issue with min/max; it depends on when you end the 24 hour period. This is the origin of the TOBS adjustment in USHCN. I did a study here comparing different choices with hourly sampling for Boulder Colorado. There are differences, but nothing to do with Nyquist.

Fig 3 again is just one day. Again there is no way you can talk about Nyquist or aliasing for a single period.

MarkW
Reply to  Nick Stokes
January 14, 2019 2:33 pm

If you miss fast moving changes in temperature then you can’t claim to have an accurate estimate of the daily average.
If you don’t have an accurate daily average then there’s no way to get an accurate monthly average.
If your monthly averages aren’t accurate, then there’s no way to claim that you have accurately plotted changes over time.

Steve O
Reply to  MarkW
January 14, 2019 3:37 pm

Let’s say your instruments could take continual measurements such that you capture every moment for a year and you compared that to my measurements, where I record the high and the low, each rounded to the nearest whole number, and divided by two.

Your computation of the average temperature would be a precise measurement, but how close do you suppose my average for the year would be to your average for the year?

William Ward
Reply to  Steve O
January 14, 2019 4:37 pm

Steve O,

You can do this with the USCRN data for a station. You can compare both methods as follows:

For “historical method” [(Tmax+Tmin)/2): Take NOAA published monthly averages for the year and add them up and divide by 12. The data for each month is arrived at (by NOAA) by averaging Tmax for the month and averaging Tmin for the month. Then these 2 numbers are averaged to give you the monthly average/mean.

For Nyquist method: Add up every sample for the station over the year and divide by the number of samples. There are 105,120 samples for the year (288*365).

Try Fallbrook CA station for example. You will find that each year from 2009 through 2017 the yearly mean error is between 1.25C and 1.5C.

Some stations have more absolute mean error than others and some stations show more error in trends and less in means. Some show large error in both and some show little error in both.

The reason for the variance is explained in the frequency analysis. The daily temperature signal as seen by plotting the 288-samples/day varies quite a bit from day-to-day and station-to-station. The shape of the signal defines the frequency content. At 2 samples per day, you always have aliasing unless you have a purely sinusoidal signal. We never see this. The amount of mean error and trend error is defined by the amount of the aliasing, where it lands in frequency (daily signal or longer-term trend), and the phase of the aliasing compared to the phase of the signal at those critical frequencies.

I encourage you and all readers to read the full paper if you want to learn more about the theory and mechanics of how aliasing manifests. I give simple graphical examples that explain this.

Steve O
Reply to  William Ward
January 15, 2019 4:18 am

I’ll definitely need to read the full paper. Thanks.

ShanghaiDan
Reply to  Steve O
January 14, 2019 5:53 pm

We see above (in the post, with real data) the error can be 1.4 deg C; when we’re talking about a 1.5 deg C change being a calamity, that much error is, itself, a calamity.

William Ward
Reply to  ShanghaiDan
January 14, 2019 6:47 pm

Amen Shanghai Dan!

Samuel C Cogar
Reply to  ShanghaiDan
January 15, 2019 8:04 am

A published report of last week’s (month’s/year’s) average near-surface temperatures ….. doesn’t have as much “value” as does a copy of last week’s newspaper. 😊 😊

Bob boder
Reply to  Steve O
January 15, 2019 10:38 am

Yep and my fancy dancy RTD probe reacts a lot quicker than that good old mercury thermometer did, on the other hand mercury thermometers don’t drift over time RTDs do and all in the same direction to boot.

Steve Richards
Reply to  Bob boder
January 15, 2019 2:26 pm

See:
https://www.nist.gov/sites/default/files/documents/calibrations/sp819.pdf

Where NIST describes why a liquid in glass thermometer needs re-calibration….

All measurement sensors require calibration.

Paramenter
Reply to  Steve O
January 16, 2019 8:05 am

Hey Steve,

Your computation of the average temperature would be a precise measurement, but how close do you suppose my average for the year would be to your average for the year?

Do you mean comparing arithmetic mean, per each year, calculated directly from the 5-min sampled data with the ‘standard’ procedure of averaging per each year monthly averages that in turn come from averaging of daily midrange values (Tmin+Tmax)/2? I did a quick comparison for boulder, Colorado 2006-2017: as expected errors propagate from daily errors higher up, for instance for 2006 the error is 0.3 C; for 2010 the error is 0.35 C; for 2012 the error is 0.25 C. Full list below:

Year Error
2006 0.31
2007 0.26
2008 0.26
2009 0.23
2010 0.35
2011 0.17
2012 0.25
2013 0.14
2014 0.07
2015 0.19
2016 0.15
2017 0.16

Crispin in Waterloo but really in Beijing
Reply to  MarkW
January 14, 2019 3:52 pm

Global warming is about energy, not temperature per se, which is acknowledged above as a proxy for energy. Without considering the humidity and air pressure as well as its temperature at the time the instruments are read, the energy has not been ascertained, merely a proxy for it.

Is this really so difficult to comprehend?

The claim is simple enough: human activities are heating the atmosphere. The heat energy in the atmosphere is not well represented by thousands of temperature measurements. If the humidity were zero or fixed, it could be, but it is not.

The whole world could “heat up” and the temperature could go down if there is a positive water vapour feedback (currently not observed).

“Climate scientists” are wandering in the realm of thermodynamicists as effectively as they wandered about in the realm of statisticians. They must hand out hats with those degrees because they are always talking through them.

Nyquist is fundamental to signal analysis, and temperature is a signal, as is humidity and air pressure. With those three the enthalpy of an air parcel can be determined accurately. If there are trends in the results at the surface, we might project them, whether up or down.

The rest is, as that say, noise. Turn up the squelch and it disappears. Wouldn’t that be nice?

William Ward
Reply to  Crispin in Waterloo but really in Beijing
January 14, 2019 4:10 pm

Crispin you nailed it!!! Everything you said hits the bullseye!

ghl
Reply to  Crispin in Waterloo but really in Beijing
January 14, 2019 4:20 pm

Crispin
What you say is true Xcept it is the temperature that extinguishes species and magnifies wildfires. Your argument may be inverted to “energy interesting, but temperature kills.”

MarkW
Reply to  ghl
January 14, 2019 5:36 pm

There are no species being “extinguished” and there has been no increase in wildfires.

Gordon
Reply to  ghl
January 14, 2019 6:02 pm

Fires in California have been tragic because Moonbeam and his cronies do not know how to manage forests.

Jim Gorman
Reply to  ghl
January 15, 2019 6:13 am

You make a statement of fact that may or may not be true. Temperature for cold blooded animals and insects may be important but I suspect that it is different for different species. Same for warm blooded animals. For them I suspect humidity plays a role also, otherwise why do weather folks always dwell on “feels like” temperatures.

I think you are relying on too many studies that quote “climate change” as the reason for species population changes without doing the hard work to actually determine how temperatures changes actually affect a species. What studies have you read that have data that shows what higher temps and lower temps do to a species so that one can determine the “best range” of temperatures. Basically the ASSUMPTION is that higher temps are bad. You never see where lower temps are either good or bad.

Phoenix44
Reply to  ghl
January 15, 2019 8:25 am

How does a higher average annual global temperature magnify a local wildfire?

Ed Reid
Reply to  ghl
January 22, 2019 9:03 am

Gordon,

Also, apparently, because PG&E and its regulators do not understand how to manage and maintain electrical infrastructure.

Louis Hooffstetter
Reply to  Crispin in Waterloo but really in Beijing
January 14, 2019 5:51 pm

“The whole world could “heat up” and the temperature could go down if there is a positive water vapour feedback…”

Crispin, as usual you are absolutely correct. But I really wish you hadn’t given the climate alarmists another angle from which to argue. Now I expect to see one or more papers published in Nature Climate Change using this excuse to explain the pause or some other nonsense.

AndyHce
Reply to  Crispin in Waterloo but really in Beijing
January 14, 2019 5:54 pm

The claim is NOT that human activity is heating anything other than in urban environments, which is quite a different thing from heating the surface fluids generally. The claim is that CO2 is causing the heating by retaining solar energy, not human activity energy. Possibly the CO2 comes from human activity.

Alan Tomlin
Reply to  Crispin in Waterloo but really in Beijing
January 15, 2019 7:48 am

Love it Crispin….especially the “wandering and hats” paragraph…..

LdB
Reply to  MarkW
January 14, 2019 5:17 pm

It sort of works like Nick wants if the heat up and cool down behaviour is identical so it averages out (the first comment by vukcevic with the bucket is the same sort of idea).

Whether the behaviour up and down is symmetrical I have no idea has anyone measured. What is amusing is they do put the sites in a white box which inverts the argument … ask Nick why a white box 🙂

The point made however is correct they are trying to turn a temperature reading to a proxy for energy in the earth system balance which Nick does not address in his answer.

So I guess it comes down to are you trying to measure temperature for human use or trying to proxy energy .. make your choice.

Reply to  MarkW
January 14, 2019 8:54 pm

As for Blackville SC, where the discrepancy between daily average of 5-minute sampling and (Tmax+Tmin)/2 is greatest: The .24 degree/decade discrepancy is between 1.11 and 1.35 degrees C per decade. This is almost 22% overstatement of warming at the station with warming rate most overstated (in degree/decade terms) by using (Tmax+Tmin)/2.

Notably, all other stations mentioned in the Figure 7 list have lesser error upward in degree/decade terms – and some of these have negative errors in degree/decade trends from using (Tmax+Tmin)/2 instead of average of more than 2 samples per day. The average of the ones listed in Figure 7 is .066 degree/decade, and I suspect these are a minority of the stations in the US Climate Reference Network.

And, isn’t USCRN supposed to be a set of pristine weather stations to be used in opposition to USHCN and US stations in GHCN?

William Ward
Reply to  Donald L. Klipstein
January 14, 2019 9:34 pm

Hi Donald,

If I understand your comment correctly, let me clarify. My apology in advance if I’m misunderstanding you.

What I show in Fig 7 is not critical of USCRN. What it shows, using USCRN data, is how 2 competing methods work to deliver accurate trends. The “Nyquist-compliant” method uses all of the 5-minute samples over the period, which may be 12 years in some cases. The linear trend is calculated from this data. I’m considering it the reference standard. If the “historical method” of using max and min values worked correctly, without error, then when the trend generated from this method is subtracted from the reference there should be no error – the result should be zero. It is not zero. I refer to error as “bias” in that figure. Each station shows a bias. A positive sign means the method gives a trend warmer than the reference. A negative sign means the method gives a trend that is cooler than the reference. But to be clear, it is the historical method that is being exposed and criticized, not USCRN or its data. Maybe I should say I’m criticizing how that data is used. We have the potential to get good results from USCRN but the method used gives results worse than the reference.

Reply to  Donald L. Klipstein
January 15, 2019 7:31 am

“I suspect these are a minority of the stations in the US Climate Reference Network.”
I calculated below all stations in ConUS with 10 years of data. The Min/max showed less warming than the integrated. But I think it was just chance. The variability of these 10 year trends is large.

William Ward
Reply to  Nick Stokes
January 16, 2019 1:48 pm

Interesting Nick.

MarkW
Reply to  Nick Stokes
January 14, 2019 2:34 pm

Anyone who believes you can get an accurate estimate of the daily average from just the high and the low, has never been outside.

Reply to  MarkW
January 14, 2019 2:48 pm

“has never been outside”
You’d have to stay outside 24 hrs to get an accurate average. But the thing about min/max is that it is a measure. It might not be the measure one would ideally choose, but we have centuries of data in those terms. We have about 30 years of AWS data. So the question is not whether it is exactly the same as hourly integration, but whether it is an adequate measure for our needs.

Again, I looked at that over a period of three years for Boulder Colorado.

Reg Nelson
Reply to  Nick Stokes
January 14, 2019 3:46 pm

True but meaningless. We may have centuries of Tmin\Tmax temperature data, but we don’t have centuries of Global Tmin\Tmax temperature data. To suggest so is simply absurd. And even if we did, the Earth is 4.5 billions of years old — a few centuries is nothing, and to make inferences from such a minuscule data record is unscientific and deliberately misleading.

MarkW
Reply to  Nick Stokes
January 14, 2019 3:53 pm

Once again, Nick admits that the data they have isn’t fit for the purpose they are using it for, but they are going to go ahead and use it because it’s all they’ve got. As the article demonstrated, daily high and low is not adequate for the purpose of trying to tease a trend of only a few hundredths of a degree out of the record.

What a sad post to hang an entire ideology on.

Greg Cavanagh
Reply to  Nick Stokes
January 14, 2019 4:19 pm

The maximum and minimum temperature of the day is of interest, and are real values; but averaging those two values gives you a nonsense value that tells you nothing.

Clyde Spencer
Reply to  Greg Cavanagh
January 14, 2019 4:34 pm

Greg
I agree that that high and low are of interest and probably of more value than the mid-range value calculated from them. The information content is actually reduced by averaging. We don’t know whether an increasing mid-range value is the result of increasing highs, lows, or both. If both, we don’t know what proportion each contributes, which may have a bearing on the mechanism or process causing the change.

But, if the public were made aware that the primary result of “Global Warming” was milder Winters, not unbearable Summer heat waves, they might not get very excited.

LdB
Reply to  Nick Stokes
January 14, 2019 5:25 pm

It would be easy enough to cover Nick just make a couple of high precision sites on existing temp sites and integrate the energy into and out of the sites at millisecond resolution and check the temperature to energy proxy. You could also do the same with sites near oceans etc to get proper values for land/ocean interface when you try and blend stuff.

William Ward
Reply to  Nick Stokes
January 14, 2019 6:01 pm

Nick,

You allude to the problem. We have centuries of data and “we” feel compelled to use it, knowing it is bad. The question is often asked, “what are we supposed to do, nothing?”. As an engineer I can answer that question. Yes – do nothing with it. It is not suitable for any honest scientific work. No engineering company would ever commission a design, especially if public safety or corporate profit were at risk, with data as flawed as the instrumental record. If a bridge were to be designed with data this bad would you put your family under it while it is load tested? If an airplane was designed and built with data this flawed would you put your family on the first flight? We have known for a long time about a long list of problems with the record. Violating Nyquist is a pretty “hard” issue – not really something anyone can honestly dismiss. We have the theory. USCRN allows us to demonstrate the extent of the problem. The full range of error cannot be fully known but we are seeing magnitudes of error that stomp all over the claimed level of changes that are supposed to cause alarm.

Are there any engineers reading this who want to say they disagree? In other words, you think engineering projects are routinely undertaken with data as bad as the instrumental record.

Duane
Reply to  William Ward
January 14, 2019 6:55 pm

As an engineer I can say with confidence that we use the data we have, not the data we’d wish for in a perfect world. We are constantly faced with the constraints of inadequate data for purposes of design.

We deal with imperfect data through several methods:

1) Statistical analysis that allows us to determine variability, and confidence limits of limited data sets

2) We use analysis of systems and components and materials to determine modeled performance under various condititions, such as “finite element analysis”, that are subject to digital modeling, including extensive analysis of identified failure modes

3) We use “safety factors”, or what some may call “fudge factors” to account for residual uncertainty that otherwise is not subject to direct analysis

4) We attempt to identify, quantify, and account for measurement errors

5) We use extensive peer review design administration processes to avoid design errors or blunders in assessing all of the above.

The bottom line is that engineering is primarily a matter of risk analysis that is built upon the notion of imperfect and insufficient data and how to deal with that while developing economical, efficient designs.

In the real world you never have sufficient or good enough data. Engineers are trained to design stuff anyway.

Curious George
Reply to  William Ward
January 14, 2019 7:06 pm

William, it is Tmin/Tmax data. It is not “bad data”. And it is NOT “2 samples a day”; it is a true Tmin and Tmax for each 24-hour period, recorded maybe an hour from each other; it has nothing to do with Nyquist (regular sampling). And if you demand nonexistent data, maybe you are not a good engineer. An engineer works with the best available data and methods. For example, a bridge designer may be highly interested in the minimum and maximum temperatures.

William Ward
Reply to  William Ward
January 14, 2019 7:22 pm

Duane,

Although you seem to counter my statement, I think you actually agree with it. I asked if any engineering project uses data as bad as the instrumental record. I didn’t say or suggest the data is perfect.

I didn’t hear you say you use uncalibrated instruments and get mail clerks to run them with out training. You didn’t say you ignore quantization error and inflate precision.

You said you do examine the confidence limits of your data and add in guard bands and safety factors. You conduct design reviews and assess your risks. You model, emulate, simulate. Note stated but I assume you build prototypes and do functional testing and reliability testing, etc.

I’m not sure why you took a counter position as I’m not saying anything different. Are you saying you think the instrumental record and the way climate science uses the record matches your processes?

If climate science used the “bad” data they have and bound the error and stopped making up data and changing it continually decade after decade then they could approach what engineers do. But the data is still aliased. Would you design with data aliased as bad as the instrumental record?

William Ward
Reply to  William Ward
January 14, 2019 7:27 pm

Curious George,

A very accurately measured Tmax and Tmin are bad for calculating mean temperature. I’m sorry if you object to the word “bad” but you need to show how it is “good” and what it is “good” for. Since mean temperatures and temperatures trends are what drives climate science you need to show they are “good” for this purpose.

Menicholas
Reply to  William Ward
January 14, 2019 7:53 pm

“If a bridge were to be designed with data this bad would you put your family under it while it is load tested? ”

Apparently some people would, with the exception that they would test it with random strangers under it.

https://www.nbcmiami.com/news/local/Six-Updates-on-Bridge-Collapse-Investigation-492009331.html

Reply to  William Ward
January 14, 2019 8:34 pm

William
“No engineering company would ever commission a design, especially if public safety or corporate profit were at risk, with data as flawed as the instrumental record.”
The engineering project, in this case, is to put a whole lot of CO2 in the air, and see what happens. We work that out with the data we have. By all means cancel the project if you think the data is inadequate.

William Ward
Reply to  William Ward
January 14, 2019 9:38 pm

Menicholas – Did I hear them say that the bridges collapsed due to too much anthropogenic CO2 reacting with their concrete mix?

Phil
Reply to  William Ward
January 14, 2019 9:47 pm

@ Nick Stokes January 14, 2019 at 8:34 pm

… put a whole lot of CO2 in the air…

Actually only about 2 parts per million per year.

William Ward
Reply to  William Ward
January 14, 2019 9:52 pm

Nick said: “The engineering project, in this case, is to put a whole lot of CO2 in the air, and see what happens. We work that out with the data we have. By all means cancel the project if you think the data is inadequate.”

I see no evidence to fear CO2. But pumping CO2 is not the project. The “project” is pumping countless billions of taxpayer dollars into the research world in a Faustian Bargain that clubs the public into a political and social agenda based upon bad science – all the while claiming to be the champions, protectors and sole wielders of science. I’d cancel that one.

Nick, I don’t believe for a nano-second that politicians actually believe in the nonsense they pedal. Or if they do they are completely incompetent. How much would it cost the developed governments of the world to blanket the world with USCRN type stations? This could have been started in the 1980s or 1990’s for sure. With good data we can analyze both the Nyquist compliant way and the historical way. How much is the US spending on the new USS Gerald R Ford – next gen carrier with full task force and compliment of F35s, submarines and Aegis destroyers? $40B? How about annually to maintain it? How much are we wasting on redundant social programs? How much are we spending on a network that allows us to collect better data globally? Why aren’t you and other in the field demanding better instruments and data? Why, because the tall peg would get beaten down. No one wants to do anything except keep the money flowing and keep the nonsensical research pumping out day after day.

Ferdberple
Reply to  William Ward
January 14, 2019 11:02 pm

For example, a bridge designer may be highly interested in the minimum and maximum temperatures.
========
Correct. But the engineer would have almost no interest in (min+max)/2. Bridges don’t fail because of averages.

MarkW
Reply to  William Ward
January 15, 2019 10:21 am

As usual Nick’s best effort amounts to a pathetic attempt to change the subject.

We do have data regarding what happens to the globe when CO2 levels increase.
CO2 levels have been 15 to 20 times above what we have now in the past and nothing bad happen. Not only did nothing bad happen, life flourished.

Duane
Reply to  William Ward
January 15, 2019 10:57 am

William,

You asked if engineers use data as bad or as imperfect as the environmental temperature record. Civil engineers use exactly that environmental data as the basis for loads or effects on a wide variety of engineered systems. I”m a civil engineer. We use the existing environmental record for air and water temperatures, humidity, wind speeds and directions, rainfall (depths, durations, and frequency of storms), water flow rates and flood stages for rivers and channels and surface runoff, seismic data, aquifer depth data, water quality data, just to name a few of the very many types of environmental data routinely used just in civil engineering design.

Most of these environmental data have relatively short records of data (very frequently just a few decades worth), with frequently missing data, or questionable data.

Fortunately, a combination of government agencies like NOAA or USGS and private standards setting organizations like ASTM have invested many decades in data collection and analysis made readily available with pre-calculated statistical analyses.

So then we use all the various tools such as the ones I listed above to provide the safest yet still economical design we can produce.

If we refused to use or settle for “bad data” or incomplete data or questionable data, in many instances we’d have no data at all. The better the data we have, the better, as in safer and more economical, designs we can produce.

William Ward
Reply to  William Ward
January 15, 2019 11:11 pm

Duane,

You said: “You asked if engineers use data as bad or as imperfect as the environmental temperature record. Civil engineers use exactly that environmental data as the basis for loads or effects on a wide variety of engineered systems…”

Again, I really don’t see any disagreement with you… for some reason it feels like you have taken that position. Civil engineers have a lot of standards they adhere to and I doubt a 1C increase in global average temperature makes you rewrite your standards or run out and reinforce a dam. Right? Sure you use rainfall data to make decisions, but I’m sure there is a lot of careful margin built in. Right? But you would not specify a concrete slump or strength or size a critical beam with methods similar to those in the instrumental record would you? I won’t labor over this, but I think/hope we agree more than disagree. I suspect you took issue with the way I made my point and that is ok, but I’m not seeing anything of substance here that is in contention.

Frank
Reply to  William Ward
January 16, 2019 9:16 pm

William Ward wrote: “I see no evidence to fear CO2.”

Obviously William knows nothing about radiative transfer of heat – the process that removes heat from our planet. An average of 390 W/m2 of thermal IR is leaving the surface of the planet, but only 240 W/m2 is reaching space. GHGs are responsible for this 160 W/m2 reduction and the current relative warmth of this planet. However, there is no doubt that rising GHGs will someday be slowing down radiative cooling by another 5 W/m2.

We don’t need a Nyquist-compliant temperature record – or any temperature record at all – to know that some danger exists and that it is potentially non-trivial. Energy that doesn’t escape to space as fast as it arrives from the sun – a radiative imbalance – must cause warming somewhere. That is simply the law of conservation of energy.

Reply to  Nick Stokes
January 15, 2019 1:09 am

“But the thing about min/max is that it is a measure” Looking into goat entrails to ‘guess’ the future is also a measure. Really, Nick, you are funny. Really funny.

Editor
Reply to  MarkW
January 14, 2019 3:19 pm

MarkW, you raise an important point most people here seems to have missed since the reliance on just two data points a day barely cover anything of the day itself.

For a recorded high of 102 degrees F at 4:40 pm doesn’t tell us how long it was over 100 degrees F, how long it was over 95 degrees F, how long it was over 90 degrees F.

A 100 degree F day can actually be hotter than a 102 degree day when it reached 90 degrees F two hours earlier and 95 degrees 1 1/2 hours earlier than the 102 degrees F day did.

It is the total amount of heat of the whole sunny day into the late evening that counts the most, not the daily high that might have lasted just 5 minutes when it was recorded.

It was a LOT hotter in the early 1990’s in my area than now due to the summer heat persisting long into the night, when it used to stay over 90 degrees late as 10:00 pm., now it never does that anymore since then often dips below 90 by around 8:30 or so. The average high still about the same, even reached a RECORD high of 112 degrees F with additional days of 110, 111 and a couple of 109, just 4 years ago.

The nights no longer so hot as it used to be.

Reply to  Sunsettommy
January 14, 2019 5:38 pm

True. I live in a hot place, Phoenix, Arizona. Some years back I started recording and plotting temperature with a sample rate of 3 minutes. It is clear to see that the hottest day in terms or max T may not be the hottest day if one considers, for example, a day that is slightly cooler by say 2 deg F; but the high is reached earlier in the day and persists for a longer time. It’s easy to see graphically. Many of the hot days peak and then cool off.

AndyHce
Reply to  Sunsettommy
January 14, 2019 6:00 pm

That is regional weather. Here, last summer, it stayed hot late very often. It was still near 100 at midnight more than a few times. Of course, things could have been different on the next block.

William Ward
Reply to  Sunsettommy
January 14, 2019 6:11 pm

Sunsettommy:

Yep: it really is about the temperature-time product, if we are going to use temperature.

There is a difference between your pizza being in the 400F oven for 1 minute and 20 minutes.

Clyde: But I do agree with you. A better use of the record (although still not good) would be to work with the max and min temperatures independently (do not average them) and add error bars to each. When things like reading error, quantization error, UHI, etc are added in for an honest range, the problem is we get a range so wide that I doubt we can say we know what has or is happening. We just know things are happening in a large range.

When violating Nyquist is factored in, I don’t think we can even really say it has warmed since the end of the Little Ice Age. I’m NOT making the claim that it didn’t warm. I’m making no claims except that the data is so bad that we need to admit we really don’t know. We have anecdotes and they may be correct, but we don’t really have the science or data to back it up honestly.

A C Osborn
Reply to  William Ward
January 15, 2019 3:52 am

This.
The other point is that as Curious George said the design of the Max/Min Thermometer was to derive the max and min readings per day, not measure the heat content, or the daily variations, just the max and min.
So in that respect it is NOT a 2 sample frequency from hundreds of samples.
It has worked well and did it’s job.
The introduction of continuously reading Electronics has introduced it’s own Errors, some of which are even worse.
As has been shown in the Australian BOM records if the data is mis-handled, ie not averaging over a long enough period the new Electronic devices pick up transient peaks.
These peaks can have many causes but do not actually impact the overall temperature.
The other classic example of course is using Electronics at Airports where they don’t measure the Climate Temperature, but how many Jet Engines pass by.

Tom Abbott
Reply to  William Ward
January 15, 2019 10:36 am

“Clyde: But I do agree with you. A better use of the record (although still not good) would be to work with the max and min temperatures independently (do not average them) and add error bars to each.”

I think so, too. If we use TMax charts, the 1930’s show to be as warm or warmer than the 21st century. And this applies to all areas of the world.

TMax charts show the true global temperature profile: The 1930’s was as warm or warmer than subsequent years. Therefore, the Earth, in the 21st century is not experiencing unprecedented warming. It was as warm or wamer than today back in the 1930’s, worldwide.

The CAGW narrative is dead. There is no “hotter and hotter” if we go by Tmax charts.

Or, CAGW is also dead if we go by unmodified surface temperature charts, which also show the 1930’s as being as warm or warmer than subsequent years.

Greg F
Reply to  Nick Stokes
January 14, 2019 2:42 pm

The Theorem tells you that you can’t resolve frequencies beyond that limit.

It also tells you that frequencies above half the sampling rate will fold back to a frequency below half the sampling rate. A frequency at .6 the sample rate will appear as a frequency at .4 the sample rate.

It isn’t a communications channel.

Irrelevant. It is sampled time series data.

Reply to  Greg F
January 14, 2019 2:52 pm

“Irrelevant.”
Of course it is relevant. You have said that “A frequency at .6 the sample rate will appear as a frequency at .4 the sample rate.”. But the only result that matters is the monthly average. All these regular frequencies, aliased or not, make virtually zero contribution to that.

Greg F
Reply to  Nick Stokes
January 14, 2019 3:15 pm

But the only result that matters is the monthly average.

Average is a low pass filter in the frequency domain. If the signal is aliased the monthly average will be wrong.

Reply to  Greg F
January 14, 2019 3:23 pm

All the frequencies in Nyquist talk are diurnal or more. Yes, monthly average is a low pass filter, and they will all be virtually obliterated, aliased or not.

Plus the effect of the locking of sampling to diurnal. What component do you think could be aliased to low frequency?

Greg Cavanagh
Reply to  Greg F
January 14, 2019 4:24 pm

Play some music through that monthly low pass filter and the song Hey Jude would be a constant note of C. Is that useful?

Reply to  Greg F
January 14, 2019 5:22 pm

“would be a constant note of C”
Do, it would be silent, or a very soft rumble. There aren’t any low frequency processes there that we want to know about. But with temperature there are.

AndyHce
Reply to  Greg F
January 14, 2019 6:18 pm

Sampling theory says the data is exactly correct for the bandwidth chosen, not approximately correct, if the sample rate is as least twice the highest frequency. Aliasing occurs only when the input signal isn’t bandwidth limited to fit the chosen sample rate.

I think one has to determine the instantaneous rate of temperature change to arrive at the optimum sample rate if one wants to come up with the most correct number. I suspect that rate varies considerably in different environments. I’m not sure that the calculated average will still be any more meaningful but the actual trend of that number over time should be more accurate, which seems to be the main point of this article.

The trend is the most important claim in today’s version of climate and the most important number in projecting a forecast of likely future results. However, it still seems pretty meaningless in view of what the real world actually seems to do.

William Ward
Reply to  Greg F
January 14, 2019 6:54 pm

Nick said: “But the only result that matters is the monthly average.”

Reply: The data is aliased upon sampling at too low of a rate. You can do all of the work you want after the sampling but the aliasing is in there. You can’t get it out later. I replied earlier but maybe you didn’t get it yet.

You can design your system and filter before sampling according to your design. You can do this if your system is electronic. You can’t filter the reading of a thermometer read “by-eye”.

Reply to  Greg F
January 14, 2019 10:15 pm

” You can do all of the work you want after the sampling but the aliasing is in there. “
The process is linear and superposable. I set out the math here. Aliasing may shift one high frequency to another. But monthly averaging obliterates them all.

Johann Wundersamer
Reply to  Nick Stokes
January 14, 2019 2:59 pm

Thought the same, Nick -after all it’s not about “real” temperatures but to determine the difference between yesterday’s measurement and today’s measurement.

William Ward
Reply to  Johann Wundersamer
January 14, 2019 6:58 pm

Johann,

We do get quoted annual record temperatures don’t we? I hope you know now that not one of those record temperature years is correct.

Your comment to Nick is about trends. I address trends. They do not escape aliasing.

Reply to  William Ward
January 14, 2019 10:18 pm

“I address trends.”
To no effect. You show that you get different results, sometimes higher, sometimes lower. But it’s all well within the expected variation. There is no evidence of systematic bias.

Scott W Bennett
Reply to  Nick Stokes
January 15, 2019 6:33 am

“There is no evidence of systematic bias. – Nick Stokes”

On the contrary, It is discussed in the literature that the bias between true monthly mean temperature (Td0) – defined as the intergral of the continuous temperature measurements in a month – and the monthly average of Tmean (Td1) is very large in some places and cannot be ignored. (Brooks, 1921; Conner and Foster, 2008; Jones et al., 1999)

The WMO(2018) say that the best statistical approximation of average daily temperature is based on the integration of continuous observations over a 24-hour period; the higher the frequency of observations, the more accurate the average; as the head post suggested!

“Td1 may exaggerate the spatial heterogeneities compared with Td0, because the impact of a variety of geographic (e.g. elevation) and transient (e.g. cloud cover) factors is greater on Tmax and Tmin (and hence in Td1) than that on the hourly averaged mean temperature, Td0 (Zeng and Wang, 2012).

Wang (2014) compared the multiyear averages of bias between Td1 and Td0 during cold seasons and warm seasons and found that the multi‐year mean bias during cold seasons in arid or semi‐arid regions could be as large as 1 °C.

WMO (1983) suggested that it is advisable to use a true mean or a corrected value to correspond to a mean of 24 observations a day. “Zeng and Wang (2012) argued from scientific, technological and historical perspectives that it is time to use the true monthly mean temperature in observations and model outputs.”

The WMO(2018) have suggested that Td1 is the least useful calculation available if attempting to improve the understanding of the climate of a particular country!

WMO Guide to Climatological Practices 1983, 2018 editions
Brooks C. 1921. True mean temperature. Mon. Weather Rev. 49: 226–229, doi: 10.1175/1520-0493(1921)492.0.CO;2.
Conner G, Foster S. 2008. Searching for the daily mean temperature. In 17th Conference on Applied Climatology, New Orleans, Louisiana.
Jones PD, New M, Parker DE, Martin S, Rigor IG. 1999. Surface air temperature and its changes over the past 150 years. Rev. Geophys. 37: 173–199, doi: 10.1029/1999RG900002.
Li, Z., K. Wang, C. Zhou, and L. Wang, 2016: Modelling the true monthly mean temperature from continuous measurements over global land. Int. J. Climatol., 36, 2103–2110, https://doi.org/10.1002/joc.4445.
Wang K. 2014. Sampling biases in datasets of historical mean air temperature over land. Sci. Rep. 4: 4637, doi: 10.1038/srep04637.
Zeng X, Wang A. 2012. What is monthly mean land surface air temperature? Eos Trans. AGU 93: 156, doi: 10.1029/2012EO150006.

Reply to  Nick Stokes
January 15, 2019 7:24 am

” It is discussed in the literature that the bias between true monthly mean temperature (Td0)”
As is clear from context, I am saying that there is no systematic bias between trends. If you think there is, please say which way it goes.

Scott W Bennett
Reply to  Nick Stokes
January 15, 2019 7:02 pm

“As is clear from context, I am saying that there is no systematic bias between trends. If you think there is, please say which way it goes. – Nick Stokes”

Thorne et al. (2016) found a consistent overestimation of temperature by the traditional method [(Tmax + Tmin)/2], for the CONUS:

Moreover, the traditional method overestimates the daily average temperature at 134 stations (62.3%) underestimates it at 76 stations (35.4%), and shows no difference at only 5 stations (2.3%)…On average, the traditional method overestimates the daily average temperature compared to hourly averaging by approximately 0.16°F, though there is strong spatial variability*.

The explanation for the long-term difference between the two methods is the underlying assumption for the twice-daily method that the diurnal curve of temperature is symmetrical. In particular, the Yule–Kendall index is positive for all 215 CONUS stations, indicating that the daily temperature curve is right skewed; that is, more hourly observations occur near the bottom of the distribution of hourly temperatures (i.e., around Tmin) than near the top of the distribution (around Tmax). – Thorne et al. 2016*

It is interesting to note from that study that spatial patterns in the annually averaged differences between the temperature-averaging methods are readily apparent. The regions of greatest difference between the two methods resemble previously defined climatic zones in the CONUS.

What concerns me most, is that this study found that the shape of the daily temperature curve was changing, such that more hours per day were spent closer to Tmin than Tmax during the (2001-15) period versus the base period (1981-2010), which doesn’t bode well for a changeover to the hourly method because it will have the effect of masking the old errors and sexing up any warming! ;-(

*Thorne, P. W., and Coauthors, 2016: Reassessing changes in diurnal temperature range: A new data set and characterization of data biases. J. Geophys. Res. Atmos., 121, 5115–5137, https://doi.org/10.1002/2015JD024583.

Scott W Bennett
Reply to  Nick Stokes
January 15, 2019 7:22 pm

“As is clear from context, I am saying that there is no systematic bias between trends. If you think there is, please say which way it goes. – Nick Stokes”

Wang (2014)* analyzed approximately 5600 weather stations globally from the NCDC and found an average difference between the two temperature-averaging methods of 0.2°C, with the traditional method overestimating the hourly average temperature. They found that asymmetry in the daily temperature curve resulted in a systematic bias. And also that Tmean resulted in a more random sampling bias by under sampling weather events such as frontal passages.

Wang, K., 2014: Sampling biases in datasets of historical mean air temperature over land. Sci. Rep., 4, 4637, https://doi.org/10.1038/srep04637

Reply to  Nick Stokes
January 15, 2019 8:44 pm

Scott W Bennett January 15, 2019 at 7:02 pm

“As is clear from context, I am saying that there is no systematic bias between trends. If you think there is, please say which way it goes. – Nick Stokes”

Thorne et al. (2016) found a consistent overestimation of temperature by the traditional method [(Tmax + Tmin)/2], for the CONUS

Scott, it seems you’re not discussing what Nick is discussing. He’s discussing trends, you’re discussing values.

The traditional method (max+min/2) definitely overestimates the true average value. However, as near as I can tell, it doesn’t make any difference in the trends. I ran the real and traditional trends on 30 US stations and found no significant difference. I also investigated the effect of adding random errors to the real data and found no difference in the trends.

I’m still looking, but haven’t found any evidence that there is a systematic effect on the trends.

w.

Phil
Reply to  Nick Stokes
January 15, 2019 9:26 pm

@ Scott W Bennett on January 15, 2019 at 7:02 pm:

Your quotes are actually from Bernhardt, et al, 2018.

Bernhardt, J., A.M. Carleton, and C. LaMagna, 2018: A Comparison of Daily Temperature-Averaging Methods: Spatial Variability and Recent Change for the CONUS. J. Climate, 31, 979–996, https://doi.org/10.1175/JCLI-D-17-0089.1

https://journals.ametsoc.org/doi/pdf/10.1175/JCLI-D-17-0089.1

S W Bennett
Reply to  Nick Stokes
January 15, 2019 10:01 pm

Willis, I’m not sure if you read my post immediately below about Wang(2014):

“They found that asymmetry in the daily temperature curve resulted in a systematic bias.”

Now I’m confused because I’m not sure what you mean by “trends” here.

I thought it was clear that several studies found long term global and regional systematic bias.

Perhaps I might have obscured the point that trends will be biased because the shape of the daily temperature curve was found not just to be skewed but also changing.

My understanding is that this will create a spurious trend in either method, unless explicitly teased out (By comparison with changes in humidity):

“Stations in the southeast CONUS experience more time in the lowest quarter of their daily temperature distribution due to higher amounts of atmospheric moisture (e.g., given by the specific humidity) and the fact that moister air warms more slowly than drier air.*”

For example, Thorne et al. (2016) using half ASOS and half manual observations – found a considerable difference in the two temperature-averaging methods in the cold-season results between Wang’s – ASOS only – observations but only a negligible difference for the warm season.

That’s me for now, I’ve exhausted my contribution to this subject! 😉

*Thorne, P. W., and Coauthors, 2016: Reassessing changes in diurnal temperature range: A new data set and characterization of data biases. J. Geophys. Res. Atmos., 121, 5115–5137, https://doi.org/10.1002/2015JD024583.

Scott W Bennett
Reply to  Nick Stokes
January 15, 2019 10:20 pm

Phil,

You are right! My mistake, referencing is not my best skill, despite appearances. In my defence there is a cross-reference that I’ve obviously confused. I have all these notes and ideas in my head and it is bloody hard to go back and remember where I got them all from. I know, I should be more rigorous but I was trying to get the ideas across as informally as possible. I figure in this day and age it is almost superfluous as any diligent reader will quickly google it.

cheers,

Scott

William Ward
Reply to  Nick Stokes
January 16, 2019 1:53 pm

The trend difference is the simple accumulation of the error. As discussed in other posts, there is a nature of the error that has a quality approaching random. I don’t think it is random – but I’ll let someone more knowledgeable try to measure that and report. It still appears to me that we see the accumulated error in the trends.

Frederick Michael
Reply to  Nick Stokes
January 14, 2019 3:06 pm

Nick is right. This paper has fundamental conceptual errors.

It is obviously wrong to claim that averaging the daily high and the low is WORSE than averaging two samples 12 hours apart. The high and the low are based on a huge number of “samples.” To treat each one as if it’s a single sample is not valid.

Let me say this a different way. The sampling theorem does not apply to statistics not based on samples. A mercury max/min thermometer uses an effectively infinite number of samples. In fact, it’s analogue; there are no samples at all.

MarkW
Reply to  Frederick Michael
January 14, 2019 3:58 pm

About as far from correct as one can be and still be speaking English.
No, a mercury max/min thermometer is not taking an infinite number of samples, it takes two samples, the high whenever that occurred and the low, whenever that occurred.
If it were taking an infinite number of samples you could reconstruct the temperature profile of the previous day, second by second.
The best you can say is that while it’s constantly taking samples, it only stores and reports two of those samples.

Frederick Michael
Reply to  MarkW
January 14, 2019 6:15 pm

Let me repeat, “It is obviously wrong to claim that averaging the daily high and low is WORSE than averaging two samples 12 hours apart.”

Is this, or is it not, obvious to you?

Ever used the sampling theorem at work?

I have.

Phil
Reply to  Frederick Michael
January 14, 2019 10:00 pm

@Frederick Michael

You are ignoring the shape of the curve. The shape of the daily temperature curve varies from one day to the next and is what is causing the error in the daily “average.” As you know, the “average” of a time series is not an average: it is a smooth or filter. Information is being discarded by the max-min thermometer. Without knowing the shape of the curve (which implies knowing exactly when the max and the min happened), the real daily smooth cannot be known. It is known that a sine wave is a bad model to assume for the daily temperature curve.

Frederick Michael
Reply to  Frederick Michael
January 15, 2019 8:30 am

Phil – The error you cite is real. That’s the problem with using (min+max)/2. It is not a perfect method.

But that has absolutely nothing to do with the sampling theorem or aliasing.

MarkW
Reply to  Frederick Michael
January 15, 2019 10:25 am

Nice dodge there.
From the rest of my comment it is obvious I was addressing your ridiculous claims that a high and a low reading is the result of an infinite number of samples.

Hugs
Reply to  Frederick Michael
January 15, 2019 11:48 am

Mercury minmax can well be considered as having a very large sampling frequency.

But in reality, very short pulses of changing temperature don’t have time to fully express in the reading. So an electronic device may react faster and have practically much more volatility. How to mimic the mercury minmax with electronic thermometer sampling the temp is a hard question not put enough effort on.

Nick says the anomaly trend is maybe not affected. Well yeah, science is not about believing stuff, it is about proving it. For me it is clear that mercury minmax can’t be compared to different electronic devices, due to a number of reasons. The best reason is the most used thermometers are simply not planned to detect changes of 0.001 degrees during a year.

The record is contaminated every time something happens near the measuring point. It is contaminated by sheltering changes, both abrupt and slow. It is contaminated by some electronic jitter and smoothing protocols.

So far from certain, but then, you can argue we should play safe. What exactly is playing safe, is disagreed, notable opinions coming from Curry, Pielke Jr, and Lomborg.

Gary Pearse
Reply to  Frederick Michael
January 14, 2019 4:03 pm

Nick would be right if the daily profile of continuous sampling were to be the same each day. One day is not a sufficien exploration. 1st commenter vukevich had the best idea which is to measure the heat in a bucket of water each day at, say 2pm (rather a fluid that doesn’t freeze). This might be automated by encapsulating the fluid with an air cupola on top fited with a pressure guage “thermometer”.

LdB
Reply to  Frederick Michael
January 14, 2019 5:39 pm

You have an average temperature but it bears no relationship to earth energy balance which you are trying to work out in climate science 🙂

To understand why consider a cold rainy day your temperature all day could have been 20deg C the cloud clears for a brief spell and the temperature rises to 24 deg C .. so that is you max. Your average will be MAX-MIN/2 so it inflates your average …. you think there is a lot more energy that day than there is. Remember at the end of the day what they use this temperature average for is to create a radiative forcing so they are proxying earth energy budget.

Matt Schilling
Reply to  LdB
January 15, 2019 7:45 am

I see this regularly where I live in upstate NY, especially on predominantly cloudy days: The weather report calls for a certain high temp and we get nowhere near it all day long. Then, late afternoon, the clouds break, and the temp moves up quickly to the predicted high, only to stay there for a short time before dropping back down as the sun sets.
We, in fact, hit the high temp called for, but we spent the majority of the day in noticeably cooler weather. The high temp for the day was almost meaningless, as the day was dominated by cooler temps.

William Ward
Reply to  Frederick Michael
January 14, 2019 6:41 pm

Frederick Michael,

Please show the math or conceptual errors. Be advised you are fighting proven mathematics of signal analysis. You challenge is to show 1) that there is no energy at 1-cycle/day, 2-cycles/day or 3-cycles/day. Good luck showing there is no energy in the signal at 1-cycle/day. Or you need to show how 1 and 3-cycles/day do not alias to 1-cycle/day. Or you need to prove why Nyquist was wrong. Or show that every electronic device ever made that converts analog and digital data and has followed Nyquist, has done so needlessly.

A mercury max/min thermometer only delivers 2-samples/day. If you think the number is infinite then you wouldn’t mind giving me an example where you get say … 3-samples/day. Give me the 3 values you get please. You will quickly see that you are mistaken. If you have a high quality, calibrated max/min thermometer next to a USCRN site and compare your 2 values (samples) after a full day (properly sampled with no TOBS), and compare it to the 288-samples that day, you will find the max and min extracted from the 288-samples match what your max/min device gives. Max/min thermometers give you 2 samples.

You can play with USCRN data for yourself and show by example that the 2 samples taken at midnight and noon (or any 2 times separated by 12 hours) will usually give a lower error mean than the average of max and min. With clock jitter you get into some strange effects. It depends upon when the samples land in time relative to the spectral content for the day.

Frederick Michael
Reply to  William Ward
January 14, 2019 7:59 pm

Thanks for your response.

The conceptual error is describing the min and the max as samples. While the (min+max)/2 method is less than perfect, it’s the right approach, given older technology. Of course, it is worse than a true average.

I do not believe that there’s no energy at frequencies higher than 1 cycle per day. Of course there is. But we are interested in a low pass filtered result, so the “energy” at higher frequencies doesn’t somehow contribute to the average temperature we seek. The min & max act as low pass filters and are appropriate.

Still, your empirical point overrules any theoretical argument I can make. I will play with the USCRN data you link to above. If it shows greater error from averaging min and max than averaging two individual samples 12 hours apart, I will gladly concede,

after changing my underwear.

Frederick Michael
Reply to  Frederick Michael
January 14, 2019 8:04 pm

Dang. Looks like I have to wait for the shutdown to end before checking the data.

Kneel
Reply to  Frederick Michael
January 14, 2019 8:23 pm

“The min & max act as low pass filters and are appropriate.”

Not even wrong.
This does NOT act as a low pass filter AT ALL. The maths of calculating the average from the full 288 samples is a low-pass filter, min and max are not.

“…the “energy” at higher frequencies doesn’t somehow contribute to the average temperature we seek.”

Yes it does – by aliasing, and that is the point.

Dave Fair
Reply to  Frederick Michael
January 14, 2019 9:14 pm

Comparing USCRN stations’ trends with the others’ over the same time period would yield useful information.

William Ward
Reply to  Frederick Michael
January 14, 2019 10:10 pm

Frederick Michael,

Your underware?!! Now that is TMI (too much information)! LOL! Thanks for the laugh.

Here is what you are missing: once you sample you have to deal with what has aliased. So if you are interested in a low pass filtered result then you need to low-pass filter before you sample. Not after – unless you sample without aliasing. If you sample without aliasing then you have all the signal has to offer and you can process it in the digital domain just as you would process it in the analog domain before you sampled it. You can also (and this is going to drive Nick crazy!) convert the sample from digital back to analog without losing any of the original data!

When we look at a thermometer with our eyes we can not filter anything. Even a max/min thermometer. There is really no practical way to filter. Filtering has to be done electronically before sampling. Then the signal can be filtered before sampling and you can use 2 samples/day if you filter appropriately.

I show in my paper what happens to trends with aliasing. How do you explain the trend bias/error otherwise? If Nick were right that you can just “extract the monthly” data despite the aliasing then you would not have the trend error.

Some of you are really holding on tight to this, but do so at your own detriment. You need to show how Nyquist doesn’t apply. You need to show how the content at 1, 2 and 3 cycles/day doesn’t alias your signal trend and daily signal. I give you the graph (full paper). Just show me how it is wrong. Otherwise use it to make your work better. I didn’t invent it, I’m just pointing it out.

Reply to  Frederick Michael
January 14, 2019 10:27 pm

“Or you need to show how 1 and 3-cycles/day do not alias to 1-cycle/day. “
Well, they can’t. You get sum and difference frequencies.

You have never dealt with the issue of locked frequencies. Sampling at regular sub-day frequencies cannot alias diurnal harmonics to low frequency values. It can go to zero, but that just reproduces the known resolution variation.

Paramenter
Reply to  Frederick Michael
January 15, 2019 3:21 am

The conceptual error is describing the min and the max as samples.

What’s wrong with that? Granted – those samples are taken usually not evenly spaced. Furthermore you can have one daily max with the same value or two, or three, we don’t know. But irregularity of sampling just adds additional burden to the original problem; not alleviates such problem.

A C Osborn
Reply to  Frederick Michael
January 15, 2019 4:07 am

The ERROR is combining them to obtain an AVERAGE of them per day.
Average them individually per week, per month, per year to obtain useful comparisions.
It is patently obvious that taking the average of the 2 readings is only a very poor approximation of the Real Average each day.
But even using the Area under the curve of all the readings of an Electronic device still does not tell you the heat content of the day unless you also have the Humidity readings as well.

Jim Gorman
Reply to  Frederick Michael
January 15, 2019 10:56 am

A C Osborn –> Bingo1 And you know what? They do have daily min, max, and avg humidity data that could be used. Go to your local NWS and you can find the daily history of humidity, it is there. I’m not a data expert on weather data, so I don’t know how long humidity has been stored and collected for, but it is there now.

I’m working on an essay about uncertainty of the averages and something I notice here is that this paper doesn’t address the errors in measurements, nor should it necessarily do so. However, those also contribute to the errors involved, to the point where most, if not all, of the trends are within the error range and should be treated as noise, not real temperature changes.

In other words, as the article says, the monthly averages are not suitable for purpose.

Clyde Spencer
Reply to  Nick Stokes
January 14, 2019 4:02 pm

Stokes,
You said, “It isn’t a communications channel.” You are stating the obvious and what is a non sequitur.

The important thing, which I suspect you are capable of understanding, but choose to ignore, is that Ward has demonstrated that the true, daily mean-temperature can and does vary from the mid-range daily temperature, and the difference is determined by the shape of the temperature time-series. In other words, the phase components of the various frequencies shape the time-series envelope and strongly effect the accuracy of the mid-range compared to the true mean. What can be concluded from this is that a trend in the change in the shapes of the envelopes can create the appearance of a temperature change in the mean that may not be valid.

Yes, I know that historically all we have to work with is two daily temperatures. But, one shouldn’t be quick to assign unwarranted accuracy and precision to data that have been demonstrated to be unfit for purpose. The mid-range value is a measure of central tendency essentially equivalent to a degenerate median. As such, it may give us some insight on the tendency of long-term temperature changes. However, an honest broker would acknowledge the shortcomings and not claim accuracy and precision of mid-range averages that have little value for estimates of accumulating energy.

Gary Pearse
Reply to  Clyde Spencer
January 14, 2019 4:45 pm

Clyde, I take your point on the invalidity of such reported accuracy of the measurements. But, since we are trying to get an early warning system for disasterous warming, it isn’t necessary to report in hundredths of a degree. I would take advantage of polar enhancement of temperature, what, 3x the avg global temp change? Put a dozen recording thermometers in each polar region and average the readings.

Ive argued before that measuring sea level in millimeters a year and even adding corrections is ridiculous if what we are worried about is sea level rise of a meter or three a century. Tide guages are sufficient and even yardsticks would suffice – hey, ax handles are good enough measuring instruments if we are talking about the Westside Hwy in Manhattan going under 10ft of water by 2008. All this baloney about accuracy to tenths and hundredths is part of the propaganda to make people feel the scientists must he right if they can measure things to a gnat hair.

Another bit of agitprop is TOBS and the station moves. Yeah there has been a bonified reason for doing this – habitat encroachment, different sampling times etc, but it also presents another ‘degree of freedom’ that might be employed by charlatans to move, remove, and adjust stations that have been running too cool, say, and to leave ones running too hot alone except for here and there to look unbiased. They did deep six a preponderance of rural stations in the “Great Thermometer Extinction Event^тм” in the USA over the past two decades. I hate myself for coming to think this way, but after the ham-handed political adjustments to the record by Hansen and Karl on the eve of their retirements, all the goodwill enjoyed by scientists earlier times has been used up and a buyer beware mood has set in re these rent seekers.

kevink
Reply to  Nick Stokes
January 14, 2019 4:18 pm

Nick Stokes wrote

“The Theorem tells you that you can’t resolve frequencies beyond that limit. But we aren’t trying to resolve high frequencies. We are trying to get a monthly average. It isn’t a communications channel.”

No, the Theorem tells you that signal frequencies above 1/2 the sample rate WILL BE CONFUSED WITH (aliased into) frequencies below 1/2 the sample rate. This adds errors to the data.

Mr. Ward is quite correct.

Cheers, Kevin

Reply to  kevink
January 14, 2019 4:48 pm

“This adds errors to the data.”
How does aliasing sub-diurnal frequencies affect the monthly average?

William Ward
Reply to  Nick Stokes
January 14, 2019 7:09 pm

Nick – I recommend you read the Full Paper. The first 11-12 pages I go over how the aliasing happens, with graphics, to illustrate how the frequency content at/near 2-cycles/day aliases to the trends.

William Ward
Reply to  kevink
January 16, 2019 2:13 pm

I appreciate your comments KevinK.

William Ward
Reply to  Nick Stokes
January 14, 2019 5:46 pm

Well hello Nick! Happy New Year. I see you have 1) not taken the time to learn signal analysis since we last spoke and 2) didn’t read either my short or full paper as you replied seconds/minutes after the link was live.

Nick said: “The Theorem tells you that you can’t resolve frequencies beyond that limit. But we aren’t trying to resolve high frequencies. We are trying to get a monthly average. It isn’t a communications channel.”
Well hello Nick! Happy New Year. I see you have 1) not taken the time to learn signal analysis since we last spoke and 2) didn’t read either my short or full paper as you replied seconds/minutes after the link was live.

Nick said: “The Theorem tells you that you can’t resolve frequencies beyond that limit. But we aren’t trying to resolve high frequencies. We are trying to get a monthly average. It isn’t a communications channel.”

My reply: No, you don’t understand sampling it seems. First, you CAN NOT (sorry to shout) separate frequencies when sampling unless you first filter out the frequencies you don’t want prior to sampling. If you ignore higher frequencies because you are not interested in them and you sample anyway, disregarding Nyquist, then those frequencies you don’t care about come crashing down on the information you do care about! There is no UNDO button for aliasing. There is no magic-super-special-climate-sciencey algorithm you can run to undo the damage if you alias. We are talking about signal analysis 101, first week of class stuff here. I have no idea why you bring up a communication channel. We are talking about a signal. The theorem doesn’t care what the signal is.

Nick said: “As for aliasing, there are no frequencies to alias. The sampling rate is locked to the diurnal frequency.”

My reply: Get the USCRN data for Cordova AK and take a look at the data for Nov 11, 2017. I use this in my paper, so I recommend we try that. Run an FFT on a day’s worth of data. Or you can run it for many days or many months or years. As I explain in my FULL paper, the frequency content down around 0-cycles/day is the very long term trend signal. [Side note: Electrical engineers usually use Hertz (Hz) to discuss frequency. For atmospheric air temperature the signals are relatively slow compared to most things in electronics. So, we would actually use micro-Hertz (uHz). This is awkward and unintuitive. So, I’m using cycles/day or samples/day. As long as we stay consistent it works.] The 10-year and 1-year signals are down very close to 0-cycles/day. So, if you want to know what can alias your long term signals or trends then you look at the spectral content of the signal at near 2-cycles/day if we are using 2-samples/day. Any energy in our signal at this frequency will come crashing down on your trends! If the amplitude of the content at 2-cycles/day is low, then the impact to trends is likely low – but phase of the signal also matters. The phase relationship between the 0-cycle and 2-cycle content will determine just how additive or subtractive the aliasing is. Figure 8 of my FULL paper shows how this content aliases in a graphical format. [Note: Don’t get figures between the full and short versions of the paper confused. The Full paper is a superset of the short paper published here.] Furthermore, at 2-samples/day, the content at 1-cycle/day and 3-cycles/day will alias to the 1-cycle/day (daily) signal – this affects daily mean values.

See image here: https://imgur.com/xaqieor

This image was not intended for public consumption, as it has some issues, but I can use it to illustrate the point. Feel free to do your own FFT. It shows an FFT for Cordova AK for Nov 11, 2017. This is the same signal I use in Figs 10 and 12 in the Full paper and Figs 1 and 3 in the WUWT paper. Note that the temperature signal looks more like a square wave than a sinusoid. Notice the many sharp transitions. These all equate to higher frequency content. The FFT image shows the signal being sampled and the positive spectral image at 12-samples/day. You can see the overlap of content. The overlap is aliasing! If the sampling were shown at 2-samples/day, then the green image would be almost on top of the blue – the aliasing would be even greater, and more aliasing would occur to the daily mean and long term trend. The frequency content is there, and the aliasing is there. You should study the frequency content of more signals to see the variation that exists. It is not easy to visually ascertain the extent that the aliasing shows up in the time domain – in the mean and trend values. The means and trends need to be calculated. The frequency charts just prove what is going on, in full compliance with the theorem.

The sampling is not locked to the diurnal frequency. Higher frequency events, like cloud cover changes, precipitation changes, moving fronts, etc., all contribute to when the max and min happen. Using max/min does absolutely give you 2-samples/day, just with what is equivalent to “clock jitter”. The fact that there is error in the periodic nature of the “clock” when max and min are the samples does not invalidate Nyquist. But the conditions violate Nyquist, and this produces error. (Summary: Nyquist not invalidated. Violated.]

Nick said: “As for the examples, Fig 1 is just 1 day. 1 period. You can’t demonstrate Nyquist, aliasing or whatever with one period.”

My reply: Of course, you can. What uses Nyquist that everyone can relate with? Digital music. Is there a limit to the length of a digitally sampled song? No. Can a short clip of a digitally sampled song contain aliasing? Of course it can. A signal is a signal.

Nick said: “Fig 2 shows the effect of sampling at different resolution. And the differences will be due to the phase (unspecified) at which sampling was done. But the min/max is not part of that sequence. It occurs at times of day depending on the signal itself.”

My reply: I’d like to see your dissertation on that (differences due to phase at which sampling was done). My Figure 3 (WUWT/Short paper) shows graphically what happens when the sample rate is decreased. It is visually obvious that much content is being missed as the sample rate declines. This gives an intuitive feel for what is happening, but Nyquist explains the theory. The Min and Max ARE a part of the samples. The Min and Max are selected from the 288-samples. If there were just 2 clock pulses corresponding to those 2 samples then you would get just those 2 values. Clock jitter. You can see that in this case the jitter is very destructive. A much better result would have been achieved just by sampling at midnight and noon!

Regarding your study on TOBS, I read it multiple times long ago. I think it is a very good study! It explains TOBS well. But TOBS has NOTHING to do with this issue. When you sample according to Nyquist you are free to redefine your day as much as you like, and it will never bite you. No TOBS ever! You will just get the correct mean for the day as defined.

Nick, signal analysis is not a controversial subject. It is used for literally every kind of technology we all enjoy and depend upon today. There are no special carve-outs or waivers for Nyquist. It rules. Violate it and the effects are well defined. We have known about it for 80+ years. What should be controversial is climate “science” not knowing about it or outright dismissing it. I’m not sure what the reasons are, and I don’t care. I’m tired of Bullsnot data being shoved on the public as evidence of some imaginary crisis. The use of the instrumental record needs to be shot in the head (figuratively speaking). It is a dumpster fire and an embarrassment that climate science propagates upon its foundation.

William Ward
Reply to  William Ward
January 14, 2019 7:04 pm

Mod – I “fat fingered” a cut-and-paste in my reply above to Nick where I start: “Well hello Nick!”

If I can get one free mercy edit of my post I would appreciate it. It should be obvious that a paragraph got copied over top of another and thus duplicated.

Sorry to Nick and others who may be trying to read it! I’m not sure how this works, but if a moderator can do a quick repair and then remove this request (or not). Thanks and my apology!

[edited ~ctm]

Reply to  William Ward
January 14, 2019 9:06 pm

William,
“If you ignore higher frequencies because you are not interested in them and you sample anyway, disregarding Nyquist, then those frequencies you don’t care about come crashing down on the information you do care about!”

No, they don’t. They can’t, to the monthly average, because of attenuation. Here is the math:

Suppose we have a frequency component exp(2πift), where f is the frequency in 1/hour, t time. Sampled values are at ft=a*n, where a is a sampling ratio (.5 the Nyquist value) and n integer. The average over 1 month is (sum Σ n=0:N, N=720*f/a, 720 hrs in month)

1/N*Σ exp(2πia*n) = (1/N)(1-exp(2πia*(N+1)))/(1-exp(2πia))

It’s linear, so that is the contribution of that frequency to the whole average. Now for close adherence to Nyquist a is small, the last term is approx i/(2πa), and the whole is an attenuated but accurate version of the integral. As a rises to .5 and beyond, it is attenuated but inaccurate. But it doesn’t matter, because the attenuation factor is about 1/(2πaN) = 1/2π/f/720. f is the frequency of the component that you think might be harmed by aliasing, less than 1, it seems. So the attenuation, relative to approx monthly frequency is 3 or (much) more orders of magnitude. It just doesn’t matter if these terms are incorrectly averaged.

William Ward
Reply to  Nick Stokes
January 14, 2019 10:25 pm

Nick – on this you are hopelessly stubborn. See Fig 4 from the short/WUWT version. It shows the daily mean error for every day in the year. Aren’t months made up from days? How can the mean be wrong by +/- 1, 2, 3, or 4C every day of the year and somehow magically the months are all just fine? Did you run an FFT? Do you see content at and around 2-cycles per day? Did you see Fig 8 of the full paper? Do you see how the aliasing works to affect the trends (content near 0-cycles/day)? Have you looked at any USCRN station data and compared the 2 competing methods for monthly mean (or median)? Did you try to calculate any long-term trends using the 2 methods?

Reply to  William Ward
January 14, 2019 10:57 pm

“Have you looked at any USCRN station data and compared the 2 competing methods for monthly mean (or median)? Did you try to calculate any long-term trends using the 2 methods?”
Yes, right here. And the point is, the results differ, but well within the expected variation for the trend of such a short period. There is no statistical difference from which you can make deductions.

“Do you see content at and around 2-cycles per day?”
I’m sure there is. But what is left after 6 db/octave attenuation? about 1/1440π.

Reply to  William Ward
January 14, 2019 10:58 pm

“1/1440π”
Oops, 1/60π. Still tiny.

Reply to  William Ward
January 15, 2019 3:07 am

“Did you try to calculate any long-term trends using the 2 methods?”
More results here.

A C Osborn
Reply to  William Ward
January 15, 2019 4:18 am

As I said up thread the problem is taking an average at all, it is only a poor approximation.
There is nothing wrong with the Max/Min readings of themselves, it is what you try and do with them that is wrong.
First of all we have the problem of the Nomenclature used here, the AVERAGE of the 2 readings is 100% accurate.
But the average of the 2 readings is NOT THE MEAN of the day’s temperatures.

Phil
Reply to  Nick Stokes
January 14, 2019 9:36 pm

Time of Observation (TOBS) should add more jitter error.

Ferdberple
Reply to  Nick Stokes
January 14, 2019 10:50 pm

There are differences, but nothing to do with Nyquist.
=========
By that logic on can drive from LA to Las Vegas, take your max speed + min speed, divide the sum by 2 and get your average speed.

So stuck in LA my min speed is 0, and somewhere out on the highway I might get up to 100, so my average for the trip will 50. And based on this result I will decide to buy a new car or not.

Frederick Michael
Reply to  Ferdberple
January 15, 2019 7:10 pm

How does that have anything to do with the Nyquist? Your example doesn’t even require sampling.

Jim G
Reply to  Nick Stokes
January 14, 2019 11:51 pm

Nick.
I am wondering if you could explain your reasoning a little better.

As was demonstrated in the example above, Tmin and Tmax are not the peaks of a sinusoidal wave.
Air temperatures over land just don’t behave that way.
With the exception of deserts, most cities will have clouds on any given day. As the clouds pass the recording station, the air will cool a bit. When it passes, it warms back up. In this case, the signal will look like the noisy square wave and not a sine.

Min/Max are meaningless on a square wave signal.

Reply to  Jim G
January 15, 2019 3:16 am

None of this has anything to do with Nyquist. But the diurnal behaviour of locations is reasonably repetitive. The mean is one measure, min/max is another. Both will respond to climate changes.

Again I’ll point to my Boulder analysis. It compares min/max with various reading times, and the integrated (in black). The min/max show a shift with reading time, so of course they don’t agree with the integrated. But they are just variably offset, and some reading times come very close. The post is here.

Reply to  Nick Stokes
January 15, 2019 2:09 am

Nick,

Signal theory applies to all signals. All time-variant sequences of numbers are signals, whether temperatures or shot records from a seismic survey.

Every time Mann splices the instrumental record onto a proxy reconstruction, he violates just about every principle of signal theory… as did Marcott’s “up-tick,” as does everyone who claims the recent warming is unprecedented based on comparisons of the instrumental record to proxy reconstructions.

Paramenter
Reply to  Nick Stokes
January 15, 2019 2:17 am

Hey Nick,

The Theorem tells you that you can’t resolve frequencies beyond that limit. But we aren’t trying to resolve high frequencies.

From where the difference between daily (Tmax+Tmin)/2 and daily true arithmetic mean (or area under the temperature curve) comes from? For particular shapes of daily temperature signal having just two samples per day yields significant error. That’s because you cannot ‘resolve’ the signal. I would say it has something to do with Nyquist.

We are trying to get a monthly average.

Below error magnitude for Boulder, CO Jan 2006-Dec 2017 between direct integration (per each month) of subhourly signal (sampled every 5 min) and NOAA monthly record based on averaging daily midrange values (Tmax+Tmin)/2. Error visible per daily records is also visible for monthly. I don’t quite understand why such daily drift should not propagate into monthly and yearly.

Boulder. Monthly vs subhourly

Paramenter
Reply to  Paramenter
January 15, 2019 2:43 am

Valid link to the chart:
Boulder. Monthly vs subhourly

Greg F
January 14, 2019 2:24 pm

NOAA averages these 20-second samples to 1-sample every 5 minutes or 288-samples/day.

An average is a pretty crude low pass filter. You would think they could do better.

MarkW
Reply to  Greg F
January 14, 2019 2:35 pm

What makes you think they want to do better?

MarkW
January 14, 2019 2:30 pm

Not only is the sampling rate grossly inadequate temporally, it is grossly inadequate spatially as well.

LdB
Reply to  MarkW
January 14, 2019 5:42 pm

Yes I agree with that Mark they need some high precision sites to integrate energy to the proxy fully and I would have a lot more confidence.

Frederick Michael
Reply to  MarkW
January 14, 2019 6:29 pm

Bingo; that’s the key point. The spatial problem is severe.

While averaging min and max is far from perfect, it’s a reasonable approach given the technology that existed just a few decades ago.

Anthony has found huge location flaws in the temperature data base. That was substantive and well documented. This Nyquist nonsense is an embarrassment, and has no place here.

MarkW
Reply to  Frederick Michael
January 15, 2019 10:35 am

They are both real flaws.
The fact remains that you cannot calculate an accurate daily average temperature from just a high and a low reading.
Nyquist is one way to calculate how many readings you need in order to calculate a quality average.

DHR
January 14, 2019 2:32 pm

“The USCRN is a small network that was completed in 2008 and it contributes very little to the overall instrumental record…”

You make the USCRN sound to be of little value. But the USCRN sites were chosen to be evenly distributed over the States and to be distant from any known human-caused heat sources such as cities, airports, highways and so forth. It seems to me that it ought to present a better average view of CONUS temperature trends for its period of operation than the Historic Climate Network of which many stations are very poorly cited and subjected to various corrections. Perhaps you could do an analysis comparing the two.

William Ward
Reply to  DHR
January 14, 2019 3:55 pm

DHR,

Thanks for your comment, I didn’t intend that my comment would make USCRN out to be of little value. Thanks for the opportunity to clarify and expand my thoughts on this. I’m a big fan of USCRN. Based upon what I know, it is a high quality network capable of doing what is needed to accurately sample a temperature signal – and I believe the siting is also good. USCRN should eliminate the long list of problems with measuring temperature. My comment was made to show that unfortunately, this high quality data is not used to calculate the global averages and trends that get reported in the datasets (HADCRUT, GISS, Berkley). Even if the max and min data from these stations were used we still have aliasing. We need to use the 5-minute samples. I wanted to be clear that USCRN provides us an ideal opportunity to compare the 2 methods but doesn’t by itself improve the reported means and trends.

Does this clarify my position?

I have done some preliminary work comparing means and trends from USCRN stations and corresponding nearby stations from the historical network (which is badly sited and suffers from UHI, thermal corruption, quantization error, reading error, infill, etc). I found wildly different results during my limited work.

Reg Nelson
Reply to  William Ward
January 14, 2019 5:07 pm

The USCRN, like the satellite temperature data and the ARGO buoy data, were supposed to be the Gold Standards of climate data. But when the results didn’t match the confirmation bias, actions were needed. So the propagandists employed the IAA method of PR. Step one I = Ignore. Step two = Attack. Step three = Adjust.

Dave Fair
Reply to  William Ward
January 14, 2019 5:37 pm

William, in your estimation does Anthony Watts’ work on the differences between well- and poorly-sited measuring stations hold water? I understand the trends are significantly different.

William Ward
Reply to  Dave Fair
January 14, 2019 7:50 pm

Hi Dave,

I’m a big fan of Anthony’s Surface Stations Project (if I remember the name…)! I really don’t know how to apply what I have done here with Nyquist to what Anthony did.

Anthony’s work on that seems to benefit from the saying “a picture paints a thousand words”. Just seeing the stations situated next to a wall of air conditioners blowing hot exhaust is very persuasive. I can’t find any logic to fault that work and I only find logic to support it. I use that in my mind as the benchmark reference for UHI/thermal corruption. I’m sorry I don’t have anything more substantial to say, except it is a very valuable work that should be effective to blunt Alarmism. But logic does not seem to work.

I have cataloged 12 significant “scientific” errors with the instrumental record:
1) Instruments not calibrated
2) Violating Nyquist
3) Reading error (parallax, meniscus, etc) – how many degrees wrong is each reading?
4) Quantization error – what do we call a reading that is between 2 digits?
5) Inflated precision – the addition of significant figures that are not in the original measurements.
6) Data infill – making up data or interpolating readings to get non-reported data.
7) UHI – ever encroaching thermal mass – giving a warming bias to nighttime temps.
8) Thermal corruption – radio power transmitters located in the Stevenson Screen under the thermistor or a station at the end of a runway blasted with jet exhaust.
9) Siting – general siting problems – may be combined with 7 and 8
10) Rural station dropout – loss of well situated stations.
11) Instrument changes – changing instruments that break with units that are not calibrated the same or instruments that are electronic where previous instruments were not. Response times likely increase adding greater likelihood to capture transients.
12) Data manipulation/alteration – special magic algorithms to fix decades old data.

A C Osborn
Reply to  William Ward
January 15, 2019 4:28 am

The problem with what you are saying about using more samples is that you can no longer compare those results to the historic data averages.
In fact even the Electronic Devices have changed in Accuracy and frequency of samples Recorded over the last 20 years or so.
Only the max & min should be compared under those circumstances.

Jim Gorman
Reply to  William Ward
January 15, 2019 11:37 am

Thank you. 3, 4, and 5 are my pet peeves and no one calling themselves a scientist should be able to ignore the measurement errors that these introduce.

Imagine if you were buying target rifles for a team and the salesman told you that all their rifles shoot within 1 minute of angle. When you get them, sure as shootin, 1/2 shoot right on, but the other half shoot at 2 minutes of angle. Did you get your money’s worth?

If you average a max temp with a min temp when the readings are only +- 0.5 degrees, do you get a mean temperature accurate to +- 0.25 degrees?

Alastair Gray
January 14, 2019 2:37 pm

As a former practising seismic geophysicist in the evil oil industry I find your arguments flawless and fully endorse them but who gives a cuss about an old doodlebugger. It does seem to make the practice of time of day correction a bit prissy. Any comments.
Maybe our Aussie pals should use this to beat their climistas over the head with

Clyde Spencer
Reply to  Alastair Gray
January 14, 2019 4:40 pm

Alastair
Perhaps this is a question best addressed to Stokes or Mosher, but how does time of day corrections matter when there are only two temperatures and they can occur at any time?

Reply to  Clyde Spencer
January 14, 2019 9:12 pm

Clyde, people used to examine the old-style min-max thermometers in the evening. At that point both the min and the max are from the same day.

But then they decided to read them in the morning … which means that the minimum is from today, but the maximum is from yesterday afternoon …

And of course, this plays havoc with using (max+min)/2 as mean temperature …

w.

Reply to  Willis Eschenbach
January 14, 2019 10:09 pm

“And of course, this plays havoc with using (max+min)/2 as mean temperature”
It doesn’t really. The monthly average is of 31 max’s and 31 min’s. It matters only a little which day they belong to. Just at the end, where one max or min might shift to the next month.

But the time of reading does matter statistically, because of the possibility of double counting. That is where TOBS comes in.

Jim G.
Reply to  Willis Eschenbach
January 14, 2019 11:58 pm

That’s something that I don’t really understand.

If you are reading today’s high, tomorrow;
Wouldn’t you just leave today’s high blank and fill it in the next morning?

As an auditor or data user;
What do you do if some folks backfilled data and others wrote it in on the same day?

It makes the dataset interesting, but limited in its usefulness.

Reply to  Jim G.
January 15, 2019 3:02 am

As an observer, you just record what you see. At the agreed time, it shows a certain max and min. It isn’t your job to say when they occurred.

Anthony Banton
Reply to  Clyde Spencer
January 15, 2019 1:47 am

” but how does time of day corrections matter when there are only two temperatures and they can occur at any time?”

Because the thermometers are reset to the temperature at THAT time by virtue of the max having a constriction the mercury has to pass and the min an indicator within, that is left behind at the lowest point. Resting them zeros the thermometers at the exact temp at the time of reading (TOBS).
On a hot day in summer that temperature is often not that far lower than the maximum of the day.
So next day, should it be cooler, the maximum thermometer is stuck back up at the reset temp of the previous evening, meaning the ‘heat’ from the previous day is recorded TWICE.

A C Osborn
Reply to  Anthony Banton
January 15, 2019 4:32 am

Yes, that COULD happen, but then you have to KNOW it happened to make a correct ADJUSTMENT.
Not adjust everything just in case it happened.

Anthony Banton
Reply to  A C Osborn
January 15, 2019 7:31 am
Gary Pearse
Reply to  Alastair Gray
January 14, 2019 5:00 pm

Actually, Wards analysis makes the algorithm/model adjustments to temperatures performed continuously to “unbias them” (BEST, and others), and the infilling over, in some cases 1200km, a sick joke. Mark Steyn remarked cogently in a Senate hearing on climate data, that we have a situation where we have a higher degree of confidence in what the weather will be like in 2100 than what the weather will be like in 1950!

n.n
January 14, 2019 2:38 pm

The low and sporadic sampling rate is one problem for “science” practiced outside of the near-domain.

Gums
January 14, 2019 2:39 pm

Salute!

It’s not only Nyquist, but the time period at certain temperatures. In other words, 16 hours at 80 degrees and 8 hours at 40 degrees has a greater effect upon criops than the reverse. Ask any gardener about growomg a tomato.

So my interest from the climate/temperature/weather folks is about how we treat the latent heat when the high temperature is not present and the low temperature is only present for a short time. Therefore, a simple average of max and min is not a good repersentation of the “climate” at the measurement station. It would seenm to me that a long high temperature would bias the overall energy equation to a high number, and vice versa. From trying to grow some “tropical” veggies, I can tellya that the latent heat in the soil at night is a big player for the growing season. The exception seems to be in the desert or up at my mountain cabin when radiation of the warm earth takes place and where a true “greenhouse” is inmvaluable.

Jez asking….
Gums…

Richard Patton
January 14, 2019 2:40 pm

Thanks for the confirmation of my suspicions about the uselessness of TMEAN. I remember one day when I was a forecaster at NAS Fallon NV we had a very cold shallow inversion. For 23+ hours of the day the temperature was about 5 deg F. Several times during the day a slight breeze would mix down warmer air from above and push the temperature up to the mid 30’s. So the “official” mean temperature for the day was 20 even though everyone knew that the true mean was more like 8 or 9 degrees F.

William Ward
Reply to  Richard Patton
January 14, 2019 8:01 pm

Hi Richard – perfect example and a real world experience!

Guy Leech
January 14, 2019 2:46 pm

This seems to be an important point put across in a somewhat over complicated way. If we want to measure the heat content of the atmosphere at the surface, which is what might be a result of AGW, a correctly constructed average temperature can be a proxy for that. It is only a useful proxy if it is averaged over very short periods of time, as explained in the post. Since the thermometer record is averaged over only two data points per day, it is not a proxy for atmospheric heat content, so is not a useful data series from which to evaluate changes in there heat content of the atmosphere over time. Wind speed & direction and cloud cover are probably significant determinants of daily minimum & maximum temperatures, and there are probably many other determinants.

William Ward
Reply to  Guy Leech
January 14, 2019 8:11 pm

Guy,

You are right it is a bit complicated, especially if you have not worked with signal analysis for a long time. It would be nice if climate scientists just did this right so we didn’t have to come along and correct them with a complex treatise. But science has been hijacked by the Alarmists and they tell us THEY OWN THE SCIENCE. And if we don’t agree with them we get little pet names like “science denier”. So unfortunately, we have to use actual math and science and shove it in their faces to show them just how wrong they are. I take no pleasure in it. I will go back to “nice mode” just as quickly as they give up the game. I find it difficult to ignore when we start to get elected officials (think Alexandrix Ocasix-Cortex) who are barely out of diapers and have no understanding of science or civics and they prescribe an insane “Green New Deal”. I think Climate Alarmism Rejectors are a bit too passive and not armed with information well enough to fight back against the insanity.

Krishna Gans
January 14, 2019 2:47 pm

Tmin and Tmax seem most usefull for DTR evolution over time, not more, not less.

Krishna Gans
Reply to  Krishna Gans
January 14, 2019 2:49 pm

Forgot a link
Only as example…

Tractor Gent
January 14, 2019 2:47 pm

So what is an appropriate sample rate of temperature? Once every 20 seconds sounds a bit arbitrary & probably ties more to equipment capability than to theoretical considerations. The sample from Alaska looks quite noisy, even with the 5 minute averaging. It would probably look a lot noisier in the full 20 sec/sample record. So, what’s the source of the noise? Is this instrumental (noise in the sensor or the amplifier & A/D converter) or is it genuine noise in the measured temperature, due to wind turbulence, say? The noise is important: if the actual noise bandwidth is greater than half the sampling frequency then there will be noise aliasing, effectively increasing the apparent noise level. I wonder if NOAA have any publicly available docs on the rationale for their choice of sampling frequency?

Tractor Gent
Reply to  Tractor Gent
January 14, 2019 2:55 pm

Just had a quick Google for info, but it looks like the furlough has got(ten) in the way 🙁

Greg Cavanagh
Reply to  Tractor Gent
January 14, 2019 4:33 pm

I very good question. I know from experience working as a surveyor in the field that random hot gusts of wind happen, I don’t know from where.

Clyde Spencer
Reply to  Greg Cavanagh
January 14, 2019 4:51 pm

Greg
And I remember a particularly hot day in California (~120 deg F, July 4th 1968) when my wife and younger brother and I were swimming in the North Fork of the American River (out of necessity to keep cool!). Every so often a really hot blast of air would come up the canyon and flash evaporate the surface of the river water. It was momentarily like being in a steam sauna and all three of us would cough after breathing the very hot and humid air. It would last for much less than a 5 minute AWS sampling interval.

William Ward
Reply to  Clyde Spencer
January 14, 2019 8:25 pm

Tractor,

Yes the government shutdown has NOAA site down. Bummer. Also, yes, good questions you ask. NOAA complicates things with the 5-minute averaging. I don’t know why they do this, except perhaps to reduce the total amount of data. I could not find a paper on it.

My first pass at this was to take the USCRN operating parameters and consider them as satisfying Nyquist. Going through a sample reduction process seems to indicate that the 288-samples per day seems to converge to under 0.1C error compared to the next rate I tried of 72-samples per day. Someone would need to decide the absolute accuracy needed. I agree that the transitions observed on some days at some stations suggest that 5-minute samples is not capturing all of the transients. Is that noise or valuable signal. Clyde’s example of the hot blast of air in the canyon would suggest we should not be averaging down to 5-minutes. Data converters are cheap and amazingly accurate. Memory is cheap. Processing power is immense. And hey, the fate of the world and survival of humanity is in the balance – so we can afford to splurge and oversample! So what if we have excess data. Decimation after the fact is easy! I just left a company (industry actually) where we were sampling microwave and millimeter wave signals in CMOS processes with performance good enough for high speed communications. Sampling temperature is glacial by comparison.

288-samples/day seems to be really good, however. I just can’t say if it is a bit of overkill or we should use all 4,320-samples/day. Either way, it doesn’t change the essence of what I present here.

thingadonta
January 14, 2019 2:50 pm

I am currently working in the desert in Australia. The other day it got to 46.1 degrees, however this was a spike reading between two 30 minute readings. The data is here:

http://www.bom.gov.au/products/IDW60801/IDW60801.94429.shtml

The highest 30 minute reading for the day was at 3:30pm at Mount Magnet airport (near a cleared hard runway-wasn’t there 50 years ago-another issue as the record is over 100 years old) and it was 45.8, but the highest recorded reading for the for day was at 46.1 sometime between 2:30-3:30pm–which spiked for a very short time-but doesn’t say exactly when.

My point being, this was likely a wind gust coming off the hard hot concrete; if they didn’t use the same ‘spike’ method 50 years ago (which I’m told they didn’t) , the max temperatures for the day would be different. So by adding these spikes to the longer term record you are splicing two different sample sets, apples and oranges, and getting an enhanced trend that isn’t there.

The argument that cold wind gust spikes would also enhance colder temperatures might also be true by using this method.

http://www.bom.gov.au/climate/dwo/IDCJDW6090.latest.shtml

William Ward
Reply to  thingadonta
January 14, 2019 8:30 pm

Thingadonta,

Another good real world example. The spikes of temperature you mention are higher frequency content. Depending upon what frequency components they contain determines the impact of the aliasing.

Sample according to Nyquist and these spikes will not inaccurately affect your mean calculations.

I’ll mention here, it would be nice if we could get away from mean calculations and just feed the sampled signals into some equations the climate scientists discover that explain how climate works! Now that would be real science. Not this sciencey-looking stuff.

A C Osborn
Reply to  William Ward
January 15, 2019 4:42 am

Not the mean, no, but they will if you are trying to analyse Max or Min and not Mean. Which is even more important data to understand Climate.

Jim Gorman
Reply to  A C Osborn
January 15, 2019 12:25 pm

+100

Ian Macdonald
January 14, 2019 2:58 pm

Since the values taken are max and min rather than values at random times of day, this is not quite the same thing as sampling a signal at discrete points along a cycle. In electronic terms it would be more like summing the outputs of two precision rectifiers, one positive voltage and the other negative voltage responding.

The problem here is that any asymmetry in the waveform, for example the warmest period of the day being only 5min whilst the coolest period covers several hours, is going to leave you with a totally wrong average value. It seeming that the short warm period has equal significance to the much longer cool period, when in fact the brief warm period is an outlier and not representative of anything.

A single thermometer embedded in a large lump of iron might be a better idea. (Water is not a good choice because of its heat of evaporation)

William Ward
Reply to  Ian Macdonald
January 14, 2019 8:39 pm

Hello Ian,

The iron idea is interesting, except it might rust. LOL! Maybe stainless steel? Either better than water.

I think we agree on fundamentals, but let me niggle about the 2 samples issue. They really are 2 samples. The max and min actually happen at a time during the day. Look in to clock jitter. This explains what happens when you add error to your sample clock rate.

Alastair Gray
January 14, 2019 3:00 pm

Reply to nick stokes if actual daily record were a boxcar sort of square sinusoid then accordingto Fourier (a godfather of global warming) you would needthe high frequencies to capture abrupt rise and fall of temperature- back to school for you laddie!

Reply to  Alastair Gray
January 14, 2019 3:26 pm

“you would need the high frequencies “
Wearily – they are computing monthly averages.

MarkW
Reply to  Nick Stokes
January 14, 2019 4:00 pm

Not relevant.
The point is that they are computing the monthly average from data that has discarded meaningful data and as a result the average isn’t as accurate as many on the alarmist side have been claiming.

The idea that you can calculate an average to a few hundredths of a degree from this mess is gross incompetence.

A C Osborn
Reply to  MarkW
January 15, 2019 4:46 am

This is the whole point, the “Average” of 2 readings is 100% correct for those 2 readings.
But is meaningless with regards to the Mean Temperature of the whole day, which is what they are trying to use it for.
The again historically it is all we have, just stop using the Average, present the Max & Min.

Tom Abbott
Reply to  A C Osborn
January 15, 2019 11:59 am

“The again historically it is all we have, just stop using the Average, present the Max & Min.”

I agree. I think they are making things way too complicated. I, personally, note the high temperature on my home thermometer and the low temperature and that’s all I need. I have no need to average the two numbers. The two numbers tell me everything I need to know.

Look at all the Tmax charts in this link below, all of which looked like they “Tmaxed out” around the 1930’s. There is no unprecedented warming in the 21st century according to Tmax.

https://wattsupwiththat.com/2018/12/15/it-is-the-change-in-temperature-compared-to-what-weve-been-used-to-that-matters-part-2/

MarkW
Reply to  MarkW
January 15, 2019 10:48 am

Another point is that historically the high and low were only recorded to the nearest whole degree.

juan slayton
January 14, 2019 3:05 pm

Temperature is not heat energy, but it is used as an approximation of heat energy.

Non-physicist here, and I’ve been puzzling about this for some time. I think a commenter some time ago compared Miami with high humidity to Phoenix with a much higher temperature and low humidity. Miami actually had significantly more heat. I suppose temperature might be a rough approximation if you assume that average humidity in a given location is somewhat stable and all you are interested in is trends, rather than absolute values, but that’s not a warranted assumption. Seems to me that energy is what we really want to know about, and measuring that would require simultaneous sampling of both temperature and humidity.

tty
Reply to  juan slayton
January 15, 2019 6:58 am

In short, what you need to measure energy in the climate system is the enthalpy.

Michael S. Kelly LS, BSA Ret.
Reply to  juan slayton
January 15, 2019 3:59 pm

Bingo! I’ve been saying that for over a year in these comments. The greenhouse effect is about retention of energy in the lower atmosphere that would normally have been radiated away to space. On Mars, the atmosphere is close to 100% carbon dioxide. Though it is extremely low pressure (and density), there is 54 times as much CO2 per unit surface are on Mars as there is on Earth. But there is no water vapor in the atmosphere, and no other greenhouse gases. On Mars, temperature is an exact indicator of atmospheric energy content.

On Earth, the presence of vast amounts of liquid water changes the game completely. The enthalpy of dry air is directly proportional to temperature. At 21 C, the enthalpy of dry air is 21 kJ/kg. At 21 C, 30% relative humidity, it is 33 kJ/kg. At 21 C, 83% relative humidity, it is 54 kJ/kg.

So at constant temperature, the difference between 83% RH and 30% RH enthalpies is equal to the total enthalpy of the dry air by itself.

Temperature, by itself, is useless in determining any shift in the energy balance of the Earth’s atmosphere.

henkie
January 14, 2019 3:05 pm

What about time constants? The time it takes to heat a blob of mercury is not comparable to the time it takes to heat up a tiny Pt sensor. You cannot use Nyquist without this information.

D. J. Hawkins
Reply to  henkie
January 14, 2019 3:46 pm

The tau for Pt thermometers is about 20 seconds, and LIG up to 60 seconds, depending on the study, hence the the 4,320 samples. Readings for Pt sensors are taken every 2 seconds however. This is a problem the Aussies have, reporting the Pt spikes as Tmax for the day without smoothing to approximate the mercury thermometer.

Alan Tomalty
Reply to  D. J. Hawkins
January 14, 2019 8:15 pm

I was wondering how they came up with 2160 as the Nyquist frequency. There is a problem here. 2160 really isn’t the frequency of the original signal. It is the minimum frequency of the measuring tool. Does the UAH temperature data operate on the same temperature measuring Tmax and Tmin of a GPS point in the atmosphere?

William Ward
Reply to  Alan Tomalty
January 14, 2019 8:51 pm

Henkie, Alan and D.J,

Good points and questions but we may be getting in deeper than we should. You would think a standard would be defined – based upon research – that specify sensor type, sensitivity, thermal mass, sensor response time, amplifier response time, front-end filtering, power supply accuracy and ripple, and drift, etc. Agreed that a different front-end will respond differently if co-located and experiencing the same stimulus. This should be a global standard, but of course every country or region will need their own standards body and each body will push for their favorite architecture and years will go by with no agreements and eventually every region will adopt their own inconsistent standards. This will leave the door open for more magic climate algorithms to be applied. Meanwhile humanity is in the crosshairs of climate disaster while the experts would fuss over minor details all selected to give themselves some kind of personal benefit.

Man, that sounds cynical…

A C Osborn
Reply to  William Ward
January 15, 2019 4:51 am

That is why we have the WMO, who do set the standard, which is ignored by the Australian BOM.
Not only that but they have also been curtailing the Lower Readings as well.
They are very bad boys down under.

tty
Reply to  Alan Tomalty
January 15, 2019 7:06 am

The satellites measure the radiation temperature which is in practice instantaneous. However they usually only measure any specific area once a day. With station-keeping satellites this is done at the same time every day, but non-station-keeping will drift, so the time-of-day will change. This is one of the reasons for the difference between RSS and UAH. RSS uses a theoretical model for daily temperature changes to correct for drift while UAH uses measured (but rather noisy) daily temperature cycle data for correction.

William Ward
Reply to  tty
January 16, 2019 2:32 pm

Alan,

You asked: “Does the UAH temperature data operate on the same temperature measuring Tmax and Tmin of a GPS point in the atmosphere?”

As I understand it, 2 satellites are involved. Each satellite passes over each location once each day, so 2 passes total for every location. Ideally the passes are spaced 12 hours apart so 2 measurements are made each day. What is measured is microwave radiation at some altitude in the troposphere. This is compared to the background microwave radiation of deep space. Then several thousand calculation are done to relate this to temperature. I’m sure someone more knowledgeable on this might correct a minor detail here, but if there are 2 samples taken in a day we have the a similar problem to the surface stations. As I explained in the paper, 2 samples taken at regular intervals will likely yield better results than finding max and min for the day. But (as has been reported in other sources) satellite measurements suffer from their own specific issues not related to undersampling.

henk
January 14, 2019 3:07 pm

What about time constants? The time it takes to heat a blob of mercury is not comparable to the time it takes to heat up a tiny Pt sensor. You cannot use Nyquist without this information.

tty
Reply to  henk
January 15, 2019 7:12 am

WMO has a standard for this, so as to make measurements comparable (but not necsessarily correct).

What it all boils down to is that before modern electronic sensors there really wasn’t any way to measure the average temperature of a medium with rapidly changing temperatures, like the atmosphere.

Krishna Gans
January 14, 2019 3:12 pm

The problem to not apply the Nyquist Rate is, that you can’t find periodic events, say a 25 year time series can’t show you periodic events > 25 years.

William Ward
January 14, 2019 3:15 pm

I’ll try to reply to all comments, but I wanted to start with a few words expressing my gratitude to a few people who helped me with this project. I’d like to thank Kip Hansen for reviewing the paper and guiding me in a number of ways to optimize the paper for for the WUWT forum. Kip’s kindness and generosity really stand out. I would also like to thank Piotr Kublicki for encouraging me to go forward with formalizing this information into a paper and for his collaboration on the numbers. In addition to doing multiple reviews, Piotr did most of the work to run the analysis on the long term trends. For the Nyquist-compliant method this involved finding the linear trend by using over 1.2 million samples for each location. I’m grateful for a few of my friends in the industry who have reviewed the paper for conceptual accuracy. These people are experts in the field of data acquisition and signal analysis. And of course, I would like to thank Anthony and Charles for taking the time to review this and to do the work to publish it on WUWT for further discussion.

Jim Gorman
Reply to  William Ward
January 15, 2019 12:45 pm

Kudos to all who participated. This is the kind of stuff that should make climate scientists say WHOA, are we going to look stupid in a few years!

Gunga Din
January 14, 2019 3:16 pm

Siting issues. This issue. (The day’s average temp for the day is in the middle of the day’s high and low with no regard to how long in the 24 hours the temp was near either?!)

Bottom line (again). We don’t really know what was the past “Global Temperature”. We don’t really know what it is now.

Who’s willing to bet Trillions of dollars on that?
Those who are out to achieve something other than “Saving the Whales … er … Planet”.

Gunga Din
Reply to  Gunga Din
January 14, 2019 3:27 pm

PS Another issue the actual number of sites and how widespread they were (and are) for any kind of “Global” temperature record.

commieBob
January 14, 2019 3:21 pm

What is the highest frequency you have to worry about?

There’s a rule of thumb for a signal that looks something like a square wave.

BW = 0.35/RT

where:

BW = bandwidth = the highest frequency you have to worry about.
RT = rise time = roughly how fast the temperature goes from 10% of the distance from Tmin to Tmax to 90%

Suppose Tmin is 0C and Tmax is 20C. 10% of the difference is 2C.
We’re interested in how fast the temperature goes from 2C to 18C.
Suppose that it takes 3 hours or 1/8 of a day.
The highest frequency we have to worry about is 0.35/(1/8) = 2.8 cycles per day.

In Figure 3 above, notice the big jump between approx. sample 125 and approx. sample 135. That’s about ten samples out of 288 or around 1/30 of a day. If we applied the rule of thumb to that we’d get a maximum frequency of 10 cycles per day. Do we have to worry about that? Maybe. If we guess that the jump was about half the distance between Tmin and Tmax, that would reduce the amplitude of that frequency component.

There’s a big gotcha. When we talk about Nyquist rates and about Fourier analysis, we’re talking about a repeating waveform. Daily temperatures are only approximately a repeating waveform. That means Nyquist is over optimistic. Based on my guess of 10 cycles per day for the signal in Figure 3, you could get away with 20 samples per day. You clearly can’t. In other words, Nyquist isn’t really what you’re aiming for. You have to do a lot better.

Reply to  commieBob
January 14, 2019 3:29 pm

“You have to do a lot better.”
To calculate a monthly average?

MarkW
Reply to  Nick Stokes
January 14, 2019 4:04 pm

If you want to calculate an accurate monthly average, you have to start with accurate daily averages.
Lacking the first, the second is meaningless.

Jim Gorman
Reply to  MarkW
January 15, 2019 12:54 pm

No wonder discussions on measurement errors fall on deaf ears. Folks, these aren’t made up facts and procedures to make people look bad like the SJW’s do. Metrology, Nyquist, systemic error analysis, etc. have been around for years. There are college courses that teach these in detail.

Climate scientists need to buck up and admit they don’t know what they don’t know! Involve other fields and let them be co-authors. You may have to share grant money, but your output will be up to accepted standards.

Clyde Spencer
Reply to  Nick Stokes
January 14, 2019 4:58 pm

Stokes
If you input daily garbage to calculate a monthly average, you will get monthly garbage!

commieBob
Reply to  Nick Stokes
January 14, 2019 5:16 pm

If you’re calculating an energy balance, you’re talking about a couple of watts per square meter of forcing. Let’s be generous and call it 3 w/m^2.

The solar constant is around 1388 w/m^2. So, 3 w/m^2 is around 0.2%.

Depending on what you’re doing, the extra precision is actually important.

IMHO, the more I think about it, the more I think invoking Nyquist is barking up the wrong tree. A straight numerical analysis demonstrates the problem more than adequately.

Your question about a monthly average makes sense if you’re thinking about Nyquist. In other words, I think invoking Nyquist actually leads us astray.

Reply to  commieBob
January 14, 2019 7:36 pm

“If you’re calculating an energy balance”
But we aren’t. And we aren’t reconstructing a sub-diurnal signal. We are calculating a monthly average temperature. That is the frequency of interest, and should be the focus of any Nyquist analysis.

commieBob
Reply to  Nick Stokes
January 15, 2019 4:36 am

The numerical analysis above clearly demonstrates that you are wrong.

A C Osborn
Reply to  Nick Stokes
January 15, 2019 4:55 am

Come on guys, we all know that these “Errors” all cancel each other out.
The Climate Scientists tell us it is so.
/Sarc off

Editor
Reply to  Nick Stokes
January 15, 2019 10:12 am

Nick ==> We have to start out with the right question: If we simply want to know a generalized “monthly temperature average”, and are willing to have large error/uncertainty bars then we can forget all this Nyquist business — of course we can. Get your GAST (land and sea) tack on your 1 to 2 degrees C uncertainty and you are home free.

But the reason we are interested in GAST is because there is an hypothesis that rising atmospheric CO2 concentrations are causing the Earth system to retain an increasing amount of the Sun’s energy — so we need to ask the question: How much energy is the Earth system retaining? Is that rising or falling? We know that incoming energy is transformed into/by all kinds of physical processes– some (and only some) of it sensible heat of the air and seas. So if we are looking at air and water temperatures for this purpose, then we need to accurately determine the energy the temperatures represent — therefore we need a more accurate view of the changes, the temperature profile, throughout the day — thus Nyquist.

MarkW
Reply to  Nick Stokes
January 15, 2019 10:53 am

Nick, you are still ignoring the fact that the daily averages that you are using to calculate your monthly average aren’t accurate.
Using bad daily averages to create a monthly average means that your monthly average is also bad.

commieBob
Reply to  Nick Stokes
January 15, 2019 1:20 pm

Probably nobody’s going to see this but …

There is a wonderful guide to Digital Signal Processing written for people who might actually want to do some Digital Signal Processing (DSP). It is The Scientist and Engineer’s Guide to Digital Signal Processing. You can read it for free.

The author does not skimp on the fundamentals. Time after time, after time, I see scientists and commenters here on WUWT and elsewhere getting the fundamentals wrong. That means that everything that follows is wrong.

When you invoke Nyquist, explicitly, or implicitly, whether you know it or not, you are talking about reconstructing the waveform you sampled. Chapter 3

So, what can you reconstruct with two samples per day? You can reconstruct a sinewave whose frequency, phase, and amplitude do not change. If the daily temperature follows a sinewave and Tmax and Tmin do not change from day to day, you’re good. If that isn’t the case, two samples per day is not enough. As William Ward points out in the article above, that leads to quite significant errors.

FabioC.
Reply to  commieBob
January 14, 2019 7:52 pm

“Suppose Tmin is 0C and Tmax is 20C. 10% of the difference is 2C.
We’re interested in how fast the temperature goes from 2C to 18C.
Suppose that it takes 3 hours or 1/8 of a day.
The highest frequency we have to worry about is 0.35/(1/8) = 2.8 cycles per day.

In Figure 3 above, notice the big jump between approx. sample 125 and approx. sample 135. That’s about ten samples out of 288 or around 1/30 of a day. If we applied the rule of thumb to that we’d get a maximum frequency of 10 cycles per day. Do we have to worry about that? Maybe. If we guess that the jump was about half the distance between Tmin and Tmax, that would reduce the amplitude of that frequency component.”

I have been doing some research in my spare time on temperature time-series and a physical integrator for temperature measurements and those above are exactly some of the issues I had to face.

If you can let me have a direct contact, I’d like to talk about those problems.

Clyde Spencer
January 14, 2019 3:35 pm

William,
You said, ” These daily readings are then averaged to calculate the daily mean temperature as Tmean = (Tmax+Tmin)/2.”

We have gone over this before. Despite NOAA calling the calculation a “mean,” it is at best a median, and more properly called the “mid-range” statistic. [ https://en.wikipedia.org/wiki/Mid-range ]

The essential point being that a mid-range does not have an associated standard deviation, standard error of the mean, or a probability distribution function. It is a simplistic measure of central tendency similar to a median, but calling it a “mean” (Which is calculated from a large number of samples.) implies that it has the associated statistics of a mean, and suggests that it is more robust and descriptive than what it is. Your Figure 1 demonstrates this deficiency clearly.

William Ward
Reply to  Clyde Spencer
January 14, 2019 9:24 pm

Hi Clyde,

You are correct sir! I struggled with whether to comply with NOAA language or circle the wagons to clarify. Kip did a good job of setting up a word counter in my head. Being verbose by nature, I was extra attentive to this. It was a challenge to be efficient with less words. I opted to leave out the detail you point out with the hope that someone like you would come along and add the point. So thank you! I also admit I get a little lax with this detail …

A C Osborn
Reply to  William Ward
January 15, 2019 4:58 am

Actually it is the Average of 2 values and nothing to do with actual means or even medians of the Temperature.

Clyde Spencer
Reply to  A C Osborn
January 15, 2019 12:10 pm

ACO

Did you bother to read the Wiki’ link I provided?

While the arithmetic procedure for calculating the misnomer is the same as a mean (i.e. dividing the sum of the values by the number of values.) It is not a mean in the usual sense. I have called it elsewhere a “degenerate median” because, unlike a typical median, which is the mid-point of a sorted list, it is (as you point out) the average of just two values, which is what even a typical median comes down to if there are an even number of entries in the list.

That is why it is best to refer to it as what is a recognized as a legitimate mathematical and statistical term, the “mid-range value,” and not give it an unwarranted status by calling it a “mean.”

steve case
January 14, 2019 3:35 pm

“Beware of averages. The average person has one breast and one testicle.” Dixie Lee Ray

The average of 49 & 51 is 50 and the average of 1 and 99 is also 50.

First they take the daily min and max and compute a daily average. Then they take the daily averages and average them up for the the year an then those annuals are averaged up from the equator to the poles and they pretend it actually means something.

Based on all that they are going to legislate policy to do what? Tell people to eat tofu, ride the bus and show up at the euthanization center on their 65th birthday?

JimG1
Reply to  steve case
January 14, 2019 4:00 pm

Then there is the case of the statistician who drowned crossing a river with average depth of one foot.

steve case
Reply to  JimG1
January 14, 2019 5:06 pm

Ha! Good one.

A C Osborn
Reply to  steve case
January 15, 2019 5:00 am

+1000 to both of you.

Editor
January 14, 2019 3:49 pm

I would disagree, for a couple of reasons.

First, the definition of the Nyquist limit:

The Nyquist-Shannon Sampling Theorem tells us that we must sample a signal at a rate that is at least 2x the highest frequency component of the signal. This is called the Nyquist Rate.

The first problem with this is that climate contains signals at just about every frequency, with periods from milliseconds to megayears. What is the “Nyquist Rate” for such a signal?

Actually, the Nyquist theorem states that we must sample a signal at a rate that is at least 2x the highest frequency component OF INTEREST in the signal. For example, if we are only interested in yearly temperature data, there is no need to sample every millisecond. Monthly data will be more than adequate.

And if we are interested in daily temperature, as his Figure 2 clearly shows, hourly data is adequate.

This brings up an interesting question—regarding temperature data, what is the highest frequency (shortest period) of interest?

To investigate this, I took a year’s worth of 5-minute data and averaged it minute by minute to give me an average day. Then I repeated this average day a number of times and ran a periodogram of that dataset. Here’s the result:

As you can see, there are significant cycles at 24, 12, and 8 hours … but very little with a shorter period (higher frequency) than that. I repeated the experiment with a number of datasets, and it is the same in all of them. Cycles down to eight hours, nothing of interest shorter than that.

As a result, since we’re interested in daily values, hourly observations seem quite adequate.

The problem he points to is actually located elsewhere in the data. The problem is that for a chaotic signal, (max + min)/2 is going to be a poor estimator of the actual mean of the period. Which means we need to sample a number of times during any period to get close to the actual mean.

And as his table shows, an hourly recording of the temperature gives only a 0.1°C error with respect to a 5-minute recording … which is clear evidence that an hourly signal is NOT a violation of the Nyquist limit.

So I’d say that he’s correct, that the average of the max and min values is not a very robust indicator of the actual mean … but for examining that question, hourly data is more than adequate.

Best to all,

w.

Phil
Reply to  Willis Eschenbach
January 14, 2019 11:54 pm

Another way to look at it is to try to estimate the area under the curve of daily temperature. The original model of a sine wave is clearly invalid. There is a distortion of a sine wave every day. Hourly data can approximate this distortion a lot better than the min-max model. 5 minute data can improve upon that, but at the expense of diminishing returns. Nyquist or no Nyquist, the min-max practice assumes a perfect sine wave as a model of the daily temperature curve and this model is invalid. In short, I agree that hourly data is much better than min-max thermometers.

Gums
Reply to  Phil
January 15, 2019 9:54 am

Salute Phil!

A great point and I would like to see that sort of record used versus simple hi-lo average.
The daily temperature is not a pure sine wave, and at my mountain cabin a simple hi-lo average is extremey misleading, especially when trying to grow certain veggies or sprouting flower seeds.

Seems to this old engineer and weatherwise gardener that we should use some sort of “area under the curve” method. Perhaps assign a simple hi-lo average to defined intervals, maybe 20 minutes, and then use the number of increments above the daily “low” in some manner to reflect the actual daily average.

I can tellya that up at my altitude that during the summer it warms up very fast from very low temperatures in the morning and then stays comfortable until after sunset. I hear that the same thing happens in the desert. So a weighted interval method would show a higher daily average during the summer when days are long and a lower average during the shorter winter days. This past summer’s data from the 4th of July was a good example I found from a nearby airfield. The high was 88 and low was 58 and average was published as 73. But the actual day was toastie! 13 hours were above the :”average”, and 10 of those hours were above 80 degrees. The exact reverse happens in the winter, and at the same site earlier this month we saw 23 for low and 62 for a high. 42 average. It was unseasonably warm, but we still had 13 hours below the average and only 11 hours above.

Gums sends…

Editor
Reply to  Willis Eschenbach
January 15, 2019 6:58 am

w. ==> Quite right …. In a pragmatic sense, we have the historic Min/Max records and the newer 5-minute (averages) record. So our choice is really between the two methods — and the 5-min records is obviously superior in light of Nyquist. Where 5-min records exist, they should be used and where only Min/Max records exist — they can be useful if they are proper acknowledged to be error-prone and acknowledged with proper uncertainty bars….

Min-Max records do not produce an accurate and precise result suitable for discerning the small changes in National or Global temperatures.

Paramenter
Reply to  Willis Eschenbach
January 15, 2019 8:10 am

Hey Willis,

As a result, since we’re interested in daily values, hourly observations seem quite adequate.

Agreed. I’ve run some comparisons between 5-min and 1-hr sampled daily signal – they yield very similar averages. But here we’re talking about 2 samples per day, or effectively just one sample (daily midrange value).

Also, Nyquist only applies to regularly spaced periodic sampling … but min/max sampling happens at irregular times.

I see it bit differently: irregular sampling makes a problem worse. If irregular sampling could alleviate Nyquist limitations that would be very cheap win: just sample any signal sparsely and irregularly and no need to worry about Shannon! Well, regardless of sampling method such method must obey Shannon in order to replicate a signal.

As you can see, there are significant cycles at 24, 12, and 8 hours … but very little with a shorter period (higher frequency) than that.

I’ve got also smaller frequency peaks around 6 and 3 hours. Still most of the signal energy resides in the yearly and daily cycles (and around).

Lots of one and two degree errors in there … the only good news is that the error distribution is normal Gaussian.

Thanks for sharing – have you run any ‘normality tests’ to determine that or in this case eyeballing is perfectly sufficient?

William Ward
Reply to  Paramenter
January 16, 2019 2:38 pm

Paramenter said: “If irregular sampling could alleviate Nyquist limitations that would be very cheap win: just sample any signal sparsely and irregularly and no need to worry about Shannon! Well, regardless of sampling method such method must obey Shannon in order to replicate a signal.”

+1E+10

January 14, 2019 4:01 pm

I noticed in Fig 6 that the trends for Blackville, SC were very large. The regularly sampled trend is 9.6°C/Cen, and the min/max is higher. It would be interesting to see the other USCRN trends, not just the differences. The period is only 11 years.

Reply to  Nick Stokes
January 14, 2019 4:39 pm

I did my own check on Blackville trends. I got slightly higher trend results (my record is missing Nov,Dec 2017).
Average 11.8±17.8 °C/Cen
Min/Max 16.5±18.4 °C/Cen
These are 1 sd errors, which obviously dwarf the difference. It is statistically insignificant, and doesn’t show that the different method made a difference.

Reg Nelson
Reply to  Nick Stokes
January 14, 2019 5:40 pm

Anecdotal evidence is somehow scientific? And be honest, Nick, if the station showed significant cooling it would thrown out or “corrected” by the likes of you.

Reply to  Reg Nelson
January 14, 2019 6:30 pm

It is WW’s choice, not mine.

Dr. S. Jeevananda Reddy
January 14, 2019 4:01 pm

The whole exercize is itself a biassed argument. The error factor [Table] is relative to 288 points of 5 minute average. Here the writer has not taken into account the drag factor in the continuous record. The drag is season specific and instrument specific. On these aspects several studies were made before accepting average as maximum plus minimum by two.

Dr. S. Jeevananda Reddy

Ian Cooper
January 14, 2019 4:06 pm

I would have thought that scientists were always looking for consistency in results when comparing the present with even the near past, let alone back to the earliest times of recording temperature. I suppose that it never occurred to anyone at the time, that when the new AWS measuring systems came on line that it might be opportune to conduct a lengthy (10 -20 year) experiment by continuing the old process alongside the new. In this way, perhaps coming up with a way to convert the old to match the new. Then again, if you didn’t know that there was this kind of issue, that thought most likely wouldn’t have crossed your mind!

I was thinking of the process used by Leif Svalgaard and colleagues when trying to tie all of our historical sun spot observations together into a cohesive and more meaningful manner. Could this be done for the temperature method to everyone’s satisfaction?

Changing our methodology due to improvements in technology doesn’t seem to always alert us to the possibility that the ‘new’ results are actually different to the old ones. Yet, from what I have read here that most certainly is the case.

Here is another example of how technology has dramatically changed the way we record natural phenomena. I have been an active auroral observer in New Zealand for just over 40 years. Up until this latest Solar Max for SSC 24 our observations were written representations of our visual impressions. This tied easily into the historical record for everywhere that the aurorae can be seen. Photographs were a bonus, and sometimes considered a poor second to the ‘actual’ visual impression of the display!

At the peak of SSC 23 the internet first played its part in both alerting people that something was happening, and therefore increasing the number & spread of observers worldwide, but also in teaching people about what they were seeing. As SSC 23 faded the sensitivity involved in Digital SLR cameras seemed to rise exponentially. This led to the strange situation as SSC 24 matured that people who had never seen aurorae were recoding displays that couldn’t be seen with the naked-eye! This unfortunately coincided with the least active SSC for at least 100 years. This manifested itself among the ‘newbies’ as a belief that aurorae are hard to see, and even when you can see them, only the camera can record the colour! The ease with which people can record the aurorae has lead to a massive increase in ‘observers,’ and I use that term loosely, who very rarely record their visual impressions, mostly because they don’t appreciate how important those impressions are as a way to keep the connection with the past going.

If there is a fundamental disconnect in the way we now record temperature compared to the past, then this should be addressed sooner rather than later!

kevink
January 14, 2019 4:15 pm

Mr. Ward is quite correct.

The proper way to prevent aliasing is to apply a low pass filter that discards all frequencies above 1/2 your sample rate. This is standard practice in electronics work. Sometimes the sensor (ie a microphone) performs this function for you.

Problem with the (max-min)/2 method is that there is no proper low pass filter that rejects “quick” spikes of extreme temps. The response time of the sensor is set by the thermal diffusivity of the mercury. This could allow a 5 minute hot blast of air from a tarmac to skew the reading.

An electronic temperature sensor (say a platinum RTD) can respond in milliseconds and normally filtering it so it cannot respond faster than 1 minute would be necessary to sample it a a 30 second rate.

An even larger problem with the official “temperature” record is the “RMS” versus “True RMS” problem. That would be a whole discussion on it’s own. The average of a max and a min temperature is NOT representative of the heat content in the atmosphere at a location.

So here we are prepared to tear down and rebuild the world’s energy supply based a a terrible temperature record that is so corrupt it is not fit for purpose at all.

Cheers, Kevin

Loren Wilson
Reply to  kevink
January 14, 2019 6:38 pm

A platinum resistance thermometer has a time constant in the range of a few seconds based on its diameter and the fluid. The fluid in this case was either flowing water at 3 feet per second or flowing air at 20 feet per second. The situation in a Stevenson screen is much slower air flow, so the time constant is much slower. I’ll do a measurement or two tomorrow with a 0.125″ diameter PRT when I get to the lab of the time constant in relatively still air. A thermistor would have a faster time constant due to the lower mass of the sensor and its sheath. Thermistors are also notorious for drifting, so unless you calibrate it often, the temperatures aren’t ready for this application.

kevink
Reply to  Loren Wilson
January 14, 2019 7:47 pm

Loren, if you investigate the “newer” surface mount chip Platinum RTD’s you see they have response times down into the 100 millisecond range. The small volume (ie thermal capacity) of the sensor does allow those response times. I may have “stretched” things a bit, but any value from 1 millisecond to 999 milliseconds is “a millisecond response time”.

Cheers, Kevin

Loren Wilson
Reply to  Loren Wilson
January 16, 2019 7:32 am

1/8″ diameter SS sheathed PRT from Omega – still air time constant on the order of 80 seconds. I did not find the specs on the ones used in the USCRN weather stations. Are they thin film, IR, or a more traditional sheathed PRT?

1sky1
January 14, 2019 4:15 pm

This is an inept rehash of an earlier guest blog by someone who fails to understand that:

1) Shannon’s sampling theorem applies to strictly periodic (fixed delta t) discrete sampling of a continuous signal
2) It doesn’t apply to the daily determination of Tmax and Tmin from the continuous, not the discretely-sampled record
3) While (Tmax + Tmin)/2 certainly differs from the true daily mean, it does so not because of any aliasing, but because of the typical asymmetry of the diurnal cycle, which incorporates phase-locked, higher-order harmonics.

William Ward
Reply to  1sky1
January 16, 2019 2:49 pm

1sky1 says: “1) Shannon’s sampling theorem applies to strictly periodic (fixed delta t) discrete sampling of a continuous signal”

Reply: max and min are samples, period = 1/2 day and with much jitter. Jitter exists on every clock. Nyquist doesn’t not carve out jitter exceptions nor limit their magnitude. Any temperature signal is continuous. You need to address and invalidate these points or your comments are not correct.

1sky1 says: “3) While (Tmax + Tmin)/2 certainly differs from the true daily mean, it does so not because of any aliasing, but because of the typical asymmetry of the diurnal cycle, which incorporates phase-locked, higher-order harmonics.”

Reply: “…because of the typical asymmetry of the diurnal cycle…” this is frequency content. Content that aliases at 2-samples/day. “…which incorporates phase-locked, higher-order harmonics.” So “phase locked higher order harmonics gets you a waiver from Mr. Nyquist?!? 35 years – many industries – many texts – never heard of such a thing. Orchestras are “phase locked” to the conductor, does that mean I can sample music sans Nyquist?!? Of course not.

1sky1
Reply to  William Ward
January 16, 2019 4:01 pm

Your failure to recognize that the daily measurement of extrema doesn’t remotely involve any clock-driven sampling (with or without “jitter”) speaks volumes. Not only are those extrema NOT equally spaced in time (the fundamental requirement for proper discrete sampling), but what triggers their irregular occurrence and recording has everything to do with asymmetric waveform of the diurnal cycle, not any clock per se.

Nyquist thus is irrelevant to the data at hand, which are NOT samples in ANY sense, but direct and exhaustive measurements of daily extrema of the thermometric signal. Pray tell, where in “35 years – many industries – many texts” did you pick up the mistaken notion that Nyquist applies to the extreme values of CONTINUOUS signals?

William Ward
Reply to  1sky1
January 16, 2019 4:41 pm

The movement of the Earth is the clock. The fact that we can get the same max min values from a Max/Min thermometer and a USCRN 288-sample/day system proves they are samples. Once you have the values it doesn’t matter what there source is. The result is indistinguishable from the process. Nyquist doesn’t somehow stop applying because the sample happens to be the “extreme values”. Once you bring the sampled data into your DSP system for processing how does your algorithm know if the data came from a Max/Min thermometer or an ADC with a clock with jitter? If you were to convert back to analog would the DAC know where the values came from? Would the DAC care that you don’t care about converting the signal back to analog and only intend to average it? No. You get the same result. They are samples, they are periodic. Nyquist applies. You are free to take the samples and use them as if they are not samples. Fortunately there are no signal analysis police to give you a citation.

1sky, if you decide to reply then the last word will be yours. Best wishes.

1sky1
Reply to  William Ward
January 16, 2019 5:30 pm

The fact that we can get the same max min values from a Max/Min thermometer and a USCRN 288-sample/day system proves they are samples.

Bass-ackwards logic! All this proves is that sufficiently frequently sampled USCRN digital systems will reveal nearly identical daily extrema as true max/min thermometers sensing the continuous temperature signal. There’s no logical way that the latter can be affected by aliasing, which is solely an artifact of discrete, equi-spaced, clock-driven sampling.

All the purely pragmatic arguments about DSP algorithms being blind to the difference between the obtained sequence of daily extrema and properly sampled time-series do not alter that intrinsic difference. In fact, the imperviousness of the extrema to aliasing and clock-jitter effects is what makes the daily mid-range value (Tmin + Tmax)/2 preferable to the four-synoptic-time average used in much of the non-Anglophone world for reporting the daily “mean” temperature to the WMO.

R Percifield
January 14, 2019 4:22 pm

My primary concern from a Nyquist standpoint is the rate of change of the reading in relationship to the sampling rate. If the rate of change is is less than 2X the sampling rate then you meet the criteria. However, that is for reconstruction of the original signal, and not for a true averaging of the system at hand. In many control systems the sampling rate is much higher than the rate of change, many times 10X or more. This is because the average over a period needs more data than the minimum in Nyquist requirements. In refrigeration we oversample at well over 20 times to get sufficient data to rule out noise, dropped signals, etc. We know the tau of the sensor and how it reacts to temperature change, so this allows us to accurately detect and respond to changes in temperature.

I would be interested in knowing what the response curve is for the sensor being used. That would give you sufficient data to determine how the system reacts to change. I have always thought that using the midpoint between max and min temps to be a very poor measurement of average temperature. Maybe someday I will do that study.

William Ward
Reply to  R Percifield
January 14, 2019 10:49 pm

R Percifield,

Sampling far over Nyquist is done for many reasons. If you have noise as you say, then in fact your bandwidth is higher and technically your Nyquist rate is also higher. Most often sample rates are much higher because 1) for control applications the ADCs are capable of running so much faster than the application needs and memory and processing are cheap, so why not do it; and 2) sampling faster relaxes your anti-aliasing filter requirements. You can use a lower cost filter and set your breakpoint farther up in frequency such that your filter doesn’t give any phase issues with your sampling. Audio is the perfect example. CD audio is sampled at 44.1ksps. Audio is considered to be 20Hz to 20kHz. Sampling at 44.1ksps means that anything above 22.05kHz will alias. So your filter has to work hard between 20kHz and 22.05kHz. If you are an audiophile you don’t want filters to play with the phase of your audio, so this is another big issue. Enter sampling at 96.1ksps or 192ksps. Filters can be much farther up in frequency and your audio can be up to 30kHz, etc. There is a titanic argument in the audio world about whether or not we “experience” sound above 20kHz. I won’t digress further…

There is the Theory of Nyquist and then the application. In the real world there are no bandwidth limited signals. Anti-aliasing filters are always needed. Some aliasing always happens. It is just small and doesn’t not affect performance if the system is designed properly.

In a well designed system – audio is a good example – you should be able to convert from analog to digital and back to analog for many generations (iterations) without any audible degradation. Of course this requires a studio with many tens – perhaps hundreds of thousands of dollars of equipment – but it is possible.

kevink
January 14, 2019 4:26 pm

“1) Shannon’s sampling theorem applies to strictly periodic (fixed delta t) discrete sampling of a continuous signal”

Seems to me measuring the temperature one a day is the definition of a strictly periodic discrete sampling of a continuous signal. Sample period is 24 hours (frequency is 1.15 e-5 Hz)

“2) It doesn’t apply to the daily determination of Tmax and Tmin from the continuous, not the discretely-sampled record”

See my first comment above

“3) While (Tmax + Tmin)/2 certainly differs from the true daily mean, it does so not because of any aliasing, but because of the typical asymmetry of the diurnal cycle, which incorporates phase-locked, higher-order harmonics.”

That would be the “RMS” versus “True-RMS” error source in the official temperature guesstimates…..

Cheers, Kevin

1sky1
Reply to  kevink
January 14, 2019 4:39 pm

Seems to me measuring the temperature one a day is the definition of a strictly periodic discrete sampling of a continuous signal.

If the temperature measurement took place at exactly the same time each day, then you would have strictly periodic sampling and great aliasing. But that’s not what is done by Max/Min thermometers that track the continuous temperature all day and register the extrema no matter at what time they occur. Those times are far from being strictly periodic in situ.

kevink
Reply to  1sky1
January 14, 2019 6:03 pm

1SKY1 wrote;

“If the temperature measurement took place at exactly the same time each day, then you would have strictly periodic sampling and great aliasing.”

You are refering to the phase of the periodic sampling, even more complicated. The periodic sampling takes place once every 24 hours. The Min/Max selection adds phase to the data, just another error source.

Cheers, Kevin

1sky1
Reply to  1sky1
January 15, 2019 4:15 pm

When people who lack even Wiki-level grasp of concept start parsing technical words, you get the above nonsense about what constitutes strictly periodic discrete sampling in signal analysis. The plain mathematical requirement is that delta t needs to be constant in such sampling. Daily extrema simply don’t occur on any FIXED hour! The attempt to characterize that empirical fact as “Min/Max selection adds phase to the data, just another error source” is akin to claiming error for the Pythagorean Theorem&