Guest Essay by Kip Hansen (with graphic data supplied by William Ward)
One of the advantages of publishing essays here at WUWT is that one’s essays get read by an enormous number of people — many of them professionals in science and engineering.
In the comment section of my most recent essay concerning GAST (Global Average Surface Temperature) anomalies (and why it is a method for Climate Science to trick itself) — it was brought up [again] that what Climate Science uses for the Daily Average temperature from any weather station is not, as we would have thought, the average of the temperatures recorded for the day (all recorded temperatures added to one another divided by the number of measurements) but are, instead, the Daily Maximum Temperature (Tmax) plus the Daily Low Temperature (Tmin) added and divided by two. It can be written out as (Tmax + Tmin)/2.
Anyone versed in the various forms of averages will recognize the latter is actually the median of Tmax and Tmin — the midpoint between the two. This is obviously also equal to the mean of the two — but since we are only dealing with a Daily Max and Daily Min for a record in which there are, in modern times, many measurements in the daily set, when we align all the measurements by magnitude and find the midpoint between the largest and the smallest we are finding a median (we do this , however, by ignoring all the other measurements altogether, and find the median of a two number set consisting of only Tmax and Tmin. )
This certainly is no secret and is the result of the historical fact that temperature records in the somewhat distant past, before the advent of automated weather stations, were kept using Min-Max recording thermometers — something like this one:
Each day at an approximately set time, the meteorologist would go out to her Stevenson screen weather station, open it up, and look in at a thermometer similar to this. She would record the Minimum and Maximum temperatures shown by the markers, often she would also record the temperature at the time of observation, and then press the reset button (seen in the middle) which would return the Min/Max markers to the tops of the mercury columns on either side. The motion of the mercury columns over the next 24 hours would move the markers to their respective new Minimums and Maximums for that period.
With only these measurements recorded, the closest to a Daily Average temperature that could be computed was the median of the two. To be able to compare modern temperatures to past temperatures, it has been necessary to use the same method to compute Daily Averages today, even though we have recorded measurements from automated weather stations every six minutes.
Nick Stokes discussed (in this linked essay) the use and problems of Min-Max thermometers as it relates to the Time of Observation Adjustments. In that same essay, he writes
Every now and then a post like this appears, in which someone discovers that the measure of daily temperature commonly used (Tmax+Tmin)/2 is not exactly what you’d get from integrating the temperature over time. It’s not. But so what? They are both just measures, and you can estimate trends with them.
And Nick Stokes is absolutely correct — one can take any time series of anything, find all sorts of averages — means, medians, modes — and find their trends over different periods of time.
In this case, we have to ask the question: What Are They Really Counting? I find myself having to refer back to this essay over and over again when writing about modern science research which seems to have somehow lost an important thread of true science — that we must take extreme care with defining what we are researching — what measurements of what property of what physical thing will tell us what we want to know?
Stokes maintains that any data of measurements of any temperature averages are apparently just as good as any other — that the median of (Tmax+Tmin)/2 is just as useful to Climate Science as a true average of more frequent temperature measurements, such as today’s six-minute records. What he has missed is that if science is to be exact and correct, it must first define its goals and metrics — exactly and carefully.
So, we have raised at least three questions:
1. What are we trying to measure with temperature records? What do we hope the calculations of monthly and annual means and their trends, and the trends of their anomalies [anomalies here always refers to anomalies from some climatic mean], will tell us?
2. What does (Tmax+Tmin)/2 really measure? Is it quantitatively different from averaging all the six-minute (or hourly) temperatures for the day? Are the two qualitatively different?
3. Does the currently-in-use (Tmax+Tmin)/2 method fulfill the purposes of any of the answers to question #1?
I will take a turn at answering these question, and readers can suggest their answers in comments.
What are we trying to measure?
The answers to question #1 depends on who you are or what field of science you are practicing.
Meteorologists measure temperature because it is one of the key metrics of their field. Their job is to know past temperatures and use them to predict future temperatures on a short term basis — tomorrow’s Hi and Lo, weekend weather conditions and seasonal predictions useful for agriculture. Temperature predictions of extremes are an important part of their job — freezing on roadways and airport runways, frost and freeze warning to agriculture, high temperatures that can affect human health and a raft of other important meteorological forecasts.
Climatologists are concerned with long-term averages of ever changing weather conditions for regions, continents and the planet as a whole. Climatologists concern themselves with the long-range averages that allow them to divide various regions into the 21 Koppen Climate Classifications and watch for changes within those regions. The Wiki explains why this field of study is difficult:
“Climate research is made difficult by the large scale, long time periods, and complex processes which govern climate. Climate is governed by physical laws which can be expressed as differential equations. These equations are coupled and nonlinear, so that approximate solutions are obtained by using numerical methods to create global climate models. Climate is sometimes modeled as a stochastic [random] process but this is generally accepted as an approximation to processes that are otherwise too complicated to analyze.” [emphasis mine — kh]
The temperatures of the oceans and the various levels of the atmosphere, and the differences between regions and atmospheric levels, are, along with a long list of other factors, drivers of weather and the long-term differences in temperature are thus of interest to climatology. The momentary equilibrium state of the planet in regards to incoming and outgoing energy from the Sun is currently one of the focuses of climatology and temperatures are part of that study.
Anthropogenic Global Warming scientists (IPCC scientists) are concerned with proving that human emissions of CO2 are causing the Earth climate system to retain increasing amounts of incoming energy from the Sun and calculate global temperatures and their changes in support of that objective. Thus, AGW scientists focus on regional and global temperature trends and the trends of temperature anomalies and other climatic factors that might support their position.
What do we hope the calculations of monthly and annual means and their trends will tell us?
Meteorologists are interested in temperature changes for their predictions, and use “means” of past temperatures to set an expected range to know and predict when things are out of these normally expected ranges. Temperature differences between localities and regions drive weather which makes these records important for their craft. Multi-year comparisons help them to make useful predictions for agriculturalists.
Climatologists want to know how the longer-term picture is changing — Is this region generally warming up, cooling off, getting more or less rain? — all of these looked at in decadal or 30-year time periods. They need trends for this. [Note: not silly auto-generated ‘trend lines’ on graphs that depend on start-and-end points — they wish to discover real changes of conditions over time.]
AGW scientists need to be able to show that the Earth is getting warmer and use temperature trends — regional and global, absolute and anomalies — in the effort to prove the AGW hypothesis that the Earth climate system is retaining more energy from the Sun due to increasing CO2 in the atmosphere.
What does (Tmax+Tmin)/2 really measure?
(Tmax+Tmin)/2, meteorology’s daily Tavg, is the median of the Daily High (Tmax) and the Daily Low (Tmin) (please see the link if you are unsure why it is the median and not the mean). The monthly TAVG is in fact the median of the Monthly Mean of Daily Maxes and the Monthly Mean of the Daily Mins. The Monthly TAVG, which is the basic input value for all of the subsequent regional, statewide, national, continental, and global calculations of average temperature (2-meter air over land), is calculated by finding the median of the means of the Tmaxs and the Tmins for the month for the station, arrived at by adding all the daily Tmaxs for the month and finding their mean (arithmetical average) and adding all the Tmins for the month, and finding their mean, and then finding the median of those two values. (This is not by a definition that is easy to find — I had to go to original GHCN records and email NCEI Customer Support for clarification).
So now that we know what the number called monthly TAVG is made of, we can take a stab at what it is a measure of.
Is it a measure of the average of temperatures for the month? Clearly not. That would be calculated by adding up the Tavg for each day and dividing by the number of days in the month. Doing that might very well give us a number surprising close to the recorded monthly TAVG — unfortunately, we have already noted that the daily Tavgs are not the average temperatures for their days but at are the medians of the daily Tmaxs and Tmins.
The featured image of this essay illustrates the problem, here it is blown up:
This illustration is from an article defining Means and Medians, we see that if the purple traces were the temperature during a day, the median would be identical for wildly different temperature profiles, but the true average, the mean, would be very different. [Note: the right hand edge of the graph is cut off, but both traces end at the same point on the right — the equivalent of a Hi for the day.] If the profile is fairly close to a “normal distribution” the Median and the Mean are close together — if not, they are quite different.
Is it quantitatively different from averaging all the six-minute (or hourly) temperatures for the day? Are the two qualitatively different?
We need to return to the Daily Tavgs to find our answer. What changes Daily Tavg? Any change in either the daily Tmax or the Tmin. If we have a daily Tavg of 72, can we know the Tmax and Tmin? No, we cannot. The Tavg for the day tells us very little about the high temperature for the day or the low temperature for the day. Tavg does not tell us much about how temperatures evolved and changed during the day.
Tmax 73, Tmin 71 = Tavg 72
Tmax 93, Tmin 51 = Tavg 72
Tmax 103, Tmin 41= Tavg 72
The first day would be a mild day and a very warm night, the second a hot day and an average sort of night. The second could have been a cloudy warmish day, with one hour of bright direct sunshine raising the high to a momentary 93 or a bright clear day that warmed to 93 by 11 am and stayed above 90 until sunset with only a short period of 51 degree temps in the very early morning. Our third example, typical of the high desert in the American Southwest, a very hot day with a cold night. (I have personally experienced 90+ degree days and frost the following night.) (Tmax+Tmin)/2 tells us only the median between two extremes of temperature, each of which could have lasted for hours or merely for minutes.
Daily Tavg, the median of Tmax and Tmin, does not tell us about the “heat content” or the temperature profile of the day. If daily Tmaxs and Tmins and Tavgs don’t tell us the temperature profile and “heat content” of their days, then the Monthly TAVG has the same fault — being the median of the mean of Tmaxs and Tmins — cannot tell us either.
Maybe a graph will help illuminate this problem.
This graph show the difference between daily Tavg (by (Tmax+Tmin)/2 method) and the true mean of daily temperatures, Tmean. We see that there are days when the difference is three or more degrees with an eye-ball average of a degree or so, with rather a lot of days in the one to two degree range. We could punch out a similar graph for Monthly TAVG and real monthly means, either of the actual daily means or from averaging (finding the mean) of all temperature records for the month).
The currently-in-use Tavg and TAVG (daily and monthly) are not the same as actual means of the temperatures during the day or the month, they are both quantitatively different and qualitatively different — they tells us different things.
So, YES, the data are qualitatively different and quantitatively different.
Does the currently-in-use (Tmax+Tmin)/2 method fulfill the purposes of any of the answers to question #1?
Let’s check by field of study:
Meteorologists measure temperatures because it is one of the key metrics of their field. The weather guys were happy with temperatures measured to the nearest full degree. One degree one way or the other was not big deal (except at near freezing). Average weather can also withstand an uncertainty of a degree or two. So, my opinion would be that (Tmax+Tmin)/2 is adequate for the weatherman, it is fit for purpose in regards to the weather and weather prediction. For weather, the weatherperson knows the temperature will vary naturally by a degree or two across his area of concern, so a prediction of “with highs in the mid-70s” is as precise as he needs to be.
Climatologists are concerned with long-term ever changing weather conditions for regions, continents and the planet as a whole. Climatologists know that past weather metrics have been less-than-precise — they accept that (Tmax+Tmin)/2 is not a measure of the energy in the climate system but it gives them an idea of temperatures on a station, region, and continental basis, close enough to judge changing climates — one degree up or down in the average summer or the winter temperature for a region is probably not a climatically important change — it is just annual or multi-annual weather. For the most part, climatologists know that only very recent temperature records get anywhere near one or two degree precision. (See my essay about Alaska for why this matters).
Anthropogenic Global Warming scientists (IPCC scientists) are concerned with proving that human emissions of CO2 are causing the Earth climate system to retain increasing amounts of incoming energy from the Sun. Here is where the differences in quantitative values, and the qualitative differences, between (Tmax+Tmin)/2 and a true Daily/Montly mean temperature comes into play.
There are those who will (correctly) argue that temperature averages (certainly the metric called GAST) are not accurate indicators of energy retention in the climate system. But before we can approach that question, we have to have correct quantitative and qualitative measures of temperature reflecting changing heat energy at weather stations. (Tmax+Tmin)/2 does not tell us whether we have had a hot day and a cool night, or a cool day and a warmish night. Temperature is an intensive property (of air and water, in this case) and not properly subject to addition and subtraction and averaging in the normal sense — temperature of an air sample (such as in an Automatic Weather Station – ASOS) — is related to but not the same as the energy (E) in the air at that location and is related to but not the same as the energy in the local climate system. Using (Tmax+Tmin)/2 and TMAX and TMIN (monthly mean values) to arrive at monthly TAVG does not even accurately reflect what the temperatures were and therefore will not, and cannot, inform us properly (accurately and precisely) about the energy in the locally measured climate system and therefore when combined across regions and continents, cannot inform us properly (accurately and precisely) about the energy in regional, continental or the global climate system — not quantitatively in absolute terms and not in the form of changes, trends, or trends of anomalies.
AGW science is about energy retention in the climate system — and the currently used mathematical methods — all the way down to the daily average level — despite the fact that, for much of the climate historical record, they are all we have — are not fit for the purpose of determining changing energy retention by the climate system to any degree of quantitative or qualitative accuracy or precision.
Weathermen and women are probably well enough served by the flawed metric as being “close enough for weather prediction”. Hurricane prediction is probably happy with temperatures within a degree or two – as long as all are comparable.
Even climate scientists, those disinterested in the Climate Wars, are happy to settle for temperatures within a degree or so — as there are a large number of other factors, most which are more important than “average temperature”, that combine to make up the climate of any region. (see again the Koppen Climate Classifications).
Only AGW activists insist that the miniscule changes wrested from the long-term climate record of the wrong metrics are truly significant for the world climate.
Bottom Line:
The methods currently used to determine both Global Temperature and Global Temperature Anomalies rely on a metric, used for historical reasons, that is unfit in many ways for the purpose of determining with accuracy or precision whether or not the Earth climate system is warming due to additional energy from the Sun being retained in the Earth’s climate system and is unfit in many ways for the purpose of determining the size of any such change and, possibly, not even fit for determining the sign of that change. The current method does not properly measure a physical property that would allow that determination.
# # # # #
Author’s Comment Policy:
The basis of this essay is much simpler than it seems. The measurements used to form GAST(anomaly) and GAST(absolute) — specifically (Tmax+Tmin)/2, whether daily or monthly) are not fit for the purpose of determining those global metrics as they are presented to the world by AGW activist scientists. They are most often used to indicate that the climate system is retaining more energy and thus warming up….but the tiny changes seen in this unfit metric over climatically significant periods of time cannot tell us that, since they do not actually measure the average temperature, even as experienced at a single weather station. The additional uncertainty from this factor increases the overall uncertainty about GAST and its anomalies to the point that the uncertainty exceeds the entire increase since the mid-20th century. This uncertainty is not eliminated through repeated smoothing and averaging of either absolute values or their anomalies.
I urge readers to reject the ever-present assertion that “if we just keep averaging averages, sooner or later the variation — whether error, uncertainty, or even just plain bad data — becomes so small as not to matter anymore”. That way leads to scientific madness.
There would be different arguments if we actually had an accurate and precise average of temperatures from weather stations. Many would still not agree that the temperature record alone indicates a change in retention of solar energy in the climate system. Energy entering the system is not auto-magically turned into sensible heat in the air at 2-meters above the ground, or in the skin temperature of the oceans. Changes in sensible heat in the air measured at 2-meters and as ocean skin temperature do not necessarily equate to increase or decrease of retained energy in the Earth’s climate system.
There will be objections to the conclusions of this essay — but the facts are what they are. Some will interpret the facts differently, place different importance values on different facts and draw different conclusions. That’s science.
# # # # #
The median is a bit undefined with only two points. Think of if you add a third point inbetween.
The median would then be that point no matter where it is placed between the two original points.
The median and the average should be used carefully.
Svend ==> “The median is a bit undefined with only two points.” Yes, but originally they only had the two points, and modernly, they ignore all the other points and just use the two points, and find the mid-point between the two (or the median of the Max and Min of the whole set).
MY opinion is that the current method results in a metric unfit for some purposes.
Svend,
But the definition of “median” should be applicable to lists of any length equal to or greater than two elements, and work for lists with both even and odd numbers of elements. A mid-point value interpolated between two extremes is not a highly informative metric for a ‘distribution,’ and that is another criticism of using it (whatever you want to call it) instead of using an arithmetic mean of a large number of samples.
Kip, In addition to your Tavg – Tmean plot it would have been nice if you had provided a plot of both Tmin – Tmax/2 and the average of the 6 minute readings to go with your explanation. Two pictures are worth a bunch of words :<)
Joe ==> This has been hashed over many times from that viewpoint — there are essays here from others. It is not just that they are different, and my how much, but rather “What are they measuring?”.
I agree with you Kip that is the problem that is never addressed.
LdB ==> You would be surprised by how often errors like this occur in modern science — many fields — counting/measuring one thing then claiming it represents a totally different thing….the worst cases happen in instances like this, in which there is an existing data set gathered for one purpose being used years later for a much different purpose. The data was not selected for the modern purpose and is not actually a measure of the modern thing claimed fore it.
Health and diet studies are far far worse than CliSci.
Kip, The only reason I suggested graphing the two together is it might better show the difference between the two metrics than does your Tavg – Tmean graph of Boulder temperatures.
I agree with you when you say: “Changes in sensible heat in the air measured at 2-meters and as ocean skin temperature do not necessarily equate to increase or decrease of retained energy in the Earth’s climate system.” I would guess that while not perfect, the Argo system could come closer to measuring long term variations in the retained energy of the system than you will ever get from land based thermometers. If you view the Earth’s climate as an electrical circuit, the ocean is nothing but a huge capacitor that dampens any long term changes in retained energy. On land, the further you are from the seashore the wider the variations in temperature. Land temperatures may work for short term local weather and climate, but the ocean controls the long term.
Joe ==> ARGO will give us a better idea of the true drivers of climate and its changes, in about twenty years, maybe. It has next to nothing to do with “ocean heat content” — in my opinion– climate is driven by the barely understood dynamics (chaotic dynamics — as in chaos theory) of ocean currents, up-dwellings and down-dwellings.
AGW scientists use absurdly “precise” estimate of alleged ocean heat content to bolster their AGW hypothesis.
Of course, it would take an instantaneous reading of the temperature, mass and specific heat of each cubic meter of the ocean to know the actual ocean heat content. However, you can come quite a bit closer with the ARGO system than with the land based thermometers. For example, you have a lot better coverage of the ocean with ARGO than you have with land based thermometers. You don’t have assume one reading represents the temperature of several hundred square kilometers. And further, with the level of heat input to the system and average specific heat of ocean water you can assume that slightly differing times of measurement don’t present near as much of a problem as the differing times of observing the land based thermometers. Granted the ARGO system hasn’t been available long enough to determine long term changes but they have screwed up the data from land based thermometers to point where it is totally unreliable and personally I thing unusable.
If hourly records are available from some location or another, do they tell a different story than using (Tmax+Tmin)/2? Does a comparison of competing data sets really show that the (Tmax+Tmin)/2 is inappropriate?
Steve O ==> Comparing the numbers will not tell us which is appropriate. That is a prior step in the scientific process…in this case left undone. The Min/Max method had been in use for a long time, and to compare modern records to the past, they have continued to use a less appropriate method that is not a measure of the thing they want to know.
“…is not a measure of the thing they want to know.”
— I don’t see a shred of evidence that that’s really the case. If you were to measure the temperature at 10am every day you’d record neither the min, the max, nor the average. It wouldn’t be anything other than the 10am temperature. Every day. In a lot of locations. Creating a large set of data.
If what you’re interested in are trends, what difference does it make? If there is a 0.6 degree change in average temperatures over a period of time, won’t the change will be apparent in the data just as clearly as if you took hourly measurements? Where is the evidence that this is not the case? None is presented.
Here is my problem with this method. Are minimum temps trending upwards? Are maximum temperatures trending up? Is one changing while the other isn’t? Why are studies on flora and fauna not using min and max temps in the region being used to determine which or both are affecting the results. It is too simplistic to simply rely on some ephemerial, fudged up “global temperature” value to determine exactly what is happening.
Jim Gorman,
The answer to your question is “Yes.” See the max/min temperature graphs here:
http://wattsupwiththat.com/2015/08/11/an-analysis-of-best-data-for-the-question-is-earth-warming-or-cooling/
Steve O ==> I am rather puzzled by your comment. If there were only 10 am measurements, and one used that record to get a trend for the month/year, you would have a trend of 10 am temperatures for that station. True enough. It would tell you almost nothing about the climatic conditions surrounding the station but you’d sure see the seasonal change of 10 am temperatures.
In science, we take measurements that we have carefully considered and designed so that the measurements will tell us about the thing we are interested in find out.
I don’t think too many scientists want to know about 10 am temperatures, at any geographical level — station, county, state, region, continent, or global. 10 am temperatures would just be a curiosity.
What Climate Scientists want to know is if climate regions are shifting from one climate type to another, if there is a continental change of importance, will the monsoons depart from their historic patterns, big things about long-term conditions.
it does, however, give you the most accurate possible representation of trends in 10am temps at that location.
that’s all you can get and that’s how to get it.
making a stew of mud by averaging multivariate data gets you clisci.
is it numerological magic or illuuuuuuusion?.
as in the animation i linked above, you plot the individual locations
then you play it back at whatever speed you find convenient to watch the trend-
if you want to see seasons, you play it slow.
if you want to see decadal variation, you speed it up.
how is this not obvious and easy?
you can watch the whole world change by the day with the ultimate possible actual resolution.
it ain’t broke and there’s no sense trying to fix it till it is.
dammit… ima do splainin one time and never again.
there is a program called Faces used by law enforcement to produce the famous ‘artist rendering’ of suspects being sought. (metaphor for ‘global temperature)
the program is a kind of digital flip book that lets you select facial features from a catalog representing a range of such features (metaphor for local temperatures)
the distinction of the program is that it produces one single number that represents the entire face (metaphor for the statistics used to produce gta)
now – how do they do that?
you can just send the number over the phone and at the other end they can reproduce the exact likeness it defines – not represent- define. as in science, not story telling.
this is how:
each selection of hair has a distinct number. (metaphor for death valley noon temp)
each selection of forehead has a distinct number. (metaphor for vostok midnight temp)
each selection of eyebrows has a distinct number. (metaphor for airport at Tuvalu at 6am)
and so forth.
do the add them up and divide the way a climate scientist do?
how would you do it right?
the answer is easy and obvious.
you concatenate.
you make a representation that is a string of unaltered values:
hair.forrid.eyebrow.eye.nose.lips.chin.facialtopiary (metaphor for nonexistent display ‘atom’ of global temperature at a moment)
you plot the stuff separately and if you want local, you zoom in; if you want regional, you zoom out.
if you want to watch seasons, you play it slow. if you want to watch millenia, speed it up.
“Temperature predictions of extremes are an important part of their job…”
Temperature predictions must consider the enthalpy (the total energy content above absolute zero) of the observed air parcels. The essay covers well enough to concept that increasing the heat energy stored in air can be done in two ways: increasing the heat capacity of a parcel of air, or increasing its temperature.
Thermals are created by rising parcels of air. It may be surprising to some that just because it is rising, it does not have to be warmer. It may have more water vapour in it than the air nearby. Moist air weighs less than dry air but holds much more heat. The moist parcel of air might also be warmer, so it rises “faster” than it would if it were the same temperature. Buoyancy and all that.
So to calculate where rising air will rise to, and how quickly, and how the parcel will cool as it rises, it is necessary to consider the humidity and the temperature.
To quantify the energy in the atmosphere, the weather forecaster, the climatologist and the physicist studying the retention of heat by back-scattered IR – all these scientists have to consider the temperature and the humidity all the time. The high temp, the low temp, the average temp, the time-averaged temp are all part of any analysis, but convey no definitive information about the state of the system because temperature is only one of the two variables needed to consider to tell us anything meaningful.
Let’s look at the amount of energy involved:
It takes about 1000 Joules to heat a kg of dry air 1 degree C.
It takes about 77,000 Joules to heat a kg of saturated air from 25 Degrees to 26 degrees.
If you are concerned about “energy being stored” in the atmosphere or a change in that quantity, temperature has much less influence than humidity. If today was “going to 30” instead of 29 because of “global warming” and a small amount of moisture is added to the air, it will be 29: no change, but slightly more humid. You cannot lean that from the temperature.
If it is significantly drier then the max will be much warmer, which tells us absolutely nothing on its own.
It is interesting that Trenberth and others are concerned with the enthalpy of the oceans (bulk temperature, mass and specific heat) but not the atmosphere (temperature only).
The GISS enthalpy set is hardly the first thing on anyone’s list of demonstrations of global warming or cooling. I suspect it is because they don’t know, and have not known, what they are doing. It makes no sense to ignore some of the essential information. They are making claims for an increase in energy in the system without showing the change in enthalpy. Instead they show the change in the temperature.
Crispin ==> Interesting,but a bit wide of the point of the essay. 🙂
Crispin, your values for the heat capacity of dry and saturated air looked a bit off to me. The value for dry air is correct, with my database giving a heat capacity of 1001 J/kg-K. Air at one standard atmosphere pressure that is saturated with water at 25°C has about 3.13 mol% water in it (equal to the vapor pressure of water divided by the total pressure – ie ideal behavior, which is a good assumption here). This is equal to 1.97 mass% water. The heat capacity of water vapor is 1864 J/kg-K. We can add the heat capacities of the two components multiplied by their concentrations to get the heat capacity of the mixture, since it is a gas, and very close to ideal. Therefore, 1001*.9803 + 1864*0.0197 = 1018 J/kg-K for saturated air.
Kip
I am not sure whether all stations are reporting Tmean as you are suggesting
here I anlysed the daily data of 10 stations here in SA:
https://www.dropbox.com/s/h7944heslj7gg7q/summary%20of%20climate%20change%20south%20africa.xlsx?dl=0
if you scroll down you get to see the daily data averaged to annual data of one station (George)
in the column order
T mean/T max/Tmin
If I take ( Tmin +Tmax )/2 it does not give me Tmean….
so what did they do?
Henry ==> Where is the data from? Look at the GHCN monthly report for that station. These don’t seem to be standard USHCN, GHCN, or MET format — so I can’t tell what they’ve done.
If you give me al ink to the real original station reports, I’ll take a look.
https://www.tutiempo.net/clima/ws-688280.html
This is the original data for George [South Africa}
Sorry Kip
I am a bit late with this.
Are you still watching comments here?
Henry ==> https://www.tutiempo.net/clima/ws-688280.html is reporting:
T Temperatura media anual (Annual Average)
TM Temperatura máxima media anual (Tmax annual average)
Tm Temperatura mínima media anual (Tmin annual average)
This is not original data by any standard. I could not find any detailed monthly temperature data at the site.
There is no indication what methods they have used or the source for their data. Without knowing the definitions of the data, we are lost.
The methods described in the essay are for the international standard used at GHCN.
Kip
what do you mean? you cannot find the daily and monthly data?
just click on the year, scroll down and you will find the individual months,
then click on the month, for example enero (January) and find the daily data.
henryp ==> Try it today, I get “Debido al uso abusivo que algunos usuarios hacen de este servicio, nos hemos visto obligados a bloquear el acceso a los datos de forma directa.”
sorry
I am flabbergasted.
They want you to pay now to see the daily data….
This was not the way it was before.
Someone has recognized the data has meaning and value in the hands of others.
True enough.
also, without the daily data access you cannot do any filling in for the years with a few day’s data missing.
henryp ==> dang!
Kip
Luckily I saved all the data from the 54 stations that I looked at.
Anyway, it does not seem too expensive to buy a subscription to the premium website. It is better than relying on the US or UK data, which I don’t trust.
Anyway, have a look at my results for New York:
https://www.dropbox.com/s/lesioxrvh24a1on/NewYorkKennedyairport.xlsx?dl=0
I have added a 4th column (F) for (Tmax + Tmin)/2 and if you scroll down you see this plotted as the purple line together with Tmean (blue), Tmax (red) and Tmin (green)
Note that I record the following trends (in New York) from the trend lines
Tmax increasing by 0.027/annum, i.e. 1.2K since 1973
Tmin increasing by 0.005/annum, i.e. 0.2K since 1973
The increase in Tmax suggests that more heat is allowed through the atmosphere, leading to higher average temperatures:
Tmean rising by 0.022K/annum, i.e. 1.0K
(Tmax+Tmin)2 rising by 0.016K/annum i.e. 0.7K
Now, my question to you is: did it get 1 degree K warmer in NY since 1973 or is it 0.7K?
I hope you can tell me?
Henry ==> The size and complexity of NY Kennedy Airport quadrupled, air traffic increased exponentially, a million square feet of terminal space is air conditioned and that heat pumped into the, square miles of asphalt and concrete added to the very local environment.
Figure the corrections to Tmax for that and we’ll see. Oh, and don’t forget the general UHI for a ka-jillion New York homes and humans.
My guess — maybe NOT ANY real rise in temps there.
Kip
no. no. NY is one of my samples.It is relevant even for just looking at Tmin (click on my name to figure it out from my final report). Tmin did not really change that much over the past 44 years.. Anyway, I would not be surprised if one of the reasons for more energy coming through the atmosphere there, i.e. Tmax rising, also pushing up Tmean, is the cleaner air….less dust and carbon soot?
So, we never disregard a weather station that has good daily data.
However, I am bit confused now by your post. With the modern technology introduced in the 70’s I always thought that Tmean was calculated from all measurements over the day, equally spread, even once a minute is possible, {with the T recorders’ equipment that I know of] .
The way I summarize the data, I was not really bothered about differences in method of measurements between stations.
But now, my question remains, what is the correct warming for NY, or the closest in your opinion, is it 0.7 or is it 1.0K since 1973 as per the reported values.
Henry ==> I will try to be gentle….airports are notoriously affected by a special type of UHI….massive concrete runways, megatons of air conditioning, most have grown exponentially since the 1950s/1980s.
In moderate climates, Tmax climbs as the Sun warms and heat the runways. If you are using data unadjusted for UHI, it may well be responsible for ALL of the warming at Kennedy Airport.
Given that, a true Tmean (mean of all the recorded five-minute reading for the station) over the 24 hour period will always be a more accurate “average” than (Tmax+Tmin)/2 — but it will just be a mathematical result — it may not have the “physical meaning” of “the correct warming for NY” that you hope to ascribe to it.
The mathematical result is based on the simple rules of mathematics we all learned in elementary school — if you have calculated correctly, you have the correct mathematical (arithmetic) result and get a nice check mark for your answer. The math result is not the same as, and may be world’s different than, the physical understanding you are seeking.
There is the much deeper question of “Do either of these values, calculated for annual, tell us anything about weather or climate at that location?” “If yes, exactly what?” (Hint == neither will tell you if more energy from the Sun has been retained in the local weather/climate/environmental system. Neither of them is a measure of that.)
I do appreciate you hard work and calculations — they add a lot to the conversation.
Henry ==> For example:
@usurbrain, ASHRAE keeps a good dataset of DHD and DCD for the major world locations and a very good dataset for North America.
Regarding measuring energy in the atmosphere, as has been stated by Crispin, temperature is a poor measuring stick without at least 2 other properties in the standard Psychrometric Chart – RH%, Dry bulb, Wet bulb, Dew point, Specific Volume, Humidity Ratio.
https://www.engineeringtoolbox.com/docs/documents/816/psychrometric_chart_29inHg.pdf
Properties these were not recorded as often as temperature min/max but I did find historical records on evaporation rates which do help fill in some blanks.
https://www.engineeringtoolbox.com/evaporation-water-surface-d_690.html
Thanks.
Over my 75+ years I have noticed that rarely does the daily low temperature go below the dew point. And those occurrences are usually related to a weather front moving in. Note a climate scientist but there is a reason. Yet I see little discussed on climate sites.
The effect of the water vapor in the atmosphere has a much greater affect than it is given credit for I’M HOME.
Thanks.
Over my 75+ years I have noticed that rarely does the daily low temperature go below the dew point. And those occurrences are usually related to a weather front moving in. Note a climate scientist but there is a reason. Yet I see little discussed on climate sites.
The effect of the water vapor in the atmosphere has a much greater affect than it is given credit for I’M HOME.
Just a comment on median and mean. In mockery of statistics, it is sometimes said that the average person has one breast and one testicle. If you are looking at testicle numbers in a population, say, that can be useful information.
But the median person probably has two breasts and no testicle. That certainly describes a more well-rounded individual, but isn’t more informative about the population. There are times when only a histogram will do.
Nick ==> And I certainly agree — the “ONE NUMBER to rule them all” approach of some segments of the climate science world is mindbogglingly wrong-headed.
Climate scientists did not invent the notion of a mean. Or median.
Nick ==> They sure didn’t, they invented GLOBAL LAND-OCEAN TEMPERATURE INDEX — which I offer with no further comment.
Standing ovation and cheering.
*Shouts of “Exactly” heard throughout the stadium*
if you dont like the index dont combine sst and sat.
combine mnat and sat.
It’s not a mockery of statistics, it’s a question to see if a person understands statistics or just learnt the formulas. The distribution of breast and testicles is a bivariate distribution and if you start doing a mean on it, the statistics police come and arrest you.
LdB ==> Or at least, they ought to.
so the daytime temperature of death valley should be averaged with the night time temperature of vostok because they are not bivariate?
Don’t let the careless terminology of “climate science” confuse the issue: the average of Tmax and Tmin is neither the “mean” nor the “median” temperature; it’s actually the MID-RANGE temperature. Whatever convenience that may provide in simplifying calculation is entirely overshadowed by the fact that it’s a bastard statistic, with no unique analytic relationship to the entire population (profile) of daily temperatures or to their true median.
Why are comments disappearing?
1sky1 ==> It is a mystery and a conundrum….sometimes they go to moderation and appear later.
Good stuff too bad I’m 5 hours late (~:
Yes the night and day temps are averaged the the annual average of the cold months and warm months are averaged and then all those averages from stations in the tropics to the poles are averaged up into one anomaly from the 1950-1980 base year that we are supposed to believe actually means something
The average of 49 and 51 is 50, 25 and 75 is 50 and 1 and 99 is 50. The point is well made.
thats why we also look at tmax, tmin, diurnal range and varience.
kip just doesnt read the science
[Kip, I’ll try to bring some of the concepts from the previous post into this discussion. I’ll try to condense it and apply it directly this new discussion so as to not repeat the last one – except essential points.]
All,
I understand the desire to use the historical record (Tmax+Tmin)/2. As Roy said, “we are forced to use what we have.” Actually, we are not forced to do anything. We chose to use it. But should we?
There is a critical piece of math and science that seems to be missing from the toolbox of climate science and that is the sampling theory of signal analysis. Lacking this knowledge, I don’t think people understand just how far off the max/min record is from a properly sampled signal. I will try to refrain (probably fail) from offering my opinion about what we should really do with the historical max/min record. But I’d like to introduce into the discussion the failure of the max/min record to comply with the Nyquist Sampling Theorem and give some examples about the extent to which this introduces error into the data.
A few brief definitions – my apology to those who know and find this unnecessary.
Signal: The waveform of continuous, time varying effect in the natural world. When we measure temperature at a point in space and time we are measuring a signal.
Sample: Any measurement of that signal that results in a discrete value (decimal, binary, hexadecimal, etc.)
Nyquist Sampling Theorem: Requires that any band-limited signal be sampled at a frequency that is at least 2x the highest frequency component in the signal. This is the Nyquist frequency.
Aliasing: Sampling a signal creates a spectral image in the frequency domain of that signal and this image is located at the sample frequency. If the sample frequency is below the Nyquist frequency, then the spectral image overlaps the signal and the sample will contain error as a function of the overlap. This error is called aliasing.
Image below of what happens when a signal is sampled at Nyquist or above:
The blue signal shows the band-limited signal being sampled. Its bandwidth is “B”. Fs (fs) is the sample frequency. In this case Fs is > = 2B. So, Nyquist is satisfied. No overlap. No Aliasing.
A properly sampled signal has **ALL** of the information from the original signal. The digital sample can be used to perfectly reconstruct the original signal in the analog domain – although that is NOT the goal. Also, all digital signal processing (DSP) done on this properly sampled signal is valid mathematically.
The image below is what happens when a signal is sampled below the Nyquist rate:
This image shows two figures. The one on the left shows more extreme aliasing and the one on the right shows less aliasing. The solid blue line represents the signal being sampled and the dashed line represents the spectral image overlapping it. The sampled output of this signal will contain this error and this error can never be extracted after the fact. Any further mathematical operations (DSP) on the sampled data will have this error. Note: finding (Tmax+Tmin)/2 is a DSP operation, although a very simple one.
The point I will get to quickly is that (Tmax+Tmin)/2 is mathematically flawed. Let me develop this with an example. I’m using USCRN data, for the Cordova, AK station from 2017 (late July through December). This analysis works with any of the stations, this is just my specific example to illustrate the effects. The amount of error varies in each example, but the fundamentals are the same. The USCRN provides an automated sample every 5 minutes. (I won’t stray into the details about how this 5-minute sample is actually an average of 10 second samples, etc.) My research leads me to believe that this network uses high-quality instrumentation and practices – certainly relative to the dumpster fire of the non-USCRN instruments. 1 sample every 5 minutes equates to 288 samples per day. Based upon Nyquist it will not alias any signal that varies slower than once every 2.5 minutes. I think this rate is more than sufficient for a very accurate sampling result.
Using this data, I calculate the mean as generated by adding up all of the 5-minute samples over the 5-month period and divide by the number of samples. For my purpose the term “mean” is defined as follows. For any given period of time, the mean is the single value of constant temperature that results in the same area under the curve as the complex time varying curve. Said another way, if we calculate the area under the curve for the complex signal, the specified mean should exactly give us that same area. The area is a product of time and temperature. This is related to thermal energy delivered in the time period specified. The result created with 288 samples per day will be (and should be) considered the “gold standard”. Assuming the instruments are calibrated and working properly, the result is accurate and precise to the limit of the instrument, does not suffer reading error and is not subject to any TOB. I then calculate the mean by using a slower sample rate. The 288 samples per day is divided down by 4, 8, 12, 24, 48, 72, 144, corresponding to sample rates of 72, 36, 24, 12, 6, 4 and 2 samples per day. The sample rate decrease was achieved while maintaining a regular clock frequency – no “jitter” was added. This experiment allows us to see how aliasing error creeps into the sampled result as the sample rate is decreased. Fundamental to this is the understanding that the temperature signal on any given day has frequency content well above the 1 cycle/day fundamental – and we can see a lot of variation to this frequency profile from day to day.
Some graphs to illustrate.
Sampled signal in the time domain (288 samples/day):
Next, the FFT of the sampled signal showing the frequency spectral content. X-axis is in samples per day – showing the frequency bins. Y-axis shows the relative energy in each band. (Only displays results out to 52 samples/day)
The next figure is a table that shows how the mean varies as you go from 288 samples per day down to 2 samples per day. 2 samples/day means there is a lot of spectral content landing on the fundamental in the form of aliasing! You can see the extent of the degradation as sample rate decreases from 288/day. Now here is a kicker: Note the 2 values at the bottom of the table. Compare 2 samples/day to (Tmax+Tmin)/2. They are both technically 2 samples/day. However, the first value is generated by complying with a proper sampling clock: we have 2 regularly timed samples. The (Tmax+Tmin) takes 2 samples whenever they happen in time. Rarely are they well aligned to a clock. The max and min values are very good since they come from the high-quality instrument, but they are a disaster as it relates to the math governing sampling! 2 samples, that occur according to a valid sample clock, *usually* yield a result closer to the gold standard than the case of averaging the high quality max and min! (I say *usually* because there is a lot of randomness to the error – but not a Gaussian distribution just for the record). The sampling time variation is known as clock jitter and it creates an erroneous result.
The next graph shows the daily error of (Tmax+Tmin)/2 as compared to the “gold standard” 288 samples/day mean. If (Tmax+Tmin)/2 were correct, then this graph would show a horizontal line at y=0. It doesn’t show that. The red arrow shows the sampling error for 11/11/2017. Note the time-domain temperature profile for 11/11/2017 is not very typical. The sampling error is large, but not the largest. We see error that exceeds +/-2.5C at some points in the record.
(Tmax+Tmin)/2 measured from uncalibrated max/min instruments, with their associated reading, quantization and TOB error are expected to yield even far worse results.
If we are going to continue to use the historical (Tmax+Tmin)/2 data, we should do so knowing that this error is present. (In addition to calibration, reading, UHI, thermal corruption, quantization, siting, and data infill and manipulation.) Even modern (Tmax+Tmin)/2 measurements are loaded with error. This includes the satellite record, assuming my understanding of how the satellite measurements are made is correct.
Furthermore, what about the future? Each of the disciplines mentioned by Kip (Meteorologists, Climatologists, etc.) can benefit going forward from data that doesn’t violate basic mathematics of sampling. Nyquist came up with his theorem in 1928! Affordable converter technology and instrumentation has been available for 40 years!
Finally, Alarmists push “records” of 0.01C on us and trends of 0.1C/decade. As if all of the other errors were not enough to arm us to fight back, Nyquist is a cannon ball right through their hearts.
“that seems to be missing from the toolbox of climate science and that is the sampling theory of signal analysis”
It starts to seem as if you have a hammer, and are finding nails everywhere. But you still aren’t relating to what climate scientists actually do. What you are demonstrating is that if you want to use integral over 24 hours as the test, then the calculation of that will depend on sampling rate. And at 2 samples per day you’ll get a bias (which will change as you change that sampling time).
Well, we get a bias with min/max, depending on time, as is well known (TOBS). That is again what I was exploring at Boulder. And the bias is fairly stable over time, and so fades with using anomaly. If people chose to use twice/day sampling for an average, they would find the same. And they would again find it necessary to make a correction if the sampling time were changed.
Nick said: “It starts to seem as if you have a hammer, and are finding nails everywhere. But you still aren’t relating to what climate scientists actually do.”
Nick, what you are saying, apparently, is what climate scientists do is ignore the laws of mathematics. This might be the first time we agree in the short time we have been communicating.
Nick said: “What you are demonstrating is that if you want to use integral over 24 hours as the test, then the calculation of that will depend on sampling rate. And at 2 samples per day you’ll get a bias (which will change as you change that sampling time).”
No, I think you are confused. It doesn’t have to do with integrals, a test or bias. It has to do with understanding the physics and mathematics of sampling.
Just because one doesn’t understand sampling doesn’t excuse one from violating the laws of sampling. There are no footnotes to Nyquist. There isn’t a special exception for climate scientists. Higher frequencies cannot be ignored when sampling just because they are not needed for the analysis. If this is done, then aliasing occurs, and that frequency content comes right back and clobbers the fundamental. The data is wrong. A number can be obtained by using the incorrect process, but this number won’t accurately relate to the physical phenomenon that took place in the physical world. The only way to eliminate frequency content that you are not interested in is to filter it out in the analog domain before sampling or filter it out digitally after you sample. But I fail to see the “science” of ignoring significant energy in a daily temperature signal. If you grab a Tmax and Tmin and do math on it, you are going to have an even lower quality result than if you grab 2 samples properly. (Tmax+Tmin)/2 is literally the worst possible way to gain information from the temperature signal, except sampling 1x/day.
I’m trying to inform everyone about a fundament flaw in the methods. I’m actually shocked that what I’m presenting is novel and not known. I’m surprised that this information isn’t creating more curiosity. But I’d rather light a candle than curse your darkness. How can I help?
William,
Interesting!
What happens when you look on a month period? This would be like “62 samples of a very long day”. Will that decrease or increase this error?
Hello MrZ,
Thanks for your comment and question. I’m thinking about trying to put together a post on this subject, where I discuss it in more detail. It’s a big subject, so if you don’t mind, let’s connect again if/when I get that out here. Aside from responding to a few people who I already started to engage on another similar post, I think it might be good for me to not further dilute Kip’s core points on this essay. Okay for you?
“I’m trying to inform everyone about a fundament flaw in the methods. I’m actually shocked that what I’m presenting is novel and not known. I’m surprised that this information isn’t creating more curiosity. But I’d rather light a candle than curse your darkness. How can I help?”
Hey William, as humbly suggested under previous Kip essay write a proper article. With all necessary details, calculations and examples. Maybe will be published here, maybe in some technical journal. Comments usually are quickly lost among other more or less sensible ones. We’ve got highly-sampled data round the clock for last few years. Identified error drift between integral of the reference temperature (sampled often) and daily (Tmin+Tmax)/2, still used as the ‘basic unit’ for purpose of temperature tracking, will also help to estimate additional error which has to be associated with historical records.
Hi Paramenter,
I was perhaps too eager to continue to develop the points through this post. Your advice is wise and well received by me. The concepts I’m presenting are important, in my opinion, but they need to be developed in an essay specifically focused on that topic. I hope to do that and publish here if possible. To the extent my (far too long) post derailed the core points of Kip’s essay, I state my humble apology.
I do appreciate your reply. I’m enjoying the discussion and the interesting points and counter points offered by all.
William
“No, I think you are confused. It doesn’t have to do with integrals, a test or bias”
The narrowness of your focus on Nyquist is excessive. Climate scientists here are not trying to reconstruct a signal. It seems you are familiar with that and want to force everything into that framework. In fact, they are trying to compute a (monthly) average, and that is ideally an integral of temperature over the period divided by the time. As always, we have to deal with finitely many points, so it is numerical integration, and might as well be done with equally spaced intervals.
If you want to think of it as a Fourier decomposition, the signal is dominated by diurnal and its harmonics. So it makes sense to sample in fractions of a day. You then get beat frequencies. These should add, as sampled, to near zero, as should all the high frequency sinusoids, with maybe a small residue at the endpoints. But some harmonics will actually coincide with the sampling frequency (or its harmonics), and the beat frequency is zero. Or put another way, it returns the same value for each period, which adds up and the result isn’t zero.
In your 5 month calc, this happens with two per day sampling. The fundamental diurnal adds to zero, but the next harmonic has one sample per cycle, which will be constant and add up. The sum itself is a sine function of the phase of the sampling relative to diurnal. That is why your test showed a large difference with 2/cycle sampling. If you sample faster, that resolves that second harmonic, and the first such alignment will be for higher harmonics. Since the signal is reasonably smooth, these attenuate, and so the result converges with faster sampling.
Nick: “The narrowness of your focus on Nyquist is excessive.”
No narrower than insisting we don’t, for example, divide by zero.
Nick: “Climate scientists here are not trying to reconstruct a signal. It seems you are familiar with that and want to force everything into that framework.”
We agree that the goal is not to reconstruct the signal. But the goal is to accurately capture the signal so that the further work you do on it is not corrupted. It’s not about forcing something into a framework. I see it differently. Ignoring or hand-waiving away Nyquist is denial of the laws of mathematics that govern sampling.
Nick: “In fact, they are trying to compute a (monthly) average, and that is ideally an integral of temperature over the period divided by the time.”
Ok. I have no objection to highly weighting the importance of the monthly average, if that is what is important to climate science. You can get that by sampling properly. And I agree, the average over any period is the integral of the signal over that time-period. But the integral accuracy will be reduced proportionally to the error in the signal being integrated. You can have the benefit of working with any time-frame average you want if you sample properly.
Nick: “As always, we have to deal with finitely many points, so it is numerical integration, and might as well be done with equally spaced intervals.”
If samples are not equally spaced in time, this is (by definition) jitter. A reconstructed signal is reduced in accuracy as a function of the jitter. The samples don’t land in time where they are supposed to with jitter. Again, while the goal is not to reconstruct, the fact that you can’t accurately reconstruct is proof that your sampled signal has error.
Nick: “If you want to think of it as a Fourier decomposition, the signal is dominated by diurnal and its harmonics.”
Yes, the signal is a summation of a finite number of sinusoids (Fourier). Don’t forget there is actually a DC content to the signal, as seen in FFT results, but we can agree that this is not important. Next you have the diurnal (1 cycle/day), but there appears to be significant content in 2 cycles/day, 3 cycles per day and so on, up to 20-30 cycles/day. Different days have different content based upon the time domain profile of the signal. At a sample rate of 2 cycles/day (even assuming periodic clock), the spectral image is almost on top of the signal. It is only shifted by 2 frequency bins in the FFT. So, the 2, 3, 4, 5 cycle/day content clobbers the 1 and 2 cycle per day signal that is sampled.
Nick: “So it makes sense to sample in fractions of a day. You then get beat frequencies. These should add, as sampled, to near zero, as should all the high frequency sinusoids, with maybe a small residue at the endpoints. But some harmonics will actually coincide with the sampling frequency (or its harmonics), and the beat frequency is zero. Or put another way, it returns the same value for each period, which adds up and the result isn’t zero.”
I don’t see how the term “beat frequency” applies here, but I won’t get hung up on it. Per my comment directly above, I don’t agree with referring to the 2, 3, 4 and 5 cycle/day content as high frequency. I’m not sure where that threshold is. I would recommend we look at energy content in each frequency bin. Someone (perhaps a committee) needs to decide what percentage of energy is significant. As an engineer, who had to make things work so as to not kill people, I don’t like to throw anything away when it comes to accuracy. Especially when the technology gives it to us for free. But if a value is stated as the standard for required energy percentage, then the work can continue, and the results can take the criticism or praise that may be due appropriately. I don’t agree that the error zeros out, but I agree that the error distribution allow for some of the error to cancel out. More on that in a second.
Nick: “In your 5-month calc, this happens with two per day sampling. The fundamental diurnal adds to zero, but the next harmonic has one sample per cycle, which will be constant and add up. The sum itself is a sine function of the phase of the sampling relative to diurnal. That is why your test showed a large difference with 2/cycle sampling. If you sample faster, that resolves that second harmonic, and the first such alignment will be for higher harmonics. Since the signal is reasonably smooth, these attenuate, and so the result converges with faster sampling.”
I think you and I are saying some similar things but using a slightly different vocabulary. As the sample rate increases, the spectral image is pushed out in frequency. The overlap decreases with increasing sample rate. At the Nyquist rate the aliasing/overlap stops. I agree that as the overlap is constrained to parts of the spectrum with very “low” energy then the error is correspondingly “low”.
But we do see that (Tmax+Tmin)/2 does produce significant error compared to a Nyquist sampled signal over short time spans. Now, if this error value is plotted over a longer period, say several months or a year, then even without doing a mathematical analysis, it is apparent to the eye that the energy is somewhat symmetrical about the 0 value. It is not completely symmetrical because the error is not Gaussian. This is because the change in profile of a day does not behave according to a Gaussian function. The jitter of (Tmax+Tmin)/2 is also not Gaussian.
In summary, the key here is just how much error remains over longer periods of time. I have not proven this but think it is reasonable to assert that the remaining error will vary from sample set to sample set. We can probably find examples that show minimal residual error and we can find examples with a large amount of residual error.
2 things guide me that may not guide you. 1) An engineering career that depended upon using all of the accuracy that the technology and economics would allow. 2) The desire to inject some mathematical/science/instrumentation sanity into the alarming claims of climate science.
Has the gap in our views diminished any through this dialog?
William,
“Don’t forget there is actually a DC content to the signal, as seen in FFT results, but we can agree that this is not important”
We certainly don’t. The DC component is the answer. It is what you are seeking. You can think of your Fourier process (F Series over a month) as one of expressing the function in terms of a constant, and a set of sinusoids that are known to integrate to zero. And that is my point about forming a month average as being essentially different to reconstructing a signal. You create the sine components only to throw them away. If aliasing turns one sine into another, you don’t care, as long as its frequency is different from zero (which should be seen as, substantially greater than once a month). It is a sophisticated low pass filter.
That is why low frequency sampling can change the answer, as in your calculation. As I said, with 2/day sampling, the 2nd, fourth etc harmonics do not sum to zero, as they should. They alias to a constant, which is misidentified as a DC component.
I use that Fourier style of integration for spatial integration of the anomalies on a sphere. I have described it here. I decompose into spherical harmonics which, like the sinusoids on a line, are orthogonal, and in particular orthogonal to the zeroth order (constant). So the integral is just the integral of that constant. I have new ideas on this that I’ll be blogging about soon. I also use the SH fit as the monthly presentation of temperature anomalies.
“I don’t see how the term “beat frequency” applies here”
sampling is also the process of multiplying the signal by a Dirac comb and integrating. The Dirac comb, regularly spaced delta functions, also has a Fourier transform, which is just the summation of the harmonics of the sampling frequency. That multiplication, as in demodulation, generates sum and difference (“beat”) frequencies, of all the combinations of harmonics. Since they are all integer multiples of diurnal, the lowest non-zero frequency is diurnal. But the zeroes matter.
“the 2nd, fourth etc harmonics do not sum to zero”
That is, harmonics of diurnal frequency.
Hi Nick,
I can see, as others have said, you are extremely knowledgeable. I’m not quarreling with you for that sake alone. I see a lot of good things you write in your last post. At some points I couldn’t follow what you were saying – it’s not easy to get all of the points across in this format when things get very detailed unless you commit a lot of time to the effort. I’ll bet the conversation would be fun if we were in the same room, and using a white board to guide the discussion.
In some ways, I think we are saying the same thing but differently. But I still think we have a few key fundamental disagreements: You think the sampling error resolves itself over long averages and I do not think it does. Let me ask you a few questions to see where we agree and disagree.
1) Do you agree that sampling according to Nyquist (using a quality clock frequency) is the absolute best, most accurate method that will yield the best possible starting point with data? Furthermore, this method, follows all mathematical laws, is immune to TOB and reading error? And that with this method any analysis is possible (daily, monthly, yearly, etc.)?
2) Do you agree that the absolute worst method is 1 sample/day?
3) Do you agree that the next worst method is measuring Tmax and Tmin and doing calculations on those 2 measurements? (Please ignore whether or not this is adequate for the task – I’m not inferring any of that from your answer to the direct question).
4) Do you agree that a not so good method, but better than measuring Tmax and Tmin, is to sample 2x/day, but according to a periodic clock? In this method it is unlikely that either Tmax or Tmin will be captured (except by luck)?
5) Do you agree that increasing the sampling rate from 2x/day up to the Nyquist frequency increases the accuracy of the sample by reducing the effect of aliasing? And that there are no benefits to sampling above Nyquist?
The questions beyond this have to do with whether or not an averaged Tmax and Tmin are good enough, but if we can agree to shelve that for a moment, we can see if we have agreement on the other 5 listed questions.
Ps it was tempting to respond about AM modulation, heterodyning, etc. I could add some information about QPSK/QAM, PSK, etc., as it relates to sampling – but I’ll refrain from going full-blown geek. Also, regarding DC in the spectrum, I didn’t communicate my thought clearly. I was trying to acknowledge DC was present but was saying it wasn’t important to the particular point I was trying to make. I agree with some of what you said about that, to the extent I followed you – I lost your drift at a certain point – hence my desire to raise the altitude to find some common ground.
William,
I have done some calculations like yours. I didn’t use Cordova, which had a lot missing. I used 2017 (full year) from Cullman AL, as you did earlier. Firstly here are my sample results. The rows are sampling 6,4 and 2 times per year. The cols are phase of sampling, relative to sample period, starting at midnight. Soo for 2 samples, it is every 1.5 hrs, etc.
___0___45___90__135__180__225__270__315_range
6 16.3 16.2 16.2 16.3 16.4 16.4 16.3 16.3 0.1
4 16.1 16.1 16.2 16.4 16.5 16.5 16.4 16.2 0.4
2 15.3 15.4 16.1 16.7 17.0 17.1 16.8 16.2 1.8
Then I calculated the first 9 fourier coefficients (1, sin(x),cos(x),sin(2x)…)
16.30 -3.70 2.43 -0.45 -0.78 -0.28 0.25 -0.12 -0.18
or as RMS
16.30 4.43 0.90 0.38 0.21
This is for the average of each 5 min over the year.
Here is a plot of the fit
From the RMS, it is clear that the 2x sampling range of 1.8 corresponds to the RMS for the 2nd harmonic of 0.9, and likewise for 4x sampling, the 0.4 range corresponds to the 0.21 4th harmonic.
Then I tried to measure the jitter of min/max sampling. I first did the min/max average using midnight reset. The mean was 16.76°C (but depends on reset time – more to do here). Then I tried averaging the harmonics sampled at those min/max points of the day:
0.76, 0.09, -0.09, -0.01.
With the mean, these add up to 17.05. I had expected this to be close to the max/min mean of 16.76, and it isn’t that close. I’m not sure why. However, I think it is clear that the big contribution to the difference is the jitter effect on the first harmonic.
Nick,
I just read about the work you did with the 2017 Cullman AL data, where you were looking at the phase information and Fourier coefficients. I see you have a strength and clear proficiency with these tools. And it is quite logical to utilize our proficiencies for analysis. It looks interesting but I’m not sure how to fit it in exactly. This is probably my deficiency – not a problem with your analysis. However, I think it is simpler and more intuitive to just stick with Nyquist analysis. I think what I did to compare the increase in error as the sample rate decreases is easier to comprehend and better illustrates what is happening. Basically, as the sample rate decreases then the integral of the sampled waveform (the mean / the Temperature-Time product / the “area under the curve”) differs from the Nyquist sampled result.
Kip just issued his epilogue, so I think that is the “last call for alcohol” so to speak. So, I’m not sure how much further to take this here, but I do hope to publish an essay in the future to focus on Nyquist specifically. Before we close, would you be willing to kindly answer my 5 questions directly? I’m honestly interested to see how close we are on this. While we may differ on whether or not Tmax and Tmin are sufficient for the purpose, I’m starting to think/hope you actually agree with most of the 5 questions – meaning you would say “yes” to the questions. I would value the answers, and if any of them are “no”, we can agree to disagree and “park it” until the next time.
What do you think?
William,
I’ll look forward to reading what you have to say. On your 5 questions:
“1) Do you agree that sampling according to Nyquist …”
Nyquist is relevant if you have a desired frequency band that you want to resolve. Then you can say whether you are “according to Nyquist”. Here the frequency band is effectively zero (integration for monthly mean). It’s true that poorly resolved high frequency processes can give spurious low frequency effects, but I’m not sure Nyquist is the best way of thinking about that.
2. “the absolute worst method is 1 sample/day”
It would be very bad. The result would entirely depend on the time of sampling, building in the diurnal range as an error.
3. “the next worst method is measuring Tmax and Tmin”
No. It locates the main features of the distribution. Errors are then in the shape in between (1sky1’s asymmetry, or my second harmonic). I think it is better than sampling twice a day.
” but better than measuring Tmax and Tmin, is to sample 2x/day”
No, but anyway, it isn’t an option. For the past, we can’t redo; for the present, we can do much better than twice a day.
“Do you agree that increasing the sampling rate from 2x/day up to the Nyquist frequency”
I’ve asked several times – what do you think the Nyquist frequency actually is for monthly averaging, and how do you work it out? In fact you are trying to measure a slow process, with a fast process running interference. regular Nyquist analysis doesn’t really cover this. But yes, your calc and mine suggests that because the diurnal cycle is such a big intrusion, resolving it to better than two samples a day will help avoid spurious effects.
I think the best way is to try to identify cycles and subtract them out, on the basis that their integrals are known to be zero. Then you can deal with the smaller residues. You won’t be making errors from discretising the cycles.
Nick,
You said: “I’ve asked several times – what do you think the Nyquist frequency actually is for monthly averaging, and how do you work it out?”
Actually, I have answered this at least once. But let me add a few thoughts. Let’s assume an electronic instrument for this example: Basically, a thermocouple feeding an instrumentation amplifier which feeds an ADC (analog-to-digital converter). The thermal mass of the front-end (the physical front-end) will act as a filter. The larger the mass the slower the response time to temperature transients. A smaller mass will have the opposite effect. I’m sure you know this. Some thermal signal content gets filtered out by this mass – but I’ll ignore this for the moment. Let’s look at the signal that gets through the filter. This could easily be attached to a calibrated analog spectrum analyzer. Air temperature signal changes would be glacial to any spectrum analyzer. It would be easy to determine the content in the signal and you could denote this is bandwidth “B”. Nyquist would be 2B (2*B). I assume NOAA has already done exercises that have led to their 5-minute sample rate. Modern instrument converters can easily sample at hundreds of thousands of samples a second, so the only reason to go slower is to not drown in sample data. Also, sampling much above Nyquist doesn’t add any benefit. Nyquist sampled data allows you to completely reconstruct, as we have discussed. Properly sampled, you can then achieve any average you want (daily, monthly, yearly). Using Tmax and Tmin you get error and you have to hope it averages out over time.
You know about the TOB problems. Going forward why not use the method that eliminates TOB and all of the potential problems. All plusses – no minuses (except kicking the orthodoxy).
I appreciate you answering the questions, but I had hoped we were closer in our understandings. It appears a big gap still exists. We can agree to disagree for now. I don’t think we are going to be able to persuade each other further. It’s too bad. Nyquist is so elegant. With instrumentation technology, processing power and data storage being of such high quality and low cost we can literally record the entire signal of every day at every station. Who knows the value that could provide in the future? Maybe one day we will actually discover some mathematical relationships between climate variables and knowing the full signal will be valuable to that development.
Actually, the vagaries of irregular daily sampling of the mid-range value (Tmax+Tmin)/2 are far removed from the vagaries of spectral aliasing produced by overly-sparse periodic sampling of the signal. The latter simply fold the bilateral power spectrum S(f) around the Nyquist frequency fN in accordion-like fashion; the sample mean value is affected ONLY IF power is aliased into zero-frequency. Because the diurnal temperature cycle is consistently asymmetric, the former produce a statistic CONSISTENTLY different from the sample mean. Thus it is misleading to refer to the mid-range “error” or to explain the discrepancy as an aliasing effect.
” The latter simply fold the bilateral power spectrum S(f) around the Nyquist frequency fN in accordion-like fashion; the sample mean value is affected ONLY IF power is aliased into zero-frequency. Because the diurnal temperature cycle is consistently asymmetric”
Exactly so, as I’ve tried to explain above. And the asymmetry of the diurnal is primarily due to the second harmonic. This aliases to zero with sampling twice a day.
But there’s aliasing only if the sampling is strictly periodic, which is NOT the case with Tmax and Tmin.
Ultimately, lacking in the early years the technology to record complete thermograms everywhere, the resort to the mid-range value proves quite reasonable.
1Sky1 and Nick,
What you both said in the above 3 posts is simply wrong.
What you are calling “asymmetry” is jitter. Jitter does not somehow allow aliasing to disappear – not in the second harmonic or any harmonic. Jitter actually increases aliasing.
Spectral overlap (aliasing) is additive, it doesn’t cancel. The spectrum usually resembles a sinc-function, (looks like an oscillating decay). In the unique case where overlap is between a spectral component and its “bilateral” twin, then the magnitudes add (it becomes more negative or more positive – not zero). In all of the other cases it is very unlikely to overlap spectral content of equal magnitude but opposite signs. No cancellation. Aliasing = error. No special exceptions for climate scientists. You can’t divide by zero either.
For convenience I have referred to integer multiples of the fundamental frequency. But there is no evidence that we are dealing with harmonics of the fundamental. Frequency content appears to be spread out in the entire band.
Here is the fact that you must contend with. The (Tmax+Tmin)/2 shows a lot of error relative to a Nyquist sampled signal. *IF* you were to convert the digital samples back to the analog domain, the Nyquist sampled signal will exactly equal the original sampled signal. Your (Tmax+Tmin)/2 samples will not. Why do math on something that has little relationship to the climate you claim to be studying? Waive your hands all you want. That isn’t going to go away.
William Ward,
+1
William,
“What you are calling “asymmetry” is jitter. Jitter does not somehow allow aliasing to disappear”
What I understood 1sky1 to mean by asymmetry is in the diurnal pattern. You can sample, say, at noon and midnight, but if the dip in the morning is deeper than the rise in the afternoon, the sample mean will be biased high. In this sense the fundamental is symmetric, the second harmonic not.
Jitter does not make aliasing disappear; it is simply aliased itself. It is like ordinary heterodyne demodulation of AM radio. The audio isn’t periodic, but when you deliberately alias the carrier to zero, the audio is in the sidebands and can be recovered. The jitter will appear as low frequency noise affecting the integrated result somewhat.
William Ward:
By claiming that both Nick Stokes and I are “simply wrong,” you show a lack of basic comprehension of what is being asserted here.
The asymmetry of the diurnal cycle is an inherent feature of surface temperature signals, arising from unequal rates of daily heating and cooling. That is the mechanism which consistently produces a peaked wave-form, whose mid-range value is invariably higher than its temporal average. This has nothing to do with discrete, periodic sampling of the continuous signal–the prerequisite for potential aliasing. The determination of daily extremes Tmax and Tmin is by no means a “sampling” in the ordinary DSP-sense of the word; it’s a totally different metric.
Au contraire! The power spectrum of densely-sampled surface temperature signals typically shows a strong peak at T = 24 hours and rapidly declining peaks at the even-numbered harmonics. With quality records, the power content in the entire baseband is very strongly dominated by this harmonic structure.
1Sky1,
I don’t think you have any understanding of sampling theory. The symmetry of a signal or lack thereof does not affect whether or not Nyquist applies. You seem to not even understand the definition of sampling. If you are measuring a continuous, band limited signal and resolving it to a discrete value at a point in time, then you are sampling. If you sample below Nyquist you get aliasing – you get an erroneous result. You are simply arguing against basic laws of mathematics which is astounding to me. There are no special carve outs (exceptions) for sampling climate related signals. Whether it is a temperature measurement of the air, a temperature measurement in a reaction vessel, the angle of a control surface on a wing or an audio signal, Nyquist applies. It doesn’t just apply – it defines the laws of mathematics. We clearly see this when comparing mean results over a day.
Sampling theory governs the physics and mathematics. But Nyquist doesn’t have a swat team on-call to kick your door down and beat you into submission should you violate the theorem. You are free to violate the mathematics and carry on as if you have not. This is ok in climate science because its all about numbers on paper. The bridge never gets built and therefore never supports a load. The airplane never gets built and therefore never flies.
The only thing you have going for your case is that in some cases this error over time diminishes due to averaging.
How ironic that pretentious red herrings are being raised by someone who repeatedly fails to grasp that the CONTINUOUS-time determination of daily EXTREMA of the temperature signal has nothing in common with DISCRETE UNDERSAMPLING that leads to spectral aliasing and potential misestimation of the signal mean. I pointed to the vertical asymmetry of temperature signal wave-form solely for the purpose of emphasizing that the MID-RANGE value (Tmax+Tmin)/2 should NOT be confused with any mathematical estimate of the signal mean.
1sky1 said: “How ironic that pretentious red herrings are being raised by someone who repeatedly fails to grasp that the …”
I can feel your affection.
We disagree and I don’t think there is much more to say now that is constructive. Maybe in another post.
Best wishes to you.
My affection is for analytic insight–not blind postures .
“My affection is for analytic insight–not blind postures.”
It looks more like your affection is for ignoring mathematical laws and blind processing of badly sampled data that has a corrupted relationship to the physical phenomenon you claim to be studying.
I agree
========
❶①❶①❶①❶①
❶①❶①❶①❶①
❶①❶①❶①❶①
❶①❶①❶①❶①
========
.
This is what Global Warming looks like.
.
Warning – may cause nightmares.
.
https://agree-to-disagree.com/this-is-what-global-warming-looks-like
.
“Stokes maintains that any data of measurements of any temperature averages are apparently just as good as any other — that the median of (Tmax+Tmin)/2 is just as useful to Climate Science as a true average of more frequent temperature measurements, such as today’s six-minute records. ”
citation please.
precision matters
you misrepresent him.
good to see roy spencer comment.
do satellites measure temperature continously
no
twice a day per location.
you measure max, you predict max with a climate model. you compare.
usefull
you measure min, ……you compare
usefull.
you average both
usefull.
are other metrics more usefull. sure.
Mosher ==> Qouted in the essay, link to his post ” Every now and then a post like this appears, in which someone discovers that the measure of daily temperature commonly used (Tmax+Tmin)/2 is not exactly what you’d get from integrating the temperature over time. It’s not. But so what? They are both just measures, and you can estimate trends with them.“
if you took the time to research
we typically use Tmean to be the integrated
temperature over the day.
tmin
tmax
tavg
tmean
this is not rocket science, yet you screw it up
Tmean is a proxy for the integrated temperature over the day ONLY if the distribution is perfectly symmetrical, which it hardly ever is. Yet, you screw it up.
Mosher ==> That is not what is shown in the GHCN Monthly files — and not as clearly explained by Customer service at NCEI when I double-checked with them to make sure — I was having trouble believing that they do what they do. NCEI confirmed the calculation of TAVG (monthly station average) exactly as I have given it.
You can look at the GHCN mpnthlies here: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/
Lot of hang down measuring contests going on. The crux of the matter is, whether two data points, (high and low) are a less reliable indicator of temperature trends than an, (extremely onerous) mean, over a 24 hour period?
To answer in the affirmative, we must prove how either, or the daily high, or low has deviated from historical trends. Since the length of sunlight at a given latitude and Julian date remains constant, I don’t see how 24 temperature readings over a day is superior to the daily high and low.
Furthermore, stations prone to UHI would carry more weight in summer days, due to extended length of sunlight.
We must not apply an infected bandage to the problem.
RobR ==> “a less reliable indicator of temperature trends ” That is only one of the questions….
The bigger question is if the (Tmax+Tmin)/2 method produces (at all later stages of calculation and interpolation)a metric that is fit for the purpose of determining AGW.
Kip,
Since, (as Dr. Spencer notes) Tmin and Tmax are the only historical data points available, there is no better existing alternative.
If you have proposal with a different strategy, by all means bring it forward. While historic twice-daily temperature were subject to the vagaries of; time, location, and paralax; I see no reason to toss them due to your statistics lesson on central tendency computation.
RobR,
You said, “Since, (as Dr. Spencer notes) Tmin and Tmax are the only historical data points available, there is no better existing alternative.”
Part of what is at issue is whether two points sampling a non-parametric temperature distribution warrants the kind of accuracy and precision claimed by NOAA and NASA with respect to how much global temperatures have increased in the last century. NOAA commonly claims two significant figures to the right of the decimal point, and NASA has claimed three!
Yes, we don’t have anything better, but that begs the question of whether it is good enough to make the claims for accuracy and precision that one commonly sees in headlines. For purposes of calculating anomalies, one can set a baseline arbitrarily, and define it as being exact. I’d suggest making the best possible estimate of pre-industrial GAST, and define it as the baseline.
I think that what Kip and William are arguing for is to stop handling the data in an anachronistic way, re-analyze the last 20 or so years,and do it properly. Admit that the historical data are inadequate to support the claims commonly made, and only make claims of high precision for modern data.
Clyde,
We have no grounds for argument, I would agree that the margin for error likely exceeds current warming.
However, this doesn’t render historic data completely useless. Any attempts to collect data using different metrics, must demonstrate superior precision. Additionally, a parallel data set of Tmax and Tmin must be concurrently maintained for fidelity with historical data.
RobR ==> There is no better existing alternative for those handicapped historical records.
There are better alternatives for modern records.
Continuing to use a metric that does not measure the thing we want to know, long after the necessity to do so has been obviated, is bad science.
What are these superior collection instruments?
Are you advocating for hourly, on the minute, or on the second data collection? If so, why is your choice more precise than thr 2x-daily method?
RobR ==> See the manual for ASOS stations: https://www.weather.gov/media/asos/aum-toc.pdf
My whole essay is about what is wrong with the Tmax/Tmin system.
Kip,
Can you not precisely explain your proposed solution in two sentences?
RobR ==> I am not King of the World — nor its Chief Scientist.
What is the problem to which you would like to to propose a solution?
Kip,
I’m simply pointing out, there’s no reason to question the utility of current methodology in the absence of a better alternative.
I’m not saying I blindly trust the precision of 2x daily collection. I’m just wondering what was the point of the statistics lesson?
What was the point of delving into your views on how different interest groups utilize temperature records. It seems like you made several loose inferences, without expressing how these problems can be fixed.
If that is the thrust of your essay: fine.
RobR ==> Oddly, the essay is about the use of a metric that is demonstrably unfit for the purposes for which it is often used.
The first step in self-correcting science is to recognize when we have something wrong. The next step is for the field to figure out how to fix it or start over and come up with a new hypothesis to test.
In Stevenson screen in met observatory, in addition to maximum and minimum thermometers, two other thermometers measure the dry and wet bulb temperatures. Up to 28.2.1949 morning observations were recorded at 0800 hr IST and there onwards it was recorded at 0830 hr IST [0300 hr GMT]; and afternoon observations were recorded at 1700 hr IST until 1-3-1949 and there onwards it was recorded at 1730 hr IST [1200 hr GMT].
Using dry and wet bulb temperatures relative humidity values for the respective times were computed using standard tables prepared by IMD.
The temperature observations are being recorded to the second place of decimal. They are adjusted to first place of decimal as follows:
If the numerical number in the first decimal is even [0, 2, 4, 6, 8], for example they are adjusted as: if the value is 0.45, it is adjusted as 0.5; if it is 0.46, it is adjusted as 0.5; if it is 0.44, it is adjusted as 0.4
If the numerical number in the first decimal is odd [1, 3, 5, 7, 9], for example they are adjusted as: if the value is 0.35, it is adjusted as 0.3; if it is 0.36, it is adjusted as 0.4; if it is 0.34, it is adjusted as 0.3.
That is: less than 0.5, 0.5 and more than 0.5 in second place follow the above rule.
These are daily mean temperature values. If we take the sum of such means over a month or year, they are monthly averages or yearly/annual averages.
Mean, Media and Mode are statistical parameters to define the homogeneity of the data series. If we have 101 data points, by plotting these from the lowest to the height, the value at 51 is the median. That is, on either side of the media 50 points will be there. If the mean of 101 points coincide with the median value, then we say that the data series are homogeneous and follow normal distribution. Then the values at different probability levels can be estimated using normal distribution test. If the mean is lower side of the media
Data set/Mean/probability level
[tmc ft]/[%]
78 year/ 2393/43
47 years /2578/58
114 years/2448/48
26 years/2400
30-years/3144 — central water commission estimated using thornthwaite water balance model — over estimate by around 20%
26 years = 1981-82 to 206-07 & 30 years = 1985-86 to 2014-15
114 years data series and 26 years data series are homogeneous as they follow normal distribution. 78, 47, 30 years data series are not homogeneous as they follow skewed distribution. This is basically because of the 132 year cyclic pattern.
Dr. S. Jeevananda Reddy
Dr. Reddy ==> And do you find that daily temperature profiles are what we would call a “normal distribution” around some central number?
How many temperatures actually occur at any weather station during the day? The physical reality is not different because we only record some of them. We acknowledge that the number of possible temperatures between Tmax and Tmin is infinite, therefore there will always be a middle value with half the set above and half the set below. This middle number will be found by (Tmax+Tmin)/2 — which gives the mid-point value.
This mid-point value (the Median) will not be the same as the true average temperature, the Mean, which is related to the amount of time each temperature was found during the day — the temperature profile.
I would be interested to know how temperatures were recorded to 2 decimal places from a Min/Max glass thermometer similar to the one pictured. …or have I misunderstood what you have said?
For the first point, it is yes. Maximum and minimum are points on the bellshape [on the left and right side]. Thermograph data can be used — you choose minute by minute or hour by hour. Try this.
Your second point — from the Stevenson Screen, four observations. Your argument shows you invented a great idea. Every meteorologist knew this. It is not a new idea. To represent local extremes, maximum and minimum are enough. Urban heat island effect — it is not difficult to remove the trend, if any.
When I was with IMD Pune in early 70s — we prepared formats to transfer the data on to puched cards. Later when IMD acquired computer, the punched cards data was transferred on to magnetic Tapes. —-. The averaging procedure , etc were programmed.
The observations and averaging pattern was decided by eminent meteorologists after detailed studies. They suggested the mean/average calculations for all stations around the world. The
daily temperature follows the Sine curve as it follows the Sun’s movement — in a daily, east to west; — in a year, south to north. The minimum occurs around just before sunrise and the maximum occurs around 3 pm.
Two places of decimal — it is the procedure followed all around the world. You can visit a met station to understand this. Even to get average: [25.5 + 34.6]/2 = 60.1/2 = 30.05 = 30.1 oC
Please visit a met station and learn how they are recording the data — averaging
Dr. Reddy ==> Averaging (or any other type of smoothing) does not increase precision or decrease uncertainty.
While the temperature profiles will be “generally” bell shaped (day night cycle, etc) the profiles will not be reliably “normal distributions” — some days, will have mornings that stay cool and only warm up in late afternoon, other heat up early and stay hot. One must not presume a normal distribution– a big error to do so.
Different climate regions (Koppen) have generally different profiles.
Most US stations are now ASOS automated stations and their records are programmatically determined.
Dr. S. Jeevananda Reddy,
But, one of the points of contention is how does one handle a set of 100 points, and more importantly, what is an appropriate name for finding the midpoint of 2 values?
Just plot on a graph the lowest to the heighest [now a days computers do this] and join them by a smooth curve. The starting point is 100% probability and the end point0% probability. 50% probability [median] gives the median. Take the sum of 100 points and divide it by 100 gives the mean/average. If the mean coincides the value at 50% probability value, median, then the it is normal distribution.
If you got data of thermograph [hourly or minute by minute], follow the above procedure. maximum and minimum are the end points. Generally the mean and median coincides as the daily graph follow sine curve.
Dr. S. Jeevananda Reddy
Dr. Reddy ==> There are comments here with data demonstrating that generalization not to be true.
Reddy,
You didn’t answer my second question.
Sorry, I answered your second question also: maximum and minimum are end points of the data set . You join the two points with a strait line. The midpoint between the two represents the median and this coincides with mean at 50% probability.
Dr. S. Jeevananda Reddy
Hi,
I think it is totally correct to say that (Tmax + Tmin)/2 is not the same as Tavg, may not vary in the same way, and may not vary the same as heat content which is yet another different thing. However, this is well known. Climate Change science needs to work on the assumption that they change in the same direction and approximately in the same magnitude in the long term, simply because we don’t have the necessary data to do the calculation correctly.
Proving that they are not the same over the short time span of a few days is trivially easy and not under discussion. It is, however, quite irrelevant. What would be interesting is to show that they vary significantly differently over a period of several years. I don’t think this has been done. Not here, for sure.
For me the discussion about why we are doing wrong by not using what we don’t have is a bit stupid, and discussing if an approximation is good enough or not without at least trying to prove that it is not in the appropriate time span is also pointless, but hey, that’s me. The key question should be why are we NOT focusing on what we DO have. That is more relevant. We do have Tmin and Tmax, SEPARATELY. Nobody is really affected by Tavg, alone. If any place in the world were to increase their Tavg by 2 degrees, but kept the temperature without variations (i.e. Tmin=Tavg=Tmax), it would be better. And if the same thing happened with Tavg reducing a bit, the same. We are affected mostly by extremes, not by the averages, we want the extremes to be less so. So I am way more interested in how Tmin evolves in cold places/months, and how Tmax evolves in hot places/months. THAT is the useful information for detecting a weather/climate crisis. And we have the data. Why do they keep insisting on focusing on Tavg without giving us Tmin, Tmax by latitude bands and months?
And this is where it gets political. They don’t give us what really affects us and they have because it would show that temperatures are getting LESS extreme, because Tmin increases way faster than Tmax, and both increase more in cold places, which is a good thing, which means no crisis. The bad (slight increase of high temperatures in hot places) is insignificant compared to the good (greater increase of low temperatures in cold places).
Nylo, pretty soon climate science will need to warn us about extreme mildness.
Nylo ==> Since the turn of the century, there is information on 5-minute intervals for most of the USHCN stations. Most of the CliSci Global numbers are still calculated from GHCN_Monthly TAVG itself calculated up from the old (Tmax_Tmin)/2 method.
For 20 years we have much better records, we just don’t use them. We could do fairly accurate temperature profiles for each day at each station…..
Those who wish ONLY to show “rising temperature trends” only care about long-term trends, pretending they have records from the 1890s/1900 to compare to.
Kip-
Thanks for bringing up this topic once again. As can be seen from the number of comments, it is a hot topic.
I am sorry to have joined the discussion so late, as it has always bothered me that we don’t accurately describe the daily temperature, yet we think we know the trend of daily temperatures.
In the middle of my engineering career, it was brought home to me that all data are samples of a distribution (As William Ward explains eloquently above). It is the shape of the distribution you need to define, if you are going to draw an inferences from the data.
I was surprised to learn from your post that the monthly Tmean was calculated from the monthly average Tmin and the monthly average Tmax. That means taking the average Tmin from (28, 30 or 31) different distributions and the average Tmax from those same distributions, and getting an average Tmean from these two numbers.
Next year when the same month is looked at, there will be a different set of (28,30 or 31) distributions. Since the average Tmean will not have been drawn from the same distributions, you would be fooling yourself to think that you could use these values to estimate the real trend.
old engineer ==> For an old guy, you have a sharp and discerning mind — and have hit the nail on the head — “you would be fooling yourself to think that you could use these values to estimate the real trend.”
People who use the historic land records of temperature, with a century or more based almost entirely on Tmax and Tmin measured by LIG thermometers in shelters, seem not to appreciate that they are not presented with a temperature that reflects the thermodynamic state of a weather site, but with a special temperature – like the daily maximum – that is set by a combination of competing factors.
Not all of these factors are climate related. Few of them can ever be reconstructed.
So it has to be said that the historic Tmax and Tmin, the backbones of land reconstructions, suffer from large and unrecoverable errors that will often make them unfit for purpose when purpose means reconstructing past temperatures for inputs into models of climate.
Tmax, for example, arises when the temperature adjacent to the thermometer switches from increasing to decreasing. The increasing component involves at least some of these:- incoming insolation as modified by the screen around the thermometer; convection of air outside and inside the screen allowing exposure to hot parcels; such convection as modified from time to time by acts like asphalt paving and grass cutting, changing the effective thermometer height above ground; radiation from the surroundings that penetrates necessary slots in the screen housing; radiation from new buildings if they are built; wind blowing from a hotter region and carrying its warming signal from afar.
On the other, cooling side of the ledger, the Tmax is set when the above factors and probably more are overcome by:- reduced insolation as the sun angle lowers; reduced insolation from clouds; reduction of radiation by shade from vegetation, if present; reduction of convective load by rainfall, if it happens; evaporative cooling of shelter, if it is rained on at critical times; cooler wind blowing from a cooler region and carrying its cooling signal from afar.
It does not seem possible to model the direction and magnitude of this variety of effects, some of which need metadata that were never captured and cannot now be replicated. Some of these effects are one-side biased, others have some possibility of cancelling of positives against negatives, but not greatly. The factors quoted here are in general not amenable to treatment by homogenization methods currently popular. Homogenization applies more to other problems, such as rounding errors from F to C, thermometer calibration and reading errors, site shifts with measured overlap effects, deterioration of shelter paintwork, etc.
The central point is that Tmax is not representative of the site temperature as would be more the case if a synthetic black body radiator was custom designed to record temperatures at very fast intervals, to integrate heat flow over a day for a daily record with a maximum. T max is a special reading with its own information content; and that content can be affected by acts like a flock of birds passing overhead. The Tmax that we have might not even reflect some or all of the UHI effect because UHI will generally happen at times of day that are not at Tmax time. And, given that the timing of Tmax can be set more by incidental than fundamental mechanisms, like time of cloud cover, corrections like TOBs for Time of Observation have no great meaning.
It seems that it is now traditional science, perceived wisdom, to ignore effects like these and to press on with the excuse that it is imperfect but it is all that we have.
The more serious point is that Tmax and Tmin are unfit for purpose and should not be used.
Geoff
Geoff ==> “The more serious point is that Tmax and Tmin are unfit for purpose and should not be used.” That’s what I think too.
We don’t need to use Tmax/Tmin for modern records. We mustn’t give Tmax/Tmin the same explanatory value we might give the data currently recorded by modern ASOS stations.
Kip,
By the time proper error bounds are put around these historic Tmax and Tmin figures, it becomes apparent that much of the sought signal is among a lot of noise, including noise where positive and negative excursions do not balance. People choose various ways to comfort themselves that there is meaning in the numbers and sometimes his becomes establishment gospel. You are quite correct, we must resist this false dogma by ways such as calling out the invalidity of the (Tmax +Tmin)/2 construct. Geoff
Geoff ==> Thank you, Geoff. A lot of kookiness going on here in comments.
Stokes et al. are ONLY looking for a trend that supports there version of climate truth and don’t seem to care that their results are a trend of something other than what their hypothesis requires. The justify this with — “but that’s all we have! anyway, it still gives us a trend.”
I became interested in this use
of the mean/median of tmax/tmin because of this
http://woodfortrees.org/graph/hadsst3nh/plot/hadsst3sh
If you look closely (or calculate SD) at the period between 1885 and 1920 and compare it with the next 40 years, you will see that the two hemispheres are closer to
being in sync when there was less data in the SH. Its hard to believe that it was an attempt at a genuine calculation of SH SST.
I pointed it out on another blog and was told that they used Krigging and asked if I don’t believe in Krigging. Silly comment but it did have me question how you can apply a method for a real intensive property to a make believe one. As an indicator of what the climate is doing, it might mean a greater error than 0.1 K if the thermometer record was adequate but of more concern is how valid is the method to get global averages from such a dogs breakfast if measurements.
Robert B ==> Haven’t heard the expression “dog’s breakfast”(outside of my immediate family) in twenty years!
A fiar description, though.
Apologies for being a bit OT here but, since you posted the data, Robert B, do you (or anyone else) have any thoughts about what happened to the NH dataset in 2003. Looks really odd to me with the apparent influence of the annual cycle showing on anomaly data from that date forwards. Probably just me being a bit thick!
http://www.woodfortrees.org/plot/hadsst3nh/from:1990/plot/hadsst3sh/from:1990
Jim Ross ==> That’s a good observation and a good question! The nature of the data changes drastically at that point…. hoipe someone has some ideas.
Kip,
Thanks very much for the response. I guess it is a bit too far off the current topic for most commenters here. For your information, it is clear from the following plot that the odd NH response is directly affecting the global data:
http://www.woodfortrees.org/plot/hadsst3nh/from:1990/plot/hadsst3sh/from:1990/plot/hadsst3gl/from:1990
What is even more bizarre is the same effect shows up on the SS2 data (which only goes to 2014):
http://www.woodfortrees.org/plot/hadsst2nh/from:1990/plot/hadsst2sh/from:1990
Why bizarre? Because this time it is the SH data that show the cyclic behaviour! I assume that this must be a labelling issue somewhere, but it does not explain the cyclic oddity.
I guess I am going to have to delve into the base data. I was just hoping that someone had already resolved this issue.
Jim Ross ==> Let me know what you find out!
Kip,
I downloaded the latest time series data (HadSST3.1.1.0) from here:
https://www.metoffice.gov.uk/hadobs/hadsst3/data/download.html
Using the monthly text files for Globe, NH and SH, copied into Excel, it is clear that for the recent data at least (1990-present, since the cyclic evidence starts in early 2003), the values provided by WFT accurately reflect the original published text data. Incidentally, I note the values quoted are the “Median global average sea-surface temperature anomaly”. The median!!
There are two linked papers available from the data download page (part I and part II), but these focus on the processes used for estimating the uncertainty ranges and these ranges track the quoted median values as would be expected. I guess that I will have to see if I can contact the Hadley Centre directly, but I was hoping that somone here at WUWT would have been familiar with this issue and could have stopped me from making a fool of myself!
Anyway, thank you for your interest, much appreciated.
Jim ==> I’d like to look at your stuff when you’ve got something to share.
Kip,
Will do (but note I am away for the next week or so). I have your email address so will use that if OK with you.
Jim
Jim==> Sure, look forward to it.
I’ve noticed the seasonal peaks start about then, years ago. Clearly not real but a poor job. Evidence of a conspiracy is not the mistake but a refusal to acknowledge and fix it (the method, not fudge it out)
Robert,
I appreciate your comment “clearly not real” as that was my initial view. I was looking for some independent confirmation of this or, alternatively, a valid explanation.
My primary reason for concern is that the global HadSST3 time series is a widely used dataset and the NH cylicity is clearly reflected in the global version (monthly), i.e. if the NH data are invalid from 2003, so are the global data.
http://www.woodfortrees.org/plot/hadsst3gl/from:1990/plot/hadsst3nh/from:1990/plot/hadsst3sh/from:1990
Jim ==> Satellite Sea Surface Temperature is NOT the same as sea surface water temperature as would be measured by boat or buoy. — some care is needed here. Satellite measures Sea SKIN Surface Temperature, the top 1-2 mm and when the sea is calm, that can be far different than the temperature of the surface (top 2 meters) of water.
Kip,
When I said I was looking for independent confirmation I was meaning by another individual who agreed that the data as published were “clearly not real”. Hence I was pleased to see Robert’s comment. I was not referring to a comparison with another dataset, e.g. satellite data, which would be based on different types of observations. So, sorry if that was unclear but thanks for the warning! For the moment, I am only interested in investigating what looks like it could be an internal bust in the HadSST3 dataset, which appears to have also been evident in the HadSST2 dataset, though likely labelled incorrectly at some point (NH vs. SH).
R Blair ==> The Mean/Median illustration is just an illustration of the idea of that the Mean of a data set varies with the profile of the data (the “distribution”) while the Median of the same data set (same max and min) does not.
It is not a graph of temperature data from anywhere — it is a general illustration of the point.
It occurs to me that the peak temperature in many locations will likely be in the form of a pulse, occurring when cloud cover parts to allow rapid solar heating. This pulse of heating may be in no way representative of the average daytime temperature. Night minima will will tend to be more of a realistic average since there is no intermittent sun to perturb the reading, and although breaks in cloud cover will allow radiative cooling, this will be a much slower process.
Thus, overall you might expect that (Tmax-Tmin)/2 will give a result warmer than the actual average temperature. Furthermore, the departure from a true average will be very dependent on cloud patterns.
-Didn’t I read somewhere that cloud patterns are (indirectly) dependent on solar particle emissions? (solar wind) This could be a link between solar activity and the perceived warming. Maybe it isn’t actual warming, but an effect of solar activity on the way that temperatures are measured.
Ian ==> Temperature profiles are much more of interest than simple Min/Maxor its median, Tavg.
” and although breaks in cloud cover will allow radiative cooling, this will be a much slower process.”
Actually no.
As someone who early routinely monitored temperatures professionally, I know a low can be quickly attained on radiation conditions occurring, especially in very dry air. Can just fall away.
You forget that hot air quickly convects aloft whilst cold air lies next the ground (2m is where a thermo is).
IOW: This is all very pedantic. There is just as much variation to the cold side of the mean as there is to the hot. So it ll “comes out in the wash”.
It is a measure that is consistent with historic records, and as such does the job well enough.
Anthony Banton,
You said, “There is just as much variation to the cold side of the mean as there is to the hot. So it ll “comes out in the wash”.” Can you justify your statement?
In my article ( https://wattsupwiththat.com/2017/04/23/the-meaning-and-utility-of-averages-as-it-applies-to-climate/ ) I claim that the global temperatures have a skewed distribution with a very long cold tail. See my graph and the comments where one of the readers duplicates my frequency distribution with some commercial software.
Clyde:
Only in that I watched it happen many times, and actually took account of it when considering conditions re ice/hoar frost formation when forecasting road conditions.
It seems intuitive to me anyway.
A sudden change in energy either sinking into the ground surface, or leaving it will impact on air above it.
Over grass and especially snow there is insulation from ground heat flux (remarkable falls in temp occur over fresh snow in radiation conditions) …. and that may happen in even a brief window of clear(er) skies.
Cold air is confined to the lowest few hundred feet (and in extreme cases the lowest 10’s feet).
Energy leaving that layer confines it’s temperature to that layer and maximises it’s effect, whereas high temps are mixed aloft via convection (height dependant on instability).
The reason is also why AGW is impacting night-time minima more than on daytime maxima.
That lowest layer of air trapped under a nocturnal inversion maximises it’s warming due to the non-condensing gas that is CO2.
https://www.sciencedaily.com/releases/2016/03/160310080530.htm
Anthony Banton ==> Agree,good enough for weathermen.
Many commentators are mostly concerned with semantics as ‘true meaning of median’ missing the point Kip (I reckon) is making that building time-series (‘anomalies’) of daily means/medians does not provide sufficient resolution to detect temperature changes over longer period of time. That may be perfectly fine for everyday usage but fancy charts that show global averaged temperature increase are questionable, at very least. I believe that is the crux of the problem. Would be nice to have this point either confirmed or falsified.
As per definition of median/mean. Apparently NCDC/NOAA does not bother much with this distinction simply calling (Tmin+Tmax)/2 as mean temperature, although arithmetic mean has a bit different definition. As per NOAA from their file specifications:
8 T_DAILY_MEAN
Mean air temperature, in degrees C, calculated using the typical historical approach: (T_DAILY_MAX + T_DAILY_MIN) / 2.
Source.
Paramenter ==> USCRN also calculates the real mean of daily temps — but this does not end up in GHCN_Monthly.
”
8 T_DAILY_MEAN [7 chars] cols 55 — 61
Mean air temperature, in degrees C, calculated using the typical
historical approach: (T_DAILY_MAX + T_DAILY_MIN) / 2. See Note F.
9 T_DAILY_AVG [7 chars] cols 63 — 69
Average air temperature, in degrees C. See Note F.”
Kip: “USCRN also calculates the real mean of daily temps”.
Yeah, I’ve noticed that. As arithmetic mean is usually synonymous with average looks like NOAA calls two different things with one description. And they don’t seems to be bothered by textbook definitions. Quite rightly so – debates about semantics are usual useless.
Thanks for another piece of interesting work. Please, keep pushing!
Paramenter ==> There is a defintions document — someone else linked it, — I have a copy I used for the essay somewhere, but I am ending off on this essay and its comments in the next few minutes. USCRN keeps both “averages” for daily values. That is a good sign. But GHCN_Monthly, used by all/most climate groups according to Nick Stokes, still sticks to the (Tmax+Tmin)/2 Tavg for the day, and TAVG for the month.
For what its worth, I had a set of 3 dataloggers. Still have.
When I first got them, one was set in the middle of a 100 acre grass field, another in a small woodland and the third in a cherry tree in my garden.
They were set to record every 5 minutes.
I have just revisited some of the data I collected and worked out some of the various ‘averages’ on this real actual data – as recorded in a field in North West England.
It really is quite amazing how small the differences are using the different averages, typically well less than 1 degC and this from dataloggers that only record to the nearest 0.5 degC.
I suggest visiting Wunderground to get more data, at 5 minute intervals. Maybe from somewhere near your home and, playing it around in Excel – maybe something odd happens in Cumbria.
What’s really needed is a comparison to somewhere that is ‘dry’ and compare it to somewhere that is ‘wetter’ – Cumbria obviously fits the latter requirement.
Is that it – thermal inertia due to water within the landscape – hence the ‘concern’ about UHI
Peta ==> Throw some 1 degree wide uncertainty bars on your averages and see if they overlap. Most will — meaning that you reallyc an’t tell if the values are different.
Remember though,that the entire claimed “anomaly”in Global Average Temp is only 0.8 degrees against 1951-1980, the start of “:AGW”. — the same less than 1 degree.