Analysis of Met Office data back to mid 1800's

“So I think there are two possibilities:
A. There’s an error in the paper and I’ve managed to find it. I consider this a remote possibility and I’d be astonished if I’m actually right and the peer reviewed paper is wrong.
B. There’s something wrong in my program in calculating the error range from the sub-sampling data.”
Don’t give up on A. Confirmation bias is a powerful thing. If it turns out to be B that’s OK. But if it is A…… then Wow. It is worth checking.

I haven’t read the report but I remember a History Channel program abot the Royal Navy loging temperature and barometric readings with every navigational sighting (at least twice a day). So here is my question does the data include those readings and if not why not? Considering the number of ships at sea at any given moment there should be thousands of readings per day all available in the Royal Navy archives. Second thought if they are used you should be able to extend the series further back in time with a near global coverage since the sun never set on the british empire just my 2 cents
Medic 1532

Why is there a coverage bias instead of just a plain old coverage error (which like many types of random measurement errors would be as likely to be positive as negative, giving a mean close to zero)? In other words, do we know that when there is only a small number of regional temperature measurements and they are put into an algorithm to get a regional average temperature, that the calculated average will always tend to be off in one direction — that is, it will be biased off the true value instead of just equally likely to be in error in either a plus or minus way from the true value? I’m sorry if I missed your explanation of this point in the original blog, but was not satisfied that I understood this aspect of what you did after reading it twice.

URL says it all
http://www.bbc.co.uk/blogs/thereporters/richardblack/2009/12/cop15_saving_the_planet_or_saving_face.html

On first look, the trend looks quite a bit like those produced with “value added” data, which is not surprising.
What I noticed is the use of standard deviation as the standard error bands is incorrect. I’m thinking you should be calculating for the standard error of the mean using the population model (n) as opposed to a sample model (n-1) because you are calculating for the entire avalilable data set. Any sites that would be excluded (for whatever reason) don’t meet the inclusion criteria and therefore would not contribute to n (they do not represent available samples).
Also, I don’t think you should use a fixed standard deviation of 1.96 as the numerator when calculating for standard error across the entire timeline (it is not clear if that is what you are doing) as each point on the timeline will have a different variance. You should re-calculate StdDev at each point and obtain your standard error from the re-calculated StdDev.

You might want to run this past William Briggs. Briggs just completed a series of postings that I suspect may have bearing on what you are trying to figure out. His blog is http://wmbriggs.com/blog/

John, having your post copied here on WUWT is like having a free one-page advertisement in the New York Times….

Leif Svalgaard (08:18:19) :
“The number of stations is not the correct parameter to use. […] The proper parameter must include the distribution of those stations. Something like the mean area around a station where there are no other stations.”
Of course, the number of stations put a limit on the mean area around each station or any other similar parameter. You should be able to estimate a lower bound of the error based on only the number of stations (i.e. the error assuming they were optimally placed).

Leif Svalgaard (08:18:19) : Says:
“The number of stations is not the correct parameter to use. Imagine that a small country [e.g. Denmark] suddenly gets the idea that more stations are good, so installs a million sensors throughout the smallish land area. Clearly that will do nothing for the error in the global average.The proper parameter must include the distribution of those stations…”
The use of confidence intervals in regards to population proportions assumes that the sample is representative of the entire population. John Graham-Cummings use of sample size to arrive at a margin of error is correct. What you and others speak of refers to sampling error (i.e. that the sampling is not representative or is biased in other ways). Sampling error would only add additional uncertainty to the estimate derived by Graham-Cummings and in no way invalidate his base margin of error estimate.

Voluntary Observing Ships (VOS) Scheme which has been running since 1853 whereby participating vessels submitted weather four times daily 00, 06, 12 and 1800 gmt.
http://vos.noaa.gov/vos_scheme.shtml

The word “Denier” is thrown around a lot in an ad hominem attack calling the AGW skeptics “Deniers”, so you think of yourselves as Holocaust Deniers”.
The word “Holocaust” comes from an ancient Greek word that when translated in English is “Burnt Offering”. The Jewish people don’t have an exclusive claim to “Holocausts”. The Chinese Burnt Offering Holocaust killed 100 million people. The Russian Burnt Offering Holocaust killed 50 million people. The Jewish Burnt Offering Holocaust killed 5 million people. I do not “Deny” that any of these “Holocausts” happened. Whenever I am accused of being an “AGW Denier” I respond by saying, I am not a Jewish Burnt Offering Holocaust Denier, and whenever I mention the word Holocaust, I always pre-qualify the word Holocaust using it’s English translation with the words “Burnt Offering”.

… not SO much.
Cheers

Having a free time I invested in 100 thermometers, just wanted to see for myself, I placed all in my fenced property, plus minus 1 hectare, I placed 3 on 30 wooden polls 2 meters high, evenly spaced starting from 50cm 1.00 and 1.50 I put 15 in the shade and 15 in direct light, ( 10 thermos. kept as spares ) after reading 2 times in daylight and 2 times at night ( same time ) at the end of 8 weeks 3 temp. on pole always showed different temp, the highest always cooler, the temp. in my garden in sunlight shows plus minus 1.03C average difference in shade 1.8 C difference from east to west, north to south shows 0.97C and 1.67C respectively, and my science is settled!

Whether this analysis is correct or not it provides this EE with a better understanding of what the temperature plots (or their creators) are trying to say.
Also, While conceding Leif’s distribution argument, it is not at all obvious to me that quantity is irrelevant. As an example it would seem that the temperature of Antarctic would be better represented with 3 (reasonably distributed) measurements than with 2 or 1.

“”” Leif Svalgaard (08:18:19) :
The number of stations is not the correct parameter to use. Imagine that a small country [e.g. Denmark] suddenly gets the idea that more stations are good, so installs a million sensors throughout the smallish land area. Clearly that will do nothing for the error in the global average.
The proper parameter must include the distribution of those stations. Something like the mean area around a station where there are no other stations. This problem has been studied at length in the literature [I can’t remember a good reference, off hand, but they exist]. “””
Well one good place to start is;- “Digital and Sampled Data Control Systems” by Julius Tou (Purdue University).
Or you can just google “nyquist sampling theorem”
The problem is NOT a problem of statistics, and it cannot be solved by statistical methods; it is a problem of sampling theory; and there is no known solution (post facto).
GISStemp and HadCRUT are accounts of GISStemp and HadCRUT respectively, which are data from a small number of thermometers. They do not represent the Earth’s surface or lower atmosphere temperature.

“Let’s see, if we raise the trunk a bit, and shrink the elephant.”
http://lifesizestatue.com/fiberglass/images/elephant_up_big2303.jpg
It looks a lot like CRU data. (Or, maybe an anteater?)
And, speaking of their data, and in particular their analysis of it…
http://hotair.com/archives/2009/12/16/video-east-anglia-crus-below-standard-computer-modeling/
“oops!”

Dr. Leif Svalgaard is right here. Take it to the limit where there are 2 temperature sensors. One is rural where rural is 98% of the land area. The other is urban where urban is the remaining 2% of the land area. The rural sensor shows no rise in day or night (max and min) temperature. The urban sensor shows no rise in the daytime but a 2 degC rise in the nighttime temperatures.
If you average min and max you get 0 deg change for rural and 1 deg for urban. The average of both is then 0.5 deg which is true for urban but not true for 98% of the (rural) land area. The temperatures need to be weighted by the area they cover.

Leif Svalgaard (10:10:20) :
I don’t think so, as values inside the sample are correlated and adding more stations does not decrease the error.
Yes. I agree that if the samples are correlated, adding more such samples would not decrease the error.
Sampling error can occur by chance or by bias in the collection of the samples. A confidence interval (CI) for a population proportion accommodates the first type of sampling error but not the second type. My point was that there is nothing wrong with deriving a CI solely on the basis of sampling error due to a small number of samples increasing the possibility of error due to chance. A resulting wide CI can, by itself, undermine possible conclusions.
A sampling error due to bias such as correlated samples or non-sampling measurement type errors is another story and if not taken into account, as you suggest, can be much more problematic than chance type errors.

Quote: “Having a free time I invested in 100 thermometers, just wanted to see for myself, I placed all in my fenced property, plus minus 1 hectare…”
Fred Lightfoot, am just curious, were you able to purchase mercury thermometers where you live? I became frustrated several years ago in my engineering job at the university, when I found that I could not replace broken mercury thermometers with an identical model, as the mercury filled models had been banned by the EH&S Dept at the university. I was required to purchase the red dye alcohol filled types – I found they were not worth a damn for accuracy, and suffered separation problems. (For separated fluid, T meniscus > T actual. There’s a warming source, i.e. corrupted data.)
I’ve never performed outdoor measurements to the extent that you did, but can tell you that a laboratory oven set to +37 deg C having an internal height of about 60cm will have a temperature gradient of about 8 deg C over the 60cm height, in the absence of forced convection. I was rather surprised the first time i discovered that, and ended up retrofitting all of our natural convection lab ovens to forced convection. Even with forced convection, the gradient was 2 to 3 deg C under the same conditions.

I think JGC needs to take a few deep breaths before hitting the return key. Anthony has been very generous in the way he has shared this blog with other people who are trying to get the message out.
Resolving problems with the temperature datasets is not going to happen overnight, and we really need to stick together on this. I wouldn’t have known about JGC’s work if it weren’t for this blog.
That said I liked what he did with putting the grids on Google Earth using KML files, and it got me thinking if it would be worthwhile doing something like Leif is suggesting, a sort of polycell distribution rather than a gridcell distribution.
Temp stations aren’t evenly distributed, but if each was allocated a polygonal area based on how close it’s neighbouring stations are, i.e. more stations equals smaller polygons, would that lead to a more accurate picture about temps?
It would get over the land-sea problem with gridcells and it would also mean that temps could be calculated by country, then just divide the total land area by the country area to get it’s contribution to the overall temperature record. Then do the same with sea temps. It could then be compared to satellite records, and if it matched them closely, then it could be a valid way of analysing the temp dataset.
I’ve glanced at the maths that would be involved in doing something like that, and it’s not pretty, but it would only have to be done once initially for all existing stations and then as new stations are added or subtracted, it would have to be redone for the neighbouring polygons.
Would this be a valid way of providing an alternative dataset, or would it be too computationally heavy for what it added?

Ok,back to obscurity, John.

Theodore de Macedo Soares (11:34:30) :
“A sampling error due to bias such as correlated samples or non-sampling measurement type errors is another story and if not taken into account, as you suggest, can be much more problematic than chance type errors.”
It seems that when you compare their very small computed error with the wide natural variation that exists, a good deal of the problem comes from the 2nd type. And, from what we’ve learned from climategate, we can add deliberate bias for good measure.

The analysis claims an error of +/- 0.8 degrees, compared to the IPCC’s +/- 0.1 degree, back in 1850.
Here’s a description of what actual temperature measurement was like back in 1850, long before the days of arguing the effects of dirt and degradation on a Stevenson screen with automatic recording:
http://climate.umn.edu/doc/twin_cities/Ft%20snelling/1850sum.htm
Even modern recording accuracy is generally +/-0.5 degrees.
http://www.srh.noaa.gov/ohx/dad/coop/EQUIPMENT.pdf
Might I suggest that both these estimates are extremely optimistic ?

That JGC seems to be a surly and ungrateful SOB. We should allow him ti luxuriate in his well-deserved obscurity.

“”” jknapp (12:52:15) :
Looking at the mets graph it seems that increasing the number of stations from around 50 (in 1850) to over 1400 (in 1960) has negligible effect on their error bars. “””
Well your observation is correct; and it wouldn’t matter if ther were 14,000 stations, it’s stil not nearly enough.
From Leif’s posting, I get the impression that in fact it is not the normal practice to associate each of these reporting stations with a certain land area around them that is presumed to have the same temperature as the thermometer (at all times).
If that is the case, then there is no way to obtain a global average that has any meaning.
I would put Tgl = S[T.A]/S[A} where my S is sigma, and A is the area associated with each sensor, and S[A] is the total surface area of the earth.
If they are NOT doing that, then it fits in the GIGO folder.
And as Anthony’s Station study has discovered; for the US stations at least, a very large fraction are on airport runways. Well of course they are ther because the airport could give a rip about the climate; they want to know the weather; and specifically, the REAL TEMPERATURE on the runway, as that is of interest to a pilot trying to land or take-off on that runway. They want to know the temperature at the time they want to land not what it might have been 24 hours ago.

Good points from George E. Smith.
Discrete measurement systems need to take into account signal bandwidth. Get it wrong, and results might not just be slightly wrong, they can be downright misleading.
Imagine the silhouette of a mountainous landscape. An aeroplane passes overhead with radar which takes merasures altitude at discrete distances. Imagine the radar takes three samples as it crosses the mountains, but (unlucky) each sample just happens to fall into three valleys. Join the points together, and we have an image of a flat landscape. Totally wrong.
The answer to “aliasing” is to reduce the distance between samples by 5 or 10 times in the above example, if this is what we have absolute confidence in our ability to spot the mountains and create a reasonable reconstruction of the landscape.
But how do we know this at the outset? We need to look at the properties of the signal and make it part of the measuring system design. This is an issue of careful design.
It is worth noting that the theoretical minimum sammpling rate is 2 times the signal bandwidth. But in practical systems, it really needs to be 5 to 10 times.
What about the spacing for sampling the putative global average temperature?
We have seen how Darwin may sit more than a thousand km away from neighbouring measuring sites, What about the spatial distribution of measurements near the poles? Or the Pacific Ocean?
I have no idea what is the spatial bandwidth of the climate system, and what distribution is necessary to avoid aliasing. I wonder what there is in the literature to give us comfort that this is not an issue with the temperature data. I haven’t seen anything.

Jordan.
Start with this paper.
EOF Based Linear Estimation Problems
by
Kwang Yul Kim
Climate System Research ProgramTexas A&M University Abstract
http://citeseerx.ist.psu.edu/viewdoc/summary;jsessionid=0D24751906E7754397CA0BD09F5053B7?doi=10.1.1.53.4547
It addresses the problems that others have raised about sampling a climate field.

Reading through all the comments here and on my blog, I still have two nagging concerns:
1. It isn’t intuitive that with so few stations in the 1800s that the error in Brohan et al. appears to stay the same across the years. Although there’s another diagram (Figure 12: top) which shows just the land anomaly trend and that does have wider earlier errors although they do not appear to be caused by the number of stations. But even that’s a bit hard to tell because I can’t see clearly how the limited coverage error is combined with the other errors. It would be good to be able to see the underlying data, but http://hadobs.org/ doesn’t appear to have the limited coverage error data.
2. I’m not sure how accurate the sub-sampling can be given that the samples are correlated (spatially).
Here’s hoping my email to the Met Office gets answered. It’s frustrating not to fully understand this paper.

“”” Jordan (14:00:30) :
Good points from George E. Smith.
Discrete measurement systems need to take into account signal bandwidth. Get it wrong, and results might not just be slightly wrong, they can be downright misleading. “””
In this case Jordan, we have a two variable system, space and time. We can be sure that the time variable contains a very solid 24 hour cyclic signal. Now a min/max thermometer is going to give you two samples a day, and that only satisfies Nyquist, if the daily cyclic signal is purely sinusoidal; but it isn’t daily temperature graphs tend to show a fairly rapid warm up in the morning, and a slower cooldown in the evening, so there is at leas a second harmonic component or 12 hour signal component. So a min/max themrometer already violates Nyquist by a factor of 2 which is all you need to fold the spectrum all the way back to zero frequency; so the correct daily average temperature is not recoverable from a min/max thermometer. And that doesn’t allow for varying cloud cover that will introduce higher signal frequencies beyond even a 12 hour sample time.
So the lcimate data records are already aliassed before you even consider the spatial sampling. Here in the bay area, we get temperature cylces over distances of 10 km or less; yet climatologists believe they can use a temperature reading to represent places 1200 km away.
The problem is that most of them seem to be statisticians, and not signal processing experts, or even Physicists.
Well no central limit theorem is going to buy you a reprieve from a Nyquist criterion violation, and no amount of linear or non-linear regression analysis, is ever going to recover the true signal which has been permanently and irretrievably corrupted by in band aliassing noise.
Other than that slight inconvenience, there isn’t any good data even improperly sampled data before about 1980, when the Argo buoys, and the polar orbit Satellites were first deployed.
So HadCRUT and GISStemp belong on the ash heap of history; they are not even worth fixing.
UHIs cause no problems unless the sampling regimen is improper.
So to me it hardly matters if the playstation video games are any good; the data that goes into them is true garbage anyway. You would think that somebody like Gavin Schmidt, would have heard of sampled data system theory, and the Nyquist Sampling theorem; but he sure doesn;’t act as if he has.

Slightly off topic but John G-C’s blog on the “hide the decline code” is the most thoughtful and credible analysis I have seen. You can read it on his blog archive for November. Thanks John 🙂

Thanks for the reference steven mosher.
I have had a quick look over the paper, but I’m not sure we’re on the same page.
The paper talks about estimation error for two distinct directions (1) find the best guage locations, or (2) work with the fixed guage locations we have. The paper examines optimal filtering from the starting point of (2).
The question (from George and myself) has more to do with (1): if we were starting with a clean sheet, what design criteria would we use to determine the spatial distribution of measuring stations for the putative gloabal T ? The sampling theorem would have a central role to play in determining the “gap” we could suffer between stations.
So the paper looks like an intersting analysis of optimal filtering, assuming we can live with T(r,t). But T(r,t) could lack meaning due to spatial aliasing.
Looking it another way – those who have questioned the MWP by talking about a “local phenomenon” are basically asserting that spatial aliasing distorts our view of the past. It’s a fair point, but works both ways.

Thanks, Colin. For those interested just in my climate change posts you can visit: http://jgc.org/blog/labels/climate%20change.html

George E. Smith
We agree on so much.
There are huge issues in the time dimension. I still take the view that where arbitrary “homogenisation” turns a negative trend into a positive, we have nothing more than an elaborate way of saying the historic data is junk.
I agree with your point that the spatial dimension may not have had adequate attention from a sampling perspective. Who has done the analysis to satisfy us that there is no distortion of the signal due to spatial aliasing?
I can appreciate many of the more detailed points you make. But let’s leave that as a challenge to the literature – show us the papers which have addressed and answered these issues. If there are none, it has to be a matter for further research. But it would leave little reason to pay attention to aggregation of the surface network.
“Well no central limit theorem is going to buy you a reprieve from a Nyquist criterion violation and no amount of linear or non-linear regression analysis, is ever going to recover the true signal which has been permanently and irretrievably corrupted by in band aliassing noise.”.
Yep. Worth repeating.

George E. Smith (10:20:39) :
Better still..
Try to reconstruct Mozart’s 19th piano concerto by taking 1 out of every 40 notes in succession from the manuscript…

The stations chosen should be those whose microenvironments did not change significantly during the observation period (decadal or century) of course.

There are many means for computing statistic uncertainties.
1- The most common way assumes that noise (or error, etc.) follows a normal law (gaussian-like). The more you get data, the more ( overall noise / number of data ) averages toward 0. Of course this is pure bet since in many cases noise does not follow normal laws … but classical statiscians often simply forget this care ! For example unknown phenomenons affect all sensors the same way at related times : thermometers boxes are repainted the same way which, boxes shape changes and reacts in a different maner depending on winds, warming urban island grows more or less simultaneously, etc.
2- Another way is to adress uncertainties related to data inhibition is to guess (extapolate, modelize, etc.) the data of one sensor (e.g. thermometor) from the data of other sensors (e.g. thermometors). This way you can compute an uncertainty related to the inhibition of one sensor, then a second one, etc. This more empirical way is muche more costly and shall be prefered only when you get time and money to design the cross-models and many sensors data.
3- An intermediate way consists into getting the delta serial on the (geographicaly averaged) mean temperatures when you inhibit the data of one or the other sensors. You can do it with 3a- simple means or preferably 3b- geographic-surface-weighted-means. Either way you get a fast evaluation of robustness against sensor inhibition.
Method 3 is much more simple, easy, fast, neutral, approximate than method 2. So it is sub-optimal on a mathematical point of view but may prove more reliable on a management point of view facing human biases (guess what I mean following CRU, Climategate etc.).
In cases 2 and 3b, strong robustness are plausible if the remaining sensors are sufficient for guessing/extrapolating (case 2) or localy-averaging/interpolating ( case 3b) the data of each previous sensor.
Although I have not read the article you make mention of, I guess (?) from some Climategate reading the way CRU is used to be working relies on geographic-surface-weighted-means (case 3b), using Voronoi/Delaunay diagramms for computing the relevant surfaces. Such diagramms associate a given location to the 3 sensors drawing the smallest triangle around it. Then some weighted average is computed (possibly with 1/distance to each of these 3 sensors).
cf. CBS article:
http://www.cbsnews.com/blogs/2009/11/24/taking_liberties/entry5761180.shtml
They do not seem to be happy with their Delaunay diagramm(s) because it has turned out to be too loose at lest in some World regions.
Regards,
Xavier DRIANCOURT
PhD machine learning, etc.
On a more genaral basis, there is a global statistic theory on robustness evaluation and uses for tuning statistic systems. See US Vladimir VAPNIK, Léon BOTTOU, et al. on this. They follow some previous USSR work on regularisation (i.e. stabilisation) of unstable systems see TYKONOV et al. on this.

“”” bill (15:15:22) :
George E. Smith (10:20:39) :
Here’s an interesting experiment for some of you computer nerds to try when you have an evening free;
Ever heard of MP3 format Mr. Smith?
or
PASC (Precision Adaptive Sub-band Coding)
or
ATRAC etc.
Yes I have; does the algorythm I suggested perform Precision Adaptive Sub-band Coding ?
The human ear is well known for being able to find intelligence in the most garbled sounds. Early encoding using things like audio spectrum inversions and such were found to still leave voice messages intelligible to a trained ear.
What we are interested in with climate data, is in being able to recover the correct signal; not one that is similar in some respects.
Can MP-3 encoding be done live in real time; or does it require fore-knowledge of what information is coming next. How would we disburse global temperature sampling stations spatially so as to be able to MP-3 encode them to reduce the amount of data ?
If we can do that it would be a good idea.

what data can you trust anymore.
take any real (unadjusted) data from the time thermometers were accurate for a baseline, and work to the present time.
we don’t need 0.1 accuracy to spot trends. trends don’t mean anything anyway. were just chasing our tail aren’t we?
if were trying to predict the future were fooling ourselves, if were trying to understand the climate lets collect data.

Jordan,
Sorry, thought you would read the references.
http://www3.interscience.wiley.com/journal/113468782/abstract?CRETRY=1&SRETRY=0
leonardo.met.tamu.edu/people/faculty/north/pdf/64.pdf

any one seen this?
from the nz climate coalition web site
http://www.investigatemagazine.com/australia/latestissue.pdf

stephen
I have had a look at two papers by North. I get the gist of his analysis, where it is coming from and where it ends. But it is still in the realms of measuring statistical aggregates, and not issues around how to reconstruct a signal from sampled data – as that’s what the above posts are about.
I’m sure we can all agree that knowledge of the statistical properties of a signal does not allow us to reconstruct the signal.
Thinking about it, failure to observe the Nyquist sampling period (in time or in space) can have an imact on how we measure statistical properties. Injudicious sampling interval could result in a need to increase the sampling interval, since more samples would be needed to counter the effects of aliasing. But other than that, I would be confident that the statistical properties should emerge from sample data, eventually.
That is not the case when we seek to reconstruct a signal from sampled data. Randomly scattering a minimum number of point measuring stations around the globe is not a sufficient condition to enable us to reconstruct a global signal.
It is essential to assess and then comply with a minimum distance between measuring points in order to avoid spatial aliasing. Failure could result in a seriously distorted impression of a trend in the global signal.
To repeat, those who claim the MWP was a “local effect” are making exactly the same point.
And – as somebody mentioned above – siting measuring stations at airports and other built-up could be another example of distortions being introduced by spatial aliasing. A way to address this kind of issue would be to design into the measuring system a decent number of point-samples between the airports and cities. With that, we would have a chance of being able to re-construct a picture of the true temperature field.
And that’s before we even think about polar ice caps and the Pacific Ocean.
In the absence of the matter being addressed in the literature, this looks like a serious gap in how we have approached the reconstruction of the putative global temperature trend.
George makes a good point – can we be satisified that the global temperature trends have not been reduced to junk due to spatial aliasing,

Toho (08:26:18) :
“I don’t see how the sampling theorem is relevant. We are not interested in exactly recreating daily temperature variations, but estimation of variations over decadal time frames”
It is important to get the temporal sampling regime right, although we also need to consider spatial sampling and risk of aliasing.
Take an event like ENSO – a phenomenon which evolves over several months, shifting thermal energy over wide regions. it is not felt equally over all parts of the globe.
Spatial distribution of the measurement network is surely a crucial factor in our attempts to create an accurate picture. Get it wrong and the picture could be completely wrecked by spatial aliasing. (Something I often wonder, when looking at 1998.)
The measurement network seems to change almost continuously (above plot of number of stations). If there was an identical “1998” at a different time, to greater or lesser extent, the temperature reconstruction would produce a different impression – solely due to changes in the network.

Jordan:
I agree with most of what you write. But my point is that it doesn’t follow from the sampling theorem. The sampling theorem is about recreating exactly. Sure, you are going to lose information when sampling, and that will cause errors in the temperature estimates. I certainly agree with that. But it has nothing to do the sampling theorem. I don’t think aliasing is a big deal by the way, because localized atmospheric energy will not stay localized for long.

“”” Toho (08:26:18) :
I don’t see how the sampling theorem is relevant. We are not interested in exactly recreating daily temperature variations, but estimation of variations over decadal time frames (besides, if you have a priori information that there is a 24-hour signal with harmonics, the sampling theorem doesn’t really apply, does it). That said, I certainly agree that there are large errors due in part to poor sampling methods, and the HadCRUT3 error estimates seem to be off by an order of magnitude or so. “””
Well then you don’t understand the sampling theorem. Standard Sampled Data theory shows that an out of band signal at a frequency B + b sampled at a rate 2B results in an in band error signal at a frequency B-b, which cannot be removed by any filter without also removing valid in band signals at the same frequency. Violation of the Nyquist criterion by a factor of only 2, as in sampling an out of band signal at a frequency of B + B or higher, results in an aliassed signal at B-B which is the average value of the function.
That means that by undersampling the data either temporally or spatially for the duration of the baseline period (30 years or whatever) that is used as the reference value for the “anomaly” calculation; that baseline average over whatever time period is corrupted by zero frequency aliassed noise components. It doesn’t matter that one may only be interested in trends; the trends are illusionary anyway. Anybody can see that plots of temperature anomalies over various time scales exhibit fractal like properties; longer and longer plotting intervals show “trends” of greater and greater extent over longer and longer time frames. I don’t understand your comment that a priori knowledge of a 24 hjour cycle with harmonics somehow dismisses the sampling theorem. Current methodology records a daily min/max temperature. That is a twice daily sampling rate, which only satisfies Nyquist in the event that the dailly cycle is a pure sinusoidal function. If it is not sinusoidal in waveform and is still periodic, there must be at least a second harmonic 12 hour periodic signal present, and 12 hour sampling of that violates Nyquist by a factor of two rendering the average contaminated by aliassing noise. Now as it turns out, min/max recording rather that simple 12 hour sampling eliminates the dgenerate case of exact twice signal frequency sampling, that can identify the signal presence, but not its amplitude, for example, when samples are taken exactly at the zero crossings, so recording zero. The min/max strategy does at least record roughly the correct amplitude, but not the correct time average, due to the presence of the 2F or higher signal components. So no, that foreknowledge does not erase the need for the sampling theorem.
Pachauri’s silly plot that Viscount Monckton called him out on in Copenhagen is a great demonstration of that fact. Statisticians are kidding themselves when they claim to be extracting additional information by applying regressions and filterings like running five year averages. We are talking about a chaotic function that never ever repeats; it only happens once; so what statistical significance is there in anything that only happens once ?
Yes it is true that the common mathematics of statistics can be applied to sets of totally unrelated numbers; and averaages, medians, standard deviations, or any other buzzword of standard statistics can be applied to number sets with no relational significance whatsoever.
That does not mean that the mere mechanics of doing that somehow will reveal “information” which was never in there in the first place.
It is often stated that White Noise contains more “information” than any other signal; it is totally unpredicatble, and no matter how long a string of white noise values you collect and process, you can never learn anything about the very next value to come along. In that sense the incoming signal is 100% information about itself.
The anomaly concept may seem advantageous, and it certainly does have some merits as to incorporating new stations into an existing network. But that is about the only merit.
Consider a solid sphere that is enclosed in a close fitting rubber (latex) skin so the skin is in contact with the sphere everywhere, but is just barely stretched, so it touches the sphere at every point.
Now take a hold of the skin at any point, and pull it away from the sphere by some small distance. The skin can then be stretched in any direction to move that point you have a hold of, over so some other point on the sphere by stretching the skin, and then the skin can be released at that point, so the skin is distorted, and points around the moved point are all moved to new locations.
It can be shown, that no matter how the skin is stretched and moved, there must always be at least two points somewhere on the skin that have not moved at all, and are still in their orighinal locations. No matter what contortions are applied there will always be at least two stationary points. Of course thoes point change for every different application of the stretch operation.
Now that simple problem in topology, is not unlike the mapping of tempearture aroound the globe and noting anomlies at each point. A zero anomaly report (at any point) is like one of the stationary points on the latex skin. The fact that some points did not move; and therfore record zero anomaly, does not grant a licence to assume that neighboring points also remained stationary, and would yield a zero anomaly, in the event that they too were actually sampled.
The complete global temperature continuous function ( at any time epoch) is exactly analagous to the latex skin on the sphere. Adjacent points can be stretched away from some stationary point, and nothing can be known about their locations (in the temperature anomaly realm) without actually taking a sample there.
In the case of temperatures the greater difference between adjacent points is like a greater stretching of the rubber skin. In weather terms, such diferences lead to the development of winds; but nothing can be learned about that unless all of those points are sampled. Anomalies yield no information that can be used to map wind patterns or virtually any other weather phenomenon, and after all, climate is supposed to be the long term average of weather.
On another issue, Bill had raised the concept of MP3 encoding, presumably as an argument counter to my suggestion to sub sample a digital data strem such as a music piece for example. Here Bill is confusing “Data compression” with “data aquisistion”
MP3 is an adaptive endoding of data THAT HAS ALREADY BEEN RECORDED.
The encoding algorithm, reacts to prior knowledge of data that has yet to arrive to be processed. similar concepts were already being applied in the 1960s in the recording of long playing 33 1/3 RPM phonograph (gramophone) records. During low level passages being cut on the master disc, the recording groove spacing was reduced to place the grooves closer together. When a louder passage was about to be recorded, the cutting lathe increased the groove spacing prior to the arrival of the loud signal, and that allowed the recording of a larger dynamic range than earlier constant spacing recording, and it also allowed different frequwncy compensation curves, that gave improeved signal to noise ratio; such as the standard RIAA recording and playback curves.
As Bill pointewd out, MP3 encoding allows data compression by factors of ten or more, which permits storing a whole lot of rather boring music on small players with play back quality, that passes for hi-fi to the unsuspecting purchasers of such music.
Just to be sure that some new technology hadn’t somehow snuck past this old fogey, I contacted a department manager at Creative; Soundblaster to some of us. They know about as much about MP3 and lookalikes as anybody.
He confirmed that there is no such thing as live real time MP3. The adaptive processing relies on prior knowledge of data yet to be processed, so some form of “buffering” is absolutely mandatory. In simple terms, the DATA must already be gathered, BEFORE you can process it, and compress it to store the pertinent information in less storage space.
So it is not a method of acquiring more information with less resources; just imagine running from place to place with a Stevenson screen to be in the right place at just the right time to record an important temperature anomnaly. Somehow basic information theory does not permit gathering more information with less resources. I’t somewhere in that whole signal to noise ratio, data rate, and channel bandwidth relationship. There’s that
Clude Shannon and his theorem in there, anothe bell Telephone Laboratories Product, Like Nyquist. How sad it is that we have lost that great National Treasure.
Signal recovery and signal processing, are somewhat different tasks than efficient data storage.
So sorry Bill, but no cigar.

Toho.
The sampling theorem is definitely not about recreating a signal exactly.
As you correctly say, sampling loses information. There is generally no prospect of exact recovery of the original signal in practical systems. A well sampled system (more-than-observing the limit of the sampling theorem) might be able to recreate a very good approximation to the original signal by interpolation between samples.
Rather than exactness, it would better to talk about the adequacy of sampling. For a bandlimited signal, the sampling theorem tells us to absolutely avoid aliasing as that will destroy our ability to reconstruct a reasonably faithful representation of the original signal.
Is aliasing a big deal in temperature reconstructions?
Perhaps the point is that we don’t know. We can share opinions on this and compare examples. That’s all good fun and helps to spread knowledge and experiences. But where is the formal analysis in the literature?
I haven’t seen it. So right now, it looks like there could be a potentially serious gap in the whole approach to recreating historic temperature series on a global scale. And that means they could be junk.
I do tend to agree with you that localized atmospheric energy will not stay localized for long. But people who claim that the MWP was “localised” would appear to have a contrary view. To repeat my example above, an El Nino event is relatively localised, and (without comfort on spatial aliasing) I wouldn’t be too ready to accept the 1998 spike was anything more than an artefact of an inadequate network.

I think it would be useful at this juncture to review the difference between standard deviation (SD) and standard error of the mean (SEM).
For example, if you want to know how tall the human race is on average, you can measure the height of N people chosen at random, and then calculate the mean. If you want to know how confident you can be in your mean, that will depend on how variable the population is (standard deviation, or SD) and how many samples you have (N). In the extreme case of a uniform population, it would be sufficient to measure a single person. It turns out that SEM = SD/sqrt(N). It bears repeating that SD is a measure of how variable your measurement is *within* the population, and SEM is how confident can be in the estimate of the true mean across the entire population.
All of this depends on the randomness of the N samples. If Japanese or Norwegians were overrepresented, your estimate would contain an error that is not expressed by SEM. The same is true for temperature. There are separate techniques that attempt to correct for sampling bias, which in the case of global temperature, is expected to be the driving source of uncertainty, and difficult to adjust for.
As an earlier person posted, rather than the number of measurements, it is the geographic distribution of measurements that does more to determine the reliability of the estimate of the mean. Reasonable people may also question the validity or relevance of global mean temperature as a rather meaningless concept, akin to the average human, with one breast and one testicle. Indeed, the last ice age was characterized less as a drop in the global average temperature, and more as a large increase in the difference between the tropics and the northern latitudes. In the terms of this discussion, SD increased more than the mean decreased.

In the time domain its been shown that (Tmax+Tmin)/2 is a good estimate. If you like go get CRN 5 minute data and see for your self. . In the spatial dimension the field is coherent over large distances. Think about it. The field can be sparsely sampled and long term trends can still be captured. That’s all you care about. Look at the records of the 4 longest temperature stations and compare them to the global average ( IPCC Ar4 ch06)
There are more important issues. Put your brain power there.

Toho (13:11:31) :
You are referencing only one point in the system and Mr. Smith is referencing the entire system. This maybe why you are not seeing things eye to eye?
If you are looking to make a mean temperature for the entire planet, you need to be thinking in terms of the entire system, and not just the particular sampling point in the system. It seems pretty silly (to me anyway, no disrespect) to argue against the idea of taking samples at evenly spaced intervals for reasons other than actual physical difficulty.
Of course, I feel that this is all a moot point and the range of the entire climate system is the important bit of knowledge. If the radiative warming theory is true the whole system should read warmer and the best way to show that, or its absence, is to show the bounds increasing. Of course, that would not prove CO2 involvement, but it would eliminate a lot of data issues.

“”” Toho (13:11:31) :
George E. Smith (23:59:30) :
“I don’t understand your comment that a priori knowledge of a 24 hjour cycle with harmonics somehow dismisses the sampling theorem.”
If you happen to have a-priori knowledge that the signal is a pure sine wave of a certain frequency you only need a single sample to be able to perfectly recreate it to prepetuity “””
This is worse than pulling teeth. If I have a sinusoidally varying signal with an exactly known frequency, and I take one sample per cycle; I get zero information about whether the amplitude of the cycle is 20 degrees C or 20 millidegrees C. I get exactly the same measurment every sample; and any process for obtaining an average, will naturally give exactly the value of that sample. Now remember I did say that the case of sampling at exactly 2.B (your are talking only 1.B) is a degenerate case with an indeterminate result. That is one of the reasons for random sampling; but that only works for a truly sinusoidal signal, which by definition has exactly the same amplitude each and every cycle. (if the amplitude changes from cycle to cycle it isn’t a sinusoidal function).
Some are saying that the min/max average is good enough. How good is good enough when we are talking hundredths of a degree.
I have looked at a whole bunch of daily “weather” maps; do it every day for the SF bay area, and daily min-max ddifferneces of 30-40 deg F are very common, with tens of degrees differences over distances of a handful of km separation.
When it comes to shorter than daily cyclic temperature changes, (we actually have clouds in California) there is no prior frequency knowledge.
One can argue that “it all comes out in the wash” and the averages are good enough. Well the climate does not depend totally on average temperature.
At the global mean temperatuyre there is no “Weather”, so there can’t be any climate either since that is defined as the average of weather.
Unfortunately, the operating Physics doesn’t pay any attention to averages. The surface emitted thermal radiation that is a major earth cooling process, follows a more 4th power of temperature law, and the spectral peak of the thermal radiation which is what is of interest to GHG capture, varies as the 5th power of the temperature.
So if you take the integral of the 4th or 5th power of any cyclic temperature curve over a complete cycle, the result is always higher than simply integrating the average; so cycles do matter, and in the case of climate temperatures there are notable annual and daily cycles that result in an always positive enhancement of the total radiant emmission from the earth, and hence estimates of whether we are warming or cooling.
It is ironic that all you statisticians think that a tenth of a degree change in the average of a variable that has a 150 degree C possible maximum range, of values on any given day is somehow significant; but you can then dismiss similar errors in your homogenised data, that result from plain and simple experimental errors; because well you aren’t really interested in the data, just the imagined “trends”.
Just don’t call it science if that is what you truly believe.
Take a look in Al Gore’s book “An Inconvenient Truth”; pages 66-67 specifically, where we have Al’s impression of atmospheric CO2 and Temperature from some ice cores, over a long period of time.
So tell us what “TREND” is depicted in those graphs of CO2 and Temperature ? I’m only interested in Climate not weather so just give me one number for the whole 600,000 years. That should be long enough for you to get some good statistical average of the trend, and I would expect a pretty small standard deviation after all that time.
Just try telling your cell phone service provider, that the sampling theorem doesn’t matter, and he should be able to give you pretty good; though “Average” service without paying any attention to the Nyquist theorem.
The CRU supporters complain that we are taking the whistle blown e-mails out of context; in other words; we aren’t getting the full story form just a few e-mail snippets. Funny how that works with e-mails but we can get perfectly good climate data from out of context “sampling”.

Toho:
I agree global reconstructions are not expressed at daily reolution, but most are monthly. Do we have the well behaved “thermal surface”? (Note, spatial resolution is the question, not time.)
My doubts remain. We know of significant regional fluctuations which prevail for many months and even years (like ENSO, AMO, PDO). What picture do we get from sparse and irregular spatial samples?
I agree with your comment that thermometer readings are just viewed as a statistic, but aliasing can introdice a nasty systematic distortion. Not necessarily a zero-mean random variable which will “melt away” in the calculation of a mean. If the network gives us a distorted picture of (say) ENSO in 1998, the same network might give a similarly distorted picture of other ENSO events.
Another example to illustrate concerns about loss of information in sampling, and why statistical analysis does not get us “out of jail”:
You’re travelling at constant speed on a bicycle, and compelled to keep your eyes closed. You are permitted to blink your eyes (open) at regular intervals to capture some information about the road ahead. This is all you have to make decisions about direction. You’re fine at (say) 10 blinks per second, but then your are required to reduce your blink rate. There will come a point when the blink rate is so low that you are unable to gather enough information about the road ahead to avoid a crash. Loss of information is too great beyond that point. The lost information is permanently lost, there is no statistical analysis or modelling from the samples to get it back. So when you are at risk of crashing, the only practical option is to increase the blink rate to something that meets your requirements.
To George:
My encounters with the sampling theorem are all founded in discrete control systems design. It is easy to show matehmatically how a stable closed loop system can be driven to instability by extending the sampling interval.
I think your familiarity of the sampling theorem will be greater than mine. However I understand the mathematical analysis usually starts with an absolutely band-limited spectrum (dropping to zero amplitude beyond an assumed maximum frequency). The sample sequence “repeat spectra” can then be completely isolated and removed in theory. Do I take it that’s the reason why you suggest the original signal can be perfectly recovered from the samples?
I asseted that perfect reconstruction is not possible – without explaining that practical systems cannot totally bandlimit a signal in the way of the theory. We can realistically reduce frequency-folding to an immaterial level (for a purpose, such as control), but that also means the samples can only ever produce an imperfect reconstruction of the continuous signal.
(OK – perhaps there is an exception of well cyclic signals, but that doesn’t add much to discussion of sampling the climate system.)
To Steven
You could well be right. Or maybe wrong – I don’t know and feel we’re not really in a position to say either way. As you;ll see from above, I feel inclined to hang onto my scepticism. The issue of aliasing needs to be formally investigated and submitted for review in the usual way. Until then, could we put an asterisk next to the global reconstructions, just to remind us of this loose end?
(Good discussion about an interesting topic.)

I forgot to add above that it is not any need to reconstruct a global temperature map that concenrs me. It is that aliassing noise can swamp even the average, with just modest undersampling, and the base time data sets used to compute temperature anomalies are just such long time averaged data that completely ignores the fact that that base average temperature is itself corrupted by noise; no matter how long the base interval is.

Toho (03:32:41) : ” … homogenized data .. will tend to hide systematic errors by making station records more appear more correlated than they are in reality.”
Yes, that’s a good point.

This graph contains raw data from NASA GIStemp – global and US.
http://i629.photobucket.com/albums/uu20/blouis79/USglobal_anomaly_vs_jetfuel-1.png
For some reason, the unsmoothed US data swings wildly compared to unsmoothed global data – the original global data source included month-by-month data too, so I think it can be reasonably trusted.
A statistical analysis of the US data would easily indicate a very wide predictive uncertainty.

George
There was no doubt some misunderstandings and dead ends in the above discussion, and the sine wave might be one of them. But I wouldn’t be too dismissive of Toho’s position.
Toho also makes this very good point (for a non-periodic signal):
” the theorem does not say the inverse, that you can’t have complete information of the signal without all the samples, or with a smaller number of sample points. All your arguments seem to be based on this inverse of the theorem, which isn’t generally true.”
Seems to make sense to me.
Toho- you have said the following on a number of ocasions: “The theorem provides a sufficient condition for reconstruction, not a necessary condition.” I don’t follow – could you please expand.
One of the things I take from the above discussion is the apparently conflicting requirements of statistical analysis (sampling error) versus the sampling theorem (errors due to aliasing) when it comes to sample autocorrelation.
If the objective is to measure statistical aggregates, autocorrelation in the samples tends to be a problem – making life generally more difficult and perhaps even obstructing an analysis. If the objective is to recreate a faithful representation of the continuous signal from a finite sample, autocorrelation between the samples would appeat to be an absolute necessity.

Gerlich and Tscheuschner (in their debunking of the CO2 greenhouse effect) argue that “there are no calculations to determine an average surface temperature of a planet” because there are too many localized random temperature variations to know which are and aren’t accounted.
I note the NASA GIStemp procedure includes “eliminination of outliers”. G&T would probably argue there are no outliers that should be eliminated, since they are all part of local variations over the globe.

George
I see more agreement in our various discussions than disagreement. Particularly on the question of whether we have an adequate sample of the climate system to support the single line which is supposed to represent the global trend in temperature.
(I might add that the concept of a global temperature is about as meaningful to me as the average one-breasted, one testacled person mentioned in a previous comment here).
Toho makes a fair point that other information can reduce the demands we would otherwise have to make on data sampling. The generality of this point should not be understated.
His (her?) first example took ihe point to an extreme – perhaps unhelpfully. However … if we know the signal is a simple sinusoid, and we also know the amplitude (or phase), it would only take one sample to give us the last unknbown to fully define the signal for an indefinite period.
Theat’s an extreme example of Toho’s (fair) point that there is a “sufficiency” versus “necessary” angle to sampling.
In response to further comment, Toho then dealt with a situation where we have less knowledge. If we only know that the signal is a sinusoid, it would only take three samples to fully define the signal. (OK, we also need to know that the samples are all within one cycle – so we would need to have at least some idea of the frequency or phase).
This is a leap that deserves acknowledgement.
I mentioned my background in control engineering, where we frequently have the luxury of a framework of “a priori” knowledge. One of my first posts on this thread talks about sampling as a design problem – the issue being to have enough initial knowledge of the system/signal to design the sampling methodology. Well that’s the sort of approach that comes naturally to tackling a control problem.
I think Toho makes much the same point from a different direction – if we know something about what we are trying to sample, we can then make decisions about how to sample it. (Toho – I hope that’s fair to what you were saying.)
At the end of all of that a question: can I get comfortable that the sparse and erratic sampling of temperatures over the last 150 years gives us the information to support the analysis at the top of this thread. Frankly? No!

George
The point about a priori information appeals to me because it has parallels in the procedure of “identification” in control system design. In reasonably well defined situations, we can understand the linkages between different parts of the controlled process to determine where measurements are required. We can often use knowledge of dynamic parameters to determine sample rate for discrete controllers, and therefore to design an observable and controllable solution. Where things are not so well defined in advance, we may need to set up some form of test to determine the required parameters empirically. I know that these are luxuries which are not generally available elsewhere – including the climate.
A situation where we have absolutely no a priori snippets of information, would lead me to question how we could even start to work out the how’s and where’s of sampling. Not least what information would we use to choose a sample frequency/disribution.
This unhappy situation does not appear to be too far from what I can see in the assessment of global average temperature. I do not think the underlying system is well enough understood to come to decisions about how best to sample it. Perhaps the greater spatial coverage of the satellite systems will help to resolve that in time.
But, IMO, the historic “instrumental” temperature record is quite another thing. Analysis of trend lines and error regions takes a remarkable degree of faith in underlying assumptions about: (i) the behaviour of the spatial field in different time scales; and (2) the signal we are getting from an inherited and changing measurement system originally set up for all sorts of other purposes.
I could take issue with those who suggest there are bigger fish to fry than concerns about sampling, spatial and perhaps even temporal aliasing. What do we have to give use the comfort that this data is more than just pile of badly sampled and misleading junk? The trends may have about as much meaning as tracing the path of a drunk man staggering around in the dark.
If talking about priorities, is there anything more important than such a fundamental question about the quality of the data?
There is no question that the historic data has some value. It is better that we have it than not. But are we allowing ourselves to be impressed by the sheer mass of data? Are we trying to convincing ourselves that the more of the data we use, the more meaning we can yield?
Dan makes a perfectly good suggestion in this thread: “don’t try to make it a global or hemispherical average, just track trends at those admittedly small sample of sites.”
But that kind of suggestion will mobilise an army of opinion, arguing that point measurements are not representive of the full spatial field. Arguments which are basically alluding to aliasing – admssion that the field does not behave in a way which can rely on sparse sampling.
So where does this get us? JGC acknowledges a problem in what he calls “coverage bias”. Would it be better for JGC to put the analysis of historical data onto the back burner until we have some pretty convincing analysis and methods which will allow us to extract meaningful information from this historical data.
Right now (I say it again) There is no convincing reason to pay any attention to those trend lines. I think Dan’s suggestion is just as convincing.

I think Gerlich and Tscheuschner talked about random variations in temperature in local regions across the globe due to cloud effects making it rather difficult to figure a global mean from a limited geographic sample. It would be interesting to compare data from home weather stations with the Hansen smoothing across geographic areas.