Guest Post by Willis Eschenbach
I have long suspected a theoretical error in the way that some climate scientists estimate the uncertainty in anomaly data. I think that I’ve found clear evidence of the error in the Berkeley Earth Surface Temperature data. I say “I think”, because as always, there certainly may be something I’ve overlooked.
Figure 1 shows their graph of the Berkeley Earth data in question. The underlying data, including error estimates, can be downloaded from here.
Figure 1. Monthly temperature anomaly data graph from Berkeley Earth. It shows their results (black) and other datasets. ORIGINAL CAPTION: Land temperature with 1- and 10-year running averages. The shaded regions are the one- and two-standard deviation uncertainties calculated including both statistical and spatial sampling errors. Prior land results from the other groups are also plotted. The NASA GISS record had a land mask applied; the HadCRU curve is the simple land average, not the hemispheric-weighted one. SOURCE
So let me see if I can explain the error I suspected. I think that the error involved in taking the anomalies is not included in their reported total errors. Here’s how the process of calculating an anomaly works.
First, you take the actual readings, month by month. Then you take the average for each month. Here’s an example, using the temperatures in Anchorage, Alaska from 1950 to 1980.
Figure 2. Anchorage temperatures, along with monthly averages.
To calculate the anomalies, from each monthly data point you subtract that month’s average. These monthly averages, called the “climatology”, are shown in the top row of Figure 2. After the month’s averages are subtracted from the actual data, whatever is left over is the “anomaly”, the difference between the actual data and the monthly average. For example, in January 1951 (top left in Figure 2) the Anchorage temperature is minus 14.9 degrees. The average for the month of January is minus 10.2 degrees. Thus the anomaly for January 1951 is -4.7 degrees—that month is 4.7 degrees colder than the average January.
What I have suspected for a while is that the error in the climatology itself is erroneously not taken into account when calculating the total error for a given month’s anomaly. Each of the numbers in the top row of Figure 2, the monthly averages that make up the climatology, has an associated error. That error has to be carried forwards when you subtract the monthly averages from the observational data. The final result, the anomaly of minus 4.5 degrees, contains two distinct sources of error.
One is error associated with that individual January 1951 average, -14.7°C. For example, the person taking the measurements may have consistently misread the thermometer, or the electronics might have drifted during that month.
The other source of error is the error in the monthly averages (the “climatology”) which are being subtracted from each value. Assuming the errors are independent, which of course may not be the case but is usually assumed, these two errors add “in quadrature”. This means that the final error is the square root of the sum of the squares of the errors.
One important corollary of this is that the final error estimate for a given month’s anomaly cannot be smaller than the error in the climatology for that month.
Now let me show you the Berkeley Earth results. To their credit, they have been very transparent and reported various details. Among the details in the data cited above are their estimate of the total, all-inclusive error for each month. And fortunately, their reported results also include the following information for each month:
Figure 3. Berkeley Earth estimated monthly land temperatures, along with their associated errors.
Since they are subtracting those values from each of the monthly temperatures to get the anomalies, the total Berkeley Earth monthly errors can never be smaller than those error values.
Here’s the problem. Figure 4 compares those monthly error values shown in Figure 3 to the actual reported total monthly errors for the 2012 monthly anomaly data from the dataset cited above:
Figure 4. Error associated with the monthly average (light and dark blue) compared to the 2012 reported total error. All data from the Berkeley Earth dataset linked above.
The light blue months are months where the reported error associated with the monthly average is larger than the reported 2012 monthly error … I don’t see how that’s possible.
Where I first suspected the error (but have never been able to show it) is in the ocean data. The reported accuracy is far too great given the number of available observations, as I showed here. I suspect that the reason is that they have not carried forwards the error in the climatology, although that’s just a guess to try to explain the unbelievable reported errors in the ocean data.
Statistics gurus, what am I missing here? Has the Berkeley Earth analysis method somehow gotten around this roadblock? Am I misunderstanding their numbers? I’m self-taught in all this stuff and I’ve been wrong before, am I off the rails here? Always more to learn.
My best to all,
w.
@Stephen Fisher Wilde says:
August 19, 2013 at 10:59 am
The conversion of energy to and fro between KE and PE is a critical issue because in each case that conversion is a negative system response to the radiative characteristics of GHGs or any other forcing element other than more mass, more gravity or more insolation.
That may be true Stephen I don’t know and not something I really care about as I said you guys even reparamterised will still argue about what is and isn’t climate change you will break into groups I expect it.
The issue I was dealing with was trying to get people to realize this perceived problem of definition of temperature and measuring point and the like is all garbage.
These terms are all junk classic physics and people are trying to give some sort of precise meaning thinking that makes the problem better. I mean temperature is not even a real thing its a group of quantum properties that sort of looks a bit like kinetic energy but really at it’s basis it goes back to a historic property that it is a group of quantum properties that made a liquid expand up a tube when you heated it.
What they lost in their argument was that they couldn’t get a common reference because classic physics is wrong and junk has been for 100 years and you can’t define your way out of it. The most important thing that temperature = energy (because energy equals quantum information) got lost in the stupidity that is classic physics. It still stuns me that people don’t realize that fact because they lost it’s meaning in the mumbo jumbo garbage that is classic physics.
Energy is the universal reference frame that works everywhere in the universe that we know and all the hard science reference to it because of that fact. If climate science is wallowing down into inane arguments about definitions then it simply needs to reference itself to energy like every other science and that was the point I was making.
Gail, good questions. Before getting to them, let me just clarify that my interest is in the minimalist physical accuracy of the measured temperatures themselves.
This means the accuracy of the thermometer (sensor) itself, under conditions of ideal siting and repair. The accuracy of any reported temperature cannot be any better than obtained under those conditions. UHI, siting irregularities, instrumental changes, etc., etc., only add to the minimum error due to the limit of instrumental accuracy. I’m interested in that maximum of accuracy and minimum of error for any given temperature measurement.
To your questions: First, any temperature measurement is unique and physically independent in terms of it being a separate measurement. It is like a single UV-visible spectrum: its inherent magnitude and properties depend in no way on any system you’ve measured before.
The magnitude and properties of the measured magnitude do depend in some way on the instrument one used, just like the UV-vis spectrum. Resolution, accuracy, linearity, and so on. Instrumental characteristics and errors don’t affect the properties of the observable. They affect the properties of the observation. Hence the need for calibration.
As with electronic spectra, one can combine temperatures from various places and in various ways — but one must keep very careful track of what one is doing, and why and what one trying to accomplish. One can linearly combine separate and independent electronic spectra to estimate the effects of an impurity, for example, but one cannot represent that composite *as* the spectrum of product plus impurity.
So, anyone can average temperatures from anywhere, but must keep careful track of exactly what that average means, and must report only that meaning. An average of temperatures from San Francisco and New York might be interesting to track and one could announce how that average changes, and how SF and NY depart from their common average over time.
The physical uncertainty in the average will depend on the accuracy of the measurements taken from each of the two thermometers. This uncertainty tells us about the error bars around the average — its reliability in other words. If the departure of an anomaly is less than the uncertainty in the average, then that anomaly has no obvious physical meaning with respect to anything going on with temperature at SF (or NY).
There is a second kind of uncertainty, which follows from your second question. This second uncertainty is not an error, but instead is a measure of the variability inherent in the system: state uncertainty.
Here’s an example: Suppose we can measure individual temperatures at SF and NY to infinite physical accuracy. We have this daily 2 pm temperature series (in C):
NY SF
25 15
22 17
18 18
25 16
28 19
Avg: 20.3(+/-)4.4 C
That (+/-)4.4 C is not an error, but it is an uncertainty. It’s a measure of how the day-by-day average bounces around. It’s a measure of the variation in the state of the system, where the system is SF + NY. Concisely, it’s state uncertainty.
Suppose after 100 years of data, the state uncertainty remains (+/-)4.4 C, then this magnitude is a measure of the inherent variability in the system at any given time over that century. Any annual SF-NY average takes its meaning only within the state variability of (+/-)4.4 C.
So, if one wanted to compare annual SF and NY anomalies vs. a centennial average of temperatures, one would have to take cognizance of the fact that the 20.3 C average by itself is not a true representation of the system. The true representation includes the inherent state variability. To evaluate the divergence of any one anomaly would require assessing it against the (+/-)4.4 C variability inherent to the system.
Turning off the infinite accuracy in our measurements, the uncertainty in any pair average temperature is the root-mean-square of the individual measurement errors. The uncertainty in the centennial average temperature is the r.m.s. of the measurement errors in all 73,000 individual measurements.
The uncertainty in any anomaly is the r.m.s. of the individual temperature measurement error and the uncertainty in the centennial average. I.e., the uncertainty in the anomaly is larger than the error in the measured temperature from which it’s derived. This uncertainty is a measure of the physical accuracy of the anomaly, the reliability of the anomaly — how much we can trust the number.
Finally, the physical meaning of an anomaly is determined by the combined physical accuracy and the state uncertainty.
That’s a long answer to your questions, but I hope it’s readable.
After all that, notice Jeff Cagle’s description of the ARGO SST methodology. No disrespect meant, but notice there’s not one word about the accuracy of the individual ARGO measurements.
ARGO buoys have never been field-calibrated. They’re calibrated on land, and then released. No one knows whether the temperatures they measure at sea reach the accuracy of the land calibrations. Re-calibrations on land show no drift, but this is not the same as knowing whether environmental exposure affects the temperature readings.
As an example using the more familiar land-station, the ARGO project is like lab-testing a land-station PRT after a year, finding that it hadn’t drifted, and deciding that therefore all the outside recorded temperatures were good to (+/-)0.1 C (lab accuracy).
Such a decision would ignore that the first-order inaccuracies in measured land temperatures all come from environmental effects that mostly impact the screen, rather than from sensor failures. Nearby drifting buoys that should record the same sea surface temperature are known to show a bias (difference) of ~0.15 C, and an r.m.s. divergence of (+/-)0.5 C. [1] Is this divergence included in SST uncertainty estimates, or is it just averaged away as 1/sqrt(N), as though it were mere Gaussian noise? Probably the latter, I’m sorry to say.
[1] W. J. Emery, et al., (2001) “Accuracy of in situ sea surface temperatures used to calibrate infrared satellite measurements” JGR 106(C2) 2387-2405.
@ur momisugly Pat
I take it that with the Argo buoys you are referring to something similar to what happened when several different temperature sensors were tested inside a Stephenson screen here in Oz. While all the sensors agreed extremely closely what the freezing and boiling points of water are when calibrated, they didn’t agree what the temperatures when placed inside the Stephenson screen. And position inside the screen affected the measured temperature as well.
Pat Frank says: @ur momisugly August 19, 2013 at 10:13 pm
…..
Thanks Pat,
I could follow your comment with out a problem and understand exactly what you are talking about.
As I said my Stat courses are decades old so I do not have the words. However after sitting through screaming matches between the analytical chemist who devised the test method and the mix room manager with out of spec batches and being the one having to straighten the mess out, I have a pretty good feel for error from the test method and variability inherent in the batch due to poor mixing or worse electrostatic forces that cause the batch to segregate the chemicals the more the batch is mixed.
Slapping some liquids in a tank and turning on a mixer or dumping some finely ground solids in a tumbler and turning it on does NOT mean you are going to get a uniform batch. Hence my cynicism when it comes to the ‘CO2 is well mixed in the atmosphere’ Assumption and the similar assumptions I see being made with temperature. That is temperature does not vary much over a wide area and a few points can effectively give you an accurate picture of the true temperature with the precision claimed. Past experience says that just doesn’t pass the smell test even though I can’t explain why.
And as LdB and others have said time and again temperature is the WRONG parameter to be measuring in the first place but we seem to be stuck with it.
1sky1
Is your point about Kriging true? It depends, and is highly dependent on what type of kriging methdology employed.
But I agree that the issue should not revolve around which gridding system is best but whether or not the raw data is reliable and unbiased. I have noticed that in this field everyone is data processing mad ignoring the most important step in science: experimental setup.
Friends:
LdB says at August 19, 2013 at 6:53 pm
{emphasis added: RSC}
I should have seen that coming. Another Myrrh has turned up.
Richard
Pat Frank:
You provide an excellent post at August 19, 2013 at 10:13 pm.
I write to draw attention to it and provide this link which jumps to it for any who may have missed it
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1395124
Richard
@Gail Combs says:
August 20, 2013 at 2:15 am
And as LdB and others have said time and again temperature is the WRONG parameter to be measuring in the first place but we seem to be stuck with it
Haha true but hey if we have brave and crazy game to try … I can give you the proper definitions but you probably need a quick intro coarse at uni. The problem is climate scientist probably don’t want to go back to uni and to be fair they do manage to get it right using classic simplifications most times 🙂
DEFINITIONS FOR THE BRAVE AND CRAZY (I think they are right classic physics always freaks me out … nah they are right I cheated and decided to copy Lasalle)
Energy of particle: energy associated with occupied quantum state of a single particle – many quantum levels available, but only one is occupied at any point in time
Thermal modes: those quantum interactions with energy gaps small enough that changes in temperature can affect a change in population of states
Non-thermal modes: those quantum interactions whose energy gaps between adjacent quantum states are too large for population to be affected by temperature
Energy expectation value : = average energy for a single molecule… averaged across all possible quantum states… and weighted by probability of each state
Total energy: N·, where N = total number of particles
Energy: the capacity to do work
Internal energy of a system (U): combined energy of all the molecular states
Heat (q): thermal transfer of energy to/from the system to the surroundings. Occurs
through random collisions of neighboring molecules.
Temperature (T): parameter that describes the energy distribution across the quantum
states available to the system
Thermal energy: kT = average Boltzmann energy level of molecules in surroundings
LdB says…
……………..
ARRGHhhh, Science Fiction Physics and Thermo! I passed those courses by the skin of my teeth and a lot of late nights studying.
When I was in school (as a chemist) we didn’t have to take Stat. at all.
>>>>>>>>>>>>>>>>>>>>>
cd says:
……t the issue should not revolve around which gridding system is best but whether or not the raw data is reliable and unbiased. I have noticed that in this field everyone is data processing mad ignoring the most important step in science: experimental setup.
………………
Correct. And I think the point J W Merks in Geostatisics: [Kriging] From Human Error to Scientific Fraud is trying to get across is you can not make a silk purse out of a sows ear.
It all goes back to the assumptions you make. You assume the data is reliable and unbiased you assume there is enough data to define the surface Kriging is describing.
Merks page Sampling paradox lists the problems we are trying to get at.
[QUOTE]
Mathematical Statistics………………. Geostatistics
Functional independence fundamental…Functional dependence ubiquitous
Weighted averages have variances…….Kriged estimates lack variances
Variances are statistically sound………..Kriging variances are pseudo variances
Spatial dependence verified……………..Spatial dependence assumed
Degrees of freedom indispensable ……Degrees of freedom dismissed
Unbiased confidence limits quantify risk..Unbiased confidence limits are lacking
Variograms display spatial dependence…Semi-variograms make pseudo science
Smoothing makes no statistical sense……Smoothing makes geostatistical sense
Mathematical statistics is a science ………Geostatistics is a scientific fraud
…wonder about the nimble workings of geostatistical minds as degrees of freedom became a burden when a small set of measured data gives a large set of calculated distance-weighted averages-cum-kriged estimates.
[UNQUOTE]
That last line is exactly what I am trying to articulate.
PG, you’ve got it by the short hairs. 🙂
Gail
I don’t know where this hatred of a generally accepted statistical methodology comes from – its not an entity trying to profligate some view of the world. Kriging, or Spatial Linear Regression, as a statistician would probably refer to it, is like any other statistical method – just that…a method. It makes assumptions that work well with one dataset and less well with another (because the underlying mathematical assumptions aren’t met). But again there are always work arounds and hence the palette of Kriging methods. It is by far and away the most sophisticated and robust gridding method because it doesn’t make any assumption beyond what the experimentally derived bivariate statistic (the variogram) tells us. So where we have sparse data it does not manufacture a trend when interpolating between control points, if the range of spatial “correlation” is exceeded. That’s one of its key strengths – it doesn’t, if you do the experimental stage and statistical stage correctly, manufacture artifacts where there isn’t any information.
My argument is that the improved “accuracy” of using more robust data processing methods is far out weighed by the benefits one would get from getting more accurate observations.
Gail, summers as an undergrad, I worked in the analytical lab for a small detergent mixing house, now defunct, called Klix Chemicals. They made large batches of powdered and liquid detergents, mostly for janitorial supply and for the military (MILSPEC was really something; detailed rules for guidance of the inept) and occasionally saponified 4000 gallons of vegetable oil to make Castile soap. I got to titrate stuff for total alkalinity, assess [phosphate], etc., and wash windows and cloth patches to test the efficacy of our products. So, I have an idea of your experience. No screaming matches as I recall, but those big mixers were something.
cd says: @ur momisugly August 20, 2013 at 8:59 am
I don’t know where this hatred of a generally accepted statistical methodology….
>>>>>>>>>>>>>>>>>
First when ever I see “a generally accepted statistical methodology” a red flag goes up because that is the same as saying the science is settled. In my work experience it meant, we know we screwed up, we know we have been doing it wrong but we are not going to admit it. Instead we are going to take the average of all our factories and call that the ” generally accepted value”and hope our customers don’t catch on.
Again, I am ‘Computer Challenged’ with a minor bit of statistics training however I have worked with statistical computer programs in industry long enough to cringe after seeing the many ways they get used incorrectly by scientists. Also I may be a light weight but Merk is not. More important he is dealing not with Climate Science where there is no real way to go back and check whether the Kriging works but in an area where that checking can be done.
I am bring this view point up to the readers of WUWT because there seems to be a general acceptance of a method few have any knowledge of and I feel this is very dangerous.
Merk says of himself:
OKAY, I am a rabble-rouser but without a discussions of pros and cons we don’t know if Merk is correct or not. Saying “.. hatred of a generally accepted statistical methodology…” doesn’t cut it as a discussion.
My husband who is trained as a physicist and does computer work mutters Nyquist Frequency. He had the care and feeding of the computers at MIT Lincoln Lab that was used by semologists and electrical engineers. He has a worse opinion of Kriging than I do. (I asked him to help me understand it.)
He says:
One of the problems that had to be over come was the belief that computers invariably represented real numbers accurately. He believed that although all the scientists in the group knew about the problem, that because of expediency the individual scientist might believe his work was immune to it.
Pat Frank, yes those big mixes give you a real appreciation of the ‘Well Mixed’ concept don’t they. (Darn it NO, you can’t cut the mixing time by fifteen minutes so you can fit in another batch….)
Coming up with the correct sampling plan (and the correct parameters to measure) was always the biggest headache in QC.
@cd >> I’m not sure why Willis is spending time on this.
Because he wants to. Does there need to be another reason?
cd:
In many regions of the globe the only century-long station records available are from major cities, whose temperatures manifest various UHI effects. No matter what “kriging” method is used in those regions, the systematic, but highly localized, urban bias is spread much more widely. The uncorrupted regional average temperature signal is rarely known with any accuracy.
In many cases, due to sparse spatio-temporal coverage, BEST winds up manufacturing a “local” time-series employing very distant urban stations that are not even subject to the same weather regimes. The claim that their kriging algorithm consistently produces accurate time-series results in 3D space is entirely specious. You can’t get such results from absent and/or faulty data.
Ken – fair enough.
Gail
You misunderstand. Statistical methodologies are developed in order to deal with specific problems – there is only a judgement for the best method based on the central aims. It really doesn’t matter what your opinion is, or your husbands, nor does his qualifications. The only thing that matters is that you use the best tools for a specific problem. For example, using the arithmetic mean is not a good measure of central tendency in a log normal distribution but it doesn’t mean it isn’t a good approach in other circumstances.
I’m not going to get into the whole “after several years working as a mathematical genius (or this or that)…”. It carries no weight to your argument. For spatial interpolation there are many methods exact vs inexact, biased vs unbiased, those that honour regional gradient and those that don’t. Kriging is in my opinion the best, and by some measure. It may not give you the most interesting pictures but then that should not be the aim of the choice of algorithm.
Again there is no single Kriging method, it is more of a paradigm that approaches spatial regression from a particular angle. For example, the quoted critique you provided mentions pseudo-variance, and indeed routine types of Kriging do this, but if you want a reliable cdf, the indicator kriging of continuous variables would be the choice.
So, since we’re all wrong, what would your choice of gridding algorithm be.
Gail
BTW what has the Nyquist Frequency to do with Kriging unless you’re trying to Krige a signal or may be using an FFT-based algorithm for solving the large linear systems that can arise from Kriging using large numbers of controls. If you can sample at (or above) the Nyquist Frequency you really need do only a spline, Kriging would be overkill!
Perhaps you can expand.
1sky1
I’m not disagreeing with you. I don’t think BEST ever claimed to eradicate the UHI effect using Kriging. I don’t think you can. As I remember it, they did suggest it did resolve the problem of data clustering – which it does.
cd:
Your post to Gail Combs at August 20, 2013 at 1:55 pm
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1395592
makes a point and asks a question which I write to address.
You say
Yes, I explained that in my post to Gail at August 19, 2013 at 1:03 pm
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1394753
And that post from me also hinted at my answer to your question; viz.
I would not have one.
I would admit that no meaningful result is obtainable.
And I would point out that a wrong result is more misleading than no result.
Richard
Richard
I don’t know what you mean by grouping data points together. Kriging works on controls, the only real data you have; conceptually each interpolation point lies on “known” correlation surfaces (defined via the variogram) centered about (and for) each control point. Therefore, there will be a unique solution that will satisfy all the surfaces for all the points given the statistical model (the variogram).
Now if you’re saying that the controls are not accurate measurements, then that is different argument to the one I am making.
cd:
I realize that you’re not disagreeing with me. What I’m pointing out to the general audience is the
flimsy basis of BEST’s claim that, after subjecting all available data to its “quality control” and “analysis,” there was no significant difference evident in trends between urban and “rural” records. Had they actually examined vetted century-long records, instead of resorting to piecemeal syntheses from scraps of data, the foolishness of such a claim would have been evident.
BTW, in any serious scientific study of spatial variability using discrete data, aliasing is no less a concern than in temporal analysis. The mathematically convenient property of smooth changes over widely homogenous fields has to be empirically established, instead of simply being decreed by analytic fiat.
cd:
Thankyou for your reply to me at August 20, 2013 at 2:58 pm
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1395639
I am answering the two points I think you want me to respond. If I have missed any then please tell me.
Firstly, you say to me
I was answering the question put to me by Gail Combs
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1394718
and I was using her terminology. I understood her phrase “grouping data points” to mean ‘obtaining an average’. And my answer addressed that
http://wattsupwiththat.com/2013/08/17/monthly-averages-anomalies-and-uncertainties/#comment-1394753
You also say to me
No. I did not mention Kriging – or any other methodology – so the use of controls was not my point.
I was making a general point in answer to your question, and my point covers every type of estimating and averaging (including Kriging).
I remind that you asked
and I answered
In other words, I would openly admit that there is no valid method available to obtain a meaningful average and, therefore, I would advise that no average should be calculated because any obtained result would be more misleading than coping with the absence of the desired average.
Richard
1sky1
Thanks for the reply. I agree with your points. However, is this practically possible. Either way their claim is a little stretched.
On your second point, and not wanting to go off on a different direction, I’m not sure exactly what you mean by homogeneous fields in relation to spatial data. But I can’t see why one would get hung up on aliasing when carrying out spatial interpolation; the aim is akin but different to signal processing. In the real world pragmatism is required, one must make some type of reasonable estimate based on limited and sparse data. In kriging the variogram can be seen just as a prior, even if an incomplete one. If the aim were to preserve signal integrity or to avoid processing artifacts resulting from aliasing, then one would require exhaustive sampling.
I think a lot of comments here seem to be asking a lot and applying standards that one might expect in a controlled laboratory environment to the global environment.