The Limits Of Uncertainty

Guest Post by Willis Eschenbach

In the comments to my post called “Inside the Acceleration Factory“, we were discussing how good the satellite measurements of sea surface heights might be. A commenter said:

Ionospheric Delay is indeed an issue. For Jason, they estimate it using a dual frequency technique. As with most everything in the world of satellite Sea Level Rise, there is probably some error in their estimate of delay, but its hard to see why any errors don’t ether cancel or resolve over a very large number of measurements to a constant bias in their estimate of sea level — which shouldn’t affect the estimate of Sea Level Rise.

Keep in mind that the satellites are making more than 1000 measurements every second and are moving their “target point” about 8km (I think) laterally every second. A lot of stuff really will average out over time.

The underlying math is simple. The uncertainty of the average (also called the “mean”) of a group of numbers is equal to the standard deviation of the numbers (a measure of how spread out the numbers are), divided by the square root of how many numbers there are. In Mathspeak, this is

$\frac{\sigma}{\sqrt{N}}$

where sigma (σ) is the standard deviation and N is how many numbers we’re analyzing.

Clearly, as the number of measurements increases, the uncertainty about the average decreases. This is all math that has been well-understood for hundreds of years. And it is on this basis that the commenter is claiming that by repeated measurements we can get very, very good results from the satellites.

With that prologue, let me show the limits of that rock-solid mathematical principle in the real world.

Suppose that I want to measure the length of a credit card.

So I get ten thousand people to use the ruler in the drawing to measure the length of the credit card in millimeters. Almost all of them give a length measurement somewhere between 85 mm and 86 mm.

That would give us a standard deviation of their answers on the order of 0.3 mm. And using the formula above for the uncertainty of the average gives us:

$\frac{0.3}{\sqrt{10000}} = 0.003$

Now … raise your hand if you think that we’ve just accurately measured the length of the credit card to the nearest three thousandths of one millimeter.

Of course not. And the answer would not be improved if we had a million measurements.

Contemplating all of that has given rise to another of my many rules of thumb, which is:

Regardless of the number of measurements, you can’t squeeze more than one additional decimal out of an average of real-world observations.

Following that rule of thumb, if you are measuring say temperatures to the nearest degree, no matter how many measurements you have, your average will be valid to the nearest tenth of a degree … but not to the nearest hundredth of a degree.

As with any rule of thumb, there may be exceptions … but in general, I think that it is true. For example, following my rule of thumb I would say that we could use repeated measurements to get an estimate of the length of the credit card to the nearest tenth of a millimeter … but I don’t think we can measure it to the nearest hundredth of a millimeter no matter how many times we wield the ruler.

Best wishes on a night of scattered showers,

w.

My general request: when you comment please quote the exact words you are referring to, so we can avoid misunderstandings.

Article Rating
Inline Feedbacks
Bruce Cobb
December 20, 2018 2:19 pm

“A lot of stuff really will average out over time.”
A septic tank does that. Averages stuff, over time. But it’s still crap.

steve case
December 20, 2018 2:30 pm

But it’s still crap.

Ha ha ha ha ha ha ha ha ha ha ha!

Mr.
December 20, 2018 5:54 pm

Can I have the Australian and Canadian rights to your quip Bruce?

pulsar
December 20, 2018 7:00 pm

“But it’s still crap”
Over time, that crap turns to natural fertilizer. All craps do that given enough time.

December 22, 2018 5:58 pm

Well then, let’s use the model mean for fertilizer.

AndyL
December 20, 2018 2:20 pm

willis
What happens if the thing you are measuring is not a single thing (like the length of a credit card) but is itself the average of (say) 1000 things? In that case, would taking 1 million, or 100 million measurements make the average more accurate?

Menicholas
December 20, 2018 4:31 pm

My understanding, and I could be wrong, is that such techniques are valid for multiple measurements of the same thing.
But not for multiple measurements of something which is in the process of varying over time, or which is different depending on which part of the thing you are measuring.
I know that a number of commenters in previous threads had this same argument regarding such things as the global temp, and averaging readings from different places or the same place on different days.
My understanding, which I freely admit may be wrong, is that you have to be measuring the same thing multiple times. If you have one ore sample, it is one thing, and results can be averaged.
You cannot average measurements taken one for each of one hundred samples. AFAIK.

Besides for that, in every single class I ever took, you cannot report any result to more significant figures that the least number of sig figs in the measurements, whenever two or more measurements are factors in the final result. Because then you are multiplying errors and claiming precision you did not measure.
I have not read any of the comments yet, but I know this is always a huge debate whenever this comes up. This is my out of the gate thoughts only…and going by memories from a long time ago.
I do not think I have done this sort of work since I got an A in analytical chemistry in college, which was a long time ago.
I am looking forward to this comment thread, because I do not know nearly as much about this as Willis or probably numerous others here.

December 20, 2018 4:51 pm

You’re right, Menicholas, and so is Willis.

The lower limit of accuracy is given by the resolution of the instrument. In the case of Willis’ ruler, the smallest division is 1 mm. The most careful measurement, using one’s most analytical eye-ball, is to 0.25 mm.

It won’t matter how many gazillion measurements are taken, the length of the credit card will never be better than, e.g., 8.6±0.25 cm. And that’s assuming all the score-marks on the ruler are regularly spaced to 10-times that accuracy.

There’s another little interesting fillip about satellite sea level. Assuming all the error is random, it will average out as per Willis’ example: 1/sqrtN. But if the true height of the sea level varies from place to place, then the measured heights will have different physically true values.

That means sea level will have a state uncertainty because its height — its physical state itself — varies. So, the reported sea level should be reported as mean±error, and mean±(physical variation). Unless all the physical variations can be shown to be normally distributed about the mean.

knr
December 21, 2018 1:56 am

Providing The measurement instrument is has the accuracy levels claimed of it and you deal with operatory error.
So in this case, to what level of precision is the ruler marked, is 1 mm really 1 mm through the whole range being used , and what measure where taken to deal with factors such has eye-sight ?
These really matter when you start to make great claim about either the data or the accuracy of the information.

You can try the average route but you still average the numbers without knowing any errors and hopping, not knowing, if the process covers the errors.

Jim HUTCHINSON
December 21, 2018 6:51 am

Remember Wittgenstein’s Ruler: Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table, you may also be using the table to measure the ruler.

December 22, 2018 5:55 pm

The ENSO state uncertainty comes to mind.

It can be seen that the range of the multivariate index or sea level show a range of some 5 “normalized values”. It would be interesting to know how these values translate into meters, but this is no random variation.

Theoretically this variation is achieved by ENSO causing more rain to fall over the ocean and less over land, causing particularly South American tropical land droughts.

The Rossby and Kelvin alternating reaction waves bouncing back and forth across the basins, and the trade wind stacking are other state uncertainties at annual scale.

Robert B
December 20, 2018 5:20 pm

The rule works if errors are perfectly random and so with millions of measurements, nearly every measurement of +0.251 is cancelled by a -0.251 measurement. You can’t round measurements to the nearest whole and expect the rule to still be true. Even if you did each measurement to many more significant figures, you use the rule assuming no systematic error eg temperature change during measurements have contracted ruler and card differently. A bit silly to just assume it when applying the rule to a million measurements even if they don’t vary spatially or with time.

Menicholas
December 20, 2018 6:21 pm

Thank you Pat and Robert for the replies.
Very good point about systematic errors…I was thinking the same thing…the ruler could be defective, or it could be at a far different temperature than when it was manufactured.
The picture could have been taken from an angle, instead of square on, etc…

Another thing I wanted to mention was being careful to use consistent terminology, and to use the words that mean what one if attempting to communicate, in particular the distinction between accuracy and precision.

About the sea level measurements…we know for sure that the level of the sea is not anything like a symmetrical oblate spheroid, and for several reasons this is true. The actual shape of the Earth is called the geoid. I will post a link to an image of this shape. It is a bumpy lumpy and irregular shape. And sea level is even more so, given that sea level is defined in terms of the gravimetric field of the Earth in each given spot of the surface, and this varies tremendously, as mentioned and depicted briefly in the minute physics video on sea level.
I am not even gonna pretend I have any special insight based on knowledge of the method used by the people interpreting the satellite data…but given all of the comments over the past few days from people who evidently do have such knowledge of at least some aspects of the method and possible confounders, and when I have gleaned about what “sea level” even means…I think I would doubt the results of satellites over tide gages and old photos of known landmarks, even if I thought the people in charge of the entire process were as unbiased as a person could possibly be…which I do not.
In common parlance, these terms are interchangeable, but we know that is not the case when discussing measurements, and errors, and statistics, and such. Mentioning this for any readers who may not be familiar with this distinction, which is anything but a trivial one. But also to remind myself, because even though I know this, I still find myself using the wrong words sometimes in my haste to get my thoughts typed out.

Menicholas
December 20, 2018 6:48 pm

I do not know if there are different version of this…but most of the images of the geoid look more or less like this…using this one because it is a rotating gif:

http://2.bp.blogspot.com/-2o3hXVPI2EM/VnTlaYLmeVI/AAAAAAAAuEA/fDXIf3NA8yE/s1600/geoid.gif

Menicholas
December 20, 2018 9:52 pm

Oops, my second comment got the last part, beginning “In common parlance…”,
out of order.
Those last sentences should be further up…here is how it was meant to read:

Thank you Pat and Robert for the replies.
Very good point about systematic errors…I was thinking the same thing…the ruler could be defective, or it could be at a far different temperature than when it was manufactured.
The picture could have been taken from an angle, instead of square on, etc…
Another thing I wanted to mention was being careful to use consistent terminology, and to use the words that mean what one if attempting to communicate, in particular the distinction between accuracy and precision.
In common parlance, these terms are interchangeable, but we know that is not the case when discussing measurements, and errors, and statistics, and such. Mentioning this for any readers who may not be familiar with this distinction, which is anything but a trivial one. But also to remind myself, because even though I know this, I still find myself using the wrong words sometimes in my haste to get my thoughts typed out.

About the sea level measurements…we know for sure that the level of the sea is not anything like a symmetrical oblate spheroid, and for several reasons this is true. The actual shape of the Earth is called the geoid. I will post a link to an image of this shape. It is a bumpy lumpy and irregular shape. And sea level is even more so, given that sea level is defined in terms of the gravimetric field of the Earth in each given spot of the surface, and this varies tremendously, as mentioned and depicted briefly in the minute physics video on sea level.
I am not even gonna pretend I have any special insight based on knowledge of the method used by the people interpreting the satellite data…but given all of the comments over the past few days from people who evidently do have such knowledge of at least some aspects of the method and possible confounders, and when I have gleaned about what “sea level” even means…I think I would doubt the results of satellites over tide gages and old photos of known landmarks, even if I thought the people in charge of the entire process were as unbiased as a person could possibly be…which I do not.

AndyL
December 21, 2018 1:29 am

I was working on a complex analogy, and realised that it was summed up by Robert’s point that each error +0.251 is balanced by a -0.251 if the errors each side of the measurement are balanced.

However, is this discussion really relevant to satellite measurements? If we are discussing tide guages then yes, measurements are taken to the nearest millimeter (or 10th or whatever they use). Satellites will measure to much greater precision than that. I just hope that the average was taken before the measurements were rounded.

Robert B
December 21, 2018 3:20 am

Its been a while since I read up on it. I remember that average distance to a swathe of surface is measured with a precision of ±2 cm with a method of judging what the conditions of the area is and choosing the right model for the wave conditions. You really can’t expect the law of large numbers to fix up any systematic errors.

Phil R
December 20, 2018 7:04 pm

Haven’t read the other responses yet, but my first reaction is, you are correct. My background is geology (so last century), but we had to take chemistry and then had geochemistry classes. Don’t get me started on the geophysics professor who had no concept of sig figs.

Menicholas
December 21, 2018 2:45 pm

I took a bunch of geology classes, starting with physical, and then took one called Geology of the National Parks, and also took Earth history. At that time I was thinking I would pursue a degree program called Interdisciplinary Natural Science, but mostly I was just taking classes in subjects I wanted to know more about and was not really thinking about a degree.
So I also took physics, history of science classes, zoology and other biology classes, and some other classes that were heavy on earth history and the history of science.
Then I started to take more classes in physical geography, since I wanted to study the weather and those were prerequisite to meteorology and climatology and hydrology.
But then I found out that physical geography, meteorology and all of those classes were not considered natural science classes, hey were in the humanities dept, and I had so many science credits it was impossible to get a degree in any of those subjects without taking over a year of extra classes. At that point I had a lot of chemistry classes, and it seemed very easy to me and hard to other people, and also it seemed like pretty much everything is based on chemistry and physical chemistry at some level, so chemistry it was.
If I had been getting good advice or I had in mind getting a degree from the get go that was going to result in a wide choice of readily available jobs, I would have been in the engineering dept, or premed/medical school.
I was just about to decide on what graduate degree to pursue when I found myself under intense pressure to help out with the family business, the plant nursery, when my dad got a brain tumor and was unable to do anything and was facing a tax nightmare. Previously, I was just building the place for them on my weekends and Holidays and a couple of summers…no plans to have anything to do with the biz, but I was the only one in the whole clan who knew anything about construction, so I just built and built until I had put up 80,000 square feet under shade and glass.
I was getting letters of employment offers from everyone from the Navy (civilian nuclear tech on a sub) to the EPA, and the nursery ting was supposed to be just season, then a year.
By the time I realized it, I had been out of school so long, could not decide on what to pursue or where to do it (I had been accepted to Penn twice…family legacy) and then wound up renovating historical buildings and shooting pool and womanizing.

Life is never what you think it will be, at least it was not for me.

LdB
December 20, 2018 7:10 pm

There is no way to answer your question AndyL, this comes back to the issue of the salary for the Greenpeace employees. Once you start abstracting thing you need to know the distribution of the thing we are talking about.

In the Visa card situation you will get most around the same sort of number and a few who make complete mistakes etc so you can apply averaging knowing that you will get closer to the answer.

Wages for example can have a heavily cauchy distribution with management with one grouping of salaries and pleb workers with a different grouping of salaries. An average in such a situation is meaningless in most analysis because it represents a salary no-one actually gets. No matter how many samples you take it doesn’t improve things because you need to first understand the distribution.

Duane
December 21, 2018 5:34 am

What Willis did not make especially clear in this post is that he is describing the precision of the standard deviation, not the precision of the measurement of card length.

So yes, collecting a million measurements and having them all fit within the described range of 1 mm, then the precision of the standard deviation becomes very tight indeed. But the standard deviation itself does not grow smaller … it remains 0.3 mm.

Same with any other statistical description based upon real world measurements, or conditions.

DocSiders
December 21, 2018 6:26 am

And amazingly, the SD from 36 measurements (samples) is almost always the same as the SD when taking a million measurements.

ChrisB
December 20, 2018 2:20 pm

if you are measuring say temperatures to the nearest degree, no matter how many measurements you have, your average will be valid to the nearest tenth of a degree … but not to the nearest hundredth of a degree.

This assertion is probably wrong.

Because the statement clearly assumes that there are 10 distinct markings in between a degree (c or f or whatever) scale. However one can always subdivide the scale to a hundred, thousand or even a million markings (increasing precision). Once thousands of observers make millions of recordings of a set temperature (lets say the freezing water at sea level) the certainty will be more than a hundredth of a degree using the above formula of Standard Error of Estimate (SEE) However it does not mean the measured value will be exactly 0 degrees C (accurate), MEAN.

In other words, there seems to be confusion about accuracy and precision in the above statement.

ChrisB
December 20, 2018 3:05 pm

“doesn’t increase either the real-world accuracy or the real-world precision”

I dont really understand what these two terms mean. Never heard “real-world” metrics before.

However, I agree with you that there is a limit to what can be measured. It is called the Planck’s length,1.6 x 10-35 m. Perhaps you meant this as your “real-world precision”

Andrew Stanbarger
December 20, 2018 4:02 pm

You are both correct, in a way, but also incorrect. I’ve had a bit of training in metrology. You both seem to be confusing accuracy and precision. The limit on the precision of any measurement [or aggregation of measurements] is the precision available to you provided by the measuring tool. The limit on the precision of the ruler in the image is 1mm. You cannot measure tenths of a millimeter, or any fraction of a millimeter, with a tool that is not marked in fractions of a millimeter. If you take a thousand measurements with that ruler, or one, you will never measure more precisely than ± 1mm. Similarly, if your radar altimeter, on one measurement, reads to the nearest tenth of a millimeter, your aggregation of fifty million measurements taken with that altimeter will never be more precise than ±.1mm. Whether it is accurate or not is another question, requiring comparison to a measurement taken of a known standard.

richard Patton
December 20, 2018 6:00 pm

Brings to mind the same issue with the “warmest the Earth has been in history by .04C garbage we heard about a couple of years ago (or was it last spring). Up until the mid 20th Century, meteorological temperatures were measured to the nearest degree (F in the case of Central England which has the longest record) because greater precision didn’t matter. And until the late 20th century the tenth of a degree was eyeballed (estimated) I know, I was an observer. There were no computers to number crunch and make predictions and + or- one degree didn’t matter. Claims of warmer or colder with greater precision than one degree F (1 degF is more precise than 1 degC) are pure garbage!!!!!

Steve O
December 21, 2018 5:14 am

“If you take a thousand measurements with that ruler, or one, you will never measure more precisely than ± 1mm.”

This is not actually correct. Andrew Preece’s comment (way, way below) gives a example from the world of electronics of how it is done.

When Kip posted a column about exactly the same thing I created a math experiment in Excel that you can repeat for yourself. You can create 1,000 instances of measurements, with randomly distributed errors around a specific known value. Round each individual measurement to the nearest round number. You can still estimate the mean with surprising accuracy. More measurements produce better estimates. With 1,000 measurements, you’ll almost always be within a two decimal places.

TimTheToolMan
December 22, 2018 1:08 am

You can create 1,000 instances of measurements, with randomly distributed errors around a specific known value. Round each individual measurement to the nearest round number.

This is the fundamental assumption where theory departs from reality. The reality is that measurements are rarely evenly distributed around the value you want to measure.

If a measuring device is capable of measuring with say 3 values, 1, 2 and 3 and you’re wanting to measure 2.75 then you’ll probably actually measure 3. If on the other hand you’re trying to measure 2.5 then you might do it if there are evenly distributed values of 2 and 3 coming from the device.

Another Paul
December 20, 2018 4:49 pm

” Never heard “real-world” metrics before.”

Measure the thickness of printer paper, I’ll provide the yardstick.

Editor
December 20, 2018 4:59 pm

Measure the thickness of 500 sheets of printer paper 400 times.

That’s more relevant.

Then try to figure out “why” the differences are present:
Too much pressure on the caliper?
Too little pressure sometimes?
Different air masses between the 500 sheets of paper in each ream?
Different nbr of sheets in different reams?
Different sheet thicknesses between reams and between paper mills?
Different calipers?
Caliper differences in successive measurements?
Different users for the calipers?
Different humidity when measured?

but remember! The alarmists would have you believe none of these occur in the sea level measurements!

Menicholas
December 20, 2018 6:44 pm

Instead of using credit cards or sheets of paper as an analogy (even though you have shown how many things can vary with even something as straightforward seeming as measuring something which seems fixed and definite), maybe a better analogy would be trying to find out how much cats weigh, and if they are gaining weight over time, and if so, is the weight gain accelerating?
Suppose one had set up scales all over the cat world and the weight of cats was measured as they ran or walked across it.
And someone else was driving around with a very accurate laser scanner which had been cleverly designed to measure the shape and size of cats very precisely, and had also devised and algorithm to translate these readings into a weight.

Or even something simpler…how long are cats from nose to tip of tail, and like the sea and the tides, the darn things are never holding still and can change the shape and length of the tail and do, continuously, do so.
And then I graphed all the results…and on the page with the graphs I said all of the values were averages which had removed seasonal and daily variations to correct for meal times and cats that got more to eat during certain times of year.

Steve Reddish
December 20, 2018 9:15 pm

I like comparing trying to measure cats with trying to determine the level of the never-still sea surface! Very apt analogy

SR

Another Paul
December 21, 2018 4:46 am

“…the darn things [cats] are never holding still…” Oh then you’ve never met my slacker cat. She will sit in your lap for hours on end.

A C Osborn
December 21, 2018 2:22 am

Andrew Stanbarger December 20, 2018 at 4:02 pm
You have also forgotten “repeatability” of the instrument.
Ever done Gauge Capability Studies?

December 20, 2018 3:41 pm

” beyond a certain limit, increasing the number of measurements doesn’t increase either the real-world accuracy or the real-world precision.”

Number of measurements increases precision but not always accuracy. You can be precisely inaccurate.

You should really should take a stat class sometime.

In general, the S.E. (the S.D. divided by (approximately) the sqrt of the N) dictates the significant figures of the measurement. You can easily add one or two decimals with 30 measurements. And with 1000 measurements, go crazy.

That said, and you do haf to use a little judgement (no math formulas involved), but we have extended really crappy measurements to nice statistical differences all the time. It’s not rocket science. Just stats 101.

If you measure the height of the two sides of your favorite table enuf times, you will always find them different. (unless you use a ruler calibrated in cm only with no tenths of a mm.)

M Courtney
December 21, 2018 4:42 am

The more measurements the more confident you can be that the actual value is within a certain range.
However, the size of that range is determined by the physical nature of the measuring process and equipment.
They are two different things. But they are both called uncertainty because they are both related to the chance that a measurement isn’t reflective of the actual value.
Hope that helps resolve the confusion.

December 21, 2018 6:33 am

Actually, with your credit card example, if the true value was 85.8 mm, more people will get 86 than 85 and even fewer 87, and even fewer 84 and 88. So even though the ruler was calibrated in mm, you can easily increase precision of the measurement.

But I think you just being purposely obtuse on this, you could easily look that up in a stat book.

Steve O
December 21, 2018 7:51 am

Willis, as long as the errors are randomly distributed, it is possible to determine the true value with surprising accuracy.

In a cell in Excel, create a “true value” between 85 and 86mm. Copy that number down into a column 1,000 rows long, and in the next column use the random function to create measurement errors and then add the errors to the true value. (Create random numbers between 1 and minus 1. Or heck, make the errors between 2 and minus 2.) In the next column, round all the measurements to the nearest whole number.

Now you have 1,000 measurements, all of which are wrong. How close do you expect the average of the 1,000 incorrect measurements to be to your true value? If you take 5 minutes to do this, I guarantee you are in for a surprise.

Matthew Schilling
December 21, 2018 8:14 am

Steve O, isn’t it possible you learned something surprising about Excel’s “Random()” function?

Gordon Dressler
December 21, 2018 6:29 pm

Willis, you posted :
“Accuracy is how close successive measurements are to the true value.
Precision is how close successive measurements are to each other.”

Well, I was taught that “accuracy” is how closely the reported measurement represents the true value of the parameter being measured. One the other hand, “precision” just represents the number of digits used in the numerical value being reported.

Thus, for example, a mechanical caliper may report a length measurement of 6.532 inches and yet be much more accurate that a scanning electron microscope that reports the same object feature as being 6.54798018* inches long if the SEM has been incorrectly calibrated or has a undetected electronic failure. (*Note: modern SEMs can indeed achieve resolutions of one nanometer, or 4e-8 inches.)

So, precision actually has nothing to do with accuracy and the number of times a given parameter is measured. And accuracy is not necessarily related to precision of the measurement, but it can be improved by statistical analysis of repeated measurements at any given level of precision.

It is the combination of using highest precision within the known/calibrated range of accuracy of the measuring device that is of utmost importance to “truthful” value reporting, whether it be for a single measurement or multiple measurements of the same parameter.

Nylo
December 22, 2018 2:37 am

if the true value was 85.8 mm, more people will get 86 than 85
That’s totally true. You can get SOME increased accuracy, but not ANY increase just by increasing the number of measurements.
In your example, you can be sure that your average will be closer to 86 than to 85. But it is NOT true that, by increasing the number of measurements, you will get exactly 80% more 86s than 85s. It may be 80% or 70% or 90%. You won’t change that by taking more and more measurements. You could increase precision to about 0,1mm as Willis said, but not more than that, being realistic.
This doesn’t mean that the maths are wrong. What it means is that the condition for the maths to work (perfectly evenly distributed errors) never happens in real world.

December 20, 2018 4:59 pm

IF SATELLITES are capable of measuring
sea level to the nearest millimeter,
or even to the nearest centimeter,
I’ll eat my hat (a scientific term).

And I would say the same thing
if the oceans had no tides,
and no waves.

In my opinion, there is no accuracy,
and 100% uncertainty. with the
satellite methodology !

Of course after the “proper adjustments,”
the conclusion will be the usual:
“It’s even worse than we thought.”

The real question is whether sea level rise
is causing property damage on the shorelines,
such as on the Maldives Islands … where
investors are building new resorts
like money was growing on trees there.

Who is getting hurt by sea level rise now?

Who is getting hurt by global warming now?

The correct answer is “the taxpayers” —
getting fleeced by leftist con men,
(harmless) global warming
and sea level rise.

Satellite sea level data
does not pass the smell test”
(another scientific term)

My climate science blog:
http://www.elOnionBloggle.Blogspot.com

Menicholas
December 20, 2018 9:03 pm

In climatespeak, “I’ll eat my hat translates to” “Is not a robust methodology”.

bonbon
December 21, 2018 5:31 am

It is incredible how the smell test, the nose, makes scientists look blind.
Smell is the most ancient sense. The smells of peppermint and caroway, quite distinct, are yet of two molecular enantiomers.
Even we, nowhere near a cat, can actually smell chirality (rotational polarity).
There is more to measurement than meets the eye!

The satellites are sensing through a plasma, atmosphere, of highly variable electromagnetic polarizablilty. Ironically they are telling us more about the atmosphere or about the ocean surface physics. It would be an expensive scandal if that data is thrown away in a mad pursuit of height accuracy.

Gordon Dressler
December 21, 2018 5:49 pm

“Jason-2 flies in a low-Earth orbit at an altitude of 1336 km. With global coverage between 66°N and 66°S latitude and a 10-day repeat of the ground track, Jason maps 95% of the world’s ice-free oceans every ten days. Sea surface height accuracy is currently 3.4 centimetres, with 2.5 expected in the future.” — source: https://www.eumetsat.int/jason/print.htm#page_1.3.0

The “Inside the Acceleration Factory“article linked in the above article’s first sentence states that C&W data analysis gives a SLR slope of 2.1 +/- 0.5 mm per year for the last 20 years using a large amount of data from one or more spacecraft instruments (presumably the Poseidon-3 dual frequency altimeter that is on Jason-2 or something with similar accuracy), having at best a 25 mm accuracy.

As to how anyone can assert that satellite radio altimetry (independent of GPS use) above oceans is accurate to +/- 1 mm or better . . . go figure.

Frank
December 22, 2018 1:30 am

Richard Greene:

If satellites weren’t capable of measuring
sea level to the nearest millimeter,
or even to the nearest centimeter,
Why did they build and deploy them.

Even though the oceans have tides.
And waves.
The area struck by the radar beam
is many waves wide.
It can’t see them.

Averaging enough points
can handle random noise.
It is systematic error
we must fear.

The GPS in your cell phone
will soon be accurate to 1 cm.

The first Grace satellites
measured their separation to 1 um
over 220 kilometers with microwaves
The second generations with visible lasers
should be much more accurate.

LIGO detected gravity waves
by detecting motions of
1/10,000 the width of a nucleus.
1/1,000 wasn’t good enough.

You are talking out of your hat.
Now eat it. (It doesn’t pass the smell test.)

For those who don’t know,
the satellites are being calibrated
by measuring the distance down
to sites with see level known by GPS.

Using one set of sites for calibration
and a second set for validation
might work.

The potential for systematic error
is high.

Tarquin Wombat-Carruthers
December 24, 2018 12:55 pm

I take real-world measurements at my holiday cottage, the address of which is:

1 Derwater Street
Tipping Point
Maldive Islands

Merry Christmas to all! Glub glub glu…..

unka
December 23, 2018 1:55 am

“beyond a certain limit” – What does define that “certain limit”? One can imagine many experiments where you can get very high precision with very crude tools. For instance think of Buffon’s needle to determine many digits of PI. It is basically a binary precision that leads to multi digit precision.

Reg Nelson
December 20, 2018 3:58 pm

Of course there were none of the subdivisions of a degree in 1850 when humans were visually eye-balling liquid mercury thermometers in a few sparse locations.

So to pretend that we know the “global temperature” was in 1850 to a tenth, or a hundredth of degree, is completely absurd, dishonest and completely unscientific.

MarkW
December 20, 2018 5:46 pm

Not to mention the number of sensors was 2 to 3 orders of magnitude too few to even begin contemplating such a thing, even if they were accurate to 0.001C.

Phil R
December 20, 2018 7:15 pm

ChrisB,

Not sure, even with my glasses, if I could differentiate the markings of the scale were “subdivide the scale to a hundred, thousand or even a million markings.”

“In theory there is no difference between theory and practice; in practice there is.”

No one could read a scale with the divisions you suggest. And you mention a confusion about accuracy and precision. your divisions could be precise, but not accurate.

ChrisB
December 20, 2018 7:59 pm

PhilR: “No one could read a scale with the divisions you suggest. ”

There is a tool called electron microscope. With it one can measure a distance of 43 picometers. That is almost a million million times smaller than a meter. https://www.youtube.com/watch?v=eSKTFXv5rdI.

As for the discussion about precision and accuracy please google it. You’ll be surprised what is out there.

A C Osborn
December 21, 2018 2:29 am

So what, you do not use an electron microscope to measure Sea Level or temperatures, in fact for anything “large” either.
So how accurate or precise is an electron microscope at measuring something 1 metre across?

LdB
December 20, 2018 7:21 pm

Willis article refers to the practical limits of measurement accuracy, plank distance is a theoretical measurement accuracy of distance which you will never achieve.

In the Willis article the ruler itself has practical limits of measurement accuracy, it for example expands and contracts with temperature as does the Visa card. So no what number you finally agree on it is only representative of a certain temperature. We have all made an assumption the ruler which was probably printed in China is actually the right scale which is why most countries have standards bodies to oversee devices that measure things.

So there are practical limits measurement accuracy of any measurement equipment and no averaging does not improve that limit because the errors lie outside the measurement distribution.

A C Osborn
December 21, 2018 2:32 am

The point about a ruler and Eye Sight is that you are measuring to 1mm and guessing or estimating anything less.

Analog Design Engineer
December 21, 2018 5:54 am

It is you who is mixing up terminology Chris. Subdividing the scale will increase resolution, not precision. Precision relates to repeated measurements. A meter could have excellent resolution (resolution is the smallest signal change that it can detect) and poor precision, although in my experience meters with excellent resolution tend to have excellent precision. The word precision is not recommended for use anymore because so many people mix up what it means. Instead the term Repeatability is recommended.

I recommend the ISO Guide to the Uncertainty of Measurement to everyone interested in this topic. An excellent guide to it with loads of examples is the SAC Technical Reference 1.

Gordon Dressler
December 22, 2018 7:45 am

“Precision relates to repeated measurements.”

I disagree. Let’s say I measure a coin with digital calipers and record its value as 20.2 mm, which may be all I need to assist in verifying its authenticity. But in reality, I could record the full readout of 20.1825 mm displayed on the caliper’s digital scale. The first numerical value is less precise than the second numerical value but both represent the exact same measurement.

I don’t need any repeat measurement to establish a given level of precision.

And the above measurement scenario tells you absolutely nothing about how accurate those numerical values are unless you know the digital caliper has been recently calibrated (or you yourself just did so) against a known standard at some time before or after that measurement.

Analog design Engineer
December 24, 2018 1:03 am

“When values obtained by repeat measurements of a particular quantity exhibit
little variability, we say that those values are precise” Les Kirkup & Bob Frenkel, ‘An Introduction to Uncertainty in Measurement’, 2006 page 33 section 3.1.9.

December 20, 2018 2:21 pm

The old ‘the errors cancel out’ anti-argument. I bet those that claim that would believe the same even if guessing the future from goat entrails. You just need many goats for accurately guessing the future 🙂

The same line of thinking goes for averaging wrong results from computer models. Why don’t they just pick results at random then average them, if they think the errors will cancel out no matter what?

On a less funny note, the central limit theorem has its limits.

Lonny Eachus
December 20, 2018 2:24 pm

Even then, as Kip Hansen pointed out here, MOST of the time, you aren’t going to get ANY reduction in uncertainty:

https://wattsupwiththat.com/2017/10/14/durable-original-measurement-uncertainty/

It only occurs under certain strict conditions, which in real-world climate data are very seldom present.

December 20, 2018 2:29 pm

Excellent explanation.
Those that have used Slide Rules easily understand this. A simple way of realizing / demonstrating this is to perform a moderately complex calculation on a slide rule (one with more than three steps) as you normally would. Write down the answer. Next, perform the first step of the calculation, write down the result of that step. Slide the slider back and forth, then set the slide on the result. Do the same for the rest of the steps. Each time reading, writing down the result of the intermediate step, sliding the slide back and forth then resetting to thath intermediate result to obtain the next result. Even with problems that only have two or three intermediate steps, the final result is drastically different. An clear example of why this happens is when you divide the cercomperference by the radius, the index is EXACTLY on PI. If your slide rule does not have a clear mark for PI you will never put the slide in the correct spot.

Also have problems over the fact that the satellite is moving, thus measuring a different spot on the ocean in a different portion of the Swell, a different wave height, and even a different ocean level, through a different wavy atmosphere.

Phil R
December 20, 2018 7:20 pm

Only my dad could use a slide rule. Messed with them a bunch when I was a kid, but could never figure how it worked.

Richard Patton
December 20, 2018 8:21 pm

I still have one NEW, in its box! My son, who is in the Oregon National Guard discovered them throwing one out and brought it home to me. I still remember how to multiply, divide and do logs but not much more. I will keep it, one of these years it will probably be worth a bunch of money.

John F. Hultquist
December 20, 2018 9:49 pm

There is a web site for that.
Look it up.
I have a yellow metal Pickett Dual Base ~1961
Yours should have the name of the maker, copyright year, & model #.
Slide rules as a topic cycle through about every 2 years.
Last time, I spent an hour looking at sites.
Give it a try.

Editor
December 20, 2018 11:14 pm

I still have the slide rules I used in the first two years of mechanical engineering, nuclear physics, statics, dynamics, and reactor design. I competed in slide rule and mathematics in high school though, so they are well-used. defintely “not in the box” shape!

December 21, 2018 1:06 am

And I have a Carmody Navigational Slide Rule. Never actually got past multiplication and division though, just kept using Norries Tables.
But at Nautical College before my Mate’s exam we had a lecturer who used one all the time. We set him up. During an exercise someone looked up from his paper and asked as if he had forgotten, “What is the square root of four?” Out came the slide rule and the answer “two nearly” had the class in stitches. The rule didn’t get used nearly as much after that.

Phil Rae
December 21, 2018 1:15 am

Usurbrain

My slide rule had PI clearly marked but it only showed up when the circumference was divided by the diameter or twice the radius! : ]

December 21, 2018 3:49 pm

MarkW
December 20, 2018 2:32 pm

To make this more like climate science. Let’s make that measuring 1000 different credit cards using 1000 different rulers by 1000 different people.

Would these measurements increase the over all accuracy at all?

MarkW
December 20, 2018 3:04 pm

Unlike thermometers, the readings from rulers won’t drift over time if they aren’t re-calibrated regularly.

Paul Blase
December 20, 2018 3:41 pm

Depends. What’s the ambient temperature and the coefficient of expansion of the ruler?

Rocketscientist
December 20, 2018 5:24 pm

That all depends on the ruler material’s sensitivity to environmental conditions.
Wood: heat and humidity
Plastic: heat and humidity
metal: heat
laser: everything

LdB
December 20, 2018 7:23 pm

They will drift because they expand and contract with temperature and many will be printed in China, with a that is close enough attitude 🙂

Louis Joseph Hooffstetter
December 20, 2018 4:44 pm

“To make this more like climate science, let’s make that measuring 1000 different credit cards using 1000 different rulers by 1000 different people. Would these measurements increase the overall accuracy at all?”

Only after the data was properly adjusted of course…

Menicholas
December 20, 2018 5:20 pm

Mark W,
Exactly!

Clyde Spencer
December 20, 2018 8:56 pm

MarkW
It is worse than you suggest. The credit card doesn’t change length over time, whereas the temperature is never the same even in the same place. Thus, the randomness of the measuring error doesn’t come into play.

Menicholas
December 20, 2018 9:09 pm

I think weighing all the cats in the world to determine if they are fattening up at an accelerating rate, may be a robust analogy.

Wayne Raymond
December 20, 2018 2:36 pm

Sooo, with respect to satellite sea level measurements, what uncertainty can be expected?

Also, I am curious why the rate of sea level change as measured by satellite seems always to be a factor of 2 higher than that measured by tide gauges. Can it be a reflection effect, like an echo, where the change in distance that the satellite measures is twice the change in sea level?

davidmhoffer
December 20, 2018 2:57 pm

There was a thread a while back showing that satellites are measuring different rates in different areas, I dimly recall noting that they in general showed higher levels of rise in the middle of the oceans than they did close to shore. I can think of a few ways that this might actually be physically possible given all the cycles at play and the short observation time we have. But let’s put that aside for the moment and assume it is true.

If the processes at play express themselves first (or more at this point in time) in the deep ocean than they do close to shore, then tide gauges, which by definition are located at the shore, would show less rise than would the satellites which are looking at a lot more area.

Latitude
December 20, 2018 3:49 pm

…Indonesia has had a positive anomaly for over 30 years
It now looks like this…

Louis Joseph Hooffstetter
December 20, 2018 4:47 pm

Yay!

Menicholas
December 20, 2018 5:25 pm

Waves are big in the open ocean, and are very chaotic.
Tide gages, I do think, are in protected harbors and such, and in laces where the sea is not as rough.
Measuring the height of a kid that is holding at least somewhat still is easier than one who is running around a schoolyard while you are measuring her.

Clyde Spencer
December 20, 2018 9:15 pm

Menicholas,
Not only are tide gauges often in harbors, but they are intentionally designed to dampen the effects of waves, and primarily be sensitive to the low-frequency tides. There is nothing in the open ocean to dampen the waves of any frequency.

Reg Nelson
December 20, 2018 4:16 pm

Originally the satellite showed a decline in sea levels. And, as they did with the ARGO data, they adjusted it to fit their preconceived ideas — the adjustment is called GIA and based on the belief that tectonic activity was hiding the “real” sea level rise.

This of course begs the question that if the sea isn’t rising where we live and where the tide gauges, why should we care.

Don K
December 20, 2018 5:25 pm

Well, yes, the GIA adjustment is clearly bogus and has to be removed to get Eustatic (apparent) sea level. But it’s only about 12% (roughly 0.35 mm/yr). The rest of the discrepancy between tidal gauges and satellites is a mystery.

Menicholas
December 20, 2018 6:59 pm

When the original readings did not agree with what they were expecting, they went in and looked at everything very carefully and identified some things that were out of whack or improperly calibrated, or maybe they rewrote the algorithm, or all of the above.
Then they got a result which not only showed what they wanted to find, but found that it was worse than we thought!
Jackpot!
They then stopped looking for things that might be out of whack or miscalibrated and stopped looking for better algorithms.
Just a hypothetical, but this is how confirmation bias, and climate science, works.

LdB
December 20, 2018 7:29 pm

From a satellite sea level measurement perspective the biggest problem is waves. You are trying to measure the surface of something has these lumpy things on. Those waves by definition distort the very surface you are trying to measure locally, the more of them the more they distort.

Patrick
December 20, 2018 2:38 pm

Many years ago, on my first day in CHEM 101 class, the instructor told us we would have points deducted for any lab or test result where we express an answer to a calculation with a higher degree of accuracy than the least accurate measurement that was used in the calculation.

This story makes me think of the announcements saying the current year is the hottest ever because the average temperature is 0.00023 degrees higher. If the weather stations providing these measurements have an accuracy of only 0.05 degrees, someone is full of bull waste products.

Do we need to send these people back to undergrad classes to remind them of the basic rules of math and science?

Louis Joseph Hooffstetter
December 20, 2018 5:05 pm

“Do we need to send these people back to undergrad classes to remind them of the basic rules of math and science?”

That would be a waste of time and money, as witch doctors don’t follow the basic rules of math and science. But we do need to keep reminding readers that the reason ‘climate scientists’ don’t follow basic freshmen chemistry science rules is because they’re witch doctors, and NOT scientists!

Tom Halla
December 20, 2018 2:45 pm

As applied to climate studies, a better analogy would be a large group of riflemen, shooting at different targets. All the riflemen have varying skills, and all the rifles/ammunition have differing inherent accuracies.
So would combining the results give any better understanding of the odds of any one target being hit?

Steve Reddish
December 20, 2018 2:46 pm

Do those satellites used for sea level measurements ever measure something that is at a known level, just for a reality check?
Every time a satellite passes over a harbor, why not take a reading, at the same degree of accuracy, at that moment?

SR

December 20, 2018 3:44 pm

Wouldn’t that mean the lake level, used for calibration, is dependent on the tide gauge base data for the lake?

Steve Reddish
December 20, 2018 4:33 pm

I wonder if the satellites say those lake levels are accelerating?

SR

Steve Reddish
December 20, 2018 4:40 pm

Wouldn’t that also mean the satellite reading would necessarily be constrained to the same degree of accuracy as can be had from the lake’s gauge?

SR

Analog Design Engineer
December 21, 2018 6:36 am

If it is then the uncertainty of the tide gauge must be included as part of the uncertainty calculation for the satellite measurement. Thus the satellite measurement uncertainty figure must be higher than lake tide gauge uncertainty figure.

Rocketscientist
December 20, 2018 5:44 pm

If they are using lake surfaces for reference, how do they know what the level of the lake is? Prior to GPS this was all done with eyeballs and transits. And even GPS has a prolate elipsoidal error envelope. It has about 6 meter radial horizontal accuracy and 10-20 meters vertical accuracy.
That’s one reason inexpensive drones won’t be making deliveries any time soon.
https://www.gps.gov/systems/gps/performance/accuracy/

Don K
December 20, 2018 6:06 pm

Large lakes often have their own “tide gauges”. Here’s a link to a web site for the Lake Champlain site at Rouses Point New York. https://water.weather.gov/ahps2/hydrograph.php?wfo=btv&gage=roun6
And another for a site at Burlington, VT https://nh.water.usgs.gov/echo_gage/measurements.htm

There would be all sorts of problems with trying to use these as calibration points. All the problems of tide gauges less tides themselves but plus changes in lake level due to precipitation, evaporation, river inflow and outflow. Plus which, the level of any point on the lake varies with which way the wind is/has been blowing.

I honestly don’t know how/if satellite RA’s are calibrated against surface targets. There are many papers on the subject and they concur that it’s a tough problem at the accuracy levels required. My vague impression is that they are checked periodically to make sure they are drifting off into fantasy land, but that there’s no overt correction for RA calibration error.

I’ll add it to my lengthy list of things to look into.

Ed MacAulay
December 21, 2018 5:05 am

Had some surveying done for property values this Fall. He just used a GPS pole for legal surveys. Next year use your smartphone!

“Smartphones’ GPS systems are going to start getting a lot more accurate. According to IEEE Spectrum, Broadcom is starting to make a mass-market GPS chip that can pinpoint a device’s accuracy to within 30 centimeters, or just under one foot. That’s compared to today’s GPS solutions, which typically have a range of three to five meters, or up to 16 feet away.”

LdB
December 20, 2018 7:43 pm

They are specific sites around the globe with there own floating laser reflective target which has GPS surface fixes.

I know Lake Issykkul (Kyrgyzstan) has one, Jason 1 and 2 had a floating target in Bass Straight off Tasmania in Australia which I assume Jason 3 also uses.

Jason 3 is now flying the same orbital path Jason 2 used to so I imagine it uses the same calibration sites.

Editor
December 20, 2018 11:21 pm

So the algea and seaweed and dirt growing underneath and top of the calibration site bouys didn’t change the “elevation” of the reflectors over those many years in the middle of Russia?

LdB
December 21, 2018 2:27 am

The buoy is GPS positioned, what you are saying makes no sense????

All that matters is the buoy is floating on the water surface are you saying it isn’t?

Editor
December 21, 2018 6:18 am

The buoy is floating on a water surface. The reflectors on the top of the buoy are held above the water level, obviously tilting as the bouy jerks and moves on its anchor chain above the bottom as random waves go by, as wind tilts the buoy slightly.

As the buoy gets physically heavier (as with every marine object with a waterline and anchor chain supporting growing biologics and dirt and scum), the buoy sinks down. This requires daily cleaning all over the surface, if you’re going to claim a “sub-millimeter” calculation accuracy in the RESULT of the “measurements” made from your instrument that is calibrated from a moving irregular source. A laser reflector on the moon, on the other hand, IS capable of giving higher accuracy because the only thing moving it are “moonquakes” and aliens.

LdB
December 21, 2018 4:31 am

I could add imagine the lake got lifted or dropped 100m in an earthquake. The GPS reading on the floating buoy measures the new position. Jason 3 sees the new height of the lake and it should match the buoy position from it’s own GPS as up/down 100m and so we know the satellite is in calibration. The only way it can fail is if the buoy isn’t floating.

LdB
December 21, 2018 8:18 am

As for accuracy Jason 3 clearly states it’s accuracy as 2.5 cm RMS so I don’t know where you get sub millimeter accuracy. Climate Science models things from the data but that ahs nothing to do directly with the Jason 3.

Editor
December 21, 2018 6:34 pm

LdB

No, earthquakes move the surface anywhere from 2 cm (mag 4, 5 or 6) to 2-3 meters sideways (mag 7.5+) and “maybe” 1/2 cm to 1-2 meters vertically in limited areas around the quake. 100 meters lake surface movement? No.

Look at the fence broken in the 1903 San Francisco quake, the road busted a few days ago in Alaska. Underwater, even Japan 8+ quake disturbed the sea floor “only” a few meters over a long line.

Well, you see, regardless of what the satellite accuracy is, the result of the satellite readings is a sea level change “accelerating” from 2.1 mm/year to 2.5 mm/year, with a ‘trend line” being analyzed to 3 and 4 decimal places to find anything but a linear (or steady) trend. But, you see, an “acceleration” had to be found!

LdB
December 22, 2018 4:37 am

The sea level measurement and acceleration you are talking about is a different thing it has nothing to do with the instrument doing the measurement., that belongs to climate scientists.

At the moment your criticisms are all over the place like a rabid dog but you clearly don’t know how it all works. I suggest you do some reading so you can voice whatever complaint you want to make.

steve case
December 20, 2018 2:50 pm

One of my favorites the first sentence in Chapter five of the IPCC’s 4th assessment report:

Climate Change 2007: Working Group I: The Physical Science Basis *
The oceans are warming. Over the period 1961 to 2003, global ocean temperature has risen by 0.10°C from the surface to a depth of 700 m

Really they can measure the global ocean to with 0.10°C Is that 2nd zero after the decimal point really significant? Let’s see, 0.10 / (2003-1961) = 0.002 – They must have some very accurate data.

* That’s a WayBack Machine link. It seems that the people at the IPCC have recently decided to make it difficult to navigate their website. Could that possibly be by design?

Darrin
December 20, 2018 2:54 pm

“Regardless of the number of measurements, you can’t squeeze more than one additional decimal out of an average of real-world observations.” I was taught that you can’t get more accurate than the least accurate number used when averaging but I can see why you might push it out one decimal place. My caveat to that is while I’ve always enjoyed math I stayed away from statistics classes.

Gunga Din
December 20, 2018 4:14 pm

Significant figures in climate science.
(To quote someone else about something else)
“They don’t know what it is, they just know what it’s called.”

Another Ian
December 20, 2018 4:47 pm

“Significant figures in climate science.”

That’s the ilk of Mann et al isn’t it? Nothing to do with real numbers.

Alan Watt, Climate Denialist Level 7
December 20, 2018 2:58 pm

And to top in all off, the “sea level” that the satellites measure is not the “sea level” that human civilization needs to care about. I hate to claim it’s not possible, but I doubt even the most sophisticated deep ocean creature will notice if the mid-Pacific ocean depth changes by a meter or two. But in any case, we don’t. We care about sea level relative to shorelines where there is significant population and fixed infrastructure.

And in those cases, if sea level rise is a problem, it’s most likely due more to land subsidence than actual change in the ocean level.

The satellite data is interesting and no doubt useful for some things, but it isn’t what we should pay attention to.

bonbon
Reply to  Alan Watt, Climate Denialist Level 7
December 21, 2018 5:58 am

There is one case where open ocean measurement is critical – tsunami warning. They start as harmless swell, until they arrive at come coastline. A lot of talk about buoy sensors went nowhere afaik after the big one in Indonesia. And there were strong rumors a deep sea ocean creature, a US Ohio Class boat was severely damaged, rolled like a toy, and had to limp home.
Some of these are caused by major earthquakes, which it so happens also have warning signatures with ionospheric effects which ironically could in theory be monitored with the same satellite radar system (if subtle effects are not averaged out). The “fukoshima” earthquake precursors were in fact measured – resistance to this earthquake precursor “smell-test” blindsides otherwise capable scientists, and gets a lot of people killed.

Gary Pearse
December 20, 2018 2:59 pm

An excellent common sense rule, Willis, thank you. Regarding sea level measurements back when estimates for 2100 were a rise of twelve feet plus (Hansen?) I argued that if that is the worry, we needn’t run down to the sea with a micrometer, a yardstick will do and if worse, then axe handles are sufficient for the job.

Also, for global av. temperature rise of 6C which seemed to be the worry, a dozen thermometers, scattered across the Arctic, where with enhancement we would have 2 or 3 times this temperature, would be a perfectly adequate warning system. Had they done this in 1988, we would know before now that another degree at most is really all we are in for.

It would be instructive to know what all the T adjustments have done to uncertainty of measurement. Indeed the algorithms are changing previous temperatures as we speak. As Mark Steyn noted at a senate hearing on data quality, ‘we don’t know what the temperature in1950 will be by 2100 and yet certainty is high about 2100’s temperature’.

December 20, 2018 4:37 pm

Even today 1950 is a lot colder than it was in 1950. Proof positive of global warming. The further we go into the future the colder the past becomes. Entropy in action. Eventually 1950 will be the date of the big bang.

Latitude
December 20, 2018 4:57 pm

“Even today 1950 is a lot colder than it was in 1950.”….LOL and true!

steve case
December 20, 2018 5:25 pm

From my file of tag lines and smart remarks:

Climatologists don’t know what the temperature is today within ±0.5°C, and don’t know what it was 100 years ago, but they know it’s 0.8°C ±0.1°C hotter now than it was then.

MarkW
December 20, 2018 5:50 pm

CO2, is a temporal gas, it steals heat from the past and moves it to the future.

Rod Evans
December 21, 2018 12:54 am

That deserves a house point, or two. Brilliant.
I am off to calibrate my hockey stick, I use it to measure climate change, I sometimes need to re-calibrate it, especially when historic matches have to be replayed.

steve case
December 21, 2018 6:50 am

CO2, is a temporal gas, it steals heat from the past and moves it to the future.

I’ve added that one to my smart remarks and tag line file.

December 20, 2018 4:37 pm

Even today 1950 is a lot colder than it was in 1950. Proof positive of global warming. The further we go into the future the colder the past becomes. Entropy in action. Eventually 1950 will be the date of the big bang.

Menicholas
December 20, 2018 9:12 pm

Yogi Berra would have been a climate scientist if he had been born 100 year later than he was.
“It gets late early out here”, “But the towels at the hotels are the best…I could barely close my suitcase”.

n.n
December 20, 2018 3:03 pm

Accuracy depends on resolution. Uncertainty depends on distribution relative to the true statistic (so-called “normal”). The average temperature is highly time and space dependent, with a potentially large spread in even a narrow frame of reference.

Louis Joseph Hooffstetter
December 20, 2018 5:39 pm

Mosher, is that you?

Astrocyte
December 20, 2018 3:04 pm

Averaging measurements WILL increase resolution for a discrete sampled signal as long there exist an appropriate random dither noise.

See the excellent paper by Walt Kester: ADC Input Noise: The Good, The Bad, and The Ugly­. Is No Noise Good Noise? Published in Analog Dialog 40-02, Feb 2006

Frank
December 20, 2018 3:14 pm

Willis wrote: “Regardless of the number of measurements, you can’t squeeze more than one additional decimal out of an average of real-world observations.”

That depends on the what type of error you are dealing with. Using EXCEL or some other software, generate 10,000 numbers with a mean of 15 (deg C) and a standard deviation of 1 (degC). You will have more than 10 digits to the right of the datapoint for each value using an add-on for EXCEL. Calculate the mean and standard deviation of the 10,000 number you actually received with all of the decimal points. Make another column of data and round your values to the nearest 0.1. Take the mean and standard deviation of the the first 100 values of rounded data and separately for 10000 values of rounded data. Do you get closer to the “right” answer using 100 rounded values or all 10,000?

If you have random noise in your data, averaging a lot of values can give you a more accurate mean. If you have systematic errors in your data, your mean won’t get closer to the right answer. For example, it you fail to look straight down and always look at a slight angle, parallax could make all of you measurements systematically bigger or smaller than the should be. If you sometimes look slightly from the left and equally likely slightly from the right, your measurements will have random noise in them, averaging can help.

IMO, because there are so many such large adjustments in converting time for the radar signal to return into a distance, the possibility of systematic error is the biggest problem. One large adjustment is from humidity and it involves meters IIRC. Humidity data is determined from re-analysis. Re-analysis is based on a set of inputs that changes over time. A gradually-changing small bias in humidity of 0.1%/yr appears capable of biasing the trend by 1 mm/yr. IIRC, there have been three major corrections of systematic errors in calculating SLR from satellite altimetry data.

Nylo
December 22, 2018 2:52 am

Exactly. What Willis is saying is basically that in real world you can never count on having completely eliminated any possible systematic errors in measurements. That only happens in the world of mathematics. Considering that less than 10% of the error of any measurement is systematic is a wrong assumption.

SMC
December 20, 2018 3:19 pm

I’m confused. The commenter said,”…resolve over a very large number of measurements to a constant bias…”, which strikes me as the gist of the comment. I don’t think he was commenting on the accuracy of the measurement but on the precision of the measurement. If the measurements are as precise as he seems to imply then the inaccuracy should be consistent. You can usually correct for inaccuracy if your measuring device is precise.

Robert of Ottawa
December 20, 2018 3:22 pm

This seems to be like what I call The False Precision falicy. A true measure of accuracy (not precicision) would be sigma/mean but of course we cannot measure the mean accurately, which is why we are resorting to statistics in the first place.

Let’s generate a random number between 1 and 100, a million times. The mean would be 50, every time, whether we measured it to 1 significant digit or 100.

MarkW
December 20, 2018 5:52 pm

Even the best random number generators are still just pseudo random number generators.

michael hart
December 20, 2018 3:35 pm

The length of the credit card is at least assumed to remain effectively constant. That is not the case for either temperature measurements or those of sea-surface height. They are both dynamic.

Menicholas
December 20, 2018 7:18 pm

Bingo!

David Stienmier
December 20, 2018 3:39 pm

Willis,

I think you are confusing resolution (1mm divisions on the ruler) with precision (how repeatable are the measurements). Also you are using an example whose precision/repeatability is smaller than it’s resolution (everyone measures between 85 and 86)

I think if you redo your analysis with something that changes more than the resolution of your measurement you will come up with different conclusion.

For example, what if I wanted to estimate the average length of pencil being used by primary grade students in a district with 100,000 students by asking them to measure their pencils and reporting to their teacher to the nearest mm. And lets say students will have up to a quarter mm error (they might round x.25 mm up or x.75 mm down but not worse).

In this example sampling more students will still improve accuracy (assuming zero bias) because the variation in individual pencils will be much greater than measurement precision even if the average over the district changes very little. With enough students I can still get better than 0.1 mm accuracy (relative to the actual mean) even with measurements rounded to the nearest mm.

David

JFD
December 20, 2018 3:44 pm

I was not in the original discussion but measuring ocean conditions such as wave heights can be misleading. I was on the deck of very large derrick barge one time off the shore of the North Island in New Zealand trying to get a fixed production platform piled to the ocean floor. The waves did not seem too high to me compared to other locations around the world. As I was pondering what to do, I noticed the wave pole on the fixed platform. The swell as 19 feet. I then timed the swell period and it was 20 seconds. I then thought about the 4000 mile fetch of open water and understood why one could not see the wave/swell height changing very much.

With the satellite moving so fast I wonder how the surface measurement is corrected for the wave heights, periods and fetches? There would be a difference between the middle of an ocean and the near shore area. As an old sea salt, Willis probably understand this aspect well.

raygun
December 20, 2018 3:48 pm

This is why we have tolerances on drawings/tooling. To elevate guess work and scrap.
In my position as a mechanical engineer for over 50 years, I had the distention of being the evil one for rejecting mechanical parts to unusually tight toleranced parts per the drawing. Many times the problem was elevated by loosening the tolerance where allowed with an Engineering Change order agreed by all involved. No need to use a micrometer, let’s say, on a credit card. Or a yard stick on a precision part.

Regards, retired mechanical engineer

December 20, 2018 3:49 pm

The issue is with the absolute uncertainty of each measurement. If this error is consistently biased in one direction and/or not random, then the error will not cancel and the uncertainty of the average converges to the bias. However; if the error is normally distributed around actual values, then the precision of the average will continue to increase as the number of samples increases.

December 20, 2018 8:33 pm

Willis,

Yes, a ruler can’t be used as a micrometer by measuring one distance over and over and taking an average, but that’s not the case with satellite based sea level measurements.

In the first case, the same instrument will always result in the same measurement of the same thing, so there’s no variability to average out. However; if you add stochastic noise to the measurement whose variability is more than the precision of the measuring device and centered around the actual distance, then there is variability to average out and the precision will continue to get better as more measurements are made.

In the second case, the same instrument makes many measurements of different distances at different places at different times and never measures the same thing twice. To the extent that the data is normally distributed around the steps of the measurements, the precision of the average will continue to get better as more measurements are made.

Menicholas
December 21, 2018 4:48 am

No one seems to have used the term “measurement resolution” in this discussion, but it is distinct from both accuracy and precision:

“In addition to accuracy and precision, measurements may also have a measurement resolution, which is the smallest change in the underlying physical quantity that produces a response in the measurement.”

https://en.wikipedia.org/wiki/Accuracy_and_precision

Max Dupilka
December 20, 2018 4:01 pm

Here is a good explanation of the “uncertainty of a mean”.

http://bulldog2.redlands.edu/fac/eric_hill/Phys233/Lab/LabRefCh6%20Uncertainty%20of%20Mean.pdf

As is stated in their text,

“the estimate of the uncertainty of the mean given by equation 6.1 has two
properties that we know (or at least intuitively expect) to be true. First, it implies that the
uncertainty of the mean is indeed smaller than the uncertainty of a single measurement (by the
factor 1/ sqrt(N)), as we would expect from the argument given in the previous section. Second, the
more measurements we take, the smaller Um (uncertainty of the mean) becomes, implying that the mean becomes a better estimate of the measurement’s true value as the number of measurements it embraces increases”

December 21, 2018 7:45 am

Max,

“Second, the more measurements we take, the smaller Um (uncertainty of the mean) becomes, implying that the mean becomes a better estimate of the measurement’s true value as the number of measurements it embraces increases”

This statement is incorrect. That should read ‘measurement’s true correspondence to the instrument’s indication.’ The point being is the averaging merely provides a more PRECISE instrument indication. The averaging does not address the underlying ACCURACY of the instrument itself.

The ACCURACY of an instrument is a finite physical characteristic. It is a function of its basic design limitations. Those limitations are typically described in its specifications and or in its metrology lab calibration statements. It is an error to assume an instrument’s ACCURACY can be improved by averaging any measurements done with it. The best that can be done, given reasonably random scattering of measured values, is produce a value that more PRECISELY represents the instrument’s measurement indication. ACCURACY is still no better than what is shown on the instrument’s calibration sticker. Also, keep in mind that no claim has ever been made by instrument manufacturer’s or cal labs that a calibration errors are randomly scattered within that range specified on the sticker.

An argument has frequently been made that absolute accuracy of the measuring instrument is not important when measuring trends. Assuming that measured values collected at different times with the same instrument will each have the same error relative to the true physical value is also false. Instrument calibration drift’s with time and not necessarily in a constant direction or rate. Some instrument types are better than others in this regard. There are also issues with hysteresis, mechanical friction, thermal history, instrument contact with measured process, etc.

The concept is simple, never claim more accuracy than the instrument specification. On top of that, you must add any degradation in that accuracy added by how the instrument is connected to what it is supposed to be measuring. Averaging measured values does not reduce those errors.

Max Dupilka
December 21, 2018 9:51 am

It has nothing to do with instrument accuracy. It is just fundamental statistics of sampling. You can choose not to believe statistic if you wish.

December 21, 2018 12:15 pm

Interesting response. You say instrument accuracy has no bearing on data collected with that instrument? Very accurate, sorta accurate, and barely working instruments are all the same for collecting data?

I definitely believe statistics when used appropriately. The trick is understanding what the result of a statistical manipulation is. Just because a particular statistical algorithm can be run does not mean its results apply to the desired end. A simple example is calculating the average life span of humans and then letting your kids play in the middle of a busy street cuz statistics say they will live decades longer. Calculating life average span is valid. Assuming it means very much for a particular individual is not valid. The same principle applies to this discussion.

Max Dupilka
December 21, 2018 1:08 pm

Maybe I don’t explain it too well so I will give it one more shot. You can never make one measurement more accurately than your instrument will allow. But, if the errors are randomly distributed around the actual value then the accuracy of the measuring device is not critical when you take a large number of measurements. However, it is very likely that the more accurate instrument also has the more likelihood of the error being in fact random, while a less accurate instrument will likely have a bias in the errors.

Maybe this equation will help, or at least help me.
We can write a measured value as

U = Ut + delta

where U is the value you measure, Ut is the true, real value of what you are measuring (which you don’t know) and delta is the error in measurement. If you take a whole bunch of N measurements and then take an average we get

sum(U)/N = sum(Ut + delta) / N = sum(Ut)/N + sum(delta)/N

Now if the errors are truly random around the real value

sum(delta) = 0

and since Ut is the one true value, sum(Ut) = N*Ut
and sum(U)/N is just the average of our measurements ie. Uavg

So with all that we can get that

Uavg = Ut for our N measurements

and voila, the average of our bunch of measurements will in fact converge to the actual value, as long as the errors are random.
We employ this method in boundary layer turbulence statistics.

Menicholas
December 23, 2018 12:51 am

Max, there is no reason to think that the errors are randomly distributed around the true value.
In fact, no one knows what the true value is, and besides the measurements are not of the same thing.
There is every reason to be sure that there is NO such known random distribution.
So everything you wrote after that is moot.
No one is declaring statistical mathematics to be invalid when used correctly.
But you seem to be saying that we can ignore the rules about when the LLN applies, and when it does not.

Jim Gorman
December 23, 2018 3:16 pm

Max –> Don’t forget we are not talking about sampling here. We are talking about one temperature measurement, taken one time, on one day.

Some folks want to say you can average daily temperatures, monthly temperatures, etc. and get better and better certainty of the mean. However, that just doesn’t work. It doesn’t help you determine what the real temperature was on any given day.

You only have to look at temperatures for two days. The first is 50 +- 0.5 and the second 51 +- 0.5. What is the average? It is 50.5 when not considering the possible values determined by the error. Does that mean each day was really 50.5, or just the first day, or just the second day? What was the average when you throw in the errors? What was the actual temperature for day 1, how about day 2?

JBom
December 20, 2018 4:02 pm

An unstated presumption, without supporting evidence, is that the standard deviation remains constant, i.e. no growth in time, i.e. no rate of change: False.

Ha ha.

Theyouk
December 20, 2018 4:09 pm

‘Certainty’ methods/discussions aside, color me skeptical (for now) on the accuracy of SLR derived from satellites. After all, look at the contested ‘accuracy’ around the height of a well-known and relatively stationary object that does not have ocean swells nor tides…Mt. Everest.
From https://www.britannica.com/place/Mount-Everest:
…A Chinese survey in 1975 obtained the figure of 29,029.24 feet (8,848.11 metres), and an Italian survey, using satellite surveying techniques, obtained a value of 29,108 feet (8,872 metres) in 1987, but questions arose about the methods used. In 1992 another Italian survey, using the Global Positioning System and laser measurement technology, yielded the figure 29,023 feet (8,846 metres) by subtracting from the measured height 6.5 feet (2 metres) of ice and snow on the summit, but the methodology used was again called into question.

In 1999 an American survey, sponsored by the (U.S.) National Geographic Society and others, took precise measurements using GPS equipment. Their finding of 29,035 feet (8,850 metres), plus or minus 6.5 feet (2 metres), was accepted by the society and by various specialists in the fields of geodesy science and cartography. The Chinese mounted another expedition in 2005 that utilized ice-penetrating radar in conjunction with GPS equipment. The result of this was what the Chinese called a “rock height” of 29,017.12 feet (8,844.43 metres), which, though widely reported in the media, was recognized only by China for the next several years. Nepal in particular disputed the Chinese figure, preferring what was termed the “snow height” of 29,028 feet. In April 2010 China and Nepal agreed to recognize the validity of both figures.

Use the long-term land-gauges that are on non-rising/non-subsiding rock for sea level measurement, I say. The rest is too close to nonsense.

nw sage
December 20, 2018 4:25 pm

“Following that rule of thumb, if you are measuring say temperatures to the nearest degree, no matter how many measurements you have, your average will be valid to the nearest tenth of a degree … but not to the nearest hundredth of a degree.”
The point that may slip by here is what is being measured: In your credit card example, you are measuring the credit card. When you switch to temperatures (which is often the case when talking about climate) you are not usually measuring temperature – per se, but the temperature OF something – air, water, etc. Your example falls apart when we realize that is nearly impossible [outside of a laboratory] to make very many temperature measurements of EXACTLY the same ‘piece’ of air under the same ambient conditions.
Conclusion: It is VERY difficult to even achieve the additional decimal point of better precision

MarkW
December 20, 2018 6:28 pm

Beyond the accuracy of the measurements themselves, there’s the difficulty of taking enough measurements so that you can claim to be taking a representative sampling of the entire earth.
Back in 1850, there was at most a couple of hundred sensors, mostly in western Europe and the eastern US.

The rest of the world was virtually unmeasured.
You might be able to say that we knew what the temperature of the eastern US and western Europe was, but to go from that to claiming that you could measure the temperature of the entire world was, much less to a tenth of a degree, is ridiculous.

December 21, 2018 10:13 am

nw sage,

Yes, there’s a big difference between measuring one invariant thing and calculating the average of many measurements of different things. In the first case, the precision and accuracy is limited to the measuring instrument and the average of multiple measurements will not improve this as the instrument will always return the same value. In the second case, the randomness of the data improves the precision and decreases the uncertainty in the average, but not for any one measurement. The result has no predictive power regarding what the next measurement will be, but accurately predicts what the average of many measurements will converge to.

December 20, 2018 4:37 pm

After reading all the comments it seems to me that the concept of uncertainty has a very uncertain meaning. Not even the mathematical average of all the various proposals seems to be certain.

Mike Smith
December 20, 2018 4:39 pm

If the raw data is poor, the average of the raw data is poor.

So this is really just another manifestation of the Garbage In, Garbage Out principle.

Don K
December 20, 2018 4:43 pm

OK Willis. So you don’t believe in statistics. That’s fine with me. There are surely practical issues with applying statistics although I’m not sure I follow your logic. And truly I don’t much care how many angels can dance on the head of a pin. I’ll leave it to Nick Stokes or his equivalent to argue with you about those details. I expect they are firing up their keyboards to discuss your disturbing lack of faith even as I type this.

But you’ve also missed an important point about satellite sea level data. It’s very noisy. You aren’t measuring a credit card, you are measuring an earthworm whose length changes constantly (within limits). And to some extent, your measuring tools are calibrated rubber bands whose lengths aren’t as constant as one might like.

How do you deal with that? You take a LOT of measurements and average. You don’t think that’s a valid procedure? Do you have a practical alternative?

If your point is that we really don’t know exactly how accurate the resulting averages are. I agree we don’t. Mostly because we don’t really know the distribution of the noise in the raw sea level data and probably don’t have all that good a grip on some of the calibration errors either

One further point. We’re mostly worried about Sea Level Change, not absolute Sea Level. How do we compute that? We subtract Sea Level estimates at some time T0 from some other time T1. That doesn’t do anything for random error since the values at T0 and T1 have uncertainties due to random error. The difference of two uncertain numbers is on average a somewhat more uncertain number. But some of our errors are probably always about the same magnitude and sign. Those are biases, not random errors. Biases largely go away when we subtract.

December 20, 2018 5:33 pm

“Some of our errors are probably always about the same magnitude and sign. Those are biases, not random errors. Biases largely go away when we subtract.” If we just could identify them ..

Phil
December 20, 2018 5:51 pm

Mr. Magoo looks at something. His eyesight is not good so it is fuzzy. With this logic all it takes is enough Mr. Magoos all looking at a fuzzy something to give you a crystal clear image of that thing. NOT!!!!

December 21, 2018 10:37 am

Modern image processing techniques would seem to contradict this. What you are talking about is something done routinely for imaging deep space. For example, the Hubble Deep Field.

pouncer
December 20, 2018 6:00 pm

Don K, you write “You aren’t measuring a credit card, you are measuring an earthworm whose length changes constantly (within limits). And to some extent, your measuring tools are calibrated rubber bands whose lengths aren’t as constant as one might like.”

And out of that process you believe statistics allows you to report an earthworm result with GREATER precision, accuracy, and to more significant figures than measuring a plastic card with a wooden ruler?

And out of your belief in statistics, what do you affirm or deny regarding the FRACTAL properties of the “edges” of either an earthworm or a credit card? Analogously, how long is the coastline? Is it conceivable in your estimation of Willis’s problem that the top length of the (not quite rectangular) credit card is different from the (nearly but not quite parallel) bottom edge, by some length within the uncertainty of the measurement process?

I don’t understand your claim with sufficient detail to determine whether or not it’s persuasive.

Don K
December 20, 2018 7:13 pm

And out of that process you believe statistics allows you to report an earthworm result with GREATER precision, accuracy, and to more significant figures than measuring a plastic card with a wooden ruler?

Actually, I’m not a true believer in statistics, because it depends on assumptions about the world that seem to me to be rarely met. But I think the Standard Error of the Mean which is I think what we are dealing with here might be one of the few things in the world that actually has a Gaussian distribution. If it actually is Gaussian, textbook statistics might just work. But that’s not what I’m claiming.

Anyway what we have here is more like good precision but lousy accuracy. We’re measuring Willis credit card with a really good micrometer. But the folks making the measurements don’t really know how to use it very well so we get substantially different measurements every time we try. Why do you believe that averaging won’t improve the results?

Fractals? Those are boundary thingees. Not very relevant I think (hope). The physical phenomenon we depend on is reflection of a microwave radio pulse from the sea surface and I think the ambiguity there is quite small. I believe it’s one of the least uncertain elements in the process. Trouble is that the sea surface is usually not very flat and the radio beam isn’t that narrow, so what the satellite “sees” is something difficult to describe or analyze.

Pouncer
December 20, 2018 7:28 pm

Don K you write: “We’re measuring Willis credit card with a really good micrometer.”

Well, not according to Willis!

HE writes: …”people … use the ruler … to measure the length of the credit card in millimeters.

Whoever “we” might be, “we” agree that “we” have lousy “accuracy”. But I guess my question to you is, do you believe increased (statistical) precision moves “our” actual understanding of a measurement from “lousy” to “good”? You correctly interpret me to believe that averaging doesn’t “improve” things. We have a measure in millimeters and I can’t see why YOU believe that CAN be improved. How do we communicate across that gap?

Clyde Spencer
December 20, 2018 9:23 pm

DonK
The surface of waves are “boundary thingees!”

Menicholas
December 23, 2018 2:02 am

Fractals?
Watch this if the subject interests you…or if it does not and you wonder what is up with that?

https://youtu.be/56gzV0od6DU

William C. Rostron
December 20, 2018 4:46 pm

I’m a little rusty on this, but I have done instrument measurement uncertainty calculations for NIST traceable Safety Related measurements, to assure that Nuclear Safety Limits are met. These aren’t your run of the mill guesswork kinds of things, but exist to show regulatory compliance under penalty of law.

Measurement Uncertainty relates to the conformance of actual process to the readout that represents that process. In other words, it’s how close the actual process is to what you say it is. In the case of temperature, it’s how close the temperature is to what the instrument readout says it is. The first time I did one of these calculations I was shocked at how uncertain things really are.

To calculate uncertainty, one must include everything that can possibly affect the reading: the analog sensor (including drift and calibration accuracy), digital conversion, readout precision, theoretical limits (that is, the degree to which a correlation formula actually represents the conversion process), and uncertainties in any physical correlations that can affect the fundamental relations. One must also add the uncertainty of the test equipment employed for calibration. Each of these terms (which is not an complete list) has various components that contribute to the uncertainty of each term. It is not uncommon for two dozen or more individual uncertainty terms to be included in a properly done uncertainty calculation.

There are two kinds of errors at play: random error and bias error. Random errors can be reduced (but not eliminated) by more measurement. But bias errors never go away and are NOT improved with more measurements. Total uncertainty is calculated by Square Root of Sum of Squares (SRSS) of all random errors linearly added to SRSS of all bias errors.

Once the uncertainty for a given measurement system is determined, the total uncertainty can be reduced only if there are multiple independent measurements; that is, only if there is more than one source for a given reading. One cannot improve uncertainty by making multiple measurements with the same device, although one can (in principle) improve precision with more measurements.

It is clear from simple algebra that the total uncertainty of a measurement is bounded by the worst term in the system. For example, if the sensor error is 1%, no amount of additional precision or accuracy in subsequent processing can make the total uncertainty better than 1%. Uncertainty is always worse than than the worst term, because SRSS can only produce positive numbers. Each additional term makes the uncertainty worse. Usually the total is dominated by a just a few big terms. Adding 0.01% accurate readout to the tail end of a 1% accurate sensor doesn’t make any real difference.

It’s kind of depressing to realize that you really don’t know what you think you know when you take a reading. In the Safety Related business, uncertainty calculations are vital to provide assurance that design limits are met and that plant operation, even during accidents, will not injure the public.

-BillR

Jim Gorman
December 20, 2018 4:55 pm

+100

Louis Joseph Hooffstetter
December 20, 2018 5:54 pm

Yeah!
What Jim Gorman said! (+100)

Menicholas
December 20, 2018 7:36 pm

Hallelujah!
Thank you William!

Jim Gorman
December 20, 2018 4:47 pm

May I point out that the term “the uncertainty of the mean” means what it says. The mean value of a number of measurements can be calculated very accurately.

It does not tell you if the individual measurements are either accurate or precise. For example, I can take a million measurements of the same credit card with a device that reads to 5 decimal places. I can calculate the uncertainty of the mean and get a very small number. Now what is the uncertainty of the mean if I tell you that the ruler is off by 5 mm? Does it change at all?

The physical measurement errors can not be removed through this method, especially if, as someone has already pointed out, you are measuring different things with different instruments . Several of us have been harping on this for a long time. Measurement errors must be propagated through out the calculations and are not removed by statistical manipulations. In other words, if you can only measure a temperature to within +- 1 degree, your average can only be accurate to within +- 1 degree.

Think about it. I’ll give you three readings, 52+-1 degree, 51+- 1 degree, and 50 +- 1 degree. What is the average? Is it (51+50+49)/3 = 50? Or is it (53+52+51)/3 = 52 Or is it perhaps somewhere in between? What is the uncertainty of the mean in this case.

Latitude
December 20, 2018 5:08 pm

Jim,, excellent point….I always adding more numbers…the bigger the margin of error

Rick C PE
December 20, 2018 4:47 pm

Willis: While I agree with your primary point, I would go a bit further. First the standard deviation of the mean should be multiplied by 2 to estimate the normal 95% confidence interval. But there is also a need to evaluate and include other effects and create an “uncertainty budget”. ISO’s ‘Guide to Expression of Uncertainty of Measurement’ (GUM) provides a detailed methodology for doing this that requires some quite heavy duty math as well as considerable training in metrology. For example there are six factors that are commonly considered in determining the MU of a high quality screw type micrometer.

I can say that when I was responsible for operations in an ISO 17025 laboratory and implementing the GUM requirements I was taken aback by how large the Measurement Uncertainty the we had to report was for much of our expensive and high precision equipment.

Jim Gorman
December 20, 2018 4:57 pm

+100

Louis Joseph Hooffstetter
December 20, 2018 5:59 pm

Yeah!
What Jim Gorman said again! (+100)
Once again, he beat me to it.

Menicholas
December 20, 2018 7:54 pm

I can recall the building and the lab where I took the lab portion of analytical chemistry (also called qualitative and quantitative analysis in previous years). The floor as concrete, six inches thick.
The scale was incredibly precise, I do not recall how many decimal places it had, but I do recall that in addition to of course having a glass enclosure to prevent air movements from corrupting the measurement…if someone walked into the room, the reading would bounce all over
the place for several tens of seconds. If you shifted your feet, it moved. And the scale was on a solid stone bench and the building had, again, a concrete floor.
No matter how well you knew the procedure of the experiment, and how carefully you followed it, if you did not have really excellent technique your result would suck. And no matter how careful you were to do everything exactly the same, the multiple trials you needed to do to get your 95% confidence interval would all be different.
Grades in that class were assigned on how well your result agreed with the accepted value, and it was tough. Some years no one got an A.
You had to know the course work and theory perfectly, and all of these limitations, and report results correctly, and be extremely careful, and do the exact same thing over multiple times, to even have any chance of getting a good result.
How are readings from satellites being published that are in complete disagreement with the established method of determining sea level changes?
How can sea level graphs now be outside the error bars of the same graphs using the same data as what was reported in 1982?

tty
December 21, 2018 1:04 pm

“First the standard deviation of the mean should be multiplied by 2 to estimate the normal 95% confidence interval.”

WARNING! This is only true for normally distributed data. Which climate data often isn’t. You can calculate 2 SD for any distribution, but only for normally distributed data does it equal 95 % confidence level.

Another Ian
December 20, 2018 4:52 pm

Just imagine the considerable tome “How to lie with statistics” would become if updated now with the benefit of “climate science”

Steve Reddish
December 20, 2018 5:12 pm

I believe the point of Willis’ post is that the most that can ever be achieved, even under the best circumstances, is 1 additional decimal place. Increasing the number of measurements won’t improve the accuracy. Therefore, any calculated measurement indicating better than that should be discounted back to the original measurements’ level of accuracy.

Thus, any claim of sea level rise acceleration, for example, that is based upon a change of mere hundreths of a millimeter when the individual measurements used for the calculations were made to tenths of a millimeter should be noted as unfounded.

Of course, as you point out nw, under less than ideal circumstances, resultant accuracy will be even less.

SR

Robert Osborn
December 20, 2018 5:32 pm

I am trying to figure out the significance of all this is somebody multiplying mean sea level rise of .003mm over a thousand years and getting 3mm of sea level rise?

Usually you don’t worry about significant digits in mean calculations. Additionally in rules of significant digits the figure .003 represents one significant digit, not three. You are supposed to ignore leading zeros. If it were recorded as .0030 it might be considered two significant digits as its suggested you derived that last zero from the observation.

Clyde Spencer
December 20, 2018 9:32 pm

The bottom line is that accuracy is related to systematic error and precision is related to random error. Random error can be reduced by multiple readings of a FIXED value, but systematic error requires comparison to a standard or an instrument of known higher accuracy AND precision.

ATheoK
December 20, 2018 5:19 pm

Willis’s point is spot on.

Other Sea level impacts…
Temperature of the water.
Direction of the wind.
Barometric pressure.
Wave peaks and troughs.
Tidal movement.
Position of the moon and sun.

And exactly how does the satellite know the local conditions while it is scanning the water?

Nor should one overlook NOAA/NASA’s modeling their sea surface heights according to isostasy and Earth’s geod structure.

And somehow, NASA/NOAA claims to measure sea level rise of 3.2 mm annually?
Not a chance.

LdB
December 20, 2018 7:49 pm

Believe it or not Jason 3 measures or has inputs for all those except maybe Barometric pressure would need to look that up. I know most of the others you mention are actually recorded in each data entry.

LdB
December 21, 2018 8:23 am

Again Jason 3 accuracy is 2.5 cm RMS it claims nothing more and is calibrated at numerous sites to ensure it stays within that.

Climate Science do analysis that infer greater accuracy but that has nothing to do with the raw data.

At the moment you are blaming an instrument about an accuracy that it doesn’t claim .. if you have a problem identify the right party.

LdB
December 21, 2018 8:34 am

FYI … read the calibration at one of the sites
https://www.mdpi.com/2072-4292/10/11/1679/htm

eyesonu
December 20, 2018 8:38 pm

I was going to make pretty much the same comment. Even if they tried to measure the deck of an aircraft carrier it would probably vary a couple of feet over a years time at the same location and much more at different locations. To claim that measurements can be made on the ocean surface to accuracy or precision within millimeters appears a little farfetched to say the least.

Editor
December 20, 2018 11:11 pm

I was going to make pretty much the same comment. Even if they tried to measure the deck of an aircraft carrier it would probably vary a couple of feet over a years time at the same location and much more at different locations.

Well, a 1000+ foot aircraft carrier would expand and contract with temperatures, the upper and lowest sections would bend several feet as the seas move (large waves) move from fore to aft, and I’ve seen the long open inside vehicle decks of cargo vessels (800 feet long) twist by feet as the bow twists and then the mid and then the stern. But the length changes because of temperature changes (-15 C to +35 C for example) are not several feet.

eyesonu
December 21, 2018 5:23 am

A simple change in atmospheric pressure alone could result in the water elev change of a few inches and with 25.4 mm per inch times say 3 inches equal to 75 mm change that would blow the millimeter claims of precision/accuracy out of the water.

I think I will go measure a piece of 20 or 40 grit sandpaper to get an accurate measurement of it’s thickness to the nearest .001 inch. I’ll report back with my measurements. Meanwhile someone else can measure the mean sea level in 20-30 ft seas to the nearest millimeter and we’ll compare results. We can use simple or complex math equations and compare the results to a baseline measurement of a fart in a high wind.

LdB
December 21, 2018 8:30 am

Jason 3 does do an adjustment for Barometric pressure

As atmospheric pressure increases and decreases, the sea surface tends to respond hydrostatically, falling or rising respectively. Generally, a 1-mbar increase in atmospheric pressure depresses the sea surface by about 1 cm. This effect is referred to as the inverse barometer (IB) effect.

The datasheet gives you the formula it uses to correct it .. again it is calibrated and stays within the 2.5cm RMS accuracy claimed.

You guys seem to have a lot of complaints about an instrument without reading a single word about what it does and how it does it.

Hivemind
December 20, 2018 5:28 pm

Your example is simple, concrete & useful to show one limitation of averaging. However, what about creating an average of the world’s temperature? So, we can average the temperature at the North Pole, New York and Jamaica.

That would be more like averaging my credit card, the size of my wallet and a passport. The number is meaningless. And yet, climate “scientists” do this all the time.

Angus McFarlane
December 20, 2018 5:30 pm

Willis your rule-of-thumb is correct. It is founded in the mathematics of “Reliability and Statistics.”
In simple terms the mathematics would be:

Total Error = Model (or System) Error + Measurement (or Data) Error

Where the above errors are probability (or random) distributions. If the system were perfect, then the system error would be zero but this rarely happens in the real world.

To use your thermometer example, the thermometer (system)would have an error of ± 0.1 °C. Therefore, no matter how many measurements we take the error would never be better than ± 0.1 °C and could be much worse, depending the size of the measurement errors.

December 20, 2018 5:55 pm

Angus, I am afraid that you are not a climatologist.

Angus McFarlane
December 20, 2018 9:32 pm

Curious George, brilliant!

Mike Smith
December 20, 2018 6:00 pm

The problems with our temperature measurements are not limited to the instrument itself. We know that the readings will be impacted greatly by the choice and condition of the site at which the thermometers are read. And we know from past work that there are huge issues with site selection and their condition.

Averaging won’t significantly reduce those errors when urban and anthropogenic effect almost always bias high.

Robert B
December 20, 2018 5:52 pm

My analogy is measuring the height of adults. Australian men are 1.76 m on average, and to keep things simple, the median is 1.75 m . Women are 1.62 m and, again to keep it simple, all under 1.75 m. Rounding 1000 measurements to the nearest 0.5 m and you get 75% of the adult population are 1. 5 m tall while 25% are 2 m tall. That means that Australian adults are 1.625 m tall on average and men are only 5 mm taller than women on average in Australia. Or if we measured the two sexes rounding off to the nearest half metre, men are 250 mm taller than women on average.

Making 1 million measurements will not make it all better.

December 20, 2018 5:52 pm

Statistics is a wild beast. Take the statement Willis quotes “its hard to see why any errors don’t ether cancel or resolve over a very large number of measurements to a constant bias.” Let’s consider a related situation, the Brownian motion – a small particle in water moved randomly by the impact of water molecules. How does the large number of impacts average? What can we say about the particle position statistically?
It turns out that if you simulate a one-dimensional case or a two-dimensional case, the particle does not move much. In a three-dimensional case, the average displacement of the particle from the original position is linear in time – I vaguely remember that it got actually measured as a way to determine Boltzmann’s constant.

December 20, 2018 5:54 pm

“A lot of stuff really will average out over time”.

Errors which go in one direction don’t, that’s why bias and extremism tends to get worse over time.

sparko
December 20, 2018 6:10 pm

Isn’t there a large systematic drift error with the height of the satellites. I do recall a proposed GRASP system to try and reduce this cumulative error to 0.1mm a year.

Jim Gorman
December 20, 2018 6:15 pm

Here is another question. Why don’t the providers of temperature data sets require that users (i.e., scientists, etc.) not only acknowledge the measurement error range contained in the data set but must also quantify how these errors affect their conclusions. In other words, evaluate their results based upon the low range of temperatures and upon the high range of the errors.

This would go a long way toward making everyone aware of the inaccuracies being ignored.

Andrew Preece
December 20, 2018 6:32 pm

This process of averaging multiple measurements to get a more accurate measurement is done all the time in electronics. Take the case of an 8-bit sampling system (ADC) which can resolve 256 different voltage levels (2^8). Imagine it has a full scale input range of 0 to 25.6 volts (unlikely numbers but it makes the maths easy). We can then resolve voltages 25.6/256 apart (0.1V apart). You might say that is the limit of our accuracy but that isn’t so.

Take an input voltage of 10.973V, just as a for instance. The nearest ADC level is 11.0V (level 110) and in the absence of noise that is what the ADC would read. It would be wrong by 0.027V. If there is some noise (dither) on the signal however the voltage will bob up and down around 10.973V randomly. However, it will apparently spend more time at the 11.0V level (level 110) than the 10.9V level (level 109). In fact it will spend time at both of these levels pro rata to the distance the real input voltage is from these levels. In this case, the signal is 0.027V from level 110 and 0.073V away from level 109. If we take a thousand measurement of the ADC we will end up with approximately 270 readings at ADC level 109 and 729 at level 110. We can use that information to narrow down the true voltage at the input to near the real value of 10.973V even though at first glance we can’t get a better accuracy than the native accuracy of the ADC, one part in 256 (or 0.1V in this particular example). In this way, 8-bit ADCs can be made equivalent to 9-bit, 10-bit, 11-bit or more, at the expense of taking multiple readings.

We can take this principle to its extreme and come up with a 1-bit ADC. This has only two output states and looks to be fairly useless at first glance. Say the input voltage range is the same as before, 25.6V. Then if the input voltage is <12.8V the output will read 0 (equivalent to 0V to 12.8V) and if it's higher it will read 1 (12.8V to 25.6V). A reolution of 12.8V, apparently rubbish. However, if we apply a large amplitude noise dither on the input voltage, the 1-bit ADC will read a signal that is at exactly 12.8V 50% of the time as a 0 and 50% of the time as a 1. Given enough readings, we will find we get equal numbers of 0s and 1s. Thus we know, despite the apparently awful resolution of a 1-bit ADC, that the input signal is at 12.80000etcV despite the only two levels we can actually read being 0V or 25.6V. If the input voltage strays away from 12.8V, say to 13.273671V then if we take enough measurements we will find we get an excess of 1s over 0s and by the proportions of these two results we can determine the input voltage to an arbitrary accuracy, limited only by the number of samples. One can turn a 1-bit ADC with dither into a 24-bit equivalent ADC just by oversampling in this way. In fact, there would be other factors that would limit the accuracy of the measurement no matter how many sample we take, but this does not impact on the primary question – that taking more samples can indeed improve accuracy in a sampled data system – it is done all the time.

Steve O
December 21, 2018 4:57 am

This is a perfect example. Thank you.

December 21, 2018 12:33 pm

Andrew,
You seem to be forgetting that your overall accuracy is still limited by the A to D converter. Yes, by dithering you can jitter the value across an ADC steps and thus estimate a value between ADC steps. This is done in many systems. However, the final accuracy, no matter how fine a division on an ADC step, is ultimately only the ADC basic accuracy. In many cases, such as signal analysis, absolute ADC accuracy is not as important as monotonicity and even step size. Dithering helps minimize ADC step intermodulation. Again, accuracy is that of the ADC, not how fine you can subdivide each ADC step.

Peter Sable
December 26, 2018 3:55 pm

One can turn a 1-bit ADC with dither into a 24-bit equivalent ADC just by oversampling in this way.

With just dithering you only get 1/2 bit per 2x oversample, or 2^48 oversampling to get 24 bits, because at best the dithering noise is uncorrelated. I don’t think there’s any system that can oversample fast enough to get 2^48 oversampling…

Delta-Sigma converters using negative correlation by feeding back an inverted signal, allowing a Hurst exponent of the quantization noise to be between 0 and 0.5, thus giving M+1/2 bits per doubling, where M is the number of feedback stages (the “order” of the delta-sigma converter)

A typical commercial delta-sigma ADC is third order giving 4.5 bits per doubling, so 32x-64x oversampling is sufficient to get 24 bits

Take the case of an 8-bit sampling system (ADC)

funny enough, Willis’s rule is correct here, you can get at most 3 additional bits (which is ~1 decimal place) of an 8 bit system because there are 8 comparators and thus differences between the comparators (manifesting as non-linearities), which means oversampling has diminishing returns.

BTW your typical 8 bit A/D in a microcontroller has terrible linearity so is already only effectively 7 bits.

This is why 1-bit delta-sigma converters are so nice. There’s only one comparator and making one comparator linear is relatively straightforward, and can be done with fairly standard CMOS processes. This is why all your computers have a delta-sigma converter for audio output.

R Percifield
December 20, 2018 7:22 pm

Willis,

Working in engineering I experience this all of the time. My assertion is the following:
Resolution:
The maximum ability of the measurement system to quantify a value. A 10 bit A to D has at best 1024 individual values 0-1023. If for example a 5V reference is used the system will divide the 5 volts into 1024 steps. This implies each step has approximately 4.88mV so the max resolution is 4.88mV. No matter how many measurements you make you will never be able to tell the difference between two numbers with less than 4.88mV.

Accuracy:
The delta from the real to measured value in the system. This is a calibration issue and independent of precision. In the A to D system above that is determined by the reference voltage accuracy and the internal offsets within the signal processing system.

Precision:
The variation of the measurements due to random error within the system. In the A to D example above the system is capable of +/- 1 Least Significant Bit.

My assertion is this:

1. For Resolution taking multiple samples will not improve resolution. If a value is between two buckets no number of samples will resolve them.

2. Accuracy is an offset and sample size will not resolve this problem, only calibration will.

3. Precision can be affected by samples assuming a predictable and stable variation. If the variation is systematic, i.e. in certain conditions the offset is different, is not predictable, or nonlinear, then an average will not work.

Thus, I would say you do not gain an extra digit by averaging, without knowing the other factors affecting the system.

R Percifield
December 21, 2018 5:05 am

Willis,

When I posted much of the intervening explanation you had stated for some reason was not displayed. I fight software people all of the time thinking that they can improve resolution by oversampling. Don’t get me started about sensor variation……. Have a good weekend and holiday.

December 20, 2018 7:43 pm

Now … raise your hand if you think that we’ve just accurately measured the length of the credit card to the nearest three thousandths of one millimeter.

Of course not. And the answer would not be improved if we had a million measurements.

I’ll raise my hand.

But then I had a science education.

And a bit of maths.

Sometimes Willis, you are dangerously wrong.

Max Dupilka
December 20, 2018 8:46 pm

I will raise my hand as well.
Here is a nice video explaining standard error of sample means and why the more samples you take the smaller the error of the sample mean will be.

Clyde Spencer
December 20, 2018 9:42 pm

Leo Smith
Are you suggesting that you are one of the few people here with a science education and exposure to mathematics?

Analog Design Engineer
December 21, 2018 7:29 am

I guess I don’t need my expensive 8.5 digit DMM anymore either. Instead I can just use a cheap handheld one & just take loads of measurements 🙂

Dr. Strangelove
December 20, 2018 8:04 pm

The standard deviation (SD) is independent of the standard error (SE). So reducing SE has no effect on SD. The SE only tells you how good your sample represents the population. For example, you want to determine the average age of a city’s population. From your sample, you calculated mean = 30 and SD = 1. How do you know SD is really 1? You did not ask all the people in the city. It could be 2. SE tells you that actual SD is close to 1 and far from 2. Therefore, your sample is good.

But your sample size does not affect SD because that is a property of the population. Another city could have SD = 0.1 or 0 (all people have the same age). SE has nothing to do whether or not you can measure the mean age to the last hour, minute or seconds.

Clyde Spencer
December 20, 2018 9:40 pm

Dr. Strangelove
Another way to look at that is that there is a fundamental relationship between the range of a normal distribution and the standard deviation. Increasing the number of samples will have a small impact on the range because most of the samples will be in the high probability central region of the distribution.

Dr. Strangelove
December 21, 2018 2:36 am

Another way to look at it is the normal distribution is a mathematical model. SE tells you how well your data fit the model. If the population is really normally distributed, the greater the sample size (N) the better the fit. But it doesn’t tell you whether or not your data is accurate. If you just made up the data, you can have a perfect fit and it would still be wrong.

Dr. Strangelove
December 21, 2018 3:59 am

The normal distribution is a continuous probability function:
P (x) = e^(a x^2 + b x + c)
It’s the base of natural log raised to the power of a quadratic equation.

When the number of trials (or samples) n is large, the normal distribution approximates the binomial distribution, which is a discrete probability function. If the probability of x is p, then the probability of getting x in n trials is a binomial distribution:
P (x) = n! / (x! (n – x)!) p^x (1 – p)^(n – x)

Scott W Bennett
December 21, 2018 4:22 am

I agree with Willis here but I keep hearing on both sides, the notion that a random distribution is an accessible absolute, that unveils uncertainty!

Everyone who actually employs this term should do the experiment in the real world, by tossing a coin for example and noting the result of even that simple 50/50 probability.

This is the way I was first introduced to “Experiment Probability” and it has stuck with me, ever since.

Lately it seems, it has become plain old probability and the “experimental” part has been dropped.

In reality, nature doesn’t have to obey the theory of probability as we commonly experience it. Only as the number of experiments – coin tosses – increase does reality approach the theory.

Convergence is not guaranteed because there is nothing in reality to stop the experiment from diverging from the theory, though it does become less and less likely as the number of “experiments” increases.

In lieu of doing the recommended experiment in real life, I just found an excellent video* that does a good job of modelling the common real world experience.

I would add that it is a cheat to use a random generator because the lesson to be learned here, is that nature only ever approaches “randomness” via large numbers. Therefore notions of ”distribution” are misleading in the “real” world!

Once more to be clear, strings of heads or tails or complex patterns are not ruled out in actuality – the experiment – despite the theory!

Scott W Bennett
December 21, 2018 4:55 am

Typo above, I meant to say experimental probability but empirical probability or relative frequency, would smell as sweet!

S W Bennett
December 23, 2018 8:31 am

I just found a much better example here:

I highly recommend this for anybody interested in probability or relative frequency / distribution, particularly from 36.00 onwards.

Coin tossing follows the Law of Long Leads or the Arcsine Law.

*In a coin toss, the symmetry does not show itself by being heads half the time, it shows itself in half the “sample paths” being above or below the line of theoretical probability most of the time!

cheers,

Scott

*Professor Raymond Flood, Probability and Its Limits

December 20, 2018 8:22 pm

Willis, Thank You for this post. This is the primary reason that the pseudo science of AGW is a farce.
Also have a look at windy.tv, almost all the weather is over and because of the oceans.
We are on planet water not planet earth.

Brian
December 20, 2018 9:09 pm

Willis,

As an experimental scientist, I can say that your arguments here are mostly wrong. The measurement precision of an instrument has effectively no impact on how precisely or accurately a quantity can be measured. To see why let’s break down the various elements of your post.

With regard to precision (random error), we can take your credit card example. Let’s suppose that people measuring the credit card all get a number between 85 and 86 mm and their results cover the full range. As part of our measurement protocol, we tell them to choose not any number between the two but one or the other. They should make that choice based on which is closer to the actual length they observe. Our data will consist, then, of a seemingly random string of 85s and 86s. We are taking the average of this string of data, some of which are 85 and the rest of which are 86.

If we have ten such measurements, the calculated averages will be numbers like 85.5 (5, 5 split), 85.6 (4, 6 split), 85.7 (3, 7 split), etc. For ten thousand measurements, however, the numbers will be 85.5000 (5000, 5000 split), 85.5001 (4999, 5001 split), 85.5002 (4998, 5002 split), etc. With an arbitrarily large number of measurements, any average between 85 and 86 is possible. But how good are these averages?

The uncertainty you mentioned (sigma/sqrt(N)), also called the “standard deviation of the mean” or the “standard error,” tells us the precision or random error of the mean thus obtained. Do an experiment with N measurements. Calculate the mean. If later experiments are done with N measurements under the same conditions, the standard deviation of the mean (SDM) tells us how much those other means will vary from the first. In general, 95% of means obtained by all N-measurement experiments will be within 2 SDMs of each other. Thus, with an arbitrarily large number of measurements, we can get means from many experiments to be as consistent as we want. There is, in other words, no fundamental limit on how precisely we can determine the result (mean) of an experiment. The measurement precision of the apparatus used is irrelevant.

Now your objection to this would likely be “But that doesn’t guarantee that the mean is the right answer. We might get 85.567 +- 0.003 mm when the actual length is 85.329 mm. The mean is consistent but consistently wrong.”

This objection would be correct, and is an example of systematic error (inaccuracy). Every experiment is subject to many systematic errors of varying sizes, and these inaccuracies cause the results to be wrong. Systematic errors are governed by a fundamental principle similar to your incorrect rule of thumb (“you can’t squeeze more than one additional decimal out of an average of real-world observations”):

No systematic error smaller than the uncertainty (random error) can be identified.

This is the point of making more and more measurements. By reducing the uncertainty of the result, we are able to potentially uncover smaller and smaller systematic errors. A small uncertainty doesn’t guarantee that the experiment is that accurate, but it does allow for the experiment to be MADE that accurate by eliminating the systematic errors. Just as a falling tide uncovers more and more objects on the beach, a falling uncertainty uncovers more and more systematic error. And like objects on a beach, this allows the systematic errors to be identified and removed.

But how are systematic errors actually identified? By doing different KINDS of experiments. If the same quantity is measured by many different techniques, those techniques are likely to have different systematic errors. For temperature, this would mean using thermometers, satellites, balloons, etc. For sea level this might be tide gauges, laser altimetry, etc. If we make all those techniques precise enough (through N measurements), they will eventually disagree. We use that disagreement to identify the systematic errors causing the discrepancies. Once the systematic errors are identified and removed, the various methods will agree to high precision. AT THAT POINT, WE KNOW THE ACTUAL RESULT TO THAT LEVEL OF UNCERTAINTY. Again, our ability to do this does not depend in any fundamental way on the measuring precision of the apparatus.

In practice, of course, this process is slow and laborious, requiring a great deal of time and resources. In practice, our ability to reduce the uncertainty may depend on available technology and manpower. But the precision of the measuring device does not place any fundamental limit on the process.

Menicholas
December 21, 2018 6:10 am

Willis, what say you to this specific contention:

“The measurement precision of an instrument has effectively no impact on how precisely or accurately a quantity can be measured.”

This seems to me to me saying having a ruler with only inches on it is no better or worse than having one with no graduations or one marked in millimeters.
I do not think any machinist or engineer would agree with this statement.

Brian
December 21, 2018 9:19 pm

Willis,

No, measuring something by multiple methods doesn’t imply that the other methods have better resolution or precision (a micrometer vs. a ruler). There’s an advantage to be gained even if the other methods have identical resolution and precision to the first. The key, again, is choosing methods that likely have different systematic errors. That’s the only way that systematic errors can be identified and eliminated.

This key point is most definitely NOT what you said or implied. Yes, you focused on using only a ruler and said that taking more measurements doesn’t help you out. But even that claim misses the point of how measurements are done in science. Any single measuring device that purports to be accurate is calibrated against other reference devices, or against reference quantities. Every scientific measuring device must be traceable to fundamental measurement standards. A calibrated device and the procedure for using it already has its systematic errors defined and quantified through the use of multiple types of experiments. So even if you are using only the ruler, it is already backed by the multiple devices and experiments I mentioned.

The bottom line is that a ruler calibrated through use of a specific measurement procedure can have systematic errors much less than the resolution of the device. And such a procedure can involve a large number of measurements. By making this large number of measurements according to the designated calibration procedure, the measurement precision is indeed reduced to a small value that is both precise and accurate. And this is all with the use of a single device.

Brian
December 22, 2018 12:31 pm

Willis,

You say “Yes, you can use all kinds of other techniques to improve your accuracy. But to do that, you have to use other techniques.”

The point you are ignoring is that ANY measuring device needs to be calibrated and it gets calibrated by using other techniques and devices. That doesn’t mean you have to use the other techniques as part of YOUR measurement. Look, you provided a ruler as your example. Presumably you weren’t assuming that the ruler is a piece of garbage. Presumably you think it works correctly according to its design. In that case, it’s been calibrated using other techniques. You can’t use the “other techniques” excuse to wiggle out of my point.

And you are just plain wrong about this. It is always possible to obtain much smaller systematic errors with a device AND A PRESCRIBED MEASUREMENT PROCEDURE than the device resolution. And once the systematic errors are small, the only thing preventing a highly accurate measurement is the random error, which is made smaller by repeated measurements. Under those conditions (where the systematic error has already been made small), high accuracy is obtained by repeated measurements.

Jim Gorman
December 22, 2018 2:36 pm

Brian –> You are conflating a real world measurement with a lab experiment. You seem to be missing the whole point. Willis’ example doesn’t exactly apply to sea level measurements by satellite nor to temperature measurements.

In both cases you get only one measurement of one thing at one time. You don’t have a static substance that you can continue to measure or to diddle with in order to get a more accurate or precise measurement.

All the ADC and other discussions are fine, but they simply don’t apply.

Brian
December 23, 2018 8:45 pm

Willis,

Getting systematic and random errors much less than the base resolution is done all the time and can be taken to extreme levels.

One good example is LIGO, the gravitational wave observatory. LIGO works on interferometry of 1-micron light, which means the base resolution (equivalent to the marking spacing on a ruler) is only 1 micron. Yet they measure deviations of space on the order of 10^-18 m, or 1 trillionth the base resolution. This is accomplished by reducing systematic errors (such as vibrations from earthquakes and traffic) to a very low level and then using statistics to reduce the random error. In this case, the repeated measurements required to gauge random error are provided by the high number of photons in the cavity. Much as I described in my first comment, this allows shifts of less than 1 millionth of a wavelength to be measured. So your statement that errors can be reduced to only about 1/10 the resolution of the device is wrong.

Micky H Corbett
December 21, 2018 3:30 am

I too started in professional life doing a PhD in experimental physics. I then moved onto safety critical software then ion propulsion for satellites. I became very familiar with metrology and had to improve my scientific method.

So, talking multiple measurements will not change anything if you cannot determine that your sample distributions are identical. After all a necessary condition for the Central Limit Theorem that underpins the error of the mean is that samples are i.i.d. – Independent and identically distributed.

If the uncertainty of individual elements of your sample are sufficiently large the uncertainty in distribution goes up. In other words noise is greater than signal. For temperature measurements this is the case when looking at changes 0.1 degrees. For satellite measurements it may also be the case if a certain range of heights are required.

More measurements do not produce less uncertainty. It is due to the CLT.

Clyde Spencer
December 20, 2018 9:50 pm

Brian
For Willis’ example, very few if any people will record 85. Almost all will record 86. Thus, you will get a long string of 86s. If nobody records an 85, you will have actually reduced the precision by reporting 86, when it is clearly less than 86!

Brian
December 21, 2018 9:29 pm

Clyde,

That’s why I said this:

“Let’s suppose that people measuring the credit card all get a number between 85 and 86 mm and their results cover the full range,”

in keeping with Willis’ own description. That’s how he got SD = 0.3. My proposed measurement procedure for getting highly precise results actually depends on individual measurements being as imprecise as the device resolution to avoid systematic error (or bias) that can creep into people’s judgement.

December 20, 2018 10:41 pm

The above is n the eyes glazing over catogary.

Two things come to mind , Julie Ceasar was asked about the loyalty of his guards. He replied yes that is of interest, but who is going to guard the guards.

Second, Joseph Stalin is said to have said, It does not matter how many vote, it does matter who counts the votes”

All of the above is wide open to cherry picking and adjustments to the data.

MJE

bonbon
December 21, 2018 6:14 am

Which is why Alexander Hamilton introduced the Electoral College, never allowed to be all in the same room at the same time. Counting-statistics may be popular, yet when this brilliant idea actually works, the winner is called “populist”.

Steve Reddish
December 20, 2018 10:50 pm

“Regardless of the number of measurements, you can’t squeeze more than one additional decimal out of an average of real-world observations.

Following that rule of thumb, if you are measuring say temperatures to the nearest degree, no matter how many measurements you have, your average will be valid to the nearest tenth of a degree ”

I think you won’t even get an additional tenth. I think you will at best, get an additional 1/2 of of a degree. Suppose you ask a bunch of people to take a reading in whole degrees from a thermometer marked off in whole degrees. If the group overwhelmingly says the thermometer indicates 35 degrees, you must record 35 degrees.

If approximately 50% say it is 35, while the others say it is 36, you can be very confident rounding the temperature to 35.5 degrees. This is because you asked people simply to decide which whole degree the thermometer indicator was closest to. If nearly equal numbers decided each way, the indicator must be very close to midway between.

If 25% say 35 degrees, while 75% say 36 degrees, you may record the temperature as 35.5 degrees, but not as something like 35.7 degrees. This is because you asked for a reading to the nearest degree. All you know for sure is that most of the group determined the indicator was closer to 36 than 35. Because some said it was closer to 35, the indicator must be still very close to midway between 35 and 36. If the indicator actually was very close to what would be 35.7, the group would have overwhelmingly noted it was closer to 36 than 35.

By asking for a reading to a whole degree, you were asking people to judge only whether the reading was more or less than .5 of a degree. Thus, you can sometimes squeeze out an extra 1/2 degree, but no finer.

Automated equipment works in the same manner because that is what it has been set up to do.

SR

Steve Reddish
December 20, 2018 10:57 pm

I forgot to put in my conclusion that multiple readings only increase the precision to 1/2 of the delineations the measuring device is marked to or capable of determining.

SR

Engineer Bob
December 20, 2018 11:59 pm

I have used a system which gets higher accuracy.

You need to add in a truly random noise source, whose standard deviation is some convenient small multiple of the measurement unit, and whose distribution is well known. For example, the thermal noise in a resistor.

Now the quantity of interest is represented by a population of different measurements. And the statistics of that population give a way to calculate the underlying value more precisely.

tty
December 21, 2018 3:30 am

You are fooling yourself Bob if you think adding noise to series of measurement improves accuracy. What it does is to allow you to use a formula that isn’t applicable. In your case if you do an infinite number of measurements it will only allow you to get back to the original accuracy before you added the noise.

tty
December 21, 2018 12:10 pm

He is still fooling himself. An infinite number of measurements would then allow him to get rid of the noise he added but would not improve the accuracy of the original measurement.

Peter Sable
December 26, 2018 4:10 pm

You are fooling yourself Bob if you think adding noise to series of measurement improves accuracy

Funny, in electronics we call this “dithering” and it’s used all the time to turn correlated noise into uncorrelated noise due to quantization errors, which is exactly what’s happening in the case of the ruler.

That quantization noise is white noise and you can get 1/2 bit resolution per halving of frequency (which is exactly analagous to the sigma/sqrt(n).

Without dithering, the quantization noise is correlated to the input signal and you are lucky if you get 3 bits or one decimal of improvement.

Of course in climate signals, unlike the ruler, there’s tons of dithering but who knows whether the dither signal is white noise.. often not. But I think making blanket statements about “at most one decimal place” isn’t correct either. Why not just estimate using the Hurst exponent that can be roughly derived from the data itself?

Peter Sable
January 1, 2019 4:02 pm

Peter, I use the Hurst exponent to correct for autocorrelation. However, I’ve never heard of using it to get a more accurate average of a number of measurements. Could you say a bit more about that?

If the Hurst exponent is between 0 and 0.5 you get a more accurate average because the signal has negative autocorrelation. This was discussed in the comments of your Hurst article.

That’s the principle behind delta-sigma converters. Negative feedback allows about one decimal place per oversample ratio in a 3-stage design. (though designers typically talk about bits, or powers of 2, instead of powers of 10).

Unless your measurement system can interact with the climate (hah!), you are not going to get negative autocorrelation from the measurement system! But I wouldn’t be surprised if negative autocorrelation appears somewhere in nature.

BTW on further reading estimating the Hurst exponent consistently requires about 20k samples, non trivial to say the least! So my proposition of “why not just estimate from the Hurst exponent” is not really practical in many cases. One might interpret that limit in a pessimistic fashion given the idea of the Null Hypothesis.

Hocus Locus
December 21, 2018 4:07 am

You need to add in a truly random noise source,

Are you suggesting something like heating the credit credit above the temperature of the ruler until the edge of the card seems centered on a ‘pip’? And a measurement system that yields information ‘at’ the pips but not in between, like a balance scale for weights (or electrical +/- comparator with precision source)? And once the balance point is reached, the degree of temperature rise and known property of the material used to ‘back estimate’ where between the pips the card edge was at ambient/ruler temperature? Proxies like turtles all the way down?

Molecules are so skittery and slippery and everything is throbbing along an edge as long as the coastline of Britain. That’s why God created the photon, to count integral sheep and get some rest.

Joe H
December 21, 2018 1:28 am

Willis, I was researching this online recently. There is conflicting information but I believe it to be correct that measurement resolution is a systematic error and not a random one. Hence it doesn’t reduce by sqrt(N) but is irreducible. What is the uncertainty here? Some say one tenth of the smallest division (i.e. 0.1mm) but I’ve seen others say that is too low and around half this resolution (i.e. 0.5mm). Either way in your example, I do not believe the credit card length, using the ruler specified, can be known with an uncertainty less than 0.1mm no matter how many people measure it.

Menicholas
December 21, 2018 5:17 am

“Observational error (or measurement error) is the difference between a measured value of a quantity and its true value.[1] In statistics, an error is not a “mistake”. Variability is an inherent part of the results of measurements and of the measurement process.
Measurement errors can be divided into two components: random error and systematic error.”
https://en.wikipedia.org/wiki/Observational_error#Systematic_versus_random_error

Joe H
December 21, 2018 12:22 pm

I’ve no idea what your reply means M. I know the difference between error and uncertainty but that has absolutely nothing to do with my post. My post is about random error (reducible) v systematic error (irreducible) and whether or not measurement minimum resolution is a random or systematic error.

Michael Carter
December 21, 2018 1:29 am

Surely the accuracy of the satellite measurements are testable?

Take a the number of passes to required establish mean sea level according to the theory, then do the same a number of times within a short time-frame (months) . If the system is accurate the precision will be tight.

Not rocket science, surely?

I have the feeling that we are not being told everything. I would love to know what variation they are really getting.

Cheers

M

Menicholas
December 21, 2018 5:03 am

“If the system is accurate the precision will be tight”

This is not the case.
From the Wikipedia on accuracy and precision:

“A measurement system can be accurate but not precise, precise but not accurate, neither, or both. For example, if an experiment contains a systematic error, then increasing the sample size generally increases precision but does not improve accuracy. The result would be a consistent yet inaccurate string of results from the flawed experiment. Eliminating the systematic error improves accuracy but does not change precision.”