The quote in the headline is direct from this article in Science News for which I’ve posted an excerpt below. I found this article interesting for two reasons. 1- It challenges use of statistical methods that have come into question in climate science recently, such as Mann’s tree ring proxy hockey stick and the Steig et al statistical assertion that Antarctica is warming. 2- It pulls no punches in pointing out an over-reliance on statistical methods can produce competing results from the same base data. Skeptics might ponder this famous quote:
“If your experiment needs statistics, you ought to have done a better experiment.” – Lord Ernest Rutherford
There are many more interesting quotes about statistics here.
– Anthony
UPDATE: Luboš Motl has a rebuttal also worth reading here. I should make it clear that my position is not that we should discard statistics, but that we shouldn’t over-rely on them to tease out signals that are so weak they may or may not be significant. Nature leaves plenty of tracks, and as Lord Rutherford points out better experiments make those tracks clear. – A
==================================
Odds Are, It’s Wrong – Science fails to face the shortcomings of statistics
March 27th, 2010; Vol.177 #7 (p. 26)
For better or for worse, science has long been married to mathematics. Generally it has been for the better. Especially since the days of Galileo and Newton, math has nurtured science. Rigorous mathematical methods have secured science’s fidelity to fact and conferred a timeless reliability to its findings.
During the past century, though, a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully. Science was seduced by statistics, the math rooted in the same principles that guarantee profits for Las Vegas casinos. Supposedly, the proper use of statistics makes relying on scientific results a safe bet. But in practice, widespread misuse of statistical methods makes science more like a crapshoot.
It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.
Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.
Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.
“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”
Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”
Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”
====================================
Read much more of this story here at Science News
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Leif Svalgaard (08:47:00) :
Basil (07:02:14) :
We, of course, basically agree.
Hey, I had intentionally misspelled ‘basically’ as ‘basilally’. And somehow it got corrected…
REPLY: No good deed goes unpunished, sorry Leif. – Anthony
Then we’re all agreed: “Liars use figures as one of their tools and when they do their figures cannot be trusted.”
Next Issue: How do we identify these people before they open their mouth or publish something?
a. Most held Elected Office
b. Most charge $100K Speaking Fees
c. Most live in Energy Inefficient Mansions
d. Most know nothing about climate change
Next issue: How do we stop these dispicable excuses for human beings?
a. 20 to Life
b. Beheading
c. Castration
d. Draw & Quarter
An example of misleading statistics:
Water vapor is around 1% of the atmosphere, a molecular constituent of air.
But the 1% figure is misleading because it is an ‘average’ of the entire volume of air in the atmosphere.
Averages are a product of statistical work-up.
But in the real atmosphere water vapor is concentrated in some volumes of air (and constantly moving and forming) and tenuous in other volumes of air.
And, these varying concentrations of water vapor do have an impact on temperature retention absorbtion & release.
Clearly, a scientist must take into account the specific concentration of water vapor in any given body of air mass.
Simply considering the average water vapor percentage will not tell the scientist how water vapor acts in the atmosphere.
One must take into account real time water vapor behavior to understand its contribution to atmosphere behavior, and, thus, climate.
Averages won’t contribute to that understanding — in fact — the average will mislead.
Because “averages” in many instances are not how the physical relationships of chemical constituents in a body of gas interact.
This thread is too long, but, I haven’t seen it stated yet:
Statistics is the science of the behavior of numbers.
Remember that.
Rarely does the real world behave like numbers, to which you can assign all sorts of characteristics, and not have any unknown behaviors in your numbers.
If you doubt the utility of statistics and science, take a couple of courses in the statistical design of experiments. It is breath taking what clever people can do if you let them design the experiments, not call in the clever people after you have done some unorganized experiment and have a bunch of trash data.
Almost nobody in medical research has any appreciation for statistical design of experiments, or even statistics.
When I was a freshly minted M.D., I and studied stats on my own, I began to read the medical literature from the viewpoint of their statistical work. Mostly just trash. Even large studies were trash. Every study was meant to prove a point. And, this was before drug companies began to pay for research.
Problem is, these people get rewarded for bad results.
My brother in law designed software for the Navy once, for a submarine. For the maiden voyage, the software designers dove with the sub. You can be confident he was sure it would work.
James F. Evans (09:30:31) :
An example of misleading statistics:
Water vapor is around 1% of the atmosphere, a molecular constituent of air.
Except that that statement is not an example of a statistics, but just a statement of fact. Not every number that is calculated or determined is ‘statistical’. Statistics is about drawing and asserting conclusions from the data, not about the data themselves.
Brent Hargreaves wrote (06:01:39) :
‘Needs’, the quote used the word “needs statistics”, not ‘requires’; small, subtle, but important difference I think …
‘Needs’ is more akin to an “if necessary” qualifier than the much stricter qualifier ‘requires’, also, if an experiment ‘needs’ statistics to ‘winnow out’ an observation, it probably:
a) isn’t clear to the naked, unaided eyeball and
b) requires the use of those ‘statistical’ methods to qualify the result to some singular number (or numbers) by which success or failure is scored.
Kinda like Climate Science; statistical techniques are needed (AKA necessary) to fudge (‘cool’) the past numbers in order to show ‘warming’ in the present … looking at raw, uncooked data (for clean, un-UHI contaminated sites, for instance) does not indicate the warming that the statistically-cooked, massaged data shows.
Therefore, statistical techniques are needed in ‘Climate Science’ to ‘prove’ their case thereby making it a bad experiment scoring by Rutherford’s rule:
.
.
My pet concern, I have seen it many times in biology/biochemistry, is when people assume the thing they are measuring is normally distributed when that is not at all clear from their data.
Leif Svalgaard (09:25:01) :
Leif Svalgaard (08:47:00) :
Basil (07:02:14) :
We, of course, basically agree.
Hey, I had intentionally misspelled ‘basically’ as ‘basilally’. And somehow it got corrected…
REPLY: No good deed goes unpunished, sorry Leif. – Anthony
Since you didn’t see it, my pun wasn’t any good to begin with 🙂
Missed you at ctm’s great party last night. Seven police cruisers were standing by outside [only half a block away] the joint.
Decades ago, Dr Edward Deming, who some call the father of Quality Control, recognized that theoretical statisticians were needed to help ensure companies correctly interpreted quality measurements. assignable causes of variability, the signifigance of trends, etc.
Its my impression climate scientists do not engage the statistical community in the formulation or review of their work. They seem to just plug and play statistical tools. Sorry, that’s what it seems. Correct me if I’m wrong.
Luboš Motl (06:26:43)
Lubos, perhaps we are reading different articles, but there’s nothing in the article that I see that says we should throw out all statistics.
Instead, it says that most of the time people misuse statistics. I agree with that completely. In climate science in particular, we are dealing with non-normal datasets that have a high Hurst coefficient, which makes most statistics unreliable.
That’s the main problem I see, that statistics are applied improperly.
PJB (05:55:15) :
“Having used statistical methods in my research, I always distilled my results down to one aspect of analysis.
Signal to noise ratio.”
I have always said that Information Theory ( and Parsing) is the only tool for analysis of data streams. Temperature data is a stream in time and space. So you are absolutely correct. It all comes down to Signal to Noise.
Last night, after I read about the launch of the latest GOES, I started thinking about some of the stuff I had been involved with in the past. Which got me to thinking about digital communications and temperature anomalies. (Okay so I’m weird.) Without understanding the nature of the noise sources and the noise distribution, any analysis becomes garbage. The simple case of calculating the noise power in a modulated analog data channel with Gaussian noise, is non-trivial and a naive analysis will result in a 2.5 dB bias! Introduce non Gaussian, or heaven forbid, non monotonic noise and the results will be completely meaningless.
Probably one area where the statistics are extremely good is in actuarial work. Insurance companies need to predict the probabilities of various outcomes with some accuracy. These will always have the most up to date thinking behind them as it is the difference between profit and loss. They will try hard to eliminate any bias.
It would be interesting to know how their premiums are changing on large weather events, such as hurricanes, cold snowbound winters, etc.
Another important thing with statistics is to design the experiment so as to provide the outcome you want.
One widely reported experiment suggested shoot-em-up games made people more violent and agressive, but their reactions were measured as soon as they came off the computer when they would have had a lot of adrenaline in their system rather than a few hours later when they would be their normal self.
Obviously it was widely used as ‘proof’ about video games rather than just an example of poor sience.
And let’s not forget Disraeli’s famous comment about statistics:
“There are three kinds of lies: lies, damned lies and statistics”
-Attributed to Benjamin Disraeli (1804-81), British statesman and Prime Minister (1868, 1874-80), in:
Mark Twain (Samuel Langhorne Clemens), U.S. writer and humorist (1835-1910), Autobiography, “Notes on Innocents Abroad”
Evans wrote: “An example of misleading statistics:
Water vapor is around 1% of the atmosphere, a molecular constituent of air.”
Leif Svalgaard (09:41:53) replied: “Except that that statement is not an example of a statistics, but just a statement of fact.”
Has Science observed & measured for H2O at every location possible to confirm that indeed that H2O does “average” out to one percent?
No. but due to the confidence in Science’s understanding of average under the circumstances, Science understands what is present (physical properties) and operative (based on what we know) specific results will happen and/or conditions will exist.
Statistics is a branch of math, no more and no less! Limitations are due to the choices we make as statisticians.
Who can argue with the Deming Method? Seems to be well-proven over time. Statistical process control in automation is standard, we cannot manufacture in today’s environment without it.
However, in terms of scientific studies, I am frustrated by small sample sizes, sampling errors, mistreatment of outlier data, tagging on regressors etc.
Climate science seems to commit all of these sins, and many more, because it is policy driven, not driven by the need for accurate and replicable results.
@joel
‘Statistics is the science of the behavior of numbers.’
Rather the frequency of numbers, since the numbers themselves never behave. :p
On the rest you are correct I think. Properly used it is a very good tool.
Another statistics quote (from my degree course ages ago): “The generation of random numbers is far too important to be left to chance.”
I remember a study (with policy implications) that assumed (to avoid intractable mathematics) that once a female is pregnant once, that female is never pregnant again.
The folks selling this stuff were no slouches. If there had been a mathematically tractable way to make realistic assumptions, I’ve no doubt they would have. The complacent attitude that has taken deep root in mainstream science modeling culture: “We tried – now let’s just go with it”.
People working in policy need to weed out the stuff that fails in “trying too hard” (unsuccessfully) to appear objective. In its lack of sobriety, the drunken “publish or perish” mill has one reliable function: Water down quality.
i.i.d. = illusory inconvenience distortion …and they told you “independent identically distributed” ….daily – as the base assumption underpinning (literally) almost everything. “Let X1, X2, X3, … ~ i.i.d. …” — and so the web of mass deception began… (mortgage meltdown, climate alarmism, … – what’s next?…)
You can prove anything you want with statistics.
For example, Mt. Everest doesn’t exist. Mountain climbers always “cherry pick” their start location to achieve the appearance of an uphill slope. Had they correctly used the longest possible trend (the entire planet) they would have know that the average slope of the earth is zero and that mountains don’t really exist. If you shrunk the earth to the size of a pool ball, it would be smoother than a pool ball is.
That argument sounds and is ridiculous, but is exactly analogous to arguments used by people on all sides of the climate debates. When you hear the term “statistically significant” you can be pretty sure that someone is pulling a fast one.
Completely agree with Luboš Motl (06:26:43), hooolllyyyy cow.
This article is complete.. eh.. nonsense (..other ‘expressions’ actually came to mind). WUWT should stay sharp.
If anybody is interested in actual statistics, I encourage you to take a look at this thread here:
http://ourchangingclimate.wordpress.com/2010/03/01/global-average-temperature-increase-giss-hadcru-and-ncdc-compared/
Statistics just challenged this:
http://scholar.google.com/scholar?q=OLS+trend+temperature+climate&hl=en&btnG=Search&as_sdt=2001&as_sdtp=on
With statistics, just as like any other formal method, if something goes wrong, you usually get the pop-up: “Error, hit any user to continue”. This however does not invalidate this (FORMAL) discipline.
For a case study exemplifying this, and involving Tamino, see also:
http://ourchangingclimate.wordpress.com/2010/03/01/global-average-temperature-increase-giss-hadcru-and-ncdc-compared/#comment-1643
I read Luboš Motl’s rebuttal and found that it wound up saying almost the same thing as the “…..stands on a flimsy foundation” article. I think Siegfried’s point is that a large number of scientists place faith in a method that they don’t adequately understand and use properly. The scientific journal editors and peers similarly are weak in understanding and application so they let the stuff go through.
What has happened in the last half century or so is that the “soft” disciplines, which used to be verbal and anecdotal wanted to move into the glow of being a science. Political Science is the king of these beasts and psychology the queen. The texts of psychology used to be prosaic insights into why people behaved the way they do – similar to detective work.
To become sciences, they had to find a way to quantify all this jive. Statistics at the 101 level became the tool for quantifying and hey, if you asked a random selection of 1000 citizens who they were going to vote for, you could report with 90%(?) confidence and were most often correct. Tree rings and things are either confounded by several variables in addition to temperature. Or the right questions aren’t being “asked”.
James F. Evans (11:24:46) :
Has Science observed & measured for H2O at every location possible to confirm that indeed that H2O does “average” out to one percent?
It doesn’t have to. We sample the concentration at enough places that we get a good average number. It is only misguided people who thinks there is something wrong with science that can misrepresent this. Anyway, your statement is not a statistical inference, so is O/T.
Willis Eschenbach (10:48:02) :
“Instead, it says that most of the time people misuse statistics. I agree with that completely. In climate science in particular, we are dealing with non-normal datasets that have a high Hurst coefficient, which makes most statistics unreliable.”
We don’t have a ‘non-normal’ dataset with a high ‘Hurst coefficient’. We have a time series containing a unit root. Nothting ‘strange’ about that, many time series contain one.
I’ve been debating (more like a ‘war’) this with AGWH-proponents for two weeks now at the above link (take a look at the last one, extensive test results posted).
Really, to use ‘AGWH’ terminology, this is a complete ‘anti-science’ article.
Bah. Stay sharp WUWT.