The quote in the headline is direct from this article in Science News for which I’ve posted an excerpt below. I found this article interesting for two reasons. 1- It challenges use of statistical methods that have come into question in climate science recently, such as Mann’s tree ring proxy hockey stick and the Steig et al statistical assertion that Antarctica is warming. 2- It pulls no punches in pointing out an over-reliance on statistical methods can produce competing results from the same base data. Skeptics might ponder this famous quote:
“If your experiment needs statistics, you ought to have done a better experiment.” – Lord Ernest Rutherford
There are many more interesting quotes about statistics here.
– Anthony
UPDATE: Luboš Motl has a rebuttal also worth reading here. I should make it clear that my position is not that we should discard statistics, but that we shouldn’t over-rely on them to tease out signals that are so weak they may or may not be significant. Nature leaves plenty of tracks, and as Lord Rutherford points out better experiments make those tracks clear. – A
==================================
Odds Are, It’s Wrong – Science fails to face the shortcomings of statistics
March 27th, 2010; Vol.177 #7 (p. 26)
For better or for worse, science has long been married to mathematics. Generally it has been for the better. Especially since the days of Galileo and Newton, math has nurtured science. Rigorous mathematical methods have secured science’s fidelity to fact and conferred a timeless reliability to its findings.
During the past century, though, a mutant form of math has deflected science’s heart from the modes of calculation that had long served so faithfully. Science was seduced by statistics, the math rooted in the same principles that guarantee profits for Las Vegas casinos. Supposedly, the proper use of statistics makes relying on scientific results a safe bet. But in practice, widespread misuse of statistical methods makes science more like a crapshoot.
It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.
Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.
Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.
“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”
Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”
Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense.”
====================================
Read much more of this story here at Science News
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Veronica,
Chances are that temperatures will remain above the GISS average for the indefinite future, even if there is no further increase in temperatures. This will potentially allow Hansen to paint his maps red – forever.
Steve Goddard (16:45:44)
I will answer this on the relevant thread …
Willis,
There is a clear UHI trend in Fort Collins and less so in Boulder. You chose to dispute it because of some irrelevant statistics generated from your spreadsheet. Meaningless statistics is the topic of this thread.
bob (19:57:06), if I may have a stab at answering your question, I think you misinterpret your quote. It is the variance that diverges to infinity, not the series.
Steve Goddard (18:36:05) : edit
I have never denied that both have UHI. It was your claim that the trend started in 1970 that I denied. If you wish to claim that my statistics are “meaningless”, you have to do so mathematically. Nothing is more statistically meaningless than a “because I said so” claim regarding statistics.
Willis,
Give me a break.
You littered the thread with graphs like this one generated from your spreadsheet, which attempted to show that the divergence was linear since 1895, implying that it had nothing to do with UHI.
http://homepage.mac.com/williseschenbach/.Pictures/boulder_ft_collins_temps.jpg
You can also drive yourswlf nuts like i do by picking up self-referencing sentences. Example: “This sentence no verb”.
Here in the leader we have a Dr Goodman quoted as “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”
If the number is more than anybody appreciates, and because he is an anybody, therefore he cannot appreciate that there is a question about it.
You can keep that one, it’s a good one.
Geoff Sherrington (22:17:26) :
“This sentence no verb”.
“This sentence is false”
Epimenides from Crete claimed that “all Cretans are liars”.
Anthony, heads up:
“Improving Predictions of Climate Change and its Impacts: Media Briefing
Wed, 17 Mar 2010 15:05:00 -0500
NSF invites reporters to participate on Monday, March 22 at 11:00 a.m., EDT
On March 22 at 11:00 a.m., EDT, officials from the National Science Foundation (NSF) and the U.S. Departments of Agriculture and Energy will discuss the launch of an interagency program aimed at generating predictions of climate change and its impacts at more localized scales and over shorter time periods than have previously been possible. This project represents an historic augmentation of …
More at http://www.nsf.gov/news/news_summ.jsp?cntn_id=116601&WT.mc_id=USNSF_51&WT.mc_ev=click
This is an NSF News item.”
Steve Goddard (20:25:49)
Not sure what your point is here, Steve. The graph is accurate, it shows the original data including any possible discontinuities. Once you take out the January 1942 discontinuity that we both agree is there, it looks like this.
Sure looks like a trend beginning to end to me. If you are saying those graphs are not accurate, post up your own.
I don’t know why the temperatures of the two towns have diverged for the last century plus. Part of it is quite possibly UHI, but the population trends don’t explain it, they don’t diverge until 1970, and before that Boulder was growing faster. I don’t know why they diverge … so sue me. You tell us why they diverge since (at a minimum) 1942.
I see no mathematical evidence of a 1920 discontinuity. What month of 1920 do you think it occurred in, and how large do you think it was? And speaking of statistical analysis, why do you think that the residuals test for discontinuities doesn’t show any 1920 discontinuity, as demonstrated here?
w.
Willis
I will assume the actual location of both weather stations cited in the graph has been cheked and that one hasn’t accidentally been installed in the kitchens of McDonalds 🙂
We have a similar disconnect locally in some of our coastal towns. One town has the coast next to it AND an estuary so is in effect bounded on two sides by water. The other is bounded by the sea only. Inland are the Highlands of Dartmoor which has a dramatic effect on weather.
Rain and sunshine and cloud are highly dependent on the direction of the winds. Our predominant winds are south westerlies which affects both town pretty equally.
If another wind direction predominates (as has happened for long periods in the last half century) -particularly one coming over the Dartmoor land mass -this has a fundamental effect on the local weather and affects one town much more than the other.
I’ve no idea of the topgraphy of the two towns you cite but could changes in prevailing wind, plus Uhi and poor location of the weather stations provide some part of the explanation?
Tonyb
So let us forget that instrumental error exists.
TonyB (01:38:49) : edit
I fear I haven’t a clue, ask Steve, he’s the one that’s making the claims. I’m just analyzing the numbers …
I know quite few scientist from bio/chem world, and as mathematician i can tell you they have no clue about statistics and they just apply random statistical tests to their data without understanding and consideration of quality and independence of data…
they just care for ‘significance’ which may or may not be there as test does not make sense quite often. If you don’t understand statistics 100% of the time don’t bother with it as you will get things wrong for sure.
http://www.timesonline.co.uk/tol/comment/columnists/guest_contributors/article7070310.ece
Willis,
With both discontinuities (1920 and 1941) removed the graph makes sense. Flat until the mid-1940s when the populations started to grow rapidly. That is the point of the UHI article and I am done trying to explain this to you and your spreadsheet.
https://spreadsheets.google.com/oimg?key=0AnKz9p_7fMvBdElxNDA4Vlh2OGhvOUdEX1N0bm1CeWc&oid=6&v=1269263839043
Steve Goddard (06:21:02) : edit
Well, since I asked you how much the discontinuity in 1920 was, and what month it occurred in, and you haven’t answered either one, I’d say you haven’t even started “trying to explain this” as you claim.
Next, the point of your UHI article was not that the difference between Boulder and Ft. Collins was “Flat until the mid-1940s when the populations started to grow rapidly.” That’s historical revisionism. You started with a chart showing 1970 on, and the statement that:
Last forty years, that would be since … umm … 1970. That’s where you started. You concluded by saying:
In other words, you didn’t say a steenkin’ thing about 1940 in your article. You claimed the temperature difference started in 1965 (not true), and that it was due to Fort Collins faster growth post 1965 (also not true, otherwise it should have gone the other way pre-1965). And you want to lecture me on the misuse of statistics?
Finally, the point of your whole article was that it was the difference in the two cities’ growth rates that was the cause of the growing temperature disparity.
But if you now claim that it started in 1940, during 1940-1970 the disparity is reversed. Boulder grew faster than Fort Collins during that time (see above) … so according to your claim, Boulder’s UHI should have grown faster 1940-1970 as well. But it didn’t, so your theory about relative population growth is dead on arrival.
Do I think that there was UHI in Boulder and Fort Collins? Yes, and I think it was greater in Fort Collins than in Boulder. But you can’t tie that to population growth as you tried to do. The math simply doesn’t work.
Willis,
Drop it, please. Fort Collins has grown much faster than Boulder, particularly around the weather station. As a result, Fort Collins has seen much more UHI effect. That is exactly what the data shows. The math is fine, it is your use of statistics that is the problem.
H.R. (06:38:17) :
channon (03:54:04) :
“Yes pure math gives what appears to be the comfort of certainty and although many pure scientists believe this to be absolutely true, most philosophers can show that no system of logic is both complete and consistent.
That being the case, that purity is only relatively true and there is an element of uncertainty inherent in all calculations and proofs. […]”
Sooo… when is it that 1pebble + 1pebble doesn’t equal 2pebbles? Did I make an error in logic somewhere?
He kicked the stone but missed the point………….
Well what has always seemed odd to me, is that everyday, it is a different experiment to measure the local temperature; so naturally you expect to get a different result from the experiment you did the day before; after all, weather comes and goes.
So what is the point of averaging the result of two entirely different experiments done under possibly different conditions. The answers are supposed to be different, and the average isn’t any more correct for either of the experiments. I don’t mind averaging the result of different runs of exactly the same experiment, run under identical conditions. The average of all of the people on earth doesn’t look any more like anyone you know, than anyone else does.
Statistics is mostly creating “information” where there is none.
One of the great debates in the election arena is whether statistics can be used to verify elections. In 2004, Kerry was predicted to be the winner by three percent based upon exit polls (of about 14,000 people nationwide as I recall — very large sample). Yet Bush was declared the winner by three percent.
The reason given by the pollsters for the discrepancy was the Bush voters declined to talk to pollsters more often than Kerry voters declined to talk with them.
On the other hand, somewhere between 75 and 80% of the votes were cast on computers with no paper trail to audit the vote. The computerized votes were counted by four or five major computer election companies several with strong ties to the Republican party. Public electioneers had turned the elections over to private corporations and public electioneers had no capability of their own to check the equipment or the software.
So who really got the most votes in 2004? Depends on who you ask. There was a huge fight between pollsters who refused to release their data to the public and a vast number of statisticians who believed the data had proven Kerry had won and the computerized voting systems were full of systemic fraud, made unprovable by no paper trail.
So is exit polling a proper use of statistics or is it not? Can it verify elections or not? Depends on who you ask and what interests they have in the outcome.
But next time you vote on a computer without a paper trail to audit it, remember trusting the software engineer to register your vote correctly is like a blind man trusting a person he does not know to mark his ballot correctly.
I mention this because if this can happen with elections, it can certainly happen with the use of statistics in other areas where a political agenda is at stake, such as climate.
Hi George how are you? Been a while.
Statistics are only properly applied to “random” processes in some way, or applied to determine what, if any, meaning “random” has. That’s all they are.
davidgmills (17:18:05),
I had not heard that computer voting companies were accused of throwing the Bush-Kerry election. [Personally, I’m all for paper ballots – with thumbprints.]
But somehow I doubt that an explosive secret like rigging a national election could be kept between competing companies, with their respective machines in tens of thousands of precincts, and considering the number of people who would have had to be in on the scam. Two people can keep a secret. Three, rarely. More than three, and you might as well blog about it. But I suppose it isn’t impossible.
That also recalls the Washington state election, which I followed at the time. Christine Gregoire lost to Dino Rossi in the first vote count by a couple of hundred votes.
State law required a re-count if the vote margin was less than 2,000, IIRC. So they did another machine count. Rossi again won, but the margin was reduced to around 40 votes. I don’t understand how computer voting can come up with different numbers in an identical machine re-count.
Anyway, Gregoire’s supporters paid for a third recount. [John Kerry personally paid $250,000.]
Surprise! Christine Gregoire ‘won’ by a hundred and some votes. She was promptly sworn in, even though Rossi produced evidence that hundreds of convicted felons had voted illegally.
Same thing happened in that long drawn out Coleman-Franken Senate race. When the votes were counted, Coleman won by a small margin, I forget how many votes exactly. Then, after more recount shenanigans in which it was shown that over 2,800 deceased individuals had ‘voted’ [the ballots were traced to an ACORN group; but no charges were ever filed], Al Franken was finally declared the winner.
Looks like some folks have learned how to game the system. That’s the new millennium democracy in America, where our elected representatives are “deemed” to have voted to pass legislation, without having to actually vote.
Steve Goddard (12:19:23) :
“Willis, Drop it, please. Fort Collins has grown much faster than Boulder, particularly around the weather station.”
Steve, that’s a rather qualitative statement. It is not common to find a UHI differential of 4 deg C in under 20 years. Might happen in China when a new megapolis was built in the bush, but it’s quite hard to see that size of change at Boulder. Any idea as to the secondary mechanism that population growth caused? From data I have looked at, which is not many cities, I rather feel that population growth in big cities increases the area of the UHI without much affecting the temperature, which seems to plateau or climb very slowly around the middle of a 1 million people city.
Geoff,
I have a thermometer on my bike, and ride in and out of downtown Fort Collins (where the station is located) all the time. If the wind is light, I always see at least two degrees difference and have seen as much as five or six.
It is very easy for me to believe the graph which shows 2.5 degrees of warming in Fort Collins, and the Colorado State Climatologist has told me that he also believes most of the warming is due to UHI.
REPLY: How do you mount the Stevenson Screen or MMTS on your bike? 😉 -A