Below is an excerpt from an excellent article in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:
Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.
Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF
h/t to WUWT reader Edward Lowe
=======================================================================
This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
If I may, I propose the name for this could be: confirmation entropy
sHx says:
January 20, 2011 at 10:02 am
dave38 says:
January 20, 2011 at 8:47 am
W Abbott says:
“….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this
It was said by Ernest Rutherford.
Wasn’t it also Rutherford who said “All science is physics. The rest is speculation”?
_________________________
It does seem to come down to that…
climatebeagle says: Any idea if “Tony from New York” was a real scientist?
…No, I don’t, but it will be interesting to find out… especially if he is!
The BBC is continuing its tradition on this issue of never letting any sceptical scientist anywhere near a camera or microphone. Rules of impartiality be damned! Let the AGW shill have the entire program to himself!
JDN,
You take the hate-filled emanations of PZ Myers as rational? That article was unconvincing.
Pharyngula caters to the disgruntled [and there are a lot of them], so it was probably fodder to that unhappy subset of the population.
In the most recent Weblog Awards for the internet’s Best Science site, it became more and more clear that WUWT was winning, so the climate alarmist blogs started piling their votes into Pharyngula, because even that blog was ahead of them.
Pharyngula lost, but it would not have done nearly as well without all the alarmist blogs monkey-piling their votes onto Pharyngula, to try and keep WUWT from winning. They openly discussed their strategy on RC when it was apparent they were going to lose badly. Which they did, as the link shows.
Normal folks can’t take much of PZ Myers. It’s hard to believe, but Myers is often more wacked out than Joe Romm.
Well… almost.☺
Look up Michelson and the speed of light. It slowly drifted to the correct value over time. Why? Well Michelson was thought to be a very careful experimenter. Confirmation entropy indeed.
Murray Duffin says:
January 20, 2011 at 7:38 am
“Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.”
This wide discrepancy suggests to me that acupuncturists in the US, Sweden and the UK are no good at it.
A couple of Feynman quotes:
“The first principle is that you must not fool yourself – and you are the easiest to fool.”
OT, but relevant to the lack of a falsifiable global warming hypothesis:
“I believe that a scientist looking at a nonscientific problem is just as dumb as the next guy.”
I miss Feynman more and more. And Ike, too.
—I recommend a new book by MELANIE PHILLIPS, who writes for the Daily Mail and the Spectator, on this woolly subject: THE WORLD TURNED UPSIDE DOWN, published by Encounter Books last year.
She covers all the scares which have had us in their grip over the past few years, pointing out that we are really not thinking straight these days and far from being rational as we all believe, we’re headin back to superstition. One chapter, for instance is ‘How Enlightenment Unravelled’
JPMiller hit the nail on the head. There are a lot of individual scientific phenomena that we understand quite well. Gravitational effects near the surface of the earth, for example. Galileo showed that a feather and an iron ball are both accelerated at the same rate. However, wher you try to do an experiment by dropping them from the same tower at the same time, the results contradict the fundamental scientific principle that the “consensus” accepts, because there are more phenomena at work here than just acceleration due to gravity.
In the case of medicine, ecology, and psychology, and many other fields, it is not possible to isolate one phenomenon and then make a broad conclusion based on your discovery about that one phemonenon, because there are usually many phenomena at play in determining an outcome. We are a long way away from understanding enough about those fields to really call them “science”. Rather, they might best be called proto-sciences, where the facts have not yet come together enough to be able to make real, testable predictions. In some cases (e.g., the social “sciences”) I suspect that this time is far in the future, and these fields should just be called speculation.
JDN – Meyers simply confirmed Lehrer’s article.
This does turn out to be a fine article, though it nearly lost me at the beginning by using the alleged superiority of the second generation of psychiatric drugs as a prime example. Those particular drug trials were simply rigged — not merely my opinion, but the FDA’s, which forbade the drug companies to advertise them as superior to the first generation. But the drug companies have bought so much of the psychiatric establishment, they didn’t have to say it in ads, their paid doctors said it in interviews with all the big publications, and the false message got spread. Now that the second generation is getting old enough to lose patent protection, we are allowed to “discover” it wasn’t ever that good, and rigged tests are being prepared for a third generation’s patents and profits. (See Robert Whitaker’s Anatomy of an Epidemic.)
So the author does under-estimate the influence of lying and corruption. Yet by the end, he does persuade me that even without those factors, we would see problems.
[snip – let’s not get into that argument related to Palin – Anthony]
Pat Frank says:
January 20, 2011 at 10:05 am
“Most of the time, scientists know what results they want, and that can influence the results they get.”
Total crock. Written by someone who has never done an experiment.
Written by someone who has never done a grant applications Pat??
Perhaps the most important implication of this excellent article is the failure of the self-correction peer review feedback loop to act early enough to cancel the influence of faulty research.
That assumes there is any impulse at all to question fashionable results. The fact is that some researchers are able to insulate themselves from any real attempt to probe their findings by pronouncing “ex cathedra” that their conclusions are too urgent and significant to be put to the test.
With the enviro reporters covering this being urged to “forget objectivity and go with your gut” the results, I theorize, are predictable.
Remember when the consensus said that complex carbohydrates were good…?
“A reversal on carbs – Fat was once the devil. Now more nutritionists are pointing accusingly at sugar and refined grains.”
http://www.latimes.com/health/la-he-carbs-20101220,0,6026869,print.story
Micky H Corbett says: “…Scientific Method is a lot like learning to play golf…”
Yeah. Looks like Michael Mann has been spending too much time working on his putts.
When does the ‘pitch’ for (healing) ‘crystals’ take place in that video?
‘morphic resonance’? turns out to be a term coined by Rupert Sheldrake in his 1981 book “A New Science of Life” … the expression to refer[ring] to what [Sheldrake] he thinks is “the basis of memory in nature….the idea of mysterious telepathy-type interconnections between organisms and of collective memories within species.”
Really? Believe in teleconnection much?
Mann et al 2007 Precipitation Teleconnections
.
“The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand. ”
If Crabbe had read Dougleas Adams he would realise that it was the white mice that are conducting the experiment , not the humans. They do this sort of thing and then analyse how we deal with it.
As I see it the problem is as it always has been. Humans expect their enquiries, whether scientific or not, to provide them with answers. The reality is that the best we can hope for from them is the ability to ask better questions.
As a scientist, I absolutely agree with that article. It is completely true. Repeatability is one of the founding pillars of science, and anything unreproducible is not science; just a phenomenon or observation (the lowest level on the scientific method, before hypothesis).
The truth is, a lot of papers are published and only YEARS LATER does another lab come along and reproduce the results. It is when this occurs that suddenly it’s found out that things weren’t what they seemed as of the first paper. The antidepressants are a perfect example. More research was done only when it became apparent the original work wasn’t living up to expectations. And low and behold, it was true.
But there’s another disconnect. Sometimes something in the lab is completely reproducible, but ONLY IN A SPECIFIC INSTANCE AND CASE. Only when the variables are exact, only when the absolutely same buffers and concentrations and test organism are used do those results occur. Once you transition from the lab to the real world, suddenly things no longer work like you thought. Even clinical trials, which are what test this transition from lab to reality, can be biased by picking a unique subject pool, and only that subject pool gets the results, while the majority of everyone else differs.
This is the hard fact of science, and why medicine is still such a black box. In science we control the variables very exactly, but in the real world the variables have high variability and even worst are often changing on the fly–rather than being held steady. And there is where so much of the disconnect creeps into the scientific literature, and when going back with new machines and testing old procedures can suddenly grant you completely different results.
Heck, in computer modeling, even the very processor you are running the model on will change the results, sometimes significantly so.
So, I believe in the end, this dichotomy of lab and variables verses nature and reality, coupled with science sometimes being lazy (you don’t get funding for doing the same experiment as someone else! You only get funding for new experiments!) about reproduction of results, is what is leading to this current shake up of all our “facts”.
The world is just so much more complex than a particular lab…
One of the most perceptive observations about empirical science that I’ve ever encountered was the following: “Science is not like going to the supermarket”. (Unfortunately, it was in an article in a popular magazine that I’ve long lost track of – in a discussion on a dubious esp experiment done with a radio audience [!] – so I can’t cite the reference.) What the author meant (which I only appreciated after a time) was that the world does not come neatly packaged like goods in a supermarket; so the image of Science as a kind of consumerism where one wanders down the aisles and selects items of knowledge from the shelves is a travesty of the actual process of gaining empirical knowledge. Unfortunately, (and there are many reasons for this, including the great success of Science and the widespread learning of causal explanations without an appreciation of what goes into their derivation) too many people have this view of science. The problem becomes worse when empirical methods are extended to areas (like epidemiology and climatology) where really adequate control is simply impossible, without appreciating how difficult it becomes to establish anything. This is certainly part of the reason why “facts are losing their truth”; our shallowness is catching up with us.
This is also a very important point. The public is all too eager to accept prepackaged sound-bytes from experts on how to handle their daily lives. Very few actually dive into the details where the real understanding comes from. It would easily appear to the layperson that “our facts are losing their truthiness” when in fact the truth they had subscribed to was simply the label on the package. They had been considering a bazooka-joe gum wrapper of factoids to be gospel and they’re only now coming into contact with reality.
As the article indicates, replication of results, by independent investigators, is important to generate confidence in ‘scientific’ findings but there can be pitfalls.
Some while ago, I was asked to ‘teach’ Design of Experiments in a two-hour slot to 1-year Masters Courses in Design, Manufacturing and the like. I thought those in receipt of this might have the best chance of remembering something for their future careers if I made most of it ‘hands-on’.
Thus I ran a quick experiment based on an 8-run, 7-factors at two levels, orthogonal array. It is by the way but this also gave the opportunity, if not the time, to debate the pros and cons of Taguchi Methods.
The 7 factors, related to a spinning top, were disc diameter, disc thickness, height of disc underside above spinning surface, radius of spin point, length of grip shaft above disc surface, radius of grip, whether grip knurled or smooth. The response was the spin time until the edge of the top first touched the surface. At the end, I summarised the individual experimental sub-groups’ results to show the size of the effect (increased response time) that each had obtained for each factor. After all, they had all done the same experiment.
In general, although the size of the effects (in seconds) often varied “between sub-groups within occasions (lecture slots)” and also between “sets of sub-groups between occasions”, the ranking of the relative importance of the design factors of the top was reasonably consistent over time. Until one day……
At this point, I should say that I don’t have the results immediately to hand but, from memory, disc diameter had the largest influence on spin time, disc thickness very little with length of the grip shaft also negligible. [Cue discussion on how to optimise material costs to maximise response time.]
As I was saying, one day amongst the usual pattern of results, two young ladies (females I think is the politically correct term) to their consternation posted the finding that a long spin shaft gave the longest spin time with most of the other factors negligible.
To their everlasting credit, they were concerned about this (though perhaps they wouldn’t have been if they didn’t know about the other replications and only had their own results to go on – a not uncommon occurrence methinks). They borrowed the equipment to research the matter further in their own time and reported back to me that they both had long fingernails which prevented them getting much spin-time at all out of tops with a short spin shaft.
Obvious, isn’t it! ?
P.S. The experiment was done as an empirical one like so many in industry have to be, though I appreciate that there is a body of theoretical knowledge in the mathematics of gyroscopes.
Anthony,
Thanks for brining this study up. I keep forgetting that many individuals feel that data mining and correlations prove causation- (at 95%CI)- vs association. The comments reminded me of the thoughts of Peter Checkland in his seminal (for me anyway) book copyrighted in 1981 entitled “Systems Thinking, Systems Practice”
http://www.amazon.com/Systems-Thinking-Practice-30-Year-Retrospective/dp/0471986062
I highly recommend a review of his work as it covers the hard and soft sciences in the context of model building.
“This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.”
Could the origins of the phenomenon be blamed on cheap Hindu Kush, I wonder. o_O
W Abbott says:
January 20, 2011 at 3:43 am
“….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.
think it was Dr Michael Kelly at one the parliamentary inquirys
All your facts are belong to us.