Below is an excerpt from an excellent article in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:
Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.
Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF
h/t to WUWT reader Edward Lowe
=======================================================================
This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
If I may, I propose the name for this could be: confirmation entropy
Fortunately for everyone, now that this phenomenon has been described and discussed, it will gradually happen less in the future…
Nature can be cruel. This may be why AGW “proof” is based entirely on computer models. With models, all of the input can be defined and there is no chance of empirical data getting in the way.
It seems obvious to me: the laws of the universe are mutating.
Has anyone considered the possibility that in at least some cases, this is a real effect and not the result of confirmation or publication bias?
I’m reminded of something that might be its complement, namely morphic resonance. If you can spare around 30 minutes of your life to listen to an interesting lecture on this topic, check out:
With respect, I think you missed the main point of the article. You are assuming that an unknown factor exists, when it’s also quite possible that all that happened is that the Edmonton result was in the far tail of the distribution of possible results. For that matter, the other two could also have been in the far tail, but in the other direction. The norm of the unknown, but actual, distribution of results might be, say, 2,000.
That’s the point of regression to the mean. If the mean of the actual population of results is 2,000 then, over time, replication will drive away the original “discovery” of an “Edmonton Effect,” whatever that might be.
And to take the point of the article further, if you could happen to identify one tiny little difference in the layout of the Edmonton experiment, the publication bias toward finding unexpected results might even get you published. Then others would do the same sort of experiment, with the null results not getting published, while the few supporting an “Edmonton Effect” do, though showing a diminished “effect.”
Meanwhile Edmonton fills up with coke heads looking for that special “Edmonton Buzz” and junk science destroys yet another economy.
All in all, a very interesting article.
“But in the Edmonton lab they moved more than five thousand additional centimetres. Similar deviations were observed in a test of anxiety.”
This goes to show the lengths test subjects will go to to get out of Edmonton.
Re Abbott’s comment:
““….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.”
I have heard this attributed to Rutherford, but I haven’t seen a citation to support that.
[Reply: It was the late British physicist Ernest Rutherford. ~dbs]
@MattN January 20, 2011 at 7:44 am:
Yes, scientific research’s history can be divided into two periods – pre-Manhattan Project and post-Manhattan Project.
In comment to Pamela’s comment…
The intolerance in science of other views was birthed by the response to creationists. There has long been an absurd fight between Biologists who are certain that random processes in ordinary organic chemicals billions of years ago led to consciousness and religious crazies who are certain that a magical hand from the sky shaped dirt into the first human being. It is a stupid fight, it’s always been a stupid fight, but neither side sees the reasoning that BOTH can be correct. Science can only tell us what we can prove, and we cannot prove/disprove the existence of magical intelligent creator, so there truly is no conflict. The whole argument would have died immediately 200 years ago if some scientist had simply kept his mouth shut and said, “Yes, uh-huh, I see how you could see it that way.” and moved on. But those of us who use our brains too much tend to impose our views on others when they appear to be wrong. Yes, human nature causes problems; sun rises in east; water is wet.
What is unforgivable is the degree to which this fight has been ratcheted up. The religious people feel like they’re being attacked, and I don’t blame them so much because they’re the unarmed people in this fight. They’re the ones who insist on believing in Santa Claus, so it’s best to just let them have their harmless beliefs rather than force a certain viewpoint on them. The scientists feel like science is being attacked, and I don’t blame as school boards have held actual hearings to determine whether an unprovable “creationism” should be taught alongside calculus.
The religious people responded to this fight how they always do, they threw money at clever people to make their case for them in a political theater.
The scientist response, however, is what should have been different. The scientists turned into activists. Regardless of how wrong someone else is, you cease being a scientist when you tell someone else what to think, you become an activist altering perception in others. As a scientist you should be trying to make the obvious plain to someone, while continuing to let them determine their own reality. The truth has a way of sneaking up on people, or smacking them in the face; but the perception of truth does not obey the whim of anyone even the smartest communicator on the planet. Scientists were choosing a horrific battle when they decided to directly battle overt human stupidity, and Einstein warned them a long time ago about it. Science *will never* win the battle against creationists or any other battle in the political arena. Let me repeat that, Science *will never* win any battle in the political arena. The reason is simple, in order for scientists to win a political battle, they must CEASE being scientists and become activists. It has happened to countless scientists thus far, and many more are seduced daily into becoming activists, rather than simple investigators of the universe.
What’s worse, is that it is now clear that this fight against creationism has made the idea of being a scientist/activist the trendy thing to do. Suddenly all these pompous PhD’s with desires for fame have a way to make a name for themselves without writing a single worthwhile paper. All they have to do is make a “principled stand” against the “Darkness and danger of belief in _____ (Insert cause here),” and the fame can roll in. You can even pick up funding this way, but who cares if you don’t? With fame you can simply write a book, throw PhD after your author name, and make money that way. It used to be the case that scientists remained locked away from the media because the only thing less palatable to a scientist than being misquoted, was being misquoted publicly by a liberal arts major.
And this is why we arrive here, ready to move on to… something. It is now OK for scientists to become activists and treat opposing scientific views in the media as if they are of the same ilk as creationism. This is because it is now OK to be a scientist and activist and never hold two opposing viewpoints in your mind at any time (something that good scientists do REGULARLY). It is all a response to trying to poke the eye of creationists. It’s stupid.., and now I’m rambling off the rails..
W Abbott says:
“….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this
It was said by Ernest Rutherford.
1. ScientistForTruth:
Quit picking on Euclid. What is taught is a couple of thousand years of revision of Euclid. The Muslim mathematicians did a lot of work on the 5 postulates. Much has been discovered since Euclid himself. Get the 3-volume Dover Euclid set and enlighten yourself. No one teaches Euclid’s formulation of the Fifth Postulate anymore.
2. etudiant:
Three sigmas? I’m sorry, but 95% confidence is not 3 sigmas.
95% is two sigmas. 3 sigmas is 99.7%.
And all data sets are not necessarily normal, either.
Please get your stats straight.
3. General observation: the lurking variable. A lurking variable is an input to the study which is not documented, not studied, and not included. In the social sciences, such lurking variables are very difficult to identify.
It is bad enough in the hard physical sciences to control all but one variable. Just ask a structural engineer about high winds and earthquakes.
The New Yorker article vividly illustrates the problem with lurking variables, chief of which is the expecation of the person establishing the study and setting forth the parameters for data collection.
In the alleged field of climate science, the lurking variable is, of course, the consistency and reliability of recorded temperatures. What happened to all of those sites which are no longer included in GISS? What, precisely, are the details of the inferential statistics used to populate the unsurveyed locations with temperatures? What, precisely, is the relation between max/min and average temperature? How is it even conceivable to infer temperatures from proxies?
What the *** is going on with the Sun? I know She is variable, but the dearth of radio emanations is quite striking. My Sky and Telescope is suspiciously silent on the question. Galileo was accused of seeing what was not there during the Maunder Minimum. His observations could not be replicated, as the Sun went very quiet. It got really cold from 1660 to 1715, too.
While I am on the subject, what is the whole carbon dioxide budget of the Earth? How does the absorption/release of carbon dioxide by the oceans depend on the PVT of the bordering atmosphere and the ocean temperature? What are the time frames for absorption/release of heat energy (all frequencies) for all gases in the atmosphere, and how do these vary with altitude and latitude?
And while I am on the subject, where is that computer model which will predict cloud cover?
And we are unlikely to learn much, due to the substantial lack of data. Just do a quick calculation of the surface area of the Earth, and figure out how many data points one would need for one each square mile or even each square kilometer. And then figure that one would need vertical data, up to 100,000 m. The cost of such a grid is out of sight!
A part of the problem, maybe a large part, is the constant attempt to reduce chaotic systems to predictable systems using simplistic, reductionist model.
Whether looking at the behavior of climates, nervous systems, or computer software, one needs to respect the unknowability of the state of the system and the immeasurability of the interactions of the many subsystems (each poorly predictable). Some things cannot be known. Quantum Mechanics has been able to quantify this “unknowability” in the realm of the very small.
In the realm of the large there is no corresponding precise mathematics. Instead, proponents of various “theories” try to glosss over the simple truth that often they do not know what they are talking about. Skepticism should be proportional to complexity and the chaotic nature of the subject.
Matt:
Achtung! Es ist Eisenhower!
No, there is nothing wrong with the Scientific Method. However, everything is wrong with Fraud.
I would venture to suggest that this is not a new phenomenon. Researchers have always interpreted their results in ways that favour their view of life and how the world works. They may also have interpreted them in ways that favour their own researches and careers. That’s the whole point of replication that the same work is done by different people who have a different take on things.
It’s more obvious now for three reasons. One, the internet and the more rapid dissemination of science and the easier access to previously published work. Two, increasingly research focuses on smaller and smaller changes in more and more complex systems. Proving to everyone’s satisfaction that an apple, when released, always falls towards the centre of earth is not complex and there are few factors involved. Proving to anybody’s satisfaction that anthropogenic CO2 emissions have warmed the earth is extremely difficult given the complex nature of climate influences, temperature measurement over the globe and doubt over earlier temperatures.
Three, the rewards for making exciting findings is much greater than it ever was.
Jeremy certainly made the best comment on this discussion. What annoys me is that most folk don’t seem to understand the philosophy of science. Even something as simple as distinguishing between a theory and a hypothesis. Both are explanations but theories can be tested unlike hypothesises. A prime example is Darwin’s explanation of the diversty of species. It is a hypothesis. It cannot be tested. This doesn’t mean it’s wrong. It’s simply an explanation that seems to work for the moment. The same is true of many theories (eg Einsteins relativity theory). They have passed the test and seem to work but this doesn’t mean they are correct. There’s always a better theory waiting to come along which will pass more tests. AGW is a hypothesis. There is no way to test it. We don’t have the evidence in any reliable form. Even our current readings of the Earth’s temperature are open to debate. We even lack a sensible theory about how the Earth’s climate actually operates. As such, predictions are merely guesses in the dark. I’m a pessimist so my money is on the next ice age arriving pretty soon.
If you give a 100 question survey to a random sample to see if they are different from the population at large, you expect to have 5 of the questions show a significant difference (at the 95% level). Thus if only 5 out of the 100 show a difference you shoud conclude that they are spurious and conclude that none of the questions truly show a difference.
We have so many people doing so many experiments that as a society we are in the position of the person doing the aforementioned survey. We expect a large number of studies to show siginificance at a 95% (or whatever level you choose) even if no real effects are there. This will result in many spurious false positives from the studies.
We cannot even measure the effect of this as the many studies that do not show a positive wil not be published, and may not even be written up.
It’s like the person doing the 100 question study just published the 5 that showed a difference and neglected to mention the 95 questions that showed no difference.
Normally, we would expect that with replication the spurious effects would be weeded out. But 5 of 100 spurious results will again show positive results when retested. and with the number of studies done, 5% of 5% of 5% still leaves a lot of spurious results out there accepted as fact.
Then when you add confirmation bias and friendly peer review, which waters down the effect of replication, It’s no wonder that many findings are spurious.
@Steve Garcia says:
January 20, 2011 at 7:26 am
Thanks for the comment. It helped crystallise some of my own views on the matter?
Archeology is definitely not a science. It is certainly a perfectly legitimate field of intellectual inquiry, but Archeology along with its cousin, History, are not sciences like Physics and Chemistry. They may follow scientific procedures, but only to a limited extent. In Archeology and History, one cannot run experiments. It is not possible to re-create the past events in controlled fashion to see how the outcomes respond to changed conditions.
This is not just common sense, it is also taught in junior high school (where I first learned of this).
Now, one of the most common defensive arguments of the CAGW science is that we do not have another Earth where we can run controlled climate experiments. The fact that there is no other Earth to use as a test subject means that climate predictions for the one and only Earth may not be as precise as those predictions based on sciences that benefit from lab experiments. Hence, 95% confidence levels, which is supposed to prove sufficient for Climate Science.
There are two problems with this. Firstly, if we do not have a second Earth to run a climate experiment, then that counts against Climate Science. Just like the impossibility of replicating the Trojan Wars count against Archeology and History as proper sciences. Some aspects of Climate Science are lab-tested and are indeed basic physics, but the model as a whole has not been tested. We still don’t, for example, what the climate sensitivity for per doubling of CO2 is, or whether feedbacks are positive or negative. Hence we have a vast level of uncertainty that can be interpreted either way depending on one’s political views, just like in Archeology and History.
Secondly, 95% confidence levels is not always a good enough level for scientific certainty. Evidently, 95% is huge level of certainty for some sciences such as Medicine. But, to my knowledge, -I’d be happy to be corrected on this- certain areas of Physics, eg, Particle Physics, seeks statistical certainty at 99.98% level while running experiments.
One day perhaps, technology will enable Climate Scientists to re-create the dynamics of the Earth’s atmosphere in a chamber where they can experiment and learn from it. Like the much maligned, much delayed, and much awaited CLOUD experiment that will hopefully help us understand how clouds are formed.
Until that happens Climate Science is not basic physics and is as hampered as Archeology and History in terms of scientific standing.
The article first discusses the disappearing efficacy of second generation anti-psychotic drugs. That problem is well known.
All studies have a baseline, and some commentators believe the problem results from comparing every new generation of drugs to placebo. The first generation drugs were proven to be more effective than placebo in double blinded trials, and so were the second generation drugs. So how does one determine whether the second generation is more effective than the first, when the two have not been tested head to head? There doesn’t seem to be much of a basis on which to make claims of superiority, beyond researcher enthusiasm and drug company advertising.
When a drug exists that is generally considered to be safe and effective, some commentators believe that newer drugs should be tested against the older, not against placebo. Only new drugs that are safer and/or more effective than their predecessors would be approved. Other commentators strongly condemn this research design because it leads to a shifting baseline over time.
It seems the more we look for confirmation bias, the more of it we find.
dave38 says:
January 20, 2011 at 8:47 am
W Abbott says:
“….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this
It was said by Ernest Rutherford.
Wasn’t it also Rutherford who said “All science is physics. The rest is speculation”?
“Most of the time, scientists know what results they want, and that can influence the results they get.”
Total crock. Written by someone who has never done an experiment.
I’m sure The New Yorker didn’t intend this as an indictment of Climate Science. But their emphasis on replicability has been at odds with the way Climate Scientists do their work, as evidenced by their resistance to publishing their raw data and correction methods, even when faced with legal ramifications.
I’m going to say something that will please my dad no end…
Scientific Method is a lot like learning to play golf.
When you first get some sort of swing you go out and try to use your driver on every tee you can within reason. Sometimes even without reason.
You tee up, swing (or do something that looks like a swing) and hit the ball more times than not. You’re having a good day.
Trouble is you don’t hit each shot very far or straight. In fact the worst ones are the 90 percenters..the ones that fly straight for 90% then curve off at the end into the trees…to which you learn about the truly versatile nature of a Wedge.
But then if you do hit a semi-decent shot..you continue along that path, taking out the driver and hitting more duff shots than not…
And it’s hard to stop getting out the driver. You believe you can hit it right. Just the next time.
Once you see reason and especially if you start to BET on your games with your mates…you magically start to increase your good shot percentage..largely because you AREN’T getting the Big Dog out every single tee shot…
Lo and behold humility on the golf course breeds results and consistency…and it is then that you realise all along what an ego-driven tool you have been from day one.
And that the course never changed while you did.
And you then learn to respect practice, consistency and how to swing properly so you can play ANY of your clubs.
Strangely this is exactly the same thing that happens in scientific investigation. It is so much better when real “reputation” is on the line to produce conservative and good science than when money is thrown around willy-nilly and ANY result is celebrated and hyped…then you start to believe you can drive successfully each time.
Trouble is you often didn’t actually hit the fairway the first time…but no-one calls you on it until much later. And then there’s all this self-examination. Oh the scientific method is failing.
It’s not. There are just a lot of tools out there who should learn about humility, calling themselves scientists when they are really just theorists without evidence.
And it’s only a bit depressing if you think just HOW many of them are out there.
Anthony:
Jonah Lehrer is a dilettante with good connections. His only scientific experience is being a middle author on a single paper in Eric Kandel’s lab. He is wrong on many points and has been effectively destroyed elsewhere: see http://scienceblogs.com/pharyngula/2010/12/science_is_not_dead.php for example.
I knew this guy was wrong when I read his “article”. Why didn’t you?