Below is an excerpt from an excellent article in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:
Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.
Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF
h/t to WUWT reader Edward Lowe
=======================================================================
This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
If I may, I propose the name for this could be: confirmation entropy
This article should be required reading for all first-year university science students.
Okay, I only read the first page, I now see there are five. I think Girma captures the gist of the perceptual error in this article.
Girma says: January 20, 2011 at 4:01 am
“But it appears that nature often gives us different answers.”
I disagree.
Nature has only one answer.
It is our interpretations that are different. and I might add, ‘limited’ by what we do not know.
The problem is lack of critical/sceptical attitude, which should be required of every scientist. The more the better. This problem has existed since there is science, but it increased with time. Meanwhile it is so huge that all established or consensus science is BS. All of it. Just try to be sceptical of any established science/theories and watch the attacks, dogma and suppression rear their ugly heads.
It can be solved, but it needs first to be recognised/acknowledged/admitted widely.
One of my favorite scientists of all time, Thomas Gold:
“The inertia of scientific thought”
http://www.suppressedscience.net/inertiaofscientificthought.html
In the scientific community, it’s widely known and practiced, de facto SOP really, that results of testing often deliver different results than expected, so the parameters of the test are altered to meet the result.
Color me surprised.
Girma says:
January 20, 2011 at 4:01 am
“But it appears that nature often gives us different answers.”
I disagree.
Nature has only one answer.
It is our interpretations that are different.
=============================================================
Of course, but this sentence is very precise because it underscores the fact that results decline over time using methods thought to “eliminate” subjective interpretation. That’s why I actually love that particular sentence.
BTW, thanks for posting this Anthony. I’ve read it a good while ago as it actually applies to my field of research in judgment/decision making, heuristics and biases in psychology. Solving this problem of, up until recently, unnoticed area of human logical fallacy could be one of the greatest contributions to modern science. It would surely revolutionize many, but not all, fields of science. I guess you could call me a skeptic of climate science BECAUSE of my preoccupation with cognitive biases. Research methodology is THE most important factor in making sense of data. Currently I am sadened to say that the climate science movement are making incredible errors in basic epistemology. History will not be kind to those who broke the most basic principles.
Could it be related at all to the fact that some “scientists” are not releasing their data in order to confirm their findings? Something about “confidentiality” .
Post-normal = lying is ok.
In the 60s an international study into lifestyle and longevity included a detailed study in a remote mountain area in Ireland, close to where I live.Their health and longevity stats were impressive. The diet was largely full dairy, fat meat, vegetables and potatoes, the lifestyle was a physical, spartan one. Their data never became part of the published study. The agenda drives the result in too many cases.
This issue simply reflects the damage generated by the pervasive misuse of basic statistics.
Most of the soft sciences use relatively modest 3 sigma deviations from the expected norm as a standard of proof. This creates lots of false positive findings which are published as gospel.
Lubos Motl in his excellent Blog The reference Frame addressed this issue very well here:
http://motls.blogspot.com/2010/03/defending-statistical-methods.html
His suggestion was that researchers should raise the bar to 5 sigma before drawing conclusions.
The difficulty is that small sample sizes and problems in controlling the extraneous variables combine to make that level of proof nearly impossible to attain. Even in very well financed sectors such as the drug industry the efficacy of most drugs is statistically not hugely different from a placebo.
That suggests that skepticism should be the default choice for any research finding, with grudging acceptance perhaps over time if it is shown to have predictive value.
W Abbott says:
January 20, 2011 at 3:43 am
“….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.
I think that was Rutherford.
Sorry off topic:
Is this a case of scammer’s getting scammed or what?
http://www.bbc.co.uk/news/science-environment-12219472
I think another thing at work here is simply the complexity of what current science is bumping up against. It seems to me that the advance of science inevitably leads to attempts to conquer ever more complex conundrums. As that process proceeds through time it gets increasingly complex to design and execute a good experiment.
Climate science is a wonderful example of this taken to extremes. The ‘experiments’ have moved into simulations because our incomplete knowledge of climate in the real world has not yielded easily to experimentation. Contrast the complexity of a climate experiment with one meant to confirm something from Newton.
I would submit that climate science has outstripped its experimental underpinnings because the grant writers can create far sexier submissions for simulation than for those that would be aimed at the root mechanisms that drive the system. It might well be true that this is a factor at work in every modern (complex) scientific endeavor.
I tend to agree with those who find the problem to be essentially one of definition. Specifically, the definition of what constitutes science, as many fields have adopted that monicker (social sciences comes most immediately to mind) to give their statistical fudgings an air of authority and certainty on par with Newtonian mechanics. I say this as an economist (a field that seems to have fallen under the rubric of social scientiest).
From the article: “Other studies claimed that females had more orgasms when their partners were symmetrical, while a paper by anthropologists at Rutgers analyzed forty Jamaican dance routines and discovered that symmetrical men were consistently rated as better dancers.”
If a ‘study’ of So You Think You Can Dance Jamaica is considered science, no wonder science is in trouble.
The Crabbe experiments deserve more careful thinking. His model, at least as described in the above article, make numerous ceteris paribus assumptions. Sure he controlled for many of the obvious possible variables – but not for the obvious one of individual differences. Just think of the variability among humans in any number of attributes – even amongst identical twins. Moreover, unless we have some sense of the baseline activity level for these mice, it is hard to understand the real import of the 10X difference in distances. Perhaps this is made clear in the original Crabbe article – I do not see a reference to the original article.
The article itself is excellent read. It is not a mistake to consider its relevance to climate science, although the article doesn’t discuss it explicitly. Interesting that the author, Jonah Lehrer, mentions a finding that challenges the Law of Gravity:
Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1Ba9jMcIC
But here is the oddity. When Judith Curry first discussed to this article back in December, she also mentioned a follow-up comment by Jonah Lehrer on another web page.
http://judithcurry.com/2010/12/14/lies-damned-lies-and-science/
And here is Jonah Lehrer talking explicitly about climate science :
Interesting that Lehrer doesn’t name as ‘denialist’ even those experimenters who cast doubt on the Laws of Gravity.
So before we pat the author on the back for this article -and we should because it is an excellent read-, let’s make note: the author is not a Skeptic but a Believer.
With regard to climate science, Jonah Lehrer is a CAGW cultist.
Steve Keohane says:
January 20, 2011 at 5:25 am
IIRC, Lilly buried 30% of the original prozac studies that showed it was ineffective without talk therapy.
=============================================================
You have perfect recollection 😉 (although I am not familiar with the percentage)
This, and many other findings like this, led to an enormously influential meta-study published in the british medical journal “the Lancet”. It gathered all research (published AND not published findings on newer generation anti-depressants) and found NO effect when put together, except for a minor effect for people with very severe depression. Remember then that placebo worked just as well, but with NO side effects which can be very harmful in some SSRIs (anti-depressants).
The time is perhaps ripe for a big meta-study conducted of all research on climate data to establish if there is an active confirmation bias in the publishing part of the field. I think there is. Even though it might still prove CO2 effects, they are likely to be seen as less severe if viewed as a whole. A problem with such a study is that much research is concluding what computers are saying. This is an enormous problem, and could perhaps lead to a fall of the “modelling paradigm” in the future if found to be very influenced by biases.
In reply to W Abbott:
“If your experiment needs statistics, you ought to have done a better experiment.”
Ernest Rutherford quotes (Baron Rutherford of Nelson. New Zealander born British Chemist who laid the groundwork for the development of nuclear physics by investigating radioactivity. Nobel Prize in 1908. 1871-1937)
This is incredible. I know that drugs are ineffective for schizophrenia. [sarc]
The new model,
http://img837.imageshack.us/img837/1986/aquecimentogobal300x210.th.jpg
If the facts are changing and losing their truth, then it wasn’t a real truth to begin with. A little over thirty years ago, climate scientists were debating the coming ice age, based on a series of facts. Today, they debate how badly we are going to cook ourselves with AGW. The facts are all still the same, but the truth behind them is still hidden. No amount of claiming a consensus and moving to a “solution” will help to uncover the truths as to how climate operates and what drives it. The target of climate science is probably not changing as fast (or at all) as the targets of drug manufacturers–climate science is very complex, and has more moving parts than we can count, but those parts stay more or less the same parts. Drug manufacturers, on the other hand, are chasing a moving target–the Red Queen effect, mentioned by another commenter.
Bottom line–science is science and the scientific method, if applied correctly, works. I have much more respect for the people who follow their curiosity instead of the money. I think the curious folks get better results.
Another comment:
If the review process in climate science in any way resembles the “bore hole” standard of realclimate.org, I am very confident there is a fundamental and rotten problem to be concerned about. It would then, in my view, be a case of serious corruption to be dealt with as a serious legal problem.
Too many scientist assume that correlation=causeation. Replicating a poorly designed experiment is not proof of causeation.
People in the UK should watch the BBC’s “Horizon” program about the “attack on science” (or rather, the credibility of scientists) next week.
[Monday, 21:00 on BBC Two (except Northern Ireland (Analogue), Wales (Analogue))]
According to the “Time Out” TV reviewer, the new head of the Royal Society, Sir Paul Nurse, will effortlessly demolish all the “deniers” claims that AGW isn’t happening in an interview with the Daily Telegraph’s “denier-in-chief” which left the journo “spluttering that he is an interpreter of interpreters”.
This is what the BBC have to say about the program:
“Nobel Prize winner Sir Paul Nurse examines why science appears to be under attack, and why public trust in key scientific theories has been eroded – from the theory that man-made climate change is warming our planet, to the safety of GM food, or that HIV causes AIDS.
“He interviews scientists and campaigners from both sides of the climate change debate, and travels to New York to meet Tony, who has HIV but doesn’t believe that that the virus is responsible for AIDS.
“This is a passionate defence of the importance of scientific evidence and the power of experiment, and a look at what scientists themselves need to do to earn trust in controversial areas of science in the 21st century.”
http://www.bbc.co.uk/programmes/b00y4yql
—
So there you have it; AGW exists because the Royal Society, the BBC and the “Time Out” TV reviewer said so.
Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?
The best article I’ve ever read in any magazine. And I would say of all the articles I’ve read in magazines, less than 5 are any good at all. I now think the other less than 4 articles sucked compared to this one.
When I was in graduate school, we did not take a single full year course on the history of research and experimental design. I took a semester to study statistical analysis. Later, I took a semester of research interpretation (similar to taking a course in literature). That was it. That I am able to sniff out poor research design and question results cannot be credited to a stellar graduate course. Worse than that, my study, published in a major journal, was chasing the lab director’s beliefs. Had I not found a positive result, there is no question in my mind that I would have been sent back to the lab with orders to look harder.
Grade inflation, too many “scientists” for academic positions or research institutions, and pal reviewers. Stop it all now: a) the subject matter must be mastered or you do not advance or graduate; b) no more government grants for “scientific” research; c) peer review process open to the public, at the very least. Transparency and accountability along with replicability. A free market system in “science”.
Page 5: “Many scientific theories continue to be considered true even after failing numerous experimental tests.”
They always said global warming was going to lead to snowier winters. Right?
The problem with global warming theory is that it never fails. Rather, the true implications of it only become obvious in hindsight. Then the theory is adjusted as if it was always thus. The science becomes settled. And then, when the next mysterious unpredicted phenomenon pops up, it gets settled again. So, at all times, it’s always settled. It’s just not the same.
“So there you have it; AGW exists because the Royal Society, the BBC and the “Time Out” TV reviewer said so.
Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?”
Horizon ceased being an objective and rational science programme a long time ago. Sadly they are tied in to the same vested interests who abuse and pervert the scientific process to achieve profits and power.
I will be willing to bet that this episode will not be interviewing scientists who have admitted to falsifying “double blind” tests and experiments in order to get the results that pharmaceutical corporations want in order to licence the selling of highly toxic chemicals in the guise of medicines.
There is nothing wrong with science and it is highly decietful of the AGW alarmists to even claim that sceptics are attacking the science of climate change.
On the contrary, most climate realists attack the perversion of the scientific method and the sloppy and dishonest deviation from the scientific method which Alarmists have become well known for.
It is the climate realists and not the alarmists which uphold and support real science. It is the Alarmists who do not.