Below is an excerpt from an excellent article in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:
Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.
Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF
h/t to WUWT reader Edward Lowe
=======================================================================
This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.
If I may, I propose the name for this could be: confirmation entropy
At the risk of mis-quoting something I am not sure I understand myself, I think this is an example of the problems with frequentist stats giving over-confidence in results. A much better explanation can be found here:
http://wmbriggs.com/blog/?p=3388&cpage=1#comment-34338
One could say that Dr Briggs is on a bit of a campaign for bayesisn stats methods, but I think he is completely correct that we “fall” for a low ‘p’ value far too quickly. There are very few experimental systems where there there are no sample selection issues and little or no thought is given to that when publishing a “significant” finding from a study. As the New Yorker article notes, when scientists know what they are expecting (or trying?) to find, it is easy to select a sample that will maximize the effect. This can happen without any ulterior motives or even without a conscious decision on the part of the scientist.
Nice to see such a good article in the New Yorker as well as a story on a real scientist telling people when follow-up results don’t gel with the original findings!
kcom says:
January 20, 2011 at 6:49 am
Page 5: “Many scientific theories continue to be considered true even after failing numerous experimental tests.”
The problem with global warming theory is that it never fails. Rather, the true implications of it only become obvious in hindsight. Then the theory is adjusted as if it was always thus. The science becomes settled. And then, when the next mysterious unpredicted phenomenon pops up, it gets settled again. So, at all times, it’s always settled. It’s just not the same.
=============================================================
Yes. It has all the flaws of what is called retrospective studies. They are fundamentally flawed as there is too much openness for biases to infect the conclusions. It can be useful, but it is widely held to be very poor at finding causes to observed effects.
Prospective studies are more in line with good scientific philosophy. You make a hyopothesis (e.g. this level of CO2 will cause these effects by this time) and discard the hypothesis if it is not matching the real world data in a meaningful way (i.e. statistical significance).
Now, to be fair, the climate is extremely tough to study, but knowing this shold make scientists extremely humble when presenting predictions due to uncertainty. This is not what we see. We see the URGE for a consensus. A trend towards labelling the wholy grail of science (‘skepticism) as “denial”. And we sometimes see absurd claims such as: 95% certainty such and such, and claims such as: “the world will warm by 6degrees celsius by April 3rd in the year 2087 with sea level rises of 787.9 mm.
I’m glad this article came up, bc this is (IMO) the biggest Achilles heel of climate science, and it is a crime against the science.
Oh, it has a name …
charlatan …
The over-reliance on ‘p’ values was even noted in the NYT (that bastion of journalistic integrity – /sarc=off):
http://www.nytimes.com/2011/01/11/science/11esp.html?_r=2
Very interesting, very little can be said to add to the article. The heart is near the end..
According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. “The scientists are so eager to pass this magical test that they start playing around with the numbers…”
In other words, decide what you’re looking for and find it in the data. Where else have we seen this sort of bias?
“Nevertheless, the data Ioannidis found were disturbing: of the thirty-four claims that had been subject to replication, forty-one per cent had either been directly contradicted or had their effect sizes significantly downgraded.
The situation is even worse when a subject is fashionable. In recent years, for instance, there have been hundreds of studies on the various genes that control the differences in disease risk between men and women.” …. “But the most troubling fact emerged when he looked at the test of replication: out of four hundred and thirty-two claims, only a single one was consistently replicable. “This doesn’t mean that none of these claims will turn out to be true,” he says. “But, given that most of them were done badly, I wouldn’t hold my breath.””
And then – According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher.[b]{ “The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,”}[/b]
Does this remind you of Dr. Hanson?
Kate says: January 20, 2011 at 6:39 am
“travels to New York to meet Tony, who has HIV but doesn’t believe that that the virus is responsible for AIDS”
“Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?”
————–
Any idea if “Tony from New York” was a real scientist? Wonder why Sir Paul Nurse didn’t travel to Berkeley to talk to Dr. Peter Duesberg if he really wanted a discussion on HIV/AIDS? Thanks for the pointer, I’ll let Dr. Duesberg know about the program, if he didn’t already.
Great article.
While I can see that those in the “hard sciences” (as I was) would feel somewhat immune to the problems brought out in the article, I think it should be clear that the more complicated the system under research (yeah, like climate), the greater the probability of being subject to these flaws and biases. Even that doesn’t mean some simple experiments aren’t capable of going awry. Remember cold fusion (in it’s simplest form as originally reported, not what it has morphed into today)? The original results were significant and got published. At least one group (GA Tech, I believe) replicated the results. It was only because the research WAS so simple that it was quickly shown to be false. If for some reason replicating the experiment were difficult, required specialized instrumentation, or collecting data from across the globe (ahem), cold fusion power plants might be under construction today.
Cold fusion segues into one last thought: at the atomic/sub-atomic levels, nature gets as squimish as the subjects in the soft-sciences. I think that area is ripe for the same problems detailed in the article.
“Palmer emphasizes that selective reporting is not the same as scientific fraud”.
“But the worst part was that when I submitted these null results I had difficulty getting them published”.
“scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness”.
“The situation is even worse when a subject is fashionable”.
This explains about 11.9% of Climate Change science. Which leaves the other 89.1% as likely the opposite of the first quote.
Actually it does have a name – it’s called activism.
@dearieme January 20, 2011 at 3:49 am
Kudos for seeing the distinction between hard and soft science. That is why each of the soft sciences had a tough time being accepted in the first place – quantification was difficult and ambiguous. They put in equivalents of proxies to try to make them look and sound like proxies, and the biggest one was statistics. I say “equivalent of proxies” because they do things like assign one answer in a questionnaire a numeric value and another answer another value (based on vague and often hidden reasoning), so when the stats are done, what can it really mean? If the assigned values are incorrectly assigned (no matter how unintentionally), is that science really a science?
I recently read (sorry I have no idea of the source now) a statistician argue that the 95% confidence thing is misunderstood by almost everyone who uses it. If that abuse (mostly unintentional) is their main claim to quantification, then is that science really a science?
Archeology. Read any archeology article and 80-90% of it is about interpretations. That is the equivalent of Olympic ice skating judging, except they don’t throw out the high and low scores – or do they? I also long ago (again the source is long since out of reach) read where archeologists toss out 85% of the C14 dates, because they don’t fit into the expected range, on the (arbitrary and spurious) premise that the samples must have been contaminated. This weeding out is bias, pure and simple, the equivalent of fudging the data or cherry picking the data. I have been arguing for some time now that archeology is not a science, even though they use a few scientific procedures. But C14 testing isn’t one of them. Just because they send off samples to be tested by a lab doesn’t make the purchasers of that service scientists. It only makes them customers. Same thing goes for the various other dating tests done in labs – it is not the archeologists who do it; they are only customers. Take that away and almost (not quite all) of archeology is left with very few claims to being a science. Comparing ceramic vase types or paleographic types or building types doesn’t make it a science, especially when those are used against C14 dating or even in the early days to calibrate C14 dates. I see them as historians, little more. They say that put 500 archeologists in a room with evidence and you come out with 1000 different opinions. How scientific is that, when they can’t look at the same real-world evidence and not come to the same conclusion about it?
Climatology is seen here as little more than guessing. The more time I spend here, the more that seems so. Being thermometologists – does that make them scientists? Anyone can read a thermometer. And statisticians can process data. The (admittedly necessary) use of proxies, that really rest on theoretical bases but cannot be tested in the real world – is that science?
Yet, when the real world evidence conflicts with the theory, and they choose to say the theory is wrong, let’s fudge the real world numbers is that science? That is in reference to Keith Briffa’s tree ring studies (see C’gate emails) in post-1960 recent years, the ones that conflicted with the instrument record. They beat him down and forced him to accept the proxies as more real than the instruments, when they should have been stepping back and deciding to use that discrepancy to learn something about proxies.
Soft sciences are ripe for being influenced by bias. Their assignation of quantified values is also a critical area of distortion. And distortion is not science.
When I was a kid, hot weather would cause me to feel light-headed (eventually becoming a bad headache), weak & listless.
Eventually at the doctor’s office, it was determined to be low blood-pressure in hot conditions (my blood-pressure was normally average). He said there wasn’t anything to do (even then, salt was already politically-incorrect).
In high-school football, they offered salt-tablets during practice in hot weather. Somehow, the listlessness & headaches disappeared! Didn’t take long to figure out the connection. Not suggesting that anyone take salt in hot weather or otherwise, but in my situation, it was an extraordinarily simple “cure”. It still works for me in hot weather.
I bet those salt tablets have long been removed from HS football practice fields. I know, Gatorade has “replaced” it.
It is just as important to not only know, but but know HOW we know. I realized this when I studied Buddhism. I realized this again when I read “Good Calories / Bad Calories” by Taubbs. No where has science been more perverted to conform to an agenda than in the Food & Drug market. (Sorry climate people, but the food science industry has you beat in years and billions) Saying red meat causes heart disease in one animal (it does when you feed it to rabbits, exclusive herbivores) and extending that to another (human, omnivore) is bad science. Also, when heads of the FDA go to Pepsi Cola after their public jobs, you have to raise an eyebrow.
I’m sure when I say to climate skeptics to consider the source, that I am preaching to the choir, but what is happening here is nothing new.
Everything is a business model. Everything.
Yes, very interesting article.
Also interesting that Lehrer has his own bias, the “consensus” bias, with regard to CAGW. The CAGW industry probably has the greatest set of built-in biases of all, including:
Perception bias
Confirmation bias
Selective reporting bias
Pal review bias
Publication bias
Funding/Career bias
Fame/Ego bias
Herd Mentality bias
Consensus bias
Academia Bias
MSM bias
I’m sure there are more.
Not all of these biases are innocent, either, as evidenced by Climategate, and with the Hockey schtick, the stated need to “get rid of” the MWP.
Palmer noted ”
“The funnel graph visually captures the distortions of selective reporting. For instance, after Palmer plotted every study of fluctuating asymmetry, he noticed that the distribution of results with smaller sample sizes wasn’t random at all but instead skewed heavily toward positive results.”
Looks very much like cherry picking from larger sample sizes, to present the smaller subset of the total sample that gave the desired results.
Now look at the problem from the layman’s point of view as a recipient of the mainstream media’s outpourings:
Breast milk good, breast milk bad, breast milk good …….etc
MMR good, MMR bad, MMR good ……….etc
Statins good, statins bad, statins good ……etc
London will be under water in ten years time, make that twenty, make that next year ….etc.
The problem of the lack of consistency in science is the ‘Ego Loop’
Nowadays, scientists have lost objectivity and scientific rigour caused by the urge to publish something/anything to satisfy their overinflated egos. They will grasp for funds from any source but the government is the best one because the pot is bottomless.
Journalists, another ego-driven ‘profession’, compound the problem of poor science by simply regurgitating the scientists’ press releases without analysis. This appears to be because they no longer have the education needed to determine pertinent questions nor to understand the answers if they had been asked. Newspapers and TV broadcasters (some of whom, despite being publicly funded) also have political agendas that influence their reporting and determines their servility.
Finally the politicians. Politicians, are amongst the leading examples of ego-mania. Unfortunately because of a low IQ problem, they haven’t got a clue about the science; any science. Their scientific advisers are hand picked ‘yes men’ who are paid handsomely to advise as required. The politicians, therefore, compound the ‘poor science’ problem further by throwing money at the purveyors of that aspect of the science they think is likely garner the most votes; thereby closing the ego-loop. And we all know what positive feedback leads to.
And the public is confused. As far as they are concerned they would rather that the scientists, journalists and politicians grew backbones and followed that well known aphorism:
“Oh Lord help me to keep my big mouth shut until I know what I am talking about”
Part of this (I would imagine) is because our engineers in this world have been very very good for the last 30-50 years. They have given scientists a huge wealth of new instrumentation to investigate the universe that we didn’t have and couldn’t dream of before now. Most of this new instrumentation is quite expensive and requires that scientists get funding from someone else to do their work. Those who have the equipment to confirm the findings of one scientist probably get their funding from the same places the original writer does. Hence, if there’s something very beneficial to those giving the funding that is being confirmed by replication, the confirmation bias can spread.
“Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.”
Paradigm blindness or paradigm paralysis has been well documented, and is very common. Scientists with a cherished belief that has a lot of ego integrity invested in it have been shown data/studies that invalidate the belief, and only days later cannot recall seeing the data/study.
Read Joel Barker
dearieme says:
January 20, 2011 at 3:49 am
Does an explanation lie, unexamined, in the phrase ” ..across a wide range of fields, from psychology to ecology”. That’s not a wide range. I expect that no experimental result of mine has diminished in accuracy, save for any where (dear God, I hope not) I simply made an unrecognised blunder. But then my field is a “hard” science not a “soft” one. Heavens, isn’t that a large part of the distinction between hard and soft?”
I do agree this could be the case. Does climate science fall in the category of “soft” science even though the principle components under it are clearly from “hard” disciplines.
Be skeptical. Be very skeptical.
Related articles:
Lies, Damned Lies, and Medical Science
Is Peer Review Broken?
Problems with Scientific Research
This is what you get with government funded science. Something Eisenhour warned us about. It sounded like a good idea at the time….
AGW science is unique in that those who contest it are beat about the head by a large organized gang extending far beyond the core group of researchers.
Climate science is fiat science.
Having read the whole article, I agree Pamela, this is excellently thought out and written. If, as others have mentioned above, the author is a true believer of CAGW, this article is a perfect example of the underlying problem of human nature, and our perception of what we divine as reality. A mind, obviously capable of rational perceptive thinking won’t apply the same to itself, i.e. cognitive dissonance. Further, I would proffer that this is a necessary survival mechanism. In order to have a sense of self, and to be able to function moment-to-moment, we can’t be questioning whether gravity works every time, or if fire burns the finger each and every time the two meet. The intellect will arrive at some conclusion, even lacking enough evidence, unless we are conscious enough to notice that it does that. Basically we gloss over a lot and go on.
“So here is my request: please shame KEVIN TRENBERTH for inciting hate speech”
I agree 100% with this author. Scientific intolerance of opposing views is at the heart of the current problems in science. It is the same problem that afflicted the church centuries ago when it was the repository of knowledge.
Scientific intolerance is no different than religious intolerance is no different than racial intolerance. History repeatedly shows how, when left unchallenged, this escalates over time with great harm to all society.
The is no evil that cannot be justified in the name of good.