![Reproducibility-Initiative_resized[1]](http://wattsupwiththat.files.wordpress.com/2014/06/reproducibility-initiative_resized1.png?w=150&resize=150%2C150)
Reproducibility — the ability to redo an experiment and get the same results — is a cornerstone of science, but it has been the subject of some troubling news lately. In recent years, researchers have reported that they could not reproduce the results from many studies, including research in oncology, drug-target validation, and sex differences in disease (and climate with Cook et al. ).
In response, journals such as Science have adopted new guidelines for certain types of studies, and members of the scientific community have published a flurry of articles and blog posts.
But, as of 2 May, a panel of experts that convened at the AAAS Forum on Science & Technology Policy was still troubled. Fortunately, while their tone was serious, the speakers also described some solutions that are moving forward.
A Far-Reaching Problem
There are many reasons that scientific results may not be reproducible, explained the speakers, who focused their remarks on the biological sciences. Sloppy research is one possible culprit, according to Story Landis, director of the National Institute for Neurological Disorders and Stroke (NINDS). Studies may be designed poorly or fail to use appropriate statistics, or the experiment’s details may be described inadequately in the published report. Researchers may also feel pressure to publish “cartoon biology” that overemphasizes the “exciting, big picture” and leaves out the more prosaic details, she said.
Brian Nosek, co-founder and director of the Center for Open Science agreed that pressure on authors contributes to a “gap between scientific values and scientific practices.” The more prestigious journals tend not to publish negative results — that is, studies in which a hypothesis is not borne out by the data — or studies whose chief aim is to replicate other findings. Researchers typically must publish in high-impact journals in order to advance their careers, and therefore have little incentive to conduct these types of studies despite their importance, Nosek said.
Studies that can’t be reproduced due to outright fraud are relatively rare, according to Robert Golub, deputy editor of the JAMA, the journal of the American Medical Association. But, even when researchers are not intentionally engaging in misconduct, non-reproducible results are troubling, the speakers agreed.
“There has been a confluence of concern from various sources within the scientific community and from outside the scientific community in the last few years that the scientific enterprise is not producing new knowledge of sufficiently high quality,” said Katrina Kelner, editor of Science Translational Medicine and organizer of the Forum session. “…This issue of reproducibility is a problem of increasingly great concern to the scientific community itself and it is, one could argue, legitimately of interest to the broader society because of the robust public support of scientific research.”
As an example of the serious consequences that non-reproducible studies can have, Landis cited a report that a drug called minocycline showed promise in mouse models of the neurodegenerative disease ALS. These findings led to a phase III clinical trial funded by the National Institutes of Health, which enrolled over 400 patients between 2003 and 2007. However, the disease actually progressed faster in patients who received the drug than in those who received a placebo. When scientists rescreened minocycline and many other compounds from 221 studies in mouse models of ALS, they found no statistically significant effects.
more: http://www.aaas.org/news/concerns-about-non-reproducible-data-mount-some-solutions-take-shape
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Simple solution: don’t publish any original studies, only replications.
On the other hand, I have heard rumours of studies which cast doubt on the idea that red wine is good for you. I hope those rumours are false. That’s taking science too far.
No surprise in human biology. 9 out 10 cancer/tumor cell line drug results reported in leading journals can not be replicated by the Big Pharma scientists when they investigate the reported results in house.
But then again, hey, lets just take 4.5C as the CO2 doubling value. Published, peer-reviewed science is never badly wrong… is it?
Duster writes “A sceptic who offers no alternative is simply a philsopher”
In terms of climate science, there is no need to offer an alternative “to CO2” because the climate has varied in the past all by itself. Warming enthusiasts acknowledge that but say this time its different. This time its CO2.
a thought for the world of scientific journals — there ought to be journals devoted solely to articles which (1) seek to replicate or falsify past study results, and (2) detail studies of any kind which had only “negative” results, i.e., no significant causation found, and (3) publish lengthier comments and responses to existing articles than journals generally permit, etc.
Sure, such journals will never be the “prestige” journals leading a field, but it ought to be far more feasible for researchers to publish these kinds of scientific articles and get some degree of professional credit, C.V. items etc.
With the advent of online and open access journals, the logistics and expense of such journals ought to be far lower.
It may be extremely difficult to find organizations and editors, reviewers, etc. willing to run such journals, but perhaps teh various professional societies could be pressed to regard this as a fundamental service to the professionalism and quality of each field of speciality.
This all may be a huge long shot, but it is the sort of thing which ought to happen!!
jimmi_the_dalek the missing part of the data is NOT the ID of the reviewers some of which have already been give away , but the time stamps which would show how much effort each article took to review . In other words the chances of people doing a honest job when they spending seconds on this job are virtual nil and that is what is suspected that Cook and his little gang went all out to ‘prove ‘ the 97% claim and to do that that fired out results to fit that need without any real checking , hence the fact that included articles that could actual be used , if reviewed probably, to support the 97%
hence the fact that included articles that could NOT actual be used , if reviewed probably, to support the 97%
There is a strong aura of ‘fuzziness’ around the use of the term ‘scientific results’. In the context of discussing the validation step of the scientific method, the term should be restricted to *empirical* measurements.
All too often lately the term has been used in the broad sense of “the results of computer modelling by programs incorporating scientific cause-effect relationships”. In this latter use, computer modelling has been used to replicate the results of other similar computer models, but this does NOT qualify as ‘reproducing’ scientific results.
Scientific truth is reproducible. Errors can be replicated, but they cannot be anchored to empirical measurements.
Mother Nature does not make errors. Mother Nature is the ultimate arbiter of scientific truth, and valid experiments and measurements are those that interrogate her directly (to preserve the metaphor) in a way that yields a result that can decide whether hypothesis A or hypothesis B is more correct.
So managers are undead engineers? This explains much…
And that’s why some of them in the CAGW field refuse to release their data – because their results are unreproducable.
First of all, thank you for the interesting post.
Being in medical and biological science since more than 20 years, I have to admit that the problem is real and it is far larger than (i think) commonly perceived.
I have more than 300 publications among which at least 200 original papers: do you really think they are all containing “top science”? No way!
In order to be funded you need publications. On the other hand, while all agree that confirmatory studies are extremely important nothing facilitate their completion. Confirmatory studies are of poor interest even for low-ranked journals, they do not help careers. and funding agencies do not pay for something already done by others.
On the contrary the studies that contradict previous findings (that could also be very healthy thing) have often few chances to be published since encounter biased review processes.
There is an intrinsically masochistic relationship between science and economics. We do not have to forget that today science and scientists have to learn to survive in a society where only the emotionally breaking news count. The so-called “scientists” (much different subjects than the ones who produced the milestone discoveries in the past) are in a sense “forced” to prime-time findings to get their research going.
Today we are judged by number of publications, impact factor, h-index etc. Forget about the old times when science funding was independent from the science results. Economics in now dominating the scene and I suspect this is not gonna change soon.
So we have to learn to live in a science system in which the real discoveries are often hidden by an overwhelming level of noise.
I expected to see a learned discussion of the bankruptcy of the academic “publish or perish” mentality when I clicked on this article, and yet, nothing! Academe is creating this sciency diarrhea as is the (wrongheaded, IMO) cultural truism that all students must go to college. Although, in one sense, I suppose that the truism is accurate, as many, if not most kids graduating from high school seem to be dumber than posts, can’t read or write well, have read nothing of import, have no sense of history or perspective, and have been brainwashed by leftist progressivist BS. How do we correct this? I would make public sector and teachers unions illegal. I’d fund public education by vouchers and let the free market make new, competitive schools. I’d end subsidized college loans. I’d install some review and accountability at universities, and fire professors who crank out or advise their post-graduates to crank out garbage science. Of course, those very same garbage producers are helping to fund the colleges and universities through their grants. So we need to scale back the Federal government’s funding of science, too. Or end it completely, perhaps, at least for a time. Let’s let market forces and competition reassert themselves in the priority mechanism of scientific research.
I don’t think there’s fraud involved; The experimenters are just fooling themselves with
“data mining”. Someone runs a test believing “A” is true. The theory is worthless, the results don’t show anything, but “A1” shows significance at the 5% level, so the experimenters publish the results for “A1”, not realizing that the NEW results DON’T show “A1” significant at the 5% level. In reality, they’ve demonstrated that (assuming A1 and A are independent)
“Either A or A1 is true at the (1 – (0.95)^2) level = 9.75%- not significant and not worth publishing.
Here’s a real world example:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1871503&http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1871503
Originally, they set out to test the hypothesis that “gullible believers” were more scientifically literate than “deniers”. THAT didn’t pan out. Publishing the actual result that DENIERS scored higher at the 5% level would have been an example matching my
(1 – (,95)^2) = 9.75% exactly, but they didn’t want to publish THAT. Instead they went into a spiel on CULTURE and how more scientific literacy increased one’s “pigheadedness” if you will.
The testers were actually throwing in a 3rd item in their data mine
1- (0.95^3))= 14.26% chance by dumb luck. Add to that the possibility that OTHER unmentioned variables may have bee tested and ruled out, but not addressed, and it’s easy to arrive at
a “5% or less probability of happening by chance alone” when the actual figure is closer to 50%.
Nicholas Schroeder says:
June 10, 2014 at 6:53 pm
Remember cold fusion? Deja vu all over again.
That is a poor example. Pons and Fleischmann have been replicated. MIT didn’t load the Palladium with sufficient Deuterium.
You might look at Elforsk’s report, expected later this month, on their six month trial of Rossi’s E-Cat HT.
Adrian Ashfield
milodonharlani says:
June 10, 2014 at 8:26 pm
“Water in the brain, good. Water on the brain, not so much.”
Try a tap on the head.
Jungle:
I had the same thing but it seems to have stopped now.
Journals need to publish research. If there isn’t enough good research out there to publish, then they publish research that isn’t so good. I once taught a basic research class for graduate students. There were about 20 in the class. Toward the end of the semester, each student chose 5 research studies from a variety of fields, analyzed them according to the principles taught in class and presented what they found. Out of more than 100 studies, only about 10 to 15 were evaluated as “clean” enough to deserve publication. Worst were found in health journals. Best were pharmaceutical studies.
Philjordan:
Do not forget Keith Briffa and his science scam involving tree rings which Steve McIntyre finally exposed through a FOI action.
“As in the latter, it was all sanctioned by peer review, and again as in climate science, the whistle blowers are mostly from outside academia. As a career academic myself, I have become convinced that the problems in academia are profound, entrenched, and pervasive, and that they will only be fixed from the outside, not from within.” ~ Michael Palmer
People who receive degrees from these colleges are also a considerable force in maintaining the paradigm under which they were educated. No one wants to admit that the degree they have is based on knowledge that has been debunked or is useless.For example, the previous claims of the health and nutrition experts, and the highly selective witch hunt for carcinogens, are all being falsified, but the Baby Boomers are exceedingly dogmatic about what they know, and conclusions they reached 40 years ago.
And so we have such preposterous legislation beginning to surface which regulates diets. Another problem resulting from the dogmatic clinging to false scientific findings is the criminalization of many neutral and even beneficial economic and social activities, such as commercial farming.
Jbird says:
… I once taught a basic research class for graduate students. …
—
What kind of topics did you teach?
@Palmer
Design, number of cases, independent ad dependent variables, control for extraneous variables, choice of statistic, use of null hypothesis, etc.
non reproducible “studies” are not studies they are experiments … that failed to validate the underlying theory …
I agree that skepticism is an important quality in a science. Without it, the reviewing scientist has
muted his critical faculties. In climate science, a skeptic is merely one who approaches the issue with his critical faculties in good order.
mpainter says:
June 11, 2014 at 9:47 am
I agree that skepticism is an important quality in a science. Without it, the reviewing scientist has muted his critical faculties. In climate science, a skeptic is merely one who approaches the issue with his critical faculties in good order.
Rather,
In climate science, a skeptic is
merelythe only one who approaches the issue with his critical faculties in good order.@Sandi says:June 10, 2014 at 9:39 pm
Sagan was soooo wrong “Science is more than a body of knowledge, it’s a way of thinking.”.
Science is a methodology, that’s all,nada nothing else, not a way of thinking (that’s philosophic) !!!