![Reproducibility-Initiative_resized[1]](http://wattsupwiththat.files.wordpress.com/2014/06/reproducibility-initiative_resized1.png?w=150&resize=150%2C150)
Reproducibility — the ability to redo an experiment and get the same results — is a cornerstone of science, but it has been the subject of some troubling news lately. In recent years, researchers have reported that they could not reproduce the results from many studies, including research in oncology, drug-target validation, and sex differences in disease (and climate with Cook et al. ).
In response, journals such as Science have adopted new guidelines for certain types of studies, and members of the scientific community have published a flurry of articles and blog posts.
But, as of 2 May, a panel of experts that convened at the AAAS Forum on Science & Technology Policy was still troubled. Fortunately, while their tone was serious, the speakers also described some solutions that are moving forward.
A Far-Reaching Problem
There are many reasons that scientific results may not be reproducible, explained the speakers, who focused their remarks on the biological sciences. Sloppy research is one possible culprit, according to Story Landis, director of the National Institute for Neurological Disorders and Stroke (NINDS). Studies may be designed poorly or fail to use appropriate statistics, or the experiment’s details may be described inadequately in the published report. Researchers may also feel pressure to publish “cartoon biology” that overemphasizes the “exciting, big picture” and leaves out the more prosaic details, she said.
Brian Nosek, co-founder and director of the Center for Open Science agreed that pressure on authors contributes to a “gap between scientific values and scientific practices.” The more prestigious journals tend not to publish negative results — that is, studies in which a hypothesis is not borne out by the data — or studies whose chief aim is to replicate other findings. Researchers typically must publish in high-impact journals in order to advance their careers, and therefore have little incentive to conduct these types of studies despite their importance, Nosek said.
Studies that can’t be reproduced due to outright fraud are relatively rare, according to Robert Golub, deputy editor of the JAMA, the journal of the American Medical Association. But, even when researchers are not intentionally engaging in misconduct, non-reproducible results are troubling, the speakers agreed.
“There has been a confluence of concern from various sources within the scientific community and from outside the scientific community in the last few years that the scientific enterprise is not producing new knowledge of sufficiently high quality,” said Katrina Kelner, editor of Science Translational Medicine and organizer of the Forum session. “…This issue of reproducibility is a problem of increasingly great concern to the scientific community itself and it is, one could argue, legitimately of interest to the broader society because of the robust public support of scientific research.”
As an example of the serious consequences that non-reproducible studies can have, Landis cited a report that a drug called minocycline showed promise in mouse models of the neurodegenerative disease ALS. These findings led to a phase III clinical trial funded by the National Institutes of Health, which enrolled over 400 patients between 2003 and 2007. However, the disease actually progressed faster in patients who received the drug than in those who received a placebo. When scientists rescreened minocycline and many other compounds from 221 studies in mouse models of ALS, they found no statistically significant effects.
more: http://www.aaas.org/news/concerns-about-non-reproducible-data-mount-some-solutions-take-shape
“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.”
yup.
if you dont have a replacement theory, the best one, no matter how flawed, rules.
Otteryd says:
June 11, 2014 at 7:21 am
As in the third tap in the Mozambique drill, after the double tap to the body?
Steven Mosher says:
June 11, 2014 at 10:26 am
The man-made GHG hypothesis has been falsified so thoroughly that it is worse than no theory at all.
OTOH, the hypothesis that solar activity (regulated by our planet’s orbital & rotational mechanics) & earth’s unusual liquid watery surface & atmosphere, plus plate tectonics, largely control climatic fluctuations has good observational support:
http://www.space.com/7195-sun-cycle-alters-earth-climate.html
Steven Mosher if your theory cannot stand up to actually being questioned , it not a theory at all your merely selling BS , peer review or not .
Plenty of theories in science and then all have to ‘put up with ‘ with ongoing critical review and accept that they may need to be modified thanks to this review , Climate ‘science’ is the first area that came up with a theory sold as ‘perfect form birth ‘ or scientific immaculate conception, that can never been wrong nor even questioned.
Reproducibility is only a requirement in the experimental sciences; in the observational sciences like astronomy or geology no flatly disconfirmatory experiments can be run. It’s more a matter of degree, with confirmatory or disconfirmatory observations adding weight or undermining a claim.
Thanks for your reply, Jbird.
—
Coincidentally, just today I received a request to review a research paper. The instructions contained the following clause:
One important concern in scientific publication right now is the apparent increase in data fabrication, data alteration, or duplicate publication. We ask that you pay attention to any signs of fraud in the text or figures, and that you inform us of any concerns about duplicate publication or unusual-looking results.
A recent opinion piece on this topic had an excellent suggestion – don’t review the results but the experimental method. That way, good papers are published whether or not the come the “correct” conclusions and the body of literature contains its own balance instead of being skewed to positive results.
http://www.genengnews.com/gen-articles/are-medical-articles-true-on-health-disease/5203/?kwrd=young
Sorry, the article had been free to view when I read it over the weekend, but is now pay-walled.
“A recent opinion piece on this topic had an excellent suggestion – don’t review the results but the experimental method”
That’s not a new suggestion, but a part of the standard review process. At least in the area I work in, the guide notes when reviewing a paper include a number of things to check, one of which would be “Are the methods used sufficient to establish the conclusion” , or words to that effect.
“if you dont have a replacement theory, the best one, no matter how flawed, rules.”
Why so? You could simply say, “We’ve got a bunch of phenomena, but we have no explanation. Every theory we’ve tried turns out to be bollocks. We just don’t know.”
Clear, honest, and justifies asking for a grant for further study.
Steven Mosher says:
June 11, 2014 at 10:26 am
“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.”
yup.
if you dont have a replacement theory, the best one, no matter how flawed, rules.
==============
Which leads us to BEST.
Same for air-to-air, or was that more exacting ?
@ur momisugly Michael Palmer…..
Maybe the pendulum will swing back toward more rigor and objectivity. A lot of damage has been done to scientific integrity, especially with the politicization of climate science.
[“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.” if you dont have a replacement theory, the best one, no matter how flawed, rules.]
Such thinking led to the creation of the Greek Gods. Zeus was the theory for lightning and Hades was the theory of what happens when we die.
If I want to read a good science fiction book, I’ll read Heinlein. If I want to know if research is garbage, then I’ll listen to the skeptic who has taken the time to find the flaws in such research. Skeptics have done more to undo bad science. Skeptics who spot shoddy science deserve our praise.
KevinK says:
I do believe that fraud is rare, but confirmation bias is rampant. The climate science community truly believes that they understand the most complex system anybody has ever tried to “model”. So they search for all the data nibbles that “confirm” their belief. And they turn away briskly from any “counter indications”. Just human nature really.
With “peer review” possibly being poor at spotting confirmation bias. Because everyone involved knows how things should be. Possibly what’s needed is more a “proof reader” or some other “outsider”.
Engineering is a totally different beast, no matter how hard I want to believe that my design will work the laws of physics are the final arbiter. Models are a useful tool in engineering, but only just one tool in the toolbox.
Hence we still “crash test” cars and bend entire aircraft wings until they snap
Well, am late for the conversation, but am happy to see that finally the reproducibility question is being discussed. It was long overdue.
Science without reproducibility is not science.
I would naively think that papers could be simply classified like:
– reproduced
– not reproduced but pending – pending validation
– no data & methodology available for reproduction – not valid
Michael Palmer says:
June 10, 2014 at 8:07 pm
The magnitude of the problem was quite shockingly highlighted in a note that appeared in the journal Nature in 2012:
….
According to the authors, who led a research lab at the pharmaceutical company Amgen, a full 47 out of 53 published experimental studies that suggested novel therapies for cancer could not be reproduced.
….
The authors, nevertheless, bravely proceed to put lipstick on this pig: “These results, although disturbing, do not mean that the entire system is flawed.” To me, with lipstick or without, this is an unmitigated disaster that is right up there with mainstream climate science.
Agree. The system of promoting “science” based on non-reproducible papers is flawed.
jimmi_the_dalek says:
June 10, 2014 at 7:06 pm
…
As far as Cook’s survey is concerned, you do not need his data. Getting the data and checking his arithmetic is a very weak form of ‘reproducibility’ , especially as the missing part of the ‘data’ is, as I understand it, the identities of the raters. If you want to challenge the results, do your own survey – with a bit of crowd sourcing you could easily read the abstracts of several thousand papers in a month.
Without the data and methodology you come to different result and then what? How does one compare that? How does one understand where the difference is coming from?