Reproducibility — the ability to redo an experiment and get the same results — is a cornerstone of science, but it has been the subject of some troubling news lately. In recent years, researchers have reported that they could not reproduce the results from many studies, including research in oncology, drug-target validation, and sex differences in disease (and climate with Cook et al. ).
In response, journals such as Science have adopted new guidelines for certain types of studies, and members of the scientific community have published a flurry of articles and blog posts.
But, as of 2 May, a panel of experts that convened at the AAAS Forum on Science & Technology Policy was still troubled. Fortunately, while their tone was serious, the speakers also described some solutions that are moving forward.
A Far-Reaching Problem
There are many reasons that scientific results may not be reproducible, explained the speakers, who focused their remarks on the biological sciences. Sloppy research is one possible culprit, according to Story Landis, director of the National Institute for Neurological Disorders and Stroke (NINDS). Studies may be designed poorly or fail to use appropriate statistics, or the experiment’s details may be described inadequately in the published report. Researchers may also feel pressure to publish “cartoon biology” that overemphasizes the “exciting, big picture” and leaves out the more prosaic details, she said.
Brian Nosek, co-founder and director of the Center for Open Science agreed that pressure on authors contributes to a “gap between scientific values and scientific practices.” The more prestigious journals tend not to publish negative results — that is, studies in which a hypothesis is not borne out by the data — or studies whose chief aim is to replicate other findings. Researchers typically must publish in high-impact journals in order to advance their careers, and therefore have little incentive to conduct these types of studies despite their importance, Nosek said.
Studies that can’t be reproduced due to outright fraud are relatively rare, according to Robert Golub, deputy editor of the JAMA, the journal of the American Medical Association. But, even when researchers are not intentionally engaging in misconduct, non-reproducible results are troubling, the speakers agreed.
“There has been a confluence of concern from various sources within the scientific community and from outside the scientific community in the last few years that the scientific enterprise is not producing new knowledge of sufficiently high quality,” said Katrina Kelner, editor of Science Translational Medicine and organizer of the Forum session. “…This issue of reproducibility is a problem of increasingly great concern to the scientific community itself and it is, one could argue, legitimately of interest to the broader society because of the robust public support of scientific research.”
As an example of the serious consequences that non-reproducible studies can have, Landis cited a report that a drug called minocycline showed promise in mouse models of the neurodegenerative disease ALS. These findings led to a phase III clinical trial funded by the National Institutes of Health, which enrolled over 400 patients between 2003 and 2007. However, the disease actually progressed faster in patients who received the drug than in those who received a placebo. When scientists rescreened minocycline and many other compounds from 221 studies in mouse models of ALS, they found no statistically significant effects.