Science Has a Reproducibility Crisis

Edward Ferrara writes:

If your Facebook feed is anything like mine, you may have recently heard about how Bill Nye–the Science Guy himself–“slammed” Tucker Carlson on the latter’s evening show on Fox. THIS. (If you live somewhere else you may have been treated to an equally smug reaction from people claiming that Carlson “won.”)

However you feel about it, the timing, coupled with Nye’s reliance on scientific consensus as a proxy for objective correctness, is somewhat serendipitous. Mounting evidence that the results of scientific studies are often not replicable has caused Nature, one of the most prolific scientific journals, to very publicly tighten its standards for submissions as of its latest issue.

In May of 2016, a survey by Nature revealed that over two thirds of researchers surveyed had tried and failed to reproduce the results of another scientist’s study. Over half of them had been unable to reproduce their own results. Fifty two percent of researchers polled said there was a “significant crisis” of reproducibility.

This is a big deal. The ability to replicate the results of studies is crucial to both scientific integrity and progress. Clinical researchers, for example, depend on reliable results from prior trials to form the building blocks of new drug advancements. In the field of cancer biology, merely 10% of results from published literature were found to be reproducible. Meanwhile, the credibility of scientific literature is understandably compromised by dubious, often sensational findings.

The root of the problem, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge, stems from today’s scientific culture. As quoted in BBC, she cites “a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

Others blame a pressure to publish. There has also been, in recent years, doubt cast on the integrity of the peer review process, especially with regard to climate science. Whatever the culprit, plans to combat issues of reproducibility are emerging. Nature has developed a checklist to serve as guidelines for authors submitting writing to the publication. Efforts shouldn’t end there, the journal argues. Reform at all levels of the scientific process could go a long way:

“Renewed attention to reporting and transparency is a small step. Much bigger underlying issues contribute to the problem, and are beyond the reach of journals alone. Too few biologists receive adequate training in statistics and other quantitative aspects of their subject. Mentoring of young scientists on matters of rigour and transparency is inconsistent at best. In academia, the ever increasing pressures to publish and chase funds provide little incentive to pursue studies and publish results that contradict or confirm previous papers. Those who document the validity or irreproducibility of a published piece of work seldom get a welcome from journals and funders, even as money and effort are wasted on false assumptions.

Tackling these issues is a long-term endeavour that will require the commitment of funders, institutions, researchers and publishers. It is encouraging that NIH institutes have led community discussions on this topic and are considering their own recommendations. We urge others to take note of these and of our initiatives, and do whatever they can to improve research reproducibility.”

More https://eddiethoughts.com/2017/03/03/science-reproducibility-crisis/