Excerpts from Robust research: Institutions must do their part for reproducibility
C. Glenn Begley, Alastair M. Buchan & Ulrich Dirnagl
Irreproducible research poses an enormous burden: it delays treatments, wastes patients’ and scientists’ time, and squanders billions of research dollars. It is also widespread. An unpublished 2015 survey by the American Society for Cell Biology found that more than two-thirds of respondents had on at least one occasion been unable to reproduce published results. Biomedical researchers from drug companies have reported that one-quarter or fewer of high-profile papers are reproducible1, 2.
Many parties are addressing the problem. Funding bodies such as the US National Institutes of Health (NIH) have announced training initiatives3 and explicitly instructed grant reviewers to consider whether experimental plans ensure rigour. New methods of data analysis and peer review have been proposed to deflate bias.
Several journals, including Nature and Science, have updated their guidelines and introduced checklists. These ask scientists whether they followed practices such as randomizing, blinding and calculating appropriate sample size. Science has also added statisticians to its panel of reviewing editors. Philanthropic and non-profit organizations have sponsored projects to improve robustness.
Funders’ policies, journal guidelines and widespread soul-searching are necessary. But they are not sufficient.
Conspicuous by their absence from these efforts are the places in which science is done: universities, hospitals, government-supported labs and independent research institutes. This has to change. Institutions must support and reward researchers who do solid — not just flashy — science and hold to account those whose methods are questionable.
…
The systems needed to promote reproducible research must come from institutions — scientists, funders and journals cannot build them on their own. These kinds of changes will require additional money, infrastructure, personnel and paperwork. The load on institutions and investigators will be real, but so is the burden of irreproducible research. Even if it is accompanied by an apparent decrease in productivity, the resulting increase in research quality will be well worth the costs.
Still, most institutions will not make the necessary moves unless forced. Funding bodies should make GIP a prerequisite for receiving a grant. The concept has gained some traction: last year, Science Foundation Ireland announced plans to conduct external audits on some of the labs that it supports.
There will not be one ideal solution. Faculty members, trainees and administrators will need to come together for honest, difficult discussions to restructure institutions. Neither scientists nor institutions should engage in mere box checking; new practices must restrain sloppiness while interfering only minimally with the many scientists who are behaving well.
Read the full article here: http://www.nature.com/news/robust-research-institutions-must-do-their-part-for-reproducibility-1.18259?WT.mc_id=SFB_NNEWS_1508_RHBox
Maybe if universities upped the ante for Ph.D. degrees we might see some change. Students should be required to take at least a year of graduate level statistical methods, Students should be required to produce original research that meets stringent reproducible results. And most of all, Lab directors/lead authors should be banned from giving Ph.D. candidates low hanging fruit projects stemming off of their own endeavors and should not be among the authors of the candidate’s research. But if you really want qualified Ph.D.’s, the candidates should be barred from any input and the product should be a stand alone single author study. It should be a test, not a coddled trip down easy street. The degree means you are capable of discovery. If the degree is awarded for lab sweeping with help from your boss, you end up with monkeys who’s acumen has not been tested.
Pamela, you hit the nail on the head with ” …monkeys whose acumen has not been tested.” It perfectly describes leading global warming activists that have the ear of our political leaders. I also think the field has gone overboard for statistical studies that obfuscate rather than enhance actual climate observations. An example is a temperature curve so festooned with annoying statistical markers so that you can’t hardly see the curve itself. I am with Earnest Rutherford here who opined that “…If your experiment needs statistics, you ought to have done a better experiment.” As to co-authors, I have never had any through the fifties and sixties and have no idea how it got so out of hand. But then again I was not subject to the academic pressures that brought forth this expansion of participatory credits. It really is out of control as Hansen’s recent paper on assessing dangerous climate change shows. He has 18 co-authors whose function apparently is more political than scientific. How else do you explain the presence of Jeffrey Sachs on his author list? He is a well-known psychologist but certainly not a climate scientist, and quite likely is meant to supply name recognition to those who do not know climate science. As far as climate science goes this article is nothing more than a pseudo-scientific fantasy about carbon reduction to save the world.
Well said.
As Mr. W. Briggs will tell you, often and clearly, statistics can make NO predictions about future experiments. It can only tell you characteristics about the actual data you have.
The 95% confidence interval was never meant to be anything but a rough guide to the variability of your data. Anything less is more or less a good guess. Robust data, or rigorous data would be in the range of 99.9999%.
If 75% of biomedical studies aren’t reproducible then it’s not a few bad apples. It’s a system that rewards notoriety over truth. For profit companies must demonstrate reproducibility every day. That’s why we trust their products. Federally funded science whether direct or via a tax subsidy can’t be reproduced because the scientist’s incentives are political not economic. Like all politicians they don’t need to be right just persuade enough people to win. But like political systems science must be seen.to function over the long hall which means reproducibility. Academic science like so much of academe turns out to be deeply corrupt and exploitative. Academe: our most corrupt sector.
Yet another indicator (as if the climate change fraud wasn’t enough) of the root problem of overpopulation in the ecological niche called “science.” Even if published results become rigidly reproducible (lol), what’s the benefit to society from all the scientific blah-blah-blah?
The big bang observations have already been made, calculus has already been invented, relativity has already been described, the germ theory of disease has already been posited, and antibiotics have already been proven. So, genius though you may be, it looks like you’re a little late to the party to make much of a dent. Furthermore, Sputnik is dead, the Cold War is over, and public funding of “science” has become an anachronism at best.
My suggestion, if the markets retest their Tech Wreck and Housing Bubble lows before the elections: it would probably be smart for 75% of so-called scientists to shore up their resumes with a more apropos job niche in mind, like a middle management spot in the fast food industry. For those left, if you’re interested in how many Higgs particles can dance on the end of a retrovirus, fine. Fund the research among yourselves and your pals. Just get your hand out of my back pocket.
Bruno & Vanini…..the fires still smoulder