Group calls for more transparency in science research, announces guidelines
From Rice University:
An international group of academic leaders, journal editors and funding-agency representatives and disciplinary leaders, including Rick Wilson, the Herbert S. Autrey Chair of Political Science and professor of statistics and psychology at Rice University, has announced guidelines to further strengthen transparency and reproducibility practices in science research reporting.
The group, the Transparency and Openness Promotion (TOP) Committee at the Center for Open Science in Charlottesville, Va., outlined its new guidelines in a story published in this week’s edition of the journal Science.
While transparency, openness and reproducibility are readily recognized as vital features of science and embraced by scientists as a norm and value in their work, a growing body of evidence suggests that those qualities are not necessarily evident today.
“A likely culprit for this disconnect is an academic reward system that insufficiently incentivizes open practices,” Wilson said. “In the present reward system, the emphasis on innovation undermines practices that support openness. Too often, publication requirements — whether actual or perceived — fail to encourage transparent, open and reproducible science.”
TOP’s objective is to translate scientific norms and values into concrete actions and change the current incentive structures to drive researchers’ behavior toward more openness.
“We know the disciplines differ in what is emphasized, so we sought to produce guidelines that focus on what is shared across disciplines,” Wilson said.
Each of eight standards has three levels of adoption in the TOP guidelines; each moves scientific communication toward greater openness, according to the article in Science. These standards are modular, facilitating adoption in whole or in part. However, they also complement each other; commitment to one standard may facilitate adoption of others.
The standards include citation standards for journals, data transparency, analytic methods (code) transparency, research materials transparency, design and analysis transparency, preregistration of studies, preregistration of analysis plans and replication.
Two standards reward researchers for the time and effort they have spent engaging in open practices. Citation standards extend current article citation norms to data, code and research materials. Regular and rigorous citation of these materials credits them as original intellectual contributions. Replication standards recognize the value of replication for independent verification of research results and identify the conditions under which replication studies will be published in the journal.
Four of the standards describe what openness means across the scientific process so that research can be reproduced and evaluated. Reproducibility increases confidence in findings and also allows scholars to learn more about what results mean. Design standards increase transparency about the research process and reduce vague or incomplete reporting of the methodology. Standards for research materials encourage the provision of all elements of that methodology, and data-sharing standards give authors an incentive to make data available in trusted repositories.
The final two standards address the values resulting from preregistration. Standards for preregistration of studies facilitate the discovery of research, even unpublished research, by ensuring that the existence of the study is recorded in a public registry. Preregistrations of analysis plans certify the distinction between confirmatory and exploratory research, or what is also called hypothesis-testing versus hypothesis-generating research. Making the distinction between confirmatory and exploratory methods transparent can enhance reproducibility.
“The guidelines are sensitive to concerns by both journals and researchers,” Wilson said. “For example, we encourage journals to state exceptions to sharing because of ethical issues, intellectual property concerns or availability of necessary resources. We encourage journals to pick and choose among the different levels and standards in order to define what they expect of the researchers.
“We acknowledge the variation in evolving norms about research transparency. Depending on the discipline or publishing format, some of the standards may not be relevant for a journal. Journal and publisher decisions can be based on many factors — including their readiness to adopt modest to stronger transparency standards for authors, internal journal operations and disciplinary norms and expectations,” Wilson said.
The present version of the guidelines is not the last word on standards for openness in science, according to the report.
“As with any research enterprise, the available empirical evidence will expand with application and use of these guidelines,” the TOP Committee wrote. “To reflect this evolutionary process, the guidelines are accompanied by a version number and will be improved as experience with them accumulates.”
An information commons and support team at the Center for Open Science is available (top@cos.io) to assist journals in selection and adoption of standards and will track adoption across journals. Adopting journals may also suggest revisions that improve the guidelines or make them more flexible or adaptable for the needs of particular subdisciplines.
To read the complete guidelines, go to https://osf.io/ud578/.
###
NOTE: The Open Atmospheric Society already embraces these initiatives. Check it out, become a member, and support open atmospheric science http://theoas.org
This is a fundamentally broken system. Band-aid solutions proposed by insiders is not the answer. All of these things should be entirely open and that requires alteration to the corrupt copyright and patent monopolies.
As a holder/coholder of 13 issued US patents on which two businesses are based, and as royalty receiving author of three books, I beg to differ with you about patents and copyrights. So does Article 1 Section 8.8 of the US Constitution.
The research transparency issue is simple. A patent filing requires sufficient disclosure that one ‘skilled in the art’ could replicate (make) the invention without undue effort. The US patent tradeoff is clear: full disclosure in return fo a time limited monopoly. Simply file before publishing. A final filing will have fuller disclosure than a typical research paper even under these open standards.
A fundamental issue with the patent system is that the vast majority of patents are simply anti-competitive in nature rather than protecting actual inventions. Dyson’s patents on vacuum cleaners are a case in point, the cyclone dust collector having existed in industry for decades prior to his marketing it as a domestic appliance. The problem becomes several orders of magnitude worse when patents on software are allowed, because it is so easy to patent a trivial and obvious piece of coding, and thereby block development of a whole range of software.
The other issue this creates is that as a developer you MUST either patent or publish every trivial feature of your work, otherwise someone else may patent it for the express purpose of holding you to random over the use of your own invention. Frankly we’d be better off without patents.
I’ve been out of the business for a good while now but years ago, when I was a development engineer for a rather large company, we were required to keep an engineering notebook to document that junk. We had to write down all of what you call ‘trivial features’,design idea, etc., preferably daily, and get someone else to initial and date the completed pages. The book could have no missing pages and each initialed page could not have free spaces where information could later be added.
These notebooks were also classified as trade secrete by the company, labeled and treated as such. According to the legal department this was sufficient to prove prior work/knowledge if/when someone else received a patent that later might interfere with our use of the idea. However, I do not know if these notebooks will still hold up under the current case law.
If they’re concerned about proprietary information, why are they publishing in the first place? The problem isn’t propriety information, it’s marketing by publication.
The insanity of the current PAL review and science orthodoxy system is best shown by this simple example:
http://www.fourmilab.ch/etexts/einstein/specrel/www/
Instead of Albert Einstein, substitute a Willis Eschenbach. (I.e., an “unknown”.) The “footnotes” on this copy are modern, the original had: 1. No references, 2. No footnotes or attributions to textbooks or even the authors own, published (essentially nil) and available works.
The concepts were “at odds” with much of “accepted” science.
The editor of the journal thought the paper had merit.
Modern publication standards: It would not be published. (Let us say Willis manages to “flesh out” his Thunderstorm/Thermostat hypothesis. Writes a good, cohesive, 30 page paper. With references and data! His chance of being published in a “peer reviewed” climate journal. ZERO. What other areas of science are so steeped in “orthodoxy” that progress is inhibited?
FORTRAN ERROR: Line 11 – Mismatched parenthesis
Fortran nightmares revisited!
Naw. We put a black marker stripe across the top of the submitted box(s) of Hollerith cards. Hunt and peck to find the errant card(s) when the OS 370 spuked them out cause of OLS mistranslations. Repunched, and resubmitted to the computer ‘gods’ using a cheat edge marked card stack. As I vaguely recall, sometimes about 0400 in the morning such things sometimes worked at the Aikins Comp Center at Harvard. Else would have to go fix the whole mess, including LRECL (remember, IBM JCL control language), and then resubmit something very sleep deprived the following night, since we never had priority before midnight.
The main underlying problem then was the damn sequential tape drives for long term memory. We were limited to 250k RAM in an IBM 370, at that time a three wired ferrite core. Ah, the memories.
“FORTRAN ERROR: Line 11” -Well, at least in Fortran it’s two brackets.
For C and its derivatives you don’t so much need a numeric keypad, as a special brackets-only keypad. Why the hell we use this gibberish in the days of computers that can just-about understand human speech, is beyond me. They had more lucid, human-readable languages in the 1950’s, and no kidding.
Nearly all fields of science. That is why there’s not outcry about Climategate, Their greatest fear is that someone turn the spotlight on them.
The Pons and Fleischmann saga illustrates that point. Wikipedia STILL lists their work as ‘pathological science’ even though it has been vindicated by several well-respected labs. Though, the reason they received such a hostile response in the first place was that they published without going through peer review first.
@Ian
That was not their fault. Their university insisted, very much against their inclinations.
I don’t think you can fix broken people with more rules, per se, *but* you can push a culture or subculture in the direction of honesty, and I hope that to some extent this works out that way.
I think other things have to be done that are unlikely to be done, but every little bit helps.
Ironic, isn’t it, that the Center For Open Science is located at UVa, which still won’t release records of Michael (Hockey Stick) Mann’s research.
Indeed, recall Peter Gleick – the ethics guru of global warming schills.
Yes, the system is broken, but this is a refreshing proposal and I hope it grows legs and arms and a couple of spleens as well. About dang time people in academia came together and officially stated what should have been standard practice all along.
Fully agree, Aphan. (I wonder why I kept thinking about ‘The Hockey Stick’ as I read this piece?)
What a charming way to introduce a blindingly obvious fact;
“A likely culprit for this disconnect is an academic reward system that insufficiently incentives open practices,” Wilson said”
Especially when they are sill paid to lie, cheat and offer up outrageous conclusions as fact from fanciful unproven theory …..
The National Science Foundation (NSF) is an independent federal agency created by Congress in 1950 “to promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense…” With an annual budget of $7.3 billion (FY 2015), we are the funding source for approximately 24 percent of all federally supported basic research conducted by America’s colleges and universities.
The director and all Board members serve six year terms. Each of them, as well as the NSF deputy director, is appointed by the President of the United States and confirmed by the U.S. Senate.
Most transparent Administration in history, might be one of the problems.
“Group calls for more transparency in science research…”
————
And will this happen in the fetid atmosphere of the obama junta?
Not bloody likely.
Especially not with the Chief Justice in his back pocket.
It’s rather ironic that the Open Access logo was chosen for this article.
These paid for submission journal probably have about the lowest standards, even of basic peer review. They have virtually no further requires concerning data reproducibility or anything else.
The main criterion for getting published is being willing to pay $999 or whatever for privileged.
In fact, why was the Open Access logo used ?!
This does not seem to mention open access and is printed in Science and addressing mainstream publishers’ journals. None of the guidelines seem to be promoting the open access model.
Why the logo? The idea of free access to papers is great but the reality of “buy yourself a peer review here” stinks. OA is the antithesis of what this article is about . Why give the false publicity and association with transparency that they don’t merit. ?
It’s a pun, y’all
It matters not a wit. Science as practiced is the handmaiden of its patron – government. With a few rare and brave exceptions, most will produce what is demanded by their sugar daddy, as it’s in their It matters not a wit. Science as practiced is the handmaiden of its patron – government. With a few rare and brave exceptions, most will produce what is demanded by their sugar daddy, as it’s in their instututional and personal professional interest to do so. It’s a political fight and skeptics are losing and will continue to lose, regardless of the scientific truth or what ordinary people think or experience. Why? Because government and its agencies, the press, Hollywood, schools and universities, scientific bodies, the UB, corporations and religious institutions all want it. Unfortunately its the good fight but a lost cause I fear, as that much power, money and authority is formidable, indeed. and personal professional interest to do so. It’s a political fight and sceptics are losing and will continue to lose, regardless of the scientific truth or what ordinary people think or experience. Why? Because government and its agencies, the press, hollywood, schools and universities, scientific bodies, the UB, corporations and religious institutions all want it. Unfortunately its the good fight but a lost cause I fear, as that much power, money and authority is formidable, indeed.
wow, sorry but that reply above was wacked out, format wise.
No, this problem far out-dates the current occupant of the White House.
It’s the left wing bias of academia in general. Soft-bellied liberal minded farts who never had guts to do anything but polish the seat of their pants on an office chair suddenly thought they could “save the world” by simply being a bit deceitful. Nobel cause [sic] corruption.
Their are many groups behind all this, each with their own motivations but science has been screwed for decades.
We are witnessing the end of the age of enlightenment.
I had expected to see a line (perhaps it is buried somewhere) demanding an accounting of funding for the paper and rejection if the funding source does not meet their approval.
Academics with their heads in clouds of theory, whose feet have never stood in a cow-pat …
“It’s a political fight and sceptics are losing and will continue to lose, regardless of the scientific truth or what ordinary people think or experience.”
They’ll win and win until they lose.
Wouldn’t it be nice if the Data Quality Act had some teeth.
“Science says”, does it?
http://liturgicalnotes.blogspot.co.uk/2015/06/science-says-does-it.html
“a growing body of evidence suggests that those qualities are not necessarily evident today.”
Begging the question fallacy.
As I pointed out in March, 2015, the Great Lakes Ice Cover went from 88.6% to 78.3% in 3 days with no rise in air temps. Now, the Arctic Ice is trending up and the Antarctic Ice is trending more ‘upper,’ but the anomaly has decreased from 0.034 to 0.014– looks like a concerted effort by all the yobs to stuff the numbers completely. Nawww, those upstanders wouldna do that, right? RIGHTTTTTT!
I work for an open access journal, and attend conferences for scholarly publishers. I can confirm that there is a definite move under way by many funders to require full publication of data sets, software code etc as a condition of the grant. There were several panel discussions about how to implement this at the Society of Scholarly Publishers convention last month. Paywall journals are of course worried about losing their exclusive rights to articles, and authors are worried about sharing their data.
I wondered whether smaller climate-focused funding orgs will have this requirement, knowing how opaque so much climate research has been. But I can confirm that within 3 years, both manuscripts and data from research funded by NIH, NSF and other major funders will have to be OA. Papers published in paywall venues like Science will have to be simultaneously published in an accessible archive.
AI, you offer a very hopeful message from the front lines. The SI requirement has already been helpful, deconstructing, for example, the Marcott mess in Science. But since Science chose not to respond to irrfutable evidence of Marcott’s scientific misconduct gleaned from his SI and dissected by Steve McIntyre and myself (separately), I will not hold my breath for this development.
Just think of it!
It is novel to want transparency and reproducibility in science?
Here is what the scientific method is to me.
1) Investigator is irrationally or rationally inspired to conceive a notion or relationship or a rule.
2) Investigator generates a hypothesis about a relationship or rule.
3) Investigator models the relationship based on some assumptions
4) Investigator designs an experiment, falsifiable if possible, and writes it down
5) Investigator executes experiment and records data
6) Investigator compares results with model and thereby the hypothesis
7) Investigator makes a conclusion about the relationship or rule and modifies hypothesis if necessary.
8) Investigator publishes his results and hopes others will read and attempt same experiment and validate work.
If #8 isn’t done, then it is not science.
What has “science” come to?
Well in Academia and government agencies there are some significant differences. Firstly, nobody is allowed to do research out of curiosity. There is nothing to fund the use of the laboratories or computers. So your numbers 1 and 2 are replaced by:
1) A research grant written to request positive proof of an assertion is put out to ‘tender’ for academic research.
2) The hypothesis to be supported by the research is supplied by the funding agency.
then having effectively bid on how they would prove the funder’s assertion the experiment progresses but must be designed to be unfalsifiable if possible.
then
7) Investigator makes the conclusion supporting the funders assertion.if this is not feasible return to 3)
8) Publish press release supporting funders assertion, some months later publish paper in Journal supported by funder, without supporting data or software and written in a way that makes reproducability difficult possibly by use of unique resource.
Forgive me. I don’t know where I went wrong. I acknowledge, respect and yield to your wise correction. OK can I have money… now?
“Garbage in, garbage out” is reproducible. Perhaps they should consider applying the scientific method. Assumptions of continuity and uniformity, reasoning through inference, and elevating correlation to the significance of causality has reproduced consensus science. The scientific method was designed to limit secular speculation and idolization. It has clearly failed to address the corruption sponsored by people’s pursuit of capital, control, and other material incentives.
I wonder why a certain Gleick is not mentioned?
I thought reproducible was intrinsic to science, as in if you aren’t providing enough information for others to reproduce your research, you re not practicing science, you are practicing sales and marketing. For example:
We have been able to discover imminent massive global species extinctions by studying the blue pumpernickel and using a significantly new algorithm that extrapolates blue pumpernickel data to the entire globe using climate change as a factor. We will release our extinction algorithm after we have completed our patent process. We expect patent issues to resolve right about the same time as mass extinctions begin.
Meanwhile our new diet health drink has been proven to increase health and reduce weight in our studies. At this time we can’t tell anyone how our studies were performed because that would threaten our patent rights. But trust us the results speak for themselves.
Although, some areas of experimentation are by nature not totally reproducible. Psychology probably contains the greatest number of examples. Does that mean we abandon such areas of research altogether, or should we instead base our findings on probabilities rather than absolutes?
Though, in all fields there arises the fallacy of ‘argument by way of ineptitude’ – as in, “I cannot reproduce your results, therefore they must be incorrect.”
Not that I’m standing-up for these charlatans, of course.
Same old wine in a different bottle. If peer review worked we’d not be having this conversation. Who thinks the same people in the same roles are going to come up with a new something that serves them worse than what they have now? Just stop.
Doesn’t get any better than this.
Thanks, dp.
We need to end peer review when there are superiors to review the work in other fields.
Climate scientists are way behind the times in their methods used to predict what would be the climate without man’s activities. Engineers, marketing and statisticians are decades if not centuries ahead of the methods used by the climate scientists to predict based on past patterns. If this prediction is wrong then any conclusions based on the predictions are invalid.
We need outsiders to publicly examine climate scientist’s claims for their computer models, again as they are by no stretch of the imagination even also rans in this area.
The scientists should not be allowed to use adjusted data for any conclusions without first having to justify it to outsiders as the only valid adjustment to me is to adjust by the difference between any reference network and the actual one where the original is no longer meeting the original conditions.
A wonderful and long overdue effort.
If widely accepted and enforced, sceience advancements will come much more quickly and with many fewer false steps.
The hallmark of transparency and reproducibly is climate science 97 % of scientists agree. (sarc)