In the February 11, 2011 issue of Science Magazine, an editorial called Making Data Maximally Available has announced a new policy regarding requirements for authors, from the pen of Bruce Alberts, the Editor-in-Chief and two Deputy Editors.
Figure 1. Why it is necessary to see all the data and code to understand a given result. XKCD, Translation here.
Dr. Alberts, as I have criticized Science in the past for your its lax enforcement of your its own policies, I wish to equally commend you for the new policy. This is a welcome although as-yet untested step. The new policy, says (formatting and emphasis mine):
Science’s policy for some time has been that “all data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science” (see http://www.sciencemag.org/site/feature/contribinfo/). Besides prohibiting references to data in unpublished papers (including those described as “in press”), we have encouraged authors to comply in one of two ways: either by depositing data in public databases that are reliably supported and likely to be maintained or, when such a database is not available, by including their data in the SOM.
However, online supplements have too often become unwieldy, and journals are not equipped to curate huge data sets. For very large databases without a plausible home, we have therefore required authors to enter into an archiving agreement, in which the author commits to archive the data on an institutional Web site, with a copy of the data held at Science. But such agreements are only a stopgap solution; more support for permanent, community-maintained archives is badly needed.
To address the growing complexity of data and analyses, Science is extending our data access requirement listed above to include computer codes involved in the creation or analysis of data.
To provide credit and reveal data sources more clearly, we will ask authors to produce a single list that combines references from the main paper and the SOM (this complete list will be available in the online version of the paper).
And to improve the SOM, we will provide a template to constrain its content to methods and data descriptions, as an aid to reviewers and readers.
We will also ask authors to provide a specific statement regarding the availability and curation of data as part of their acknowledgements, requesting that reviewers consider this a responsibility of the authors.
We recognize that exceptions may be needed to these general requirements; for example, to preserve the privacy of individuals, or in some cases when data or materials are obtained from third parties, and/or for security reasons. But we expect these exceptions to be rare.
This is indeed excellent news. It is a huge step, from Michael Mann claiming that to ask for his code was “intimidation” and he would not reveal it, to Science Magazine requiring code as a condition of publication. As someone who has condemned actions of past Editors of Science Magazine, I give full marks and plaudits to Bruce Alberts and the Deputy Editors of Science Magazine for bringing your requirements in line with the computer-based nature of so much of contemporary science. I encourage Nature and all other scientific journals to adopt this policy.
If I had to pick one person who was responsible for this sea change at Science Magazine, I would point to the long and untiring fight by Steve McIntyre for full disclosure of scientific data, code, and results. Much of my own activism in this field is a direct result of watching him struggle for good science. With rare exceptions he has remained calm and restrained despite an unending string of vitriolic personal attacks and public slurs by many who fatuously claimed to be his scientific betters, and by scientists who foolishly claimed that his math was wrong and should be ignored. He has written a number of letters to the Editors of Science and NSF officials and individuals encouraging them to follow their own policies and require authors to archive datasets.
His generally unruffled demeanor and unwillingness to engage in the gutter tactics of too many leading AGW scientific luminaries has inspired in me an occasionally successful attempt to become less cowboy and more … more … well … let me call it “more Canadian”. Science Magazine owes him a debt, as do we all.
Next, Dr. Alberts, Science Magazine had a very poor record of compliance with your own stated policies under the previous Editorship of Donald Kennedy. Some of these issues are still unresolved, with the magazine repeatedly refusing to take a principled stand and require the authors to either put up or shut up — that is to say, produce the data required under those policies (stated above) in effect at the time of publication, or withdraw the papers. I’m sure Steve M. can tell you exactly which ones are still pending. The fact that Donald Kennedy was willing to give his friends a free pass is absolutely no reason for you to continue his reprehensible actions. In fact, it should give you more reason to clean this up, because you would be removing a stain from your reputation.
Finally, it is important to note that there is a new guardian who will be overseeing your future actions. That guardian does not sleep, does not rest, and looks at everything. It is something called the “AEI”. Some people say that the AEI is a net-roaming, self-replicating spiderbot species named the “Argus-Eyed Intarweb”. Others stoutly maintain it is a more mystical entity, the “All-seeing Eye of the Internet”.
In either case, as I have too often found to my cost, the AEI doesn’t miss anything. Something might not get noticed on day one, but sooner rather than later the AEI gets around to it, and my errors get exposed, my overstatements get trimmed back, my misunderstandings get explained, or my goose gets cooked.
I say this because as we saw with Donald Kennedy, having a policy and rigorously enforcing a policy are very different things. You have the opportunity to clean up past mistakes and to hold authors to true scientific principles. The AEI is watching to see if you do that, and it sees through walls, I’ve got the bite marks on my aspiration to prove it.
My best to you, and congratulations again, very well done and long overdue.
w.
While I too welcome this policy statement, I also know from watching Steve McIntyre’s travails with Science that policy is as flexible as the editor wishes it to be.
I shall remain wary until I see it in action through the tenure of more than one Science editor.
The first place to start is to answer why Science refused to publish criticisms of the notorious article by Naomi Oreskes in the December, 2004 issue of Science on the “scientific consensus” behind anthropogenic global warming. At present, I am not aware of anyone who has been able to replicate her results. Answers, Dr. Kennedy…
I remember back in 2007 when the rallying cry was,”Free the Code!”
I doubt that the source would have a problem with you using it, but you should still properly credit the cartoon you used.
See the original at http://xkcd.com/327/
(You’re welcome)
Nice letter, Willis.
There’s a place in the world for both cowboys and Canadians; restraint is sometimes a virtue, but too often in Canada it’s a result of a misplaced fear of offending the other. (This is no comment on Steve McIntyre, btw, who is fearless in his quest for truth!) I appreciate and wish I could emulate your direct cowboy approach, and hope this letter gets an appropriate response.
Science needs to wake up to the cold hand (eye?) of reality.
Rob S says:
February 25, 2011 at 8:04 pm
Thanks, fixed. I knew the strip but didn’t have the URL for the cartoon.
w.
An old-timer, eh? 😉
Good post, [Willis]. Indeed Steve McIntyre is owed a great deal of gratitude for his work. And he has done it all without responding in kind to the various slanders and slurs thrown at him.
I’m glad to see this new policy, but I do believe there are still some outstanding issues of non-compliance with the old policy at Science. Are they going to correct that as well? If I am correct, I am certain Steve McIntyre will know the specifics. I hope he revisits the issues.
Although he often amused us, for example by being completely unaware to what degree he was covered in chalk dust, we never questioned his integrity. Good job Dr. Alberts.
The new policy, encompassing both data and computer programs, is certainly welcome. As is the proposal that supplementary material explicate all data processing steps. But I have reservations. You’ll note that the editorial states that under the previous policy, authors were “encouraged” to comply with data storage policy. Certainly they were not held to it.
Time will tell if the new policy is truly a requirement, or merely a suggestion.
I guess Mann et al won’t be publishing in Science anymore. Perhaps they can redefine “peer-review” to mean published on Realclimate.
Hmm, I wonder if I’m related…
old Bobby Drop Tables.
An old favorite of mine that I used to show in database classes in a classroom that is now a pile of rubble in Christchurch NZ.
We shall see. I personally feel these people left science for politics 20 years ago. They are now shallow, stupid, and uneducated. All about money and not science.
The individuals who are interested in real science are now foreign students, coming over to pick over the bones of American academia. To take back home.
Innocent minds want to know, Willis, just where do you store your “aspiration”?
<P-)
#
#
evanmjones says:
February 25, 2011 at 7:54 pm
I remember back in 2007 when the rallying cry was,”Free the Code!”
#####
your welcome. First posted as a tagline to all my comments at RC.
The first scientist to join in our campaign was Judith Curry.
Good post, i am welcome this new policy of science magazine
bellows in india
I’m with you Willis. I can’t agree more that we need a more Canadian attitude in science in general and climatology specifically. And that is not because I am a Canadian, which I am. I am also a US citizen and find myself continually embarrassed by the actions and attitudes of many of my countrymen. I had great respect for Steve McIntyre from our mutual days in the mining industry. His efforts on the “climate good data brief” simply enhanced that respect.
evanmjones says:
February 25, 2011 at 7:54 pm
I remember back in 2007 when the rallying cry was,”Free the Code!”
==============================================
No doubt, I’m not holding my breath. As was pointed out, we’ll see how rigorously this new policy gets enforced.
> Finally, it is important to note that there is a new guardian who will be
> overseeing your future actions. That guardian does not sleep, does not rest,
> and looks at everything. It is something called the “AEI”. Some people say
> that the AEI is a net-roaming, self-replicating spiderbot species named the
> “Argus-Eyed Intarweb”. Others stoutly maintain it is a more mystical entity,
> the “All-seeing Eye of the Internet”.
===============================================
This is why linux has prograssed so much in 20 years. See http://en.wikipedia.org/wiki/Linus's_law
Linus’s Law is a claim about software development, named in honor of Linus Torvalds and formulated by Eric S. Raymond in his essay “The Cathedral and the Bazaar”. The law states that “given enough eyeballs, all bugs are shallow”; or more formally: “Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone.” Presenting the code to multiple developers with the purpose of reaching consensus about its acceptance is a simple form of software reviewing. Researchers and practitioners have repeatedly shown the effectiveness of the reviewing process in finding bugs and security issues, and also that reviews may be more efficient than testing.
===============================================
In the Open Source world, it is possible (even if not likely) that a 14-year-old kid will find a bug and submit a patch that ends up being accepted into the linux kernel or into a program. We’d be a lot better off today if Briffa’s data had been inspected with a fine-tooth comb when the hockey stick was first released. To those scientists who object to this… if you believe that your data/algorithms/code can’t stand up to public scrutiny by laymen…
a) what does this say about the quality of your paper?
b) what does this say about the quality of the current peer-review process that lets your paper get published in the first place with those problems?
When data and code are published is called science. When not, it is called magic.
Hey, mods or Anthony or anyone…….does anyone know how to get in contact with Dr. Curry?
There’s a matter that I believe it is fairly important an fairly urgent that needs Dr. Curry’s attention. Please e-mail me.
All of this talk about “old-timers” from 2007 makes me feel like Methuselah…
So, members of the Team and other AGW cult followers will either be given some form of special dispensation, so they can continue on as before, or Nature will cease publishing unfounded climate scare stories.
Five bucks says it’s the former.
It still baffles the mind that an honest scientist wouldn’tqant to freely give up their code and/or data to others for scrutiny. In my job it’s a requirement. Every experiment has to go through “second chemist review” which means another chemist not connected with the program checks all data tables, math, code, etc. before it can be officially released. Slows the process down but there are always mistakes found and corrected.
Total data access is not required unless the results are so obviously model conceived. We wish to know that the data is based on real time observations of the real world without any fudging of the data to push the results towards the researcher’s preconceived ideas. (Most research results can show that preconceived ideas lean results towards those ideas.) We also need to know that the methods used in the data analysis is correct, clear and easily explained or understood by peers.
If your research involves the real world where observation and measurement gives the required data why use a model. Models only prove one thing, laziness. If your research involves observation of impossibly small things, like particle physics does, then a model might help understanding what is happening.
Willis
Thank you for yet another graciously-worded piece that is exactly what needs to be heard. And thank you for crediting Steve McIntyre so well.
Before the recent events (Steig-O’Donnell, Judith Curry’s pieces, etc) I was seriously concerned that Steve had lost heart through the continual and relentless bad grace with which he has been treated by “the Team”, and the continuing efforts of the media and establishment to support “the Team” and refuse to do the necessary sanitization like asking Jones WHY he asked his mates to delete emails, investigate the science itself, investigate the IPCC setup and accountability, etc.
Thank God you’re still a cowboy, Willis. There are virtues too, like calling a spade a spade, having hands-on allround passionate interest, and not being so far up your ivory tower you’re afraid to say sorry.
That’s a very “in” joke — but once it clicked — it was hilarious!
Willis, I wrote a poem about a cowboy but he is nothing like you 🙂
Corporate Cowboy:
We’re here to plan a takeover of the T.N. Mine.
I need a finance document by half past nine.
I want some strategies for public opinion,
A list of tradeoffs for the workers’ union.
Well need some wampum for the indiginous tribe,
Offer them jobs and offer a bribe.
Jane- call the Governor, I need him here,
Tell him to meet me at six for a beer.
I was once a lonesome cowboy,
I fought at the O.K.Corral,
But now I’m a corporate cowboy
And I’m doing very well,
So stand aside sister and brother, I’m after wealth and fame,
Rival companies better take cover, power’s the name of the game.
Yep! Third world countries are the new frontier,
No regulations and nothing to fear
From do-good activists crowding the scene,
Preaching rights and an environment that’s clean.
None of them know what makes the world turn,
It’s crash through or crash, that’s what they need to learn.
Shiny bums in libraries doing research
Will never get action, they’re just stupid jerks.
I was once a lonesome cowboy,
I fought at the O.K.Corral,
But now I’m a corporate cowboy
And I’m doing very well.
So stand aside sister and brother,
I’m after wealth and fame, Rival companies better take cover, power’s the name of the game.
Jane-polish my spurs and fetch my horse.
Well Beth Cooper,
Polish your pencil and write a prose about Willis.
“We recognize that exceptions may be needed to these general requirements; for example, to preserve the privacy of individuals, or in some cases when data or materials are obtained from third parties, and/or for security reasons. But we expect these exceptions to be rare.”
These loopholes have been used disingenuously in the past. I don’t understand how you can receive a paper for publication and then when you ask “where is the data?” are told that it can’t be supplied because of one of these “exceptions” publish it anyway.
“Show your work” should not be an optional condition in science.
“I guess Mann et al won’t be publishing in Science anymore. Perhaps they can redefine “peer-review” to mean published on Realclimate.”
And since being an anonymous reviewer on a paper that’s directly challenging one of yours is not considered bad form then they can probably redefine peer review on Real Climate to mean they get to be an anonymous reviewer on their own papers. There couldn’t be anything unethical about that.
Excellent comment Willis.
In the same issue of Science, there ia an article by Prof. Jonathan Overpeck and colleagues “Climate Data Challenges in the 21st Century”. They say many of the right things:
– “…climate scientists must not only share data among themselves, but they must also meet a growing obligation to facilitate access to data for those outside their community and, in doing so, respond to this broader user community to ensure that the data are as useful as possible”.
– “These efforts [national climate assessments and adaption/mitigation efforts] will succeed only if climate data are made readily accessible in forms useful to scientists and nonscientists alike”.
– “Open and free availability of model data, observations, and the software used for processing is crucial to all aspects of the new paradigm. Governments that currently restrict either model output or observed data distribution must be convinced that it is in the best interests of everyone that all climate data be made openly available to all users, including those engaged in research and applications. International agreements must eliminate data restrictions, just as journals and funding agencies should require easy access to all data associated with the papers they publish and the work they fund”.
The work that Steve McIntyre has done to highlight these issues is not referenced in this paper either.
See http://www.sciencemag.org/content/331/6018/700.abstract
Have no faith that this step, perhapsin the right direction on paper, will change anything. Culture is a really hard thing to change.
DEEBEE says:
February 26, 2011 at 6:06 am
Have no faith that this step, perhapsin the right direction on paper, will change anything. Culture is a really hard thing to change.
——————————–
I have faith in this, but then I know Bruce Alberts personally. I was a post-doc at UCSF when he was doing his most famous work. He’s a brilliant scientist and a great person. I’m proud to say that one of my personal favorite scientific works from my own work was submitted to PNAS by him.
In the context of this post though, I had another interaction with him somewhat later in my career, where he really showed me the rigor of his scientific analysis. The story’s too long to go into detail, except to say that it was similar to the above, albeit on a smaller scale.
Welcome news, IF Science Magazine follows their newly stated policy.
Should the Nobel committee ever wish to resurrect the Nobel Peace Prize as something of value, after years of depreciating it to little more than junk, they should give it to Steve McIntyre.
“We recognize that exceptions may be needed to these general requirements; for example, to preserve the privacy of individuals, or in some cases when data or materials are obtained from third parties, and/or for security reasons. But we expect these exceptions to be rare.”
—————————————————————
Oh, dear. Unless they disclose when these ‘exceptions’ are made, and why, we are only slightly forrader than we were.”Security reasons’ means whatever they want it to mean. And, why should materials obtained from third parties get an across-the-board waiver? If that is the basis of conclusions/recommendations, surely it is not good enough to say – We can’t tell you, for privacy reasons? Of course it would be wrong to identify individuals, but there is a lot that can be exposed to scrutiny short of that.
The good news is – a journal which claims to pontificate on ‘the science’ is feeling the pressure and has appointed an apparently ethical editor.
The bad news is – the rest of them don’t give a rats’.
It is great to hear from Dr Alberts’ former students. He must indeed be something special to attract comments on WUWT, given the large population of former students on the globe, in the US, let alone on WUWT.
But, as has been discussed in PNS (Post Normal Science) threads, being a good person or a good scientist is not the be-all and end-all. In the current environment, he needs to be able to stand up to aggressive challenges from people who are convinced that they have the One True Way to Save the Planet. How he goes on that score is the critical issue.
I do think that there is a market niche (alas) for publications that insist on transparency.
Why not tell Willis to go back to school and study some basic science instead of trying to snipe at the big boys over things that are far beyond his ken.
REPLY: Why not put your name to your words, if your truly believe them, instead of hiding behind a bogus name? It is easy to criticize, any coward can do it from anonymity. It takes character however to put your name to your words and stand behind them. – Anthony Watts
Amused said
“Why not tell Willis to go back to school and study some basic science instead of trying to snipe at the big boys over things that are far beyond his ken.”
The only thing anyone should be ” Amused ” about is that this individual apparently has an IQ that is apparently high enough to learn how to type.
When you make blanket statements and personal attacks like that without any evidence, you show that not only are you unwilling to listen to someone else’s opinion, but you are also willfully blind, and part of the problem of this bastardization of Science and it’s principals and methods.
Then again, I know that back at RC and Sceptical Science, which is where I’m sure you came from, that this is standard practice.
Diego Cruz says:
February 26, 2011 at 12:21 am
””When data and code are published is called science. When not, it is called magic.”’
almost correct, the last word could perhaps be better replaced with ‘malpractice’ or ‘fraud’ or even ‘religion’! ….hmmmm….
That is a wonderful cartoon. Really captures the Left’s heart felt feelings about everyone else; that is, because the little trick was based on some cleverness and ingenuity, it is excusable and everyone will just have to deal with it. And, of course, it is your fault because your data checking is inadequate.
Here’s hoping for a copy-paste job of this policy to AGU and AMS journals. And thorough enforcement.
> Dr. Alberts, as I have criticized Science in the past for your lax enforcement of
> your own policies, I wish to equally commend you for the new policy.
I think your letter would be less alienating and more effective, if it were to separate characterizations of the journal’s past actions from the actions of Alberts himself.
I recommend the following changes:
“Dr. Alberts, as I have criticized Science in the past for ITS lax enforcement of ITS own policies, I wish to equally commend you for the new policy.”
“Next, Dr. Alberts, Science Magazine had a very poor record of compliance with ITS own stated policies under the previous Editorship of Donald Kennedy.”
“In fact, it should [] give more reason to clean this up, because THAT would be removing a stain from THE JOURNAL’S reputation.”
“THE JOURNAL HAS the opportunity to clean up past mistakes and to hold authors to true scientific principles. The AEI is watching to see if THE JOURNAL DOES that, and it sees through walls[.] I’ve got the bite marks on my aspiration to prove it.”
Amused. says:
February 26, 2011 at 10:11 am
First, Anthony is 100% correct. I downweight all anonymous comments severely. If you don’t have the blanquillos to stand behind your words, why would I initially give weight to what you think? I don’t ignore anonymous comments, I’ll take scientific truth written on a bathroom wall, but I initially downweight anonymous comments. After a while you get to know who’s anonymously talking science and who’s talking trash.
But anonymous insults like yours? Sorry, they carry no weight at all . Zero. Zip.
I encourage all posters to stand up and be recognized under their own name. I understand that there are still people in business or education situations where if you admit to climate heresy people throw decaying fruit at you or something, I understand that. But short of that, if you truly believe in what you are saying, why not take authorship of it?
In addition to Anthony’s comment, Amused, let me add that this is a scientific blog. If you think that I am wrong about something, you may very well be correct — I’ve been wrong more times than I care to recall.
To show that I am wrong, however, requires much more than a smart mouth. If you want to show that I don’t understand the “basic science” as you say, what you need to do is:
1. QUOTE WHAT I SAID that you think is in error. I can’t talk about my sins of commission or omission unless the topic is crystal clear.
2. State your objections to what I said. However, to move the scale, you need to offer more than an opinion, particularly an anonymous opinion. You need to provide things like facts. Observations. Previous scientific studies. References to textbooks. Logical expositions, with each step solidly established. Citations to recognized authorities. That kind of thing.
Do that, and you can get some traction here. Otherwise, you’re just spinning your wheels.
w.
Amused. says:
February 26, 2011 at 10:11 am
Why not tell Willis to go back to school and study some basic science instead of trying to snipe at the big boys over things that are far beyond his ken.
Willis Eschenbach says:
February 26, 2011 at 7:29 pm
First, Anthony is 100% correct. I downweight all anonymous comments severely. If you don’t have the balls to stand behind your words, why would I initially give weight what you think? I don’t ignore anonymous comments, I’ll take scientific truth written on a bathroom wall, but I initially downweight anonymous comments. After a while you get to know who’s anonymously talking science and who’s talking trash.
– – – – – –
Willis,
In principle I share both Anthony’s and your views on anonymity, though I would describe them differently.
One of my several lines of reasoning criticizing anonymity revolves around ownership of ideas. If someone does not give their identity, it raises consequences in attributing ownership of their ideas and reckoning with plagiarism, misquoting, etc.
Another line of reasoning about why anonymity is controversial is dual personas. It leads to no restriction on one person representing themselves as many people, often with inconsistent views, personal specifics and professions. That is not intellectually honest.
I have several other lines of thought about controversies in anonymity . . . but that was a starter.
John
I heartily support Dr. Alberts’ and Science’s board for making this decision.
It behooves us to eagerly voice our approval, when those whom we’ve disagreed with in the past make decisions and take actions that we are happy to see. I hope such agreement is taken as an act of conciliation on our part. Contrary to some people’s opinion of skeptics, we are not just contrarians, and we do recognize when common ground is found.
That common ground is the good of science.
I found right away that skeptics (who I am proud to count myself among) have focused much of their efforts against what they saw as actions and agendas that were detrimental to science. Thus, when those we’ve opposed do something for the good of science, frankly, it is time for a toast! 🙂
This IS a big step, especially if Dr Alberts is able to follow through on it.
It would be additionally excellent if Dr Alberts might address the past lack of code having been submitted. It may not be feasible, but an ex post facto rule might be good. And a big step forward might well also be a published list of those articles which do not meet the new rule.
If all the code used in climate research were currently available, the replication process would have a great capability to solidify (or refute) what work has been done.
Kev-in-Uk says: February 26, 2011 at 12:31 pm
Diego Cruz says: February 26, 2011 at 12:21 am
When data and code are published is called science. When not, it is called magic.’
almost correct, the last word could perhaps be better replaced with ‘malpractice’ or ‘fraud’ or even ‘religion’! ….hmmmm….
I would go for ‘malpractice’.
It is the failure of a professional person, as a physician or lawyer, to render proper services through reprehensible ignorance or negligence or through criminal intent, especially when injury or loss follows.
http://dictionary.reference.com/browse/malpractice
It is the consequence of their actions that is being decribed and needs to stated in public discussion. The cause, motive and intention can be determined later through discusion, which needs to confirm malpractice has occurred, not question it.
It might even help Scientists to promote themselves as part of a profession.
In most cases, it should be adequate to describe the algorithm in a way that anyone could program it in any language and get the same results. But it can’t hurt to provide the exact code as well.
I suspect that many authors would be embarrassed at how sloppy and opaque their code is. In many cases, they might be using some code that was written years ago by someone who has since left the organization, but which seems to work. Forcing them to provide it might induce them to clean it up and comment it to where they understood it themselves.
Hu McCulloch says:
February 27, 2011 at 8:05 am
Many thanks, Hu. The problem, as I had hoped to illustrate with the cartoon, is that often despite a very clear description of the algorithm the code ends up doing something unintended. Michael Mann’s hockeystick code had this flaw. These kinds of problems are time-consuming and nearly impossible to find without the code.
Which is why I disagree that “In most cases, it should be adequate to describe the algorithm in a way that anyone could program it in any language and get the same results.” Without the code, we simply don’t know if that is true, because despite a flawless description, the code may contain hidden flaws that have subtly influenced the outcome.
My best to you,
w.