Guest Post by Willis Eschenbach
From an interesting post entitled “Trust and Don’t Bother To Verify” on Judith Curry’s excellent blog , I’ve taken the following quote:
Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.
I’ve written before about the data and code archiving policies of the journal Science, and how they are not enforced for certain favored papers. In this regard, consider the case of Pinsky et al. This was a study that said that fishes were moving in the direction of the “climate velocity”. As a fisherman, I’m always interested in such studies. Their results appeared too regular to me, and I wanted to check their work. However, I found that neither their data nor their code was available. So last month, I wrote to the good folk at Science to see if they would enforce their own policies.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
Mmmm. Upon re-reading it, in retrospect I see that I was not as polite as I might have liked … but then I’ve grown bone-weary of Science not following its own data and code archiving policies for certain climate articles. In response to my email, I got … nothing. Zero. Zip. Nada word from anyone at Science.
Undaunted, I persevered. After waiting for two weeks, I wrote again, and this time I copied it around the organization:
From: Willis Eschenbach
Subject: Fwd: TO: Dr. Marcia McNutt
Date: October 1, 2013 11:24:03 PM PDT
To: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Friends:
I sent the following message two weeks ago to Dr. McNutt. However, it seems to have miscarried.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
I have not received a reply. Perhaps Dr. McNutt was not the proper person to address this to. So I am sending it to other addresses, in the hopes of getting some reply. I’m sorry to bother you, but if you could pass this to someone who could explain why you are not following your own written policies in this instance.
Many thanks,
w.
This time, I actually got a response, the very next day:
From: Andrew Sugden
Subject: Re: FW: TO: Dr. Marcia McNutt
Date: October 2, 2013 2:59:33 PM PDT
To: Willis Eschenbach
Dear Dr Eschenbach
Thank you for your message to Dr McNutt. I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.
Your sincerely
Andrew Sugden
And the following day, I replied:
From: Willis Eschenbach <willis@surfacetemps.org>
Subject: Re: TO: Dr. Marcia McNutt
Date: October 3, 2013 9:48:34 AM PDT
To: Andrew Sugden <asugden@science-int.co.uk>
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dr. Sugden, thank you most kindly for your reply. However, I fear that I’ve double-checked the paper and the SI, and there is far, far too little information, either in the paper itself or in the Supplementary Information, to allow their results to be confirmed, replicated, or falsified.
Here’s an example. It just happens to be the first area on their list, their study of the Eastern Bering Sea. The source of the data is given as being the RACE survey … but other than that we know nothing.
For example. The RACE survey covers 112 species … which of these species did they actually look at, and which ones did they leave out of their survey? Then they say they didn’t look at all tows … so which individual tows did they look at, and which did they leave out of their survey? Their only information on the subject is as follows:
While surveys were conducted in a variety of seasons (Table S1), we analyze each survey separately and use season-specific temperature data to account for these differences. We restricted our analysis to tows without gear and duration problems, to taxa that were resolved at least to genus, and to taxa that were sampled at least once per year to reduce effects from changes in taxonomic recording or resolution.
Unfortunately, that is far from enough information to be able to tell if their results are real or not.
Look, Dr. Sugden, this is not rocket science. To verify if what they have reported is a real effect, what we readers of Science need is very, very simple. It is a list in plain text that looks like this:
Year Month Day Tow# Species Catch Lat Start Long Start Lat End Long End Depth Temperature Result 1998 3 12 116 capelin 17.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Utilized1998 3 12 116 sculpin 1.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Excluded, uncertain identification…
Without that list showing exactly which data was used, and which data was excluded, and why, their results cannot be falsified … and unfalsifiable claims are not science, and not worth reporting in Science magazine
What they have done is just waved their hands and pointed at a huge pile of data, and said, We got our data from that pile … I’m sorry, but in 2013 that doesn’t cut it. To check their work, we need to know, not where they got their data, but exactly what data was used and what data was excluded. For all we know, there were transcription errors, or bugs in their computer code, or incorrectly categorized results, could be anything … but there’s no way to tell.
Nor is this an onerous requirement. The block of data representing the entire analysis would be a few megabytes. And presumably, in order to analyze the data, it’s all on the computer. So outputting a list of the data that was actually used or excluded is a few minutes work for a junior analyst.
I fear Science magazine and your Reviewers have dropped the ball on this one, Dr. Sugden. You have not done your due diligence and required the archiving of the data actually used in the study. Without that, you’re just publishing an anecdote, a charming fairy tale told by Dr. Pinsky.
It’s an interesting anecdote, to be sure … but it’s not science.
Please let me know what your magazine intends to do in this case. As it stands, you’ve published something which is totally unfalsifiable, in direct contravention of your own policies. Here are your relevant policies:
Data and materials availability
All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …
Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or electron microscopy maps for macromolecular structures, and climate data) must be deposited in an approved database, and an accession number or a specific access address must be included in the published paper. We encourage compliance with MIBBI guidelines (Minimum Information for Biological and Biomedical Investigations).
Details include but are not limited to:
- …
- Climate data. Data should be archived in the NOAA climate repository or other public databases.
- Ecological data. We recommend deposition of data in Dryad.
Clearly, the information that they provided falls woefully short of that required by your policies. No archive of their data. And pointing at a huge pile of data is not sufficient to let me “understand, assess, and extend the conclusions” as your policies require. I don’t have a clue what in the huge pile of data they used and what they excluded, so the information they gave about the location of the huge pile of data is useless.
The requirements, your own requirement, are bozo-simple, and easy to comply with. All they need to do is archive the collection of data that they actually used or rejected, and archive the computer code that they used to analyze that data.
They have done neither one …
Please let me know your plan of action on this, both for this paper and in general. As it stands, your magazine is passing off the unverifiable, unfalsifiable anecdotes recounted by Pinsky et al. as if they were real science. This is not the first time that your magazine has done that … and I don’t think that’s good for you personally as a scientist, for the reputation of Science magazine, or for science itself. People are trusting science less and less these days … and the publication of unverified anecdotes as if they were real studies is one of the reasons.
Your requirements for data and code archiving are simple and transparent. Now … you just have to enforce them.
Thanks for your assistance in all of this,
w.
Perhaps overly verbose but I wanted them to understand the issue. I waited almost two weeks, and when I’d gotten nothing, I wrote back:
From: Willis Eschenbach
Subject: Re: TO: Dr. Marcia McNutt
Date: October 14, 2013 11:00:05 AM PDT
To: Andrew Sugden
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Dr. Sugden;
As I detailed in my attached letter, neither the data nor the computer code for the Pinsky et al. study on the migration of fishes in response to climate velocity is available in a usable form.
While the data is publicly available, there is no detailed list or other means to identify the data actually used in the Pinsky study. Without that, in fact their data is not available—it is a needle in a haystack of needles. And without that, the study cannot be replicated, and thus it should not be published.
In addition, the computer code is nowhere to be found.
Both of these violate your express policies, as detailed below.
It’s been almost two weeks now since my attached letter was sent … I’m sorry to bother you again, but is there any progress in this matter? Or should I just submit this to the Journal of Irreproducible Results? Hey, just kidding … but it is very frustrating to try to see if there are flaws in published science, only to find out that Science itself is not following its own published policies.
My apologies for copying this around, but it may be that I’m not talking to the person in authority regarding this question. Do you have plans to rectify your omission in the Pinsky study, and require that they archive the actual data and code used? And if so, what are the plans?
Or are you going to do the Pontius Pilate?
In any case, any information that you have would be most welcome.
Many thanks for your assistance in this matter.
w.
PS—Please, do not tell me to contact the scientists directly. This is 2013. The exact data and code that the scientists used should be available at 2AM their time to a teenaged researcher in Ghana who doesn’t even speak the scientists’ language. That’s the reason you have a policy requiring the authors to archive or specifically identify their data, and to post their code. Pinsky et al. have done neither one.
That was sent on the 14th. Today’s the 21st. So I figured, at this point it’s been almost three weeks without an answer … might as well post up the story.
Now, would I have caught more flies with honey than with vinegar? Perhaps … perhaps not.
But the issue is not the quality or politeness of my asking for them to follow their own policies. Look, I know I can be abrasive at times, and that Dr. McNutt has no reason to like me, but that’s not the issue.
The issue is whether the journal Science follows their own policies regarding the archiving of data and code, or not. If you don’t like the way I’m asking them to do it, well, perhaps you might ask them yourself. I may be overly passionate, I might be going about it wrong, but at least I’m working in my own poor way to push both Science and science in the direction of more transparency through the archiving of data and code.
Sadly,
w.
They are not going to let you have their data….you only want to find something wrong with it. (Somehow that sounds slightly familiar!)
Head meet wall.
Willis, I think you did it perfectly.
As you pointed out to them, this is not the first time, so it is perfectly reasonable of you to be a little annoyed and to show that. You were, in fact, quite polite. You thanked Dr Sugden and outlined very clearly what you were about.
It would be nice to think he is trying to follow it up and wants to get back to you with a positive result, but it rather sounds like he’s been told to bin it and ignore you.
Are there any other subscribers to Science Magazine who can team up on this matter of data storage? What would it take to organize a few hundred – a few thousand would be better – to write in about it all at the same time?
I’m not a subscriber, and I’m just thinking aloud, but this is the place where most people meet, so… is it an idea worth following up? It just seems to me this sort of thing needs larger masses objecting to poor quality control going on – there and anywhere else (including MSM buildings – forget the politicians, they are immune and the news doesn’t cover it unless it’s Green. Actually marching on an editors office might get a better result. Alas, now I’m getting side-tracked. Sorry).
Good luck. I hope you get the result you and so many others want. That they dare to claim the policy and so badly ignore it, well, that’s fraudulent, isn’t it? To claim something is available when it isn’t? To put on a pretence of scrupulous attention to detail and data storage when it’s hollow?
Why should they let you see their data when all you want to do is find something wrong with it?
Well done. Whilst your use of words is “forthright” – it does not give any excuse whatsoever for the lack of meaningful response.
Science, paper without reliability. Dont read bullsh*t
I don’t think you encourage action with the tone and sarcasm in your letters. Why not write them as professionally as possible so they are taken as seriously as they should be? Stick to the facts and be clear and concise.
LOL @ur momisugly jorgekafkazar
Science ? how low have they fallen … perhaps to the level near toilet paper but at least about equal in quality to Scientific American or National Enquirer ?
Willis, you just gotta get with the In-Crowd. All that’s required is to check your soul at the door.
Governments looking to cut research budgets without appearing anti-science would do well to defund researchers who do not publish data or code.
Journals prove time and again that they’re just trade magazines with undeserved reputations.
I agree with Bill Jamison. Only address the issue based on their policy. Starting comparisons to a “teenage researcher in Ghana” puts it in the personal attack mode. It just adds more extra curricular information that obscures your central point.
They gave the data source, and they gave the rules that the scientists followed. That would, at a base line be sufficient. I’d say “thank you” to them, and ask for the code next. Then if they provided that, I would grab a random sample from the data that complies with the scientists data selection rule and see what you get. If it doesn’t match, then show them your work as a counter point, and ask for the scientists data set.
Not a scientist myself, but those are roughly the steps I’d taken.
P.S. Thanks for all the great articles!
Perhaps they should change the name of the journal to Non-Science pronounced, of course, nonsense. 😉
Willis: well done. I mourn the erosion of standards –the blatant hypocrisy– of scientific journalism. Two thoughts. First, many of the papers in these journals arise from work funded by NIH or NSF or other government sources. The underlying grants were predicated on the grantees complying with a raft of terms and conditions. Private donors likewise impose such terms. Perhaps a careful sifting of these –most would be posted for public review by applicants– could point to violations of policies on data access. Even if there are no present violations, grantors and donors can be approached to ask them to show leadership (and avoid wasting their money) by refusing to fund work that isn’t transparent. Certainly, there are loud cries that studies funded by industry should disclose raw data, so why not studies funded by the public or by philanthropies?
Second, again depending on grant fine print, the failure to document one’s work so that it can be audited for financial integrity –that real work of real value was in fact performed in exchange for the money paid– might be a violation of law, even criminal law. The False Claims Act is a terrible weapon and has I think been misapplied; but what’s sauce for the goose might be sauce for the gander as well.
To JorgeF and others who object to Willis’s “tone”: The policy is there, in writing. Someone who needs the requested data in order to evaluate the results of the “study” is entitled to have it. If the article doesn’t include it, it should not have been published. Willis’s request is perfectly reasonable, regardless of language. All of the sympathy for the poor folks at Science who receive such letters is misplaced; the issue is what is owed to the reader of the article, which according to the written policy is full and usable disclosure of specific data used and methods of use. If a researcher cannot verify or falsify results based on the authors’ data as shown in the article, the result is not science.
BioBob said:
“Willis, you just gotta get with the In-Crowd. All that’s required is to check your soul at the door.”
Superb! A brilliant summation of everything that is wrong with Alarmist ‘science’. If you don’t mind Bob, I’ll be passing your comment on to my friends over the next couple of weeks.
After one failure to reply at all and one fobbing off, I think some obstreperous rudeness is entirely in order. It’s likely to get you a much wider coverage and exposure — which is exactly what this sort of hypocritical practice needs.
All part of the time-honored tactic of announcing “standards” and then ignoring them, but taking full credit for them as if they were more than wind and words.
Willis,
Glad to see that you have a Doctorate at last; well according to Andrew Sugden, anyway.
Was your thesis on the subject of Ancient and Modern Mechanical Devices on Mainland Britain?
As long as Science is making some nice profits who gives a damn about maintaining standards anyway [ sarc / ]
.
Pre-coffee, so I might be missing something. But the problem is *not* that the dataset isn’t publicly available and mentioned. But that they didn’t properly document their methodology in the paper? (Not that it’s any different for replication.)
It’s true that it would be more embarrassing for them to ignore a reasonably polite request. But one does what it can and you did plenty. Thank you.
Why should they let you see their data when all you want to do is find something wrong with it?
Data is data, if there is something wrong with it they shouldn’t have used it in the first place! One discredited survey, can only lead to subsequent ones!
As for your tone Willis, I don’t have a problem with it, these people are creating studies that are directly causing our energy bills here in the UK to go up by 10% together with the blight on our lovely countryside of windmills, that no-one wants.
I would imagine the tone of the letter I received if I refused to pay my energy bills would not be as pleasant as the tone of your e-mails to these publishers!
Impressive effort!
My experience with peer review processes is that:
1. The reviewer (who is always a busy person) is asked to do the review in very short time. It is true (I do not publish or review in “Nature”) that usually extra time is given if demanded, but the pressure is important.
2. No data or code is offered by default.
If I wanted to check the conclusions and data in a paper, I would not write to the journal. I would write to the author. Most authors are only too willing to help (and yes I know there have been exceptions) and this would have been the more practical approach.
It’s very difficult to keep a civil tongue when faced with this sort of grossly unreasonable behaviour. I think you (Willis) showed a lot more restraint than would many given your previous with these organisations.
The point is this: it’s not just you that can’t see inside the ‘fish paper’. There will be tax funded fish people who can’t tell if this paper adds anything to the condition and they will already know that this is the case. The act of publishing becomes akin to the situation in Britain for many years where MP’s submitted expenses that everyone from top to bottom knew were bogus. But that’s top to bottom on the inside – outwardly it resembled proper practice.
These journals exist simply to provide the appearance of propriety so that the university funding cycle can continue.
You can look below the surface Willis because there’s nothing there and all of the in-crowd already know that. You are just being naive to think that a darn pesky citizen scientist can join in their game – there’s nothing to join in. The paper is not intended to ever be read, merely to be published in order to justify more money.
What’s needed is something to keep these organisations honest and that something would have to be able to directly affect their sources of funding. Is it possible to name and shame their university?
What would I know? As an engineer had I not fully documented everything I would just have been fired. That’s how industry deals with competence and ego issues. It largely works.