Guest Post by Willis Eschenbach
From an interesting post entitled “Trust and Don’t Bother To Verify” on Judith Curry’s excellent blog , I’ve taken the following quote:
Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.
I’ve written before about the data and code archiving policies of the journal Science, and how they are not enforced for certain favored papers. In this regard, consider the case of Pinsky et al. This was a study that said that fishes were moving in the direction of the “climate velocity”. As a fisherman, I’m always interested in such studies. Their results appeared too regular to me, and I wanted to check their work. However, I found that neither their data nor their code was available. So last month, I wrote to the good folk at Science to see if they would enforce their own policies.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
Mmmm. Upon re-reading it, in retrospect I see that I was not as polite as I might have liked … but then I’ve grown bone-weary of Science not following its own data and code archiving policies for certain climate articles. In response to my email, I got … nothing. Zero. Zip. Nada word from anyone at Science.
Undaunted, I persevered. After waiting for two weeks, I wrote again, and this time I copied it around the organization:
From: Willis Eschenbach
Subject: Fwd: TO: Dr. Marcia McNutt
Date: October 1, 2013 11:24:03 PM PDT
To: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Friends:
I sent the following message two weeks ago to Dr. McNutt. However, it seems to have miscarried.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
I have not received a reply. Perhaps Dr. McNutt was not the proper person to address this to. So I am sending it to other addresses, in the hopes of getting some reply. I’m sorry to bother you, but if you could pass this to someone who could explain why you are not following your own written policies in this instance.
Many thanks,
w.
This time, I actually got a response, the very next day:
From: Andrew Sugden
Subject: Re: FW: TO: Dr. Marcia McNutt
Date: October 2, 2013 2:59:33 PM PDT
To: Willis Eschenbach
Dear Dr Eschenbach
Thank you for your message to Dr McNutt. I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.
Your sincerely
Andrew Sugden
And the following day, I replied:
From: Willis Eschenbach <willis@surfacetemps.org>
Subject: Re: TO: Dr. Marcia McNutt
Date: October 3, 2013 9:48:34 AM PDT
To: Andrew Sugden <asugden@science-int.co.uk>
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dr. Sugden, thank you most kindly for your reply. However, I fear that I’ve double-checked the paper and the SI, and there is far, far too little information, either in the paper itself or in the Supplementary Information, to allow their results to be confirmed, replicated, or falsified.
Here’s an example. It just happens to be the first area on their list, their study of the Eastern Bering Sea. The source of the data is given as being the RACE survey … but other than that we know nothing.
For example. The RACE survey covers 112 species … which of these species did they actually look at, and which ones did they leave out of their survey? Then they say they didn’t look at all tows … so which individual tows did they look at, and which did they leave out of their survey? Their only information on the subject is as follows:
While surveys were conducted in a variety of seasons (Table S1), we analyze each survey separately and use season-specific temperature data to account for these differences. We restricted our analysis to tows without gear and duration problems, to taxa that were resolved at least to genus, and to taxa that were sampled at least once per year to reduce effects from changes in taxonomic recording or resolution.
Unfortunately, that is far from enough information to be able to tell if their results are real or not.
Look, Dr. Sugden, this is not rocket science. To verify if what they have reported is a real effect, what we readers of Science need is very, very simple. It is a list in plain text that looks like this:
Year Month Day Tow# Species Catch Lat Start Long Start Lat End Long End Depth Temperature Result 1998 3 12 116 capelin 17.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Utilized1998 3 12 116 sculpin 1.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Excluded, uncertain identification…
Without that list showing exactly which data was used, and which data was excluded, and why, their results cannot be falsified … and unfalsifiable claims are not science, and not worth reporting in Science magazine
What they have done is just waved their hands and pointed at a huge pile of data, and said, We got our data from that pile … I’m sorry, but in 2013 that doesn’t cut it. To check their work, we need to know, not where they got their data, but exactly what data was used and what data was excluded. For all we know, there were transcription errors, or bugs in their computer code, or incorrectly categorized results, could be anything … but there’s no way to tell.
Nor is this an onerous requirement. The block of data representing the entire analysis would be a few megabytes. And presumably, in order to analyze the data, it’s all on the computer. So outputting a list of the data that was actually used or excluded is a few minutes work for a junior analyst.
I fear Science magazine and your Reviewers have dropped the ball on this one, Dr. Sugden. You have not done your due diligence and required the archiving of the data actually used in the study. Without that, you’re just publishing an anecdote, a charming fairy tale told by Dr. Pinsky.
It’s an interesting anecdote, to be sure … but it’s not science.
Please let me know what your magazine intends to do in this case. As it stands, you’ve published something which is totally unfalsifiable, in direct contravention of your own policies. Here are your relevant policies:
Data and materials availability
All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …
Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or electron microscopy maps for macromolecular structures, and climate data) must be deposited in an approved database, and an accession number or a specific access address must be included in the published paper. We encourage compliance with MIBBI guidelines (Minimum Information for Biological and Biomedical Investigations).
Details include but are not limited to:
- …
- Climate data. Data should be archived in the NOAA climate repository or other public databases.
- Ecological data. We recommend deposition of data in Dryad.
Clearly, the information that they provided falls woefully short of that required by your policies. No archive of their data. And pointing at a huge pile of data is not sufficient to let me “understand, assess, and extend the conclusions” as your policies require. I don’t have a clue what in the huge pile of data they used and what they excluded, so the information they gave about the location of the huge pile of data is useless.
The requirements, your own requirement, are bozo-simple, and easy to comply with. All they need to do is archive the collection of data that they actually used or rejected, and archive the computer code that they used to analyze that data.
They have done neither one …
Please let me know your plan of action on this, both for this paper and in general. As it stands, your magazine is passing off the unverifiable, unfalsifiable anecdotes recounted by Pinsky et al. as if they were real science. This is not the first time that your magazine has done that … and I don’t think that’s good for you personally as a scientist, for the reputation of Science magazine, or for science itself. People are trusting science less and less these days … and the publication of unverified anecdotes as if they were real studies is one of the reasons.
Your requirements for data and code archiving are simple and transparent. Now … you just have to enforce them.
Thanks for your assistance in all of this,
w.
Perhaps overly verbose but I wanted them to understand the issue. I waited almost two weeks, and when I’d gotten nothing, I wrote back:
From: Willis Eschenbach
Subject: Re: TO: Dr. Marcia McNutt
Date: October 14, 2013 11:00:05 AM PDT
To: Andrew Sugden
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Dr. Sugden;
As I detailed in my attached letter, neither the data nor the computer code for the Pinsky et al. study on the migration of fishes in response to climate velocity is available in a usable form.
While the data is publicly available, there is no detailed list or other means to identify the data actually used in the Pinsky study. Without that, in fact their data is not available—it is a needle in a haystack of needles. And without that, the study cannot be replicated, and thus it should not be published.
In addition, the computer code is nowhere to be found.
Both of these violate your express policies, as detailed below.
It’s been almost two weeks now since my attached letter was sent … I’m sorry to bother you again, but is there any progress in this matter? Or should I just submit this to the Journal of Irreproducible Results? Hey, just kidding … but it is very frustrating to try to see if there are flaws in published science, only to find out that Science itself is not following its own published policies.
My apologies for copying this around, but it may be that I’m not talking to the person in authority regarding this question. Do you have plans to rectify your omission in the Pinsky study, and require that they archive the actual data and code used? And if so, what are the plans?
Or are you going to do the Pontius Pilate?
In any case, any information that you have would be most welcome.
Many thanks for your assistance in this matter.
w.
PS—Please, do not tell me to contact the scientists directly. This is 2013. The exact data and code that the scientists used should be available at 2AM their time to a teenaged researcher in Ghana who doesn’t even speak the scientists’ language. That’s the reason you have a policy requiring the authors to archive or specifically identify their data, and to post their code. Pinsky et al. have done neither one.
That was sent on the 14th. Today’s the 21st. So I figured, at this point it’s been almost three weeks without an answer … might as well post up the story.
Now, would I have caught more flies with honey than with vinegar? Perhaps … perhaps not.
But the issue is not the quality or politeness of my asking for them to follow their own policies. Look, I know I can be abrasive at times, and that Dr. McNutt has no reason to like me, but that’s not the issue.
The issue is whether the journal Science follows their own policies regarding the archiving of data and code, or not. If you don’t like the way I’m asking them to do it, well, perhaps you might ask them yourself. I may be overly passionate, I might be going about it wrong, but at least I’m working in my own poor way to push both Science and science in the direction of more transparency through the archiving of data and code.
Sadly,
w.
data as used. code as run.
what the hell do these people not understand.
I’ve had similar problems with scaffetta and vaughan Pratt.
This is not that hard. Turn in the data as you used it and the code as you ran it.
That way I can
A. Check the work quickly
B. Build on your science if it good
C. help correct it if its not.
I would suugest that people at WUWT start filling email boxes at Science. with polite requests for the data and the code. Grass roots effort. people rely too much on guys like willis and steve Mc and me to make these requests. The more people who ask or complain in a nice way the better chance we have of changing things. We might not agree about the climate but we can agree about the importance of supplying code and data. On this there should be no sides to the debate. both sides, all sides. Show your data and your code or stop wasting our time.
Wmasaw failed to see the tongue-in-cheek nature of Andrew Harding’s comment. Some people don’t get irony, but in this case perhaps Wmaswa does not know the history behind this.
The question “Why should they let you see their data when all you want to do is find something wrong with it?” is adapted almost word for word from a response by the hockey stick team (can’t remember who exactly) to requests for data. It was the standard response before requests backed by the FOIA started to be sent and before Climategate. Looks like some people have not learned anything.
– – – – – – – – –
JohnWho,
Appreciate getting the comment from you. Thanks.
Can you please explain why you think my comment was ‘attacking’ Willis? That is a sincere question on my part; I am not trying to be rhetorical; I really would like to know what you think constitutes ‘attacking’ on my part.
As to your point suggesting that I request data/methodology from Science Magazine, no. I have my own priorities wrt the climate science situation, as I am sure you do, and Science Magazine is not one of mine at this time.
John
“”Or are you going to do the Pontius Pilate?”
As usual Willis, you’re going to have to decide whether your mission is to actually get people to do what you want (in this case provide data), or be satisfied with showing them up. I understand and share the impulse to do the latter, but it’s not the way to go if you’re interested in results..
“Things fall apart; the centre cannot hold… ”
That means different things in different contexts, but what I mean by it here is that “scientific” journals no longer are doing the job they were created to do, advancing science. Now they are a place for the “ins” to meet requirements for tenure, or research contracts, or whatever. Real science is carefully excluded, because it is usually done by someone outside the circle. And I’m not just talking about “Science” but as far as I know, it’s true of all of them.
Those doing real science need a place to publish their findings, but “science” journals are not that place. The “place” will end up being on-line, With all the problems that will have, it’s still the only way. It might be on a web-site maintained by a university, but they are going to go away too, because even more than the science journals are no longer about science, the universities are no longer about learning, and also not needed for it. Probably each scientist will have to set up his own web-site, publish his results there, and try to get the attention of others in his field, probably by sending them emails.
Anyway trying to reform Science is a waste of time, because it’s really already dead.
What scientists mean when they write… ☺
It seems to me that many posters are missing the point that Willis was making. That is the magazine (sorry “Journal”) has a stated policy …..
“All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …” Notice “All data necessary to understand, assess and extend the conclusions……any reader of Science. All computer codes involved in the creation or analysis………..”
The Journal has not followed that policy. Whether he could have used incomplete data or a different computer code of his own is irrelevant. That would not be meeting the policy.
The magazine (for that is what it is) is LIE ing!!!!!!!!!
You could of course ask other esteemed establishments if they have a study of climate and its effects on fish. If they point you at this study, just use your ammunition to prove that is not a valid study.
If you paper arrives at the “correct” conclusion, why would you need to produce data? “Everybody knows” the conclusion is “correct”, right?
@ur momisugly John Whitman
Whether “attacking” is the right term or not is not something I wish to discuss further. To me, your tone was certainly not praising and appeared, again, to me, to go beyond simple constructive criticism.
Things “to me” and things “to you” may often be different – such is what makes the world go round.
You said: “I have my own priorities wrt the climate science situation, …”
and for you efforts in those endeavors, based on what I’ve seen from you, I am entirely supportive and appreciative.
Regards,
another John
Steven Mosher says:
October 22, 2013 at 8:06 am
data as used. code as run.
what the hell do these people not understand.
Agreed.
Moreover, it is not like they are trying to advance science and serve society worldwide or something.
Oh, wait, from the sciencemag.org website:
“The Science family of sites is published by the nonprofit AAAS, whose mission is to advance science and serve society worldwide.”
Never mind.
🙂
Seems to me that Willis is onto something big here, and that Science is playing with fire. Some commenters have suggested an escalation of letter writing. Might I suggest an alternative: an escalation of evaluation of studies for compliance with journal policy? This could be crowdsourced and would be a great service to science and humanity. Expand it to other journals as well so that Science doesn’t feel (unduly) picked-on. Do it well, publish the results, data and methodology, call a press conference or Stossel or 60 Minutes and open the can for all to see and smell. Would this not put serious fear into the journals’ hearts? Better they should start playing nice with Willis, but too late; there are bigger fish to fry here.
dbstealey says:October 22, 2013 at 8:36 am
What scientists mean when they write… Thanks, hilarious.
fred berple: “The missing data is the list showing what RACE data was specifically included/excluded. ”
Ok, wasn’t missing anything pre-coffee. Thing is, what you select from the dataset, why, and how you deal with outliers are not minor bits of statistical arcanum. That’s all part of methodology in statistics.
And if that is not there, and apparently it is not, then the issue isn’t that the journal Science isn’t upholding it’s data policy. It is that it is publishing papers that are *facially invalid* as they cannot be replicated even in principle. Nor can it be properly peer reviewed as a discussion of the peer reviewers cannot validate how *completely normal* considerations of statistics were dealt with.
The idea that they’re not producing they’re source dataset is simply wrong and misguided.
Willis was quite correct to write:
Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
If anything, Willis was overly polite; businesslike, and to the point. No name calling, just facts.
Science is clearly in the wrong. But Dr. McNutt’s response is to circle the wagons. She will not easily give in, because her pride and ego are now in play — and we see how that shakes out when someone lacks character. The truth takes a back seat to the line drawn by McNutt and Science [to which I subscribed for more than twenty years, as I watched it become more and more political].
You could not find another case where there is such a clear distinction between right and wrong; between honesty and propaganda. Between ethical conduct, and an unethical refusal to abide by a clear, unambiguous, written policy.
That corrupt attitude comes straight from the top. We can see it in our national affairs, where the guy at the top deliberately disregards the law. It is an indication that we are entering another Dark Age. The current cycle must go to its inevitable extreme, before we have another Enlightenment and Renaissance.
Even if Science gives in and eventually provides the requested data, and methodology, and code, getting it will be harder than pulling teeth. And it will only apply to this particular paper. Because it is clear that Science has an unwritten policy of deliberately ignoring its written policy. They do this all the time, because they are pushing an agenda, not engaging in the search for knowledge. This paper slyly purports to blame AGW for the movement of fish. That is what is important to Science, not the scientific method, and certainly not the truth.
The unethical scoundrels who have managed to get the reins of power have an agenda. They do not want the truth, they only want to advance their agenda. McNutt has demonstrated her lack of character in the past, and this is simply another example. She disregards written policy because she has ulterior motives.
McNutt has sold her soul to the devil, and there is no getting it back. She is bought and paid for. If she suddenly began to enforce the policy requiring the public archiving of all data, code and methods, Science would simply replace her with someone else who would push their agenda.
It is the system itself that is rotten. This is just another example. Honesty and ethical conduct are the first casualties in this descent into mass ignorance. The truth is now the enemy.
Hi Willis,
I’ve not glanced at the comments yet, and so this comment may repeat something said upstream. It is counterproductive to lecture the editors or whoever is in charge at Science as to the nature of science needing to be falsifiable in order to be ‘science’. You should instead address the problem simply in terms of Science having a responsibility of making data available so that other researchers can ,replicate the data analysis. This approach would appear more compatible with what I think most working ‘professional’ scientists would request.
Keep up the good work!
The tar baby that is big climate science marches on, tainting all who embrace it.
For those who think W.E.’s tone too harsh, it is wise to remember that this is not the first time something like this has happened. There is a certain level of duplicity going back for years and exposed clearly in the climategate revelations.
Imagine that you have been shortchanged at a cash register. No big deal right? You point out the error calmly and the error is rectified. Now suppose that every time you shop at a certain store the same short-change ‘mistake’ is made. At this point you watch the cashiers very carefully, and you are not at all shy about showing what you are doing and what you think of the process.
That’s what I read in WE’s tone.
Of course the real solution is to take the business to a reputable establishment.
oldspanky says:
October 22, 2013 at 6:09 am
“about what an intromittent organ he is”
Perhaps I should adopt that nickname rather than my current one.
AAAS Membership Department
PO Box 96177
WASHINGTON DC 20077-7054
USA
Dear Sirs,
Thank you for the invitation to become a member. I was considering this very seriously, if only because I was exasperated with Scientific American, to which I have subscribed since I was a boy. However, their reliance on journalists rather than scientists had become so much the norm that I finally gave in and started to look elsewhere.
Membership of your Association was quite high on my list. Most of my research showed that, via Science, I might be better informed of broad developments in science than either Scientific American or Nature. I was on the point of subscribing when I read a blog by Willis Eschenbach in Watts Up With That. Suffice it to say that I was horrified that scientific standards have so slipped that you felt able to ignore a reasoned complaint regarding data availability and the associated ability to test the hypothesis independently.
I shall not be taking up membership until such time as you can demonstrate a greater level of adherence to your written policies than you have shown in this matter. I value honesty very highly. Compromise for the sake of political expediency is a personal anathema.
Yours faithfully
ferd berple said @ur momisugly October 22, 2013 at 6:45 am
Sadly, Ferd, very little replication happens in science these days. There’s no funding for it. And even if funding were available, the journals won’t publish a paper unless it has something original to say (or at least that’s policy) and a replication is by definition not original research.
Where published research gets found out is when someone attempts to use the results as input for their own research and anomalous results become clear. It’s in the nature of the industry I’m afraid. And yes, Willis is correct to challenge Science/McNutt’s intransigence in not applying Science stated policy. Perhaps they need to change their policy to one of “We refuse to require any of our published scientists to provide any data, or methodology needed to replicate their work”.
A question, if anyone knows: Is Science equally lax in applying their written policy when it comes to studies in areas other than climate change?
I think you were very polite in all your emails. Good work Willis, I can’t imagine the time and work going into this trying to “extract water from a stone” but I and many others are grateful for your persistence in this and many other of your pursuits in the true name of science.
Kudos to you,
J.Philip
Willis, you could have dripped words of purest honey , sung to an anthem plucked from Orpheus’s lyre, with a backing chorus of sirens and seraphim, and I’d still bet the usual 5 Quatloos that you are not seeing any data or code as long as there’s an aperture in your fundament. These are
little bureaucratic leprechauns defending their pot of gold. The only thing that will bring them round is a metaphorical smack in the mouth hard enough to put them on the floor.
– – – – – – – – –
JohnWho,
Thank you for answering my question and your interaction. Appreciate the quickness too.
Yes, sometimes I am profoundly critical of Willis; constructive or not, it is the case. And sometimes I am profoundly in agreement.
In this matter of his communication to Science Magazine, I think his very important and appropriate and reasonable request appears, due to his non-professional approach, as a harangue at a professional person (Dr. Marcia McNutt). Especially so given some of the issues with his August open letter to Dr. Marcia McNutt.
Could it be taken as Willis having an ongoing pattern of initiating anti-professional behavior toward Dr. Marcia McNutt? I shrug in answer to my own question.
John
PS – JohnWho, and thanks for your kind words in your last paragraph. I should not deserve them. As you may notice my path sometimes has only a few or no intellectual companions. : )