Guest Post by Willis Eschenbach
From an interesting post entitled “Trust and Don’t Bother To Verify” on Judith Curry’s excellent blog , I’ve taken the following quote:
Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.
I’ve written before about the data and code archiving policies of the journal Science, and how they are not enforced for certain favored papers. In this regard, consider the case of Pinsky et al. This was a study that said that fishes were moving in the direction of the “climate velocity”. As a fisherman, I’m always interested in such studies. Their results appeared too regular to me, and I wanted to check their work. However, I found that neither their data nor their code was available. So last month, I wrote to the good folk at Science to see if they would enforce their own policies.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
Mmmm. Upon re-reading it, in retrospect I see that I was not as polite as I might have liked … but then I’ve grown bone-weary of Science not following its own data and code archiving policies for certain climate articles. In response to my email, I got … nothing. Zero. Zip. Nada word from anyone at Science.
Undaunted, I persevered. After waiting for two weeks, I wrote again, and this time I copied it around the organization:
From: Willis Eschenbach
Subject: Fwd: TO: Dr. Marcia McNutt
Date: October 1, 2013 11:24:03 PM PDT
To: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Friends:
I sent the following message two weeks ago to Dr. McNutt. However, it seems to have miscarried.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
I have not received a reply. Perhaps Dr. McNutt was not the proper person to address this to. So I am sending it to other addresses, in the hopes of getting some reply. I’m sorry to bother you, but if you could pass this to someone who could explain why you are not following your own written policies in this instance.
Many thanks,
w.
This time, I actually got a response, the very next day:
From: Andrew Sugden
Subject: Re: FW: TO: Dr. Marcia McNutt
Date: October 2, 2013 2:59:33 PM PDT
To: Willis Eschenbach
Dear Dr Eschenbach
Thank you for your message to Dr McNutt. I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.
Your sincerely
Andrew Sugden
And the following day, I replied:
From: Willis Eschenbach <willis@surfacetemps.org>
Subject: Re: TO: Dr. Marcia McNutt
Date: October 3, 2013 9:48:34 AM PDT
To: Andrew Sugden <asugden@science-int.co.uk>
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dr. Sugden, thank you most kindly for your reply. However, I fear that I’ve double-checked the paper and the SI, and there is far, far too little information, either in the paper itself or in the Supplementary Information, to allow their results to be confirmed, replicated, or falsified.
Here’s an example. It just happens to be the first area on their list, their study of the Eastern Bering Sea. The source of the data is given as being the RACE survey … but other than that we know nothing.
For example. The RACE survey covers 112 species … which of these species did they actually look at, and which ones did they leave out of their survey? Then they say they didn’t look at all tows … so which individual tows did they look at, and which did they leave out of their survey? Their only information on the subject is as follows:
While surveys were conducted in a variety of seasons (Table S1), we analyze each survey separately and use season-specific temperature data to account for these differences. We restricted our analysis to tows without gear and duration problems, to taxa that were resolved at least to genus, and to taxa that were sampled at least once per year to reduce effects from changes in taxonomic recording or resolution.
Unfortunately, that is far from enough information to be able to tell if their results are real or not.
Look, Dr. Sugden, this is not rocket science. To verify if what they have reported is a real effect, what we readers of Science need is very, very simple. It is a list in plain text that looks like this:
Year Month Day Tow# Species Catch Lat Start Long Start Lat End Long End Depth Temperature Result 1998 3 12 116 capelin 17.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Utilized1998 3 12 116 sculpin 1.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Excluded, uncertain identification…
Without that list showing exactly which data was used, and which data was excluded, and why, their results cannot be falsified … and unfalsifiable claims are not science, and not worth reporting in Science magazine
What they have done is just waved their hands and pointed at a huge pile of data, and said, We got our data from that pile … I’m sorry, but in 2013 that doesn’t cut it. To check their work, we need to know, not where they got their data, but exactly what data was used and what data was excluded. For all we know, there were transcription errors, or bugs in their computer code, or incorrectly categorized results, could be anything … but there’s no way to tell.
Nor is this an onerous requirement. The block of data representing the entire analysis would be a few megabytes. And presumably, in order to analyze the data, it’s all on the computer. So outputting a list of the data that was actually used or excluded is a few minutes work for a junior analyst.
I fear Science magazine and your Reviewers have dropped the ball on this one, Dr. Sugden. You have not done your due diligence and required the archiving of the data actually used in the study. Without that, you’re just publishing an anecdote, a charming fairy tale told by Dr. Pinsky.
It’s an interesting anecdote, to be sure … but it’s not science.
Please let me know what your magazine intends to do in this case. As it stands, you’ve published something which is totally unfalsifiable, in direct contravention of your own policies. Here are your relevant policies:
Data and materials availability
All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …
Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or electron microscopy maps for macromolecular structures, and climate data) must be deposited in an approved database, and an accession number or a specific access address must be included in the published paper. We encourage compliance with MIBBI guidelines (Minimum Information for Biological and Biomedical Investigations).
Details include but are not limited to:
- …
- Climate data. Data should be archived in the NOAA climate repository or other public databases.
- Ecological data. We recommend deposition of data in Dryad.
Clearly, the information that they provided falls woefully short of that required by your policies. No archive of their data. And pointing at a huge pile of data is not sufficient to let me “understand, assess, and extend the conclusions” as your policies require. I don’t have a clue what in the huge pile of data they used and what they excluded, so the information they gave about the location of the huge pile of data is useless.
The requirements, your own requirement, are bozo-simple, and easy to comply with. All they need to do is archive the collection of data that they actually used or rejected, and archive the computer code that they used to analyze that data.
They have done neither one …
Please let me know your plan of action on this, both for this paper and in general. As it stands, your magazine is passing off the unverifiable, unfalsifiable anecdotes recounted by Pinsky et al. as if they were real science. This is not the first time that your magazine has done that … and I don’t think that’s good for you personally as a scientist, for the reputation of Science magazine, or for science itself. People are trusting science less and less these days … and the publication of unverified anecdotes as if they were real studies is one of the reasons.
Your requirements for data and code archiving are simple and transparent. Now … you just have to enforce them.
Thanks for your assistance in all of this,
w.
Perhaps overly verbose but I wanted them to understand the issue. I waited almost two weeks, and when I’d gotten nothing, I wrote back:
From: Willis Eschenbach
Subject: Re: TO: Dr. Marcia McNutt
Date: October 14, 2013 11:00:05 AM PDT
To: Andrew Sugden
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Dr. Sugden;
As I detailed in my attached letter, neither the data nor the computer code for the Pinsky et al. study on the migration of fishes in response to climate velocity is available in a usable form.
While the data is publicly available, there is no detailed list or other means to identify the data actually used in the Pinsky study. Without that, in fact their data is not available—it is a needle in a haystack of needles. And without that, the study cannot be replicated, and thus it should not be published.
In addition, the computer code is nowhere to be found.
Both of these violate your express policies, as detailed below.
It’s been almost two weeks now since my attached letter was sent … I’m sorry to bother you again, but is there any progress in this matter? Or should I just submit this to the Journal of Irreproducible Results? Hey, just kidding … but it is very frustrating to try to see if there are flaws in published science, only to find out that Science itself is not following its own published policies.
My apologies for copying this around, but it may be that I’m not talking to the person in authority regarding this question. Do you have plans to rectify your omission in the Pinsky study, and require that they archive the actual data and code used? And if so, what are the plans?
Or are you going to do the Pontius Pilate?
In any case, any information that you have would be most welcome.
Many thanks for your assistance in this matter.
w.
PS—Please, do not tell me to contact the scientists directly. This is 2013. The exact data and code that the scientists used should be available at 2AM their time to a teenaged researcher in Ghana who doesn’t even speak the scientists’ language. That’s the reason you have a policy requiring the authors to archive or specifically identify their data, and to post their code. Pinsky et al. have done neither one.
That was sent on the 14th. Today’s the 21st. So I figured, at this point it’s been almost three weeks without an answer … might as well post up the story.
Now, would I have caught more flies with honey than with vinegar? Perhaps … perhaps not.
But the issue is not the quality or politeness of my asking for them to follow their own policies. Look, I know I can be abrasive at times, and that Dr. McNutt has no reason to like me, but that’s not the issue.
The issue is whether the journal Science follows their own policies regarding the archiving of data and code, or not. If you don’t like the way I’m asking them to do it, well, perhaps you might ask them yourself. I may be overly passionate, I might be going about it wrong, but at least I’m working in my own poor way to push both Science and science in the direction of more transparency through the archiving of data and code.
Sadly,
w.
The issue of papers being published with unsupported data is a very serious one. What if Government and other bodies take costly action on what they have read in a paper, or taxes are applied, or even, in this case, fishing for certain species banned, in order to mitigate the problem ‘exposed’ in the paper? If the paper is flawed according to the data available then the authors, together with the magazine should be prosecuted and, if found guilty, heavily fined for publishing bogus papers on the back of bogus scientific study, a study that could lead to expensive problems that do not exist in reality..
I feel the scientists and other experts who exchange their views on this wonderful site, should keep up the pressure until something is done at government and publication level to enforce the rule that authors and publications,should make raw data avilable with every paper published, a deficiency exposed so well by Willis, that could have far reaching and even dangerous implications.
I am sure that Willis’ tone in a private communication looked unprofessional, but the worst that should have happened should have been a few private words around the office about what an intromittent organ he is, followed by a properly diligent response to a substantive complaint. Willis’ is entitled to express himself as unprofessionally as he likes; the editors of Science don’t get a choice whether they do their job not. In fact they don’t get a choice whether they do their duty or not.
If all available data is given, presumably another scientist could reverse engineer the method for getting the same results, or do an independent study which generates other results. Apparently Pinsky et al. used the data selectively, and this raises the question whether their method of exclusion was actually right,as it may well have been. So if they don’t explain their method of selection, the reviewing scientist, in this case Willis, will not be in a good position to determine whether their results are correct.
“Shut Up”, Dr. McNutt explained.
My 2 cents is I would use a more polite tone. Describe *exactly* what you need and why you need it and thank them for their help in advance.
The “modern” traditions of science date back decades, and IMHO it shows. If a software engineer tried to get away with saying his code had been peer reviewed therefore it’s good, people would laugh in his face. But in science using peer review to support the claim of validity is par for the course.
Scientists work in a culture that facilitates bad results. Getting them to even be aware of that is an educational process. As Napolian said, there’s no need to invoke malice when ignorance will suffice. Modern scientists are, generally speaking, ignorant of the procedures needed to create testable, reproducible results.
It’s up to the public to teach them.
This is a funny video that can explain with some humour what Willis is asked to do with “raw” data. It’s about the invention of the train. Leonardo Da Vinci met two strange guys coming from the future. The two guys present themselves as scientists and colleagues and ask leonardo to build a train with just “raw” data… Enjoy
I am afraid this is but one example of what is happening, as The Team defend The Cause. The AR5 is proving to be a pack of scientific lies. This is being exposed on the blogsphere, but nowhere else. The Team are desperate to keep it that way, and not let the MSM and our politicians know that the warmists have been lying through their teeth for years. All we are going to get on this sort of issue, into the indefinite future, is stalling tactics.
James says:
October 22, 2013 at 4:06 am
They have made the raw data available that is all you should need.
===========
No, they have not made the data available. The missing data is the list showing what RACE data was specifically included/excluded. Without this data there is no way to replicate the results. Unless replicated, there can be no confidence the conclusions are valid.
Every published paper should come with a huge bold red stamp on the front “NOT REPLICATED”. Then, after the paper is replicated by multiple independent sources, the stamp can be replaced by a green “REPLICATED” stamp, along with the paper describing the replication.
Magazines like Science can then move into the industry of publishing replications, which in many ways tells us more about the validity of the science than does peer review.
Peer Review has been shown time and time again to be near worthless. It results in publication bias, because the reviewers are reinforcing their own beliefs, some of which are true and some of which are false.
For example, say you came up with a fantastic proof that quantum mechanics was dead wrong. You submitted the paper for publishing and it was rejected as nonsense, because it flies in the face of the current consensus in physics. So no one hears about your proof.
This consensus reinforcement holds back scientific progress, because no one knows for sure which beliefs are false. The proof that a scientific belief is false may well lie in a paper rejected by Peer Review because it was contrary to current beliefs.
Thus, we must rely on replication.
Correction:
Willis, your italics from
should be changed?
Thanks for your efforts Willis. Unfortunately the response of the journal ‘Science’ sounds like politics and some of the comments to your post sound like groupies acceptance of that type of politics. Keep trying to shine the light. Nasty things are done in the dark. The rotten buggers doing the nasty things tend to run for shadows when the light goes on. Keep after them.
The question that Science needs to honestly ask itself is this: Could we reproduce the Pinsky results based on where we pointed Willis to and nothing else? If they can’t then Pinsky is falsified OR it’s not science.
ferd berple says:
October 22, 2013 at 6:45 am
James says:
October 22, 2013 at 4:06 am
They have made the raw data available that is all you should need.
===========
No, they have not made the data available. The missing data is the list showing what RACE data was specifically included/excluded. Without this data there is no way to replicate the results. Unless replicated, there can be no confidence the conclusions are valid.
Every published paper should come with a huge bold red stamp on the front “NOT REPLICATED”. Then, after the paper is replicated by multiple independent sources, the stamp can be replaced by a green “REPLICATED” stamp, along with the paper describing the replication.
Magazines like Science can then move into the industry of publishing replications, which in many ways tells us more about the validity of the science than does peer review.
Peer Review has been shown time and time again to be near worthless. It results in publication bias, because the reviewers are reinforcing their own beliefs, some of which are true and some of which are false.
For example, say you came up with a fantastic proof that quantum mechanics was dead wrong. You submitted the paper for publishing and it was rejected as nonsense, because it flies in the face of the current consensus in physics. So no one hears about your proof.
This consensus reinforcement holds back scientific progress, because no one knows for sure which beliefs are false. The proof that a scientific belief is false may well lie in a paper rejected by Peer Review because it was contrary to current beliefs.
Thus, we must rely on replication.
Tim says:
ferd berple hit the nail on the head. What we have is a travesty representing itself as science or putting it another way: the journals are travestying science. That is very sad.
Willis,
Perhaps you should write a second data / methodology request letter to Dr. Marcia McNutt at Science Magazine, where you include a very sincere apology for any impoliteness / rudeness in your first data / methodology request letter. Ask again with a strictly professional demeanor.
John
PS – Willis, unfortunately your track record on unprofessional communications to Dr. Marcia McNutt goes back to the ill-advised wording and rather rambling context of your open letter to her on August 4, 2013 upon her assuming editor duty at Science Magazine. Note: I was always impressed with Steve McIntyre’s persistently professional and very polite communications over the many years in requesting data / methodology from the scientists whose research he was trying to audit / replicate. He is a great role model.
I’ve asked for data very, very nicely…. still didn’t get it…
(journal, authors & university, got as far as APS – ref Lewandwosky)
Got named in a paper as showing conspiracy ideation, (an example of punitive psychology) in response… by the same author – Lewandowsky.
I have nothing but contempt for the system now.
The same people who make their living playing fast and lose with the scientific method repeatedly call this forum “anti-science.” How they have learned from the politicians as the science has become politicized – accuse your opponent of exactly what you are doing.
As a graduate student in Chemistry, I had to document each reaction in a chemical synthesis that was published in ACS journals. If mistakes were made, other noted them and sent comments to the principal author. My advisor and I had to issue a correction to the paper even thought it was a simple misassignment of a peak in an analytical spectrum. But that is what was done in the 1970’s and ’80’s. When we found that researcher published irreproducible results, their reputations were tarnished and word spread quickly. not pretty, but necessary. I hope this is still the case in chemistry, but it is not in other scientific endevours.
Um… who is it who pressures the magazines to have data retention policies such as these?
Perhaps the learned societies? If so, how about a letter to them asking if this is the kind of thing that is acceptable, and would meet their guidelines?
I’m sure there is a way to embarrass Science. We just need to find it. Perhaps a tame politician…?
It is a ripoff.
“SCIENCE” is sold to subscribers who take the title at face value.
The content does not reflect the title–not even close.
Hence, those paying for it are being robbed and lied to.
“SCIENCE”–a den of thieves and liars.
(And many of you thought Willis was harsh…)
3 weeks is not enough time. I think typically it takes about 7 years to get data, and only if it is reused in another journal with stricter adherence to policies.
M. Schneider says:
October 22, 2013 at 3:58 am
It’s high time for database of “scientific” journals rated on an ABCDF scale. A link to said database should be top, front and center..
– Why aren’t we doing this?
———————
Perhaps another WUWT Reference Page is in order…
Interesting. SCIENCE’s reply:
” I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.”
seems to show that they took Willis’ query seriously.
(bold mine)
Willis’ now appears to have complied with the bolded section.
It is indeed disappointing that a reader of SCIENCE, any reader, should have to point out to them that “…data necessary to understand, assess, and extend the conclusions of the manuscript…” and “All computer codes involved in the creation or analysis of data” are not available in the manner required by the journal.
Should not that be one of the primary items “checked off” by the editorial staff?
Willis,
You are clearly a gifted talent in many areas. Getting people to do what you would like them to do does not appear to be one of those gifts. When you want people to do something like correct a deep mistake as described in your post, unless you have actual power over them, you treat them nicely…at least enough to draw them into a dialog. One of the great self-tricks that AGW prmoters develp is the ability to dismiss or ignore that which challenges their mission. You are an outsider, powerless from their perspective, and act more like a gadfly. that you are pointing out something that is correct and reasonable is completely irrelevant to these people. They have turf to protect and their faith to promote.
Losing your temper only gives them an excuse to dismiss you.
From one John to another:
John Whitman says:
October 22, 2013 at 7:16 am
Willis,
Perhaps you should write a second data / methodology request letter to Dr. Marcia McNutt at Science Magazine, where you include a very sincere apology for any impoliteness / rudeness in your first data / methodology request letter. Ask again with a strictly professional demeanor.
John
Perhaps, instead of attacking Willis, YOU could write a data/methodology request letter (written using a strictly professional demeanor) to Science Magazine (address it to Dr. Marcia McNutt if you wish) requesting the same information based on the same problem.
Then, in the interest of science (and “Science” magazine), you can report back the results.
Just a suggestion.
“…researchers have been inundated over the past few years with what feel like unreasonable and malicious demands for their raw data. They fear the hacking of their emails is the culmination of a concerted attack by data terrorists.”
https://web.archive.org/web/20091127140130/http://www.newscientist.com/article/mg20427363.200-researchers-must-stay-on-the-moral-high-ground.html
I think Willis has made a mistake here in how he has dealt with this. When asking someone for information that you know they don’t want to give, or that may paint them in a bad light, you must always, always remain the most calm and polite person in the room. Chase them up twice a day, but always do so in respectful terms. You know from the outset that the fight will be long and hard anyway, the secret is not to give them a single excuse to refuse or delay your request. Not one.
People are people and no-one ever thinks they are the one being unreasonable. Being rude, abrupt or sarcastic only reinforces their view that your request is malicious, and something they can legitimately ignore for as long as possible. Stand your ground by all means, but state the facts, and only the facts, clearly and simply no matter how many times you have to do it. Refer back to previous delays and refusals, but betray no emotion at all. Tell them that their delays appear unreasonable, or unjustifiable, but always with (if necessary) icy calm.
I have learnt this the hard way, working with clients, suppliers and industry regulators.
Good luck though Willis and I hope you get there in the end.