Guest Post by Willis Eschenbach
From an interesting post entitled “Trust and Don’t Bother To Verify” on Judith Curry’s excellent blog , I’ve taken the following quote:
Journals’ growing insistence that at least some raw data be made available seems to count for little: a recent review by Dr Ioannidis which showed that only 143 of 351 randomly selected papers published in the world’s 50 leading journals and covered by some data-sharing policy actually complied.
I’ve written before about the data and code archiving policies of the journal Science, and how they are not enforced for certain favored papers. In this regard, consider the case of Pinsky et al. This was a study that said that fishes were moving in the direction of the “climate velocity”. As a fisherman, I’m always interested in such studies. Their results appeared too regular to me, and I wanted to check their work. However, I found that neither their data nor their code was available. So last month, I wrote to the good folk at Science to see if they would enforce their own policies.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
Mmmm. Upon re-reading it, in retrospect I see that I was not as polite as I might have liked … but then I’ve grown bone-weary of Science not following its own data and code archiving policies for certain climate articles. In response to my email, I got … nothing. Zero. Zip. Nada word from anyone at Science.
Undaunted, I persevered. After waiting for two weeks, I wrote again, and this time I copied it around the organization:
From: Willis Eschenbach
Subject: Fwd: TO: Dr. Marcia McNutt
Date: October 1, 2013 11:24:03 PM PDT
To: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Friends:
I sent the following message two weeks ago to Dr. McNutt. However, it seems to have miscarried.
From: Willis Eschenbach
Subject: TO: Dr. Marcia McNutt
Date: September 14, 2013 6:30:37 AM PDT
To: Science Editorial <science_editors@aaas.org>
Dear Dr. McNutt:
I have commented publicly in the past on Science magazine not following its own data archiving policy, but only for the favored few with whom the editors agree.
This issue has come up again with the recent publication of the Pinsky et al. study on the migration of fishes in response to climate velocity. Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
Since that means that their study is not replicable or auditable, it also means their study is not science … so what is it doing in your magazine?
I assume that you will rectify this oversight as soon as possible.
Best regards,
w.
I have not received a reply. Perhaps Dr. McNutt was not the proper person to address this to. So I am sending it to other addresses, in the hopes of getting some reply. I’m sorry to bother you, but if you could pass this to someone who could explain why you are not following your own written policies in this instance.
Many thanks,
w.
This time, I actually got a response, the very next day:
From: Andrew Sugden
Subject: Re: FW: TO: Dr. Marcia McNutt
Date: October 2, 2013 2:59:33 PM PDT
To: Willis Eschenbach
Dear Dr Eschenbach
Thank you for your message to Dr McNutt. I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.
Your sincerely
Andrew Sugden
And the following day, I replied:
From: Willis Eschenbach <willis@surfacetemps.org>
Subject: Re: TO: Dr. Marcia McNutt
Date: October 3, 2013 9:48:34 AM PDT
To: Andrew Sugden <asugden@science-int.co.uk>
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dr. Sugden, thank you most kindly for your reply. However, I fear that I’ve double-checked the paper and the SI, and there is far, far too little information, either in the paper itself or in the Supplementary Information, to allow their results to be confirmed, replicated, or falsified.
Here’s an example. It just happens to be the first area on their list, their study of the Eastern Bering Sea. The source of the data is given as being the RACE survey … but other than that we know nothing.
For example. The RACE survey covers 112 species … which of these species did they actually look at, and which ones did they leave out of their survey? Then they say they didn’t look at all tows … so which individual tows did they look at, and which did they leave out of their survey? Their only information on the subject is as follows:
While surveys were conducted in a variety of seasons (Table S1), we analyze each survey separately and use season-specific temperature data to account for these differences. We restricted our analysis to tows without gear and duration problems, to taxa that were resolved at least to genus, and to taxa that were sampled at least once per year to reduce effects from changes in taxonomic recording or resolution.
Unfortunately, that is far from enough information to be able to tell if their results are real or not.
Look, Dr. Sugden, this is not rocket science. To verify if what they have reported is a real effect, what we readers of Science need is very, very simple. It is a list in plain text that looks like this:
Year Month Day Tow# Species Catch Lat Start Long Start Lat End Long End Depth Temperature Result 1998 3 12 116 capelin 17.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Utilized1998 3 12 116 sculpin 1.6 kg 56.712N 176.55E 56.914N 177.25E 72-75m 11.6-11.9°C Excluded, uncertain identification…
Without that list showing exactly which data was used, and which data was excluded, and why, their results cannot be falsified … and unfalsifiable claims are not science, and not worth reporting in Science magazine
What they have done is just waved their hands and pointed at a huge pile of data, and said, We got our data from that pile … I’m sorry, but in 2013 that doesn’t cut it. To check their work, we need to know, not where they got their data, but exactly what data was used and what data was excluded. For all we know, there were transcription errors, or bugs in their computer code, or incorrectly categorized results, could be anything … but there’s no way to tell.
Nor is this an onerous requirement. The block of data representing the entire analysis would be a few megabytes. And presumably, in order to analyze the data, it’s all on the computer. So outputting a list of the data that was actually used or excluded is a few minutes work for a junior analyst.
I fear Science magazine and your Reviewers have dropped the ball on this one, Dr. Sugden. You have not done your due diligence and required the archiving of the data actually used in the study. Without that, you’re just publishing an anecdote, a charming fairy tale told by Dr. Pinsky.
It’s an interesting anecdote, to be sure … but it’s not science.
Please let me know what your magazine intends to do in this case. As it stands, you’ve published something which is totally unfalsifiable, in direct contravention of your own policies. Here are your relevant policies:
Data and materials availability
All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …
Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, appropriate data sets (including microarray data, protein or DNA sequences, atomic coordinates or electron microscopy maps for macromolecular structures, and climate data) must be deposited in an approved database, and an accession number or a specific access address must be included in the published paper. We encourage compliance with MIBBI guidelines (Minimum Information for Biological and Biomedical Investigations).
Details include but are not limited to:
- …
- Climate data. Data should be archived in the NOAA climate repository or other public databases.
- Ecological data. We recommend deposition of data in Dryad.
Clearly, the information that they provided falls woefully short of that required by your policies. No archive of their data. And pointing at a huge pile of data is not sufficient to let me “understand, assess, and extend the conclusions” as your policies require. I don’t have a clue what in the huge pile of data they used and what they excluded, so the information they gave about the location of the huge pile of data is useless.
The requirements, your own requirement, are bozo-simple, and easy to comply with. All they need to do is archive the collection of data that they actually used or rejected, and archive the computer code that they used to analyze that data.
They have done neither one …
Please let me know your plan of action on this, both for this paper and in general. As it stands, your magazine is passing off the unverifiable, unfalsifiable anecdotes recounted by Pinsky et al. as if they were real science. This is not the first time that your magazine has done that … and I don’t think that’s good for you personally as a scientist, for the reputation of Science magazine, or for science itself. People are trusting science less and less these days … and the publication of unverified anecdotes as if they were real studies is one of the reasons.
Your requirements for data and code archiving are simple and transparent. Now … you just have to enforce them.
Thanks for your assistance in all of this,
w.
Perhaps overly verbose but I wanted them to understand the issue. I waited almost two weeks, and when I’d gotten nothing, I wrote back:
From: Willis Eschenbach
Subject: Re: TO: Dr. Marcia McNutt
Date: October 14, 2013 11:00:05 AM PDT
To: Andrew Sugden
Cc: Science Editorial <science_editors@aaas.org>, science_letters <science_letters@aaas.org>, science_bookrevs@aaas.org, Science News <science_news@aaas.org>, gchin@aaas.org, hjsmith@aaas.org
Dear Dr. Sugden;
As I detailed in my attached letter, neither the data nor the computer code for the Pinsky et al. study on the migration of fishes in response to climate velocity is available in a usable form.
While the data is publicly available, there is no detailed list or other means to identify the data actually used in the Pinsky study. Without that, in fact their data is not available—it is a needle in a haystack of needles. And without that, the study cannot be replicated, and thus it should not be published.
In addition, the computer code is nowhere to be found.
Both of these violate your express policies, as detailed below.
It’s been almost two weeks now since my attached letter was sent … I’m sorry to bother you again, but is there any progress in this matter? Or should I just submit this to the Journal of Irreproducible Results? Hey, just kidding … but it is very frustrating to try to see if there are flaws in published science, only to find out that Science itself is not following its own published policies.
My apologies for copying this around, but it may be that I’m not talking to the person in authority regarding this question. Do you have plans to rectify your omission in the Pinsky study, and require that they archive the actual data and code used? And if so, what are the plans?
Or are you going to do the Pontius Pilate?
In any case, any information that you have would be most welcome.
Many thanks for your assistance in this matter.
w.
PS—Please, do not tell me to contact the scientists directly. This is 2013. The exact data and code that the scientists used should be available at 2AM their time to a teenaged researcher in Ghana who doesn’t even speak the scientists’ language. That’s the reason you have a policy requiring the authors to archive or specifically identify their data, and to post their code. Pinsky et al. have done neither one.
That was sent on the 14th. Today’s the 21st. So I figured, at this point it’s been almost three weeks without an answer … might as well post up the story.
Now, would I have caught more flies with honey than with vinegar? Perhaps … perhaps not.
But the issue is not the quality or politeness of my asking for them to follow their own policies. Look, I know I can be abrasive at times, and that Dr. McNutt has no reason to like me, but that’s not the issue.
The issue is whether the journal Science follows their own policies regarding the archiving of data and code, or not. If you don’t like the way I’m asking them to do it, well, perhaps you might ask them yourself. I may be overly passionate, I might be going about it wrong, but at least I’m working in my own poor way to push both Science and science in the direction of more transparency through the archiving of data and code.
Sadly,
w.
They are not going to let you have their data….you only want to find something wrong with it. (Somehow that sounds slightly familiar!)
Head meet wall.
Willis, I think you did it perfectly.
As you pointed out to them, this is not the first time, so it is perfectly reasonable of you to be a little annoyed and to show that. You were, in fact, quite polite. You thanked Dr Sugden and outlined very clearly what you were about.
It would be nice to think he is trying to follow it up and wants to get back to you with a positive result, but it rather sounds like he’s been told to bin it and ignore you.
Are there any other subscribers to Science Magazine who can team up on this matter of data storage? What would it take to organize a few hundred – a few thousand would be better – to write in about it all at the same time?
I’m not a subscriber, and I’m just thinking aloud, but this is the place where most people meet, so… is it an idea worth following up? It just seems to me this sort of thing needs larger masses objecting to poor quality control going on – there and anywhere else (including MSM buildings – forget the politicians, they are immune and the news doesn’t cover it unless it’s Green. Actually marching on an editors office might get a better result. Alas, now I’m getting side-tracked. Sorry).
Good luck. I hope you get the result you and so many others want. That they dare to claim the policy and so badly ignore it, well, that’s fraudulent, isn’t it? To claim something is available when it isn’t? To put on a pretence of scrupulous attention to detail and data storage when it’s hollow?
Why should they let you see their data when all you want to do is find something wrong with it?
Well done. Whilst your use of words is “forthright” – it does not give any excuse whatsoever for the lack of meaningful response.
Science, paper without reliability. Dont read bullsh*t
I don’t think you encourage action with the tone and sarcasm in your letters. Why not write them as professionally as possible so they are taken as seriously as they should be? Stick to the facts and be clear and concise.
LOL @ jorgekafkazar
Science ? how low have they fallen … perhaps to the level near toilet paper but at least about equal in quality to Scientific American or National Enquirer ?
Willis, you just gotta get with the In-Crowd. All that’s required is to check your soul at the door.
Governments looking to cut research budgets without appearing anti-science would do well to defund researchers who do not publish data or code.
Journals prove time and again that they’re just trade magazines with undeserved reputations.
I agree with Bill Jamison. Only address the issue based on their policy. Starting comparisons to a “teenage researcher in Ghana” puts it in the personal attack mode. It just adds more extra curricular information that obscures your central point.
They gave the data source, and they gave the rules that the scientists followed. That would, at a base line be sufficient. I’d say “thank you” to them, and ask for the code next. Then if they provided that, I would grab a random sample from the data that complies with the scientists data selection rule and see what you get. If it doesn’t match, then show them your work as a counter point, and ask for the scientists data set.
Not a scientist myself, but those are roughly the steps I’d taken.
P.S. Thanks for all the great articles!
Perhaps they should change the name of the journal to Non-Science pronounced, of course, nonsense. 😉
Willis: well done. I mourn the erosion of standards –the blatant hypocrisy– of scientific journalism. Two thoughts. First, many of the papers in these journals arise from work funded by NIH or NSF or other government sources. The underlying grants were predicated on the grantees complying with a raft of terms and conditions. Private donors likewise impose such terms. Perhaps a careful sifting of these –most would be posted for public review by applicants– could point to violations of policies on data access. Even if there are no present violations, grantors and donors can be approached to ask them to show leadership (and avoid wasting their money) by refusing to fund work that isn’t transparent. Certainly, there are loud cries that studies funded by industry should disclose raw data, so why not studies funded by the public or by philanthropies?
Second, again depending on grant fine print, the failure to document one’s work so that it can be audited for financial integrity –that real work of real value was in fact performed in exchange for the money paid– might be a violation of law, even criminal law. The False Claims Act is a terrible weapon and has I think been misapplied; but what’s sauce for the goose might be sauce for the gander as well.
To JorgeF and others who object to Willis’s “tone”: The policy is there, in writing. Someone who needs the requested data in order to evaluate the results of the “study” is entitled to have it. If the article doesn’t include it, it should not have been published. Willis’s request is perfectly reasonable, regardless of language. All of the sympathy for the poor folks at Science who receive such letters is misplaced; the issue is what is owed to the reader of the article, which according to the written policy is full and usable disclosure of specific data used and methods of use. If a researcher cannot verify or falsify results based on the authors’ data as shown in the article, the result is not science.
BioBob said:
“Willis, you just gotta get with the In-Crowd. All that’s required is to check your soul at the door.”
Superb! A brilliant summation of everything that is wrong with Alarmist ‘science’. If you don’t mind Bob, I’ll be passing your comment on to my friends over the next couple of weeks.
After one failure to reply at all and one fobbing off, I think some obstreperous rudeness is entirely in order. It’s likely to get you a much wider coverage and exposure — which is exactly what this sort of hypocritical practice needs.
All part of the time-honored tactic of announcing “standards” and then ignoring them, but taking full credit for them as if they were more than wind and words.
Willis,
Glad to see that you have a Doctorate at last; well according to Andrew Sugden, anyway.
Was your thesis on the subject of Ancient and Modern Mechanical Devices on Mainland Britain?
As long as Science is making some nice profits who gives a damn about maintaining standards anyway [ sarc / ]
.
Pre-coffee, so I might be missing something. But the problem is *not* that the dataset isn’t publicly available and mentioned. But that they didn’t properly document their methodology in the paper? (Not that it’s any different for replication.)
It’s true that it would be more embarrassing for them to ignore a reasonably polite request. But one does what it can and you did plenty. Thank you.
Why should they let you see their data when all you want to do is find something wrong with it?
Data is data, if there is something wrong with it they shouldn’t have used it in the first place! One discredited survey, can only lead to subsequent ones!
As for your tone Willis, I don’t have a problem with it, these people are creating studies that are directly causing our energy bills here in the UK to go up by 10% together with the blight on our lovely countryside of windmills, that no-one wants.
I would imagine the tone of the letter I received if I refused to pay my energy bills would not be as pleasant as the tone of your e-mails to these publishers!
Impressive effort!
My experience with peer review processes is that:
1. The reviewer (who is always a busy person) is asked to do the review in very short time. It is true (I do not publish or review in “Nature”) that usually extra time is given if demanded, but the pressure is important.
2. No data or code is offered by default.
If I wanted to check the conclusions and data in a paper, I would not write to the journal. I would write to the author. Most authors are only too willing to help (and yes I know there have been exceptions) and this would have been the more practical approach.
It’s very difficult to keep a civil tongue when faced with this sort of grossly unreasonable behaviour. I think you (Willis) showed a lot more restraint than would many given your previous with these organisations.
The point is this: it’s not just you that can’t see inside the ‘fish paper’. There will be tax funded fish people who can’t tell if this paper adds anything to the condition and they will already know that this is the case. The act of publishing becomes akin to the situation in Britain for many years where MP’s submitted expenses that everyone from top to bottom knew were bogus. But that’s top to bottom on the inside – outwardly it resembled proper practice.
These journals exist simply to provide the appearance of propriety so that the university funding cycle can continue.
You can look below the surface Willis because there’s nothing there and all of the in-crowd already know that. You are just being naive to think that a darn pesky citizen scientist can join in their game – there’s nothing to join in. The paper is not intended to ever be read, merely to be published in order to justify more money.
What’s needed is something to keep these organisations honest and that something would have to be able to directly affect their sources of funding. Is it possible to name and shame their university?
What would I know? As an engineer had I not fully documented everything I would just have been fired. That’s how industry deals with competence and ego issues. It largely works.
Andrew Harding hits it right on the head. As the individual who posted this comment – “Why should they let you see their data when all you want to do is find something wrong with it?” is apparently happy to accept anything that is claimed to be correct, rather then wasting it’s time checking the results. That appeal to authority just does not work anymore and rightly so. We have already witnessed what damage that practice has generated. So it’s either put up the data and the required information or throw the lot into the garbage bin where it belongs. Finished.
Let’s face it, the editors at “Science” sold their souls for warmism long ago. Science has nothing to do with it. You are the enemy.
Two words: [snip . . site rules . . mod] magazine
It’s high time for database of “scientific” journals rated on an ABCDF scale. A link to said database should be top, front and center..
— Why aren’t we doing this?
I disagree.
They have made the raw data available that is all you should need. An important part of science is reproducibility of results. However, in order to make the reproductions useful they shouldn’t just be a rework of the original methodology. It is the result that is important. If they were to give you a step by step walkthrough. It would very likely bias the person trying to reproduce the result to apply the same methodology and that isn’t as useful as someone who thinks for themselves about how they’d analysis the data.
Also if you are just looking in detail at what someone else has done. The likely result is “I wouldn’t have done it that way I think you are wrong”. This is not useful. The most useful thing Willis could do is take the raw data do his own analysis and present the result. If it’s different then this prompts the debate about why and which way is better. If Willis does indeed have the better solution then we end up with a new piece of science that is an improvement on the previous version and this is what we want in the end.
James may disagree but he is wrong.
You have to have the plans for the machine before you can figure out if it is doing what it is supposed to do in the way that it is claimed.
A paper should be like a patent and contain all knowledge required to replicate the invention. What you have instead is the equivalent of a patent disclosure which is just a bunch of claims.
Once you have access to a patent you can build it to see if it actually works and does what is claimed for it.
Note well that in order for a patent to be granted it is not required that the invention actually work – just Google “fuel saving patent”.
I read this Economist article on the plane and guessed that it would attract blog attention – rightly so. It is good that Willis has focused the issue on a specific Science paper.
Willis
I couldn’t agree more.
Without that, you’re just publishing an anecdote, a charming fairy tale told by Dr. Pinsky.
Excellent! And you’re 100% right of course. Problem is that one may need to vet their data against the raw source.
Like it says on the logo:
SCIENCE
my
AAAS !
I think they want you to go to the same pile of data, do your own study, and write your own paper that contradicts the results of Pinsky etal.
The first thing you need to do is realize that there has been some warming in the last 40 or so years (following cooling, which followed warming, etc.). Each warming or cooling cycle would likely cause a small shift in average location of fish. So what? There is no supporting evidence for the cause of warming (or cooling), so blaming it on human activity (other than overfishing) is without basis. This is a useless paper whether it is a valid study or not.
jorgekafkazar on October 22, 2013 at 1:14 am said;
“Why should they let you see their data when all you want to do is find something wrong with it?”
I think a couple of commenters missed the humour in what jorgekafkazar has posted. It’s a parody of a comment made by that arch-warmunist Phil Jones of UEA CRU.
James says:
October 22, 2013 at 4:06 am
I disagree.
They have made the raw data available that is all you should need. An important part of science is reproducibility of results. However, in order to make the reproductions useful they shouldn’t just be a rework of the original methodology.
===
Well that is one argument but it’s not the written policy of Science, which requires code.
SOOOO right to challenge them, Willis – that article was ‘BBC Science’ (i.e. ‘Headline it on the BBC News – but don’t whatever you do, verify it’….)
You tell ’em!
“… would I have caught more flies with honey than with vinegar?”
Perhaps, but do you really want the flies?
“James says:
October 22, 2013 at 4:06 am
I disagree.
They have made the raw data available that is all you should need.”
They being who? Raw data, since when? If we “take” the UEA CRU “data”, we know the RAW data was lost in the mid 1990’s in office moves. CET data disproves AGW.
Excellent approach Willis.
If one can be said to have met the “data requirement” by certifying, “the data I used is somewhere in pile A”, then the requirement can also logically be met by certifying, “I used data from somewhere”. All this reminds me of the famously evasive testimony by one of our politicians that, “It all depends on what the definition of ‘is’ is.” In both the politician’s and the SCIENCE journal’s responses, one can only conclude that, “Someone is hiding the pea!”
Police Arrest Procedure, make an omission or an error and criminal walks free
Thread winner!
For those suggesting that Willis just needs the pile of unanalyzed raw data to perform his own study, I would guess that he wants to find out if the authors cherry-picked the data to arrive at a foregone conclusion. So he needs to know their selection process and what analytic tools they used. As Willis points out, replication is the heart of science, and falsification is its brain.
If the study were federally-funded, maybe it’s time for a FOIA demand?
/Mr Lynn
Willis, what you could do is annoy them to death. Ask AAAS again for the data. Tell them you went public with their lack of response, and provide a link to this article. And then also tell them that you will be emailing them every day until you finally provide an answer. And make sure you email every day. If, after a long period of time, they still do not respond, go public again on as many venues as you can. You need to go public because there is a chance they put your emails on auto-delete. Shame and annoy them to follow their own requirements.
Where is the AAAS office? Someone who lives nearby could volunteer to try and talk to the people who work there and ask them why they are not following their own requirements. The key is volunteering, so the person cannot be accused of being paid to do this. Just annoy them until AAAS complies.
I’m with you Willis. After years of dealing with frivolous ‘science’ claims, and adversarial attitudes when trying to get to a reasonable bottom of things, it is hard to just play nice.
Jimmi_the_dalek says:
“Most authors are only too willing to help (and yes I know there have been exceptions)…”
The exceptions are fraudulent scientists. The editors of ‘Science’ and other scientific journals are not supposed to be enablers of fraud, they are supposed to be guardians of good science. Yet they routinely publish studies linking human activity to detrimental climate change, while actively stifling legitimate scientific debate. Thank you Willis for exposing yet another poisonous practice that permeates our current science culture. Irreproducible research is bullshit! The cure is to constantly point out the bullshit and hypocrisy and rub the miscreants’ noses in it.
Great stuff Willis, keep it up mate.
As to criticism of your approach, I consider that considering the gravity of the issue and the offhand fashion with which the representatives attempted to brush you off, you were most restrained.
The issue of papers being published with unsupported data is a very serious one. What if Government and other bodies take costly action on what they have read in a paper, or taxes are applied, or even, in this case, fishing for certain species banned, in order to mitigate the problem ‘exposed’ in the paper? If the paper is flawed according to the data available then the authors, together with the magazine should be prosecuted and, if found guilty, heavily fined for publishing bogus papers on the back of bogus scientific study, a study that could lead to expensive problems that do not exist in reality..
I feel the scientists and other experts who exchange their views on this wonderful site, should keep up the pressure until something is done at government and publication level to enforce the rule that authors and publications,should make raw data avilable with every paper published, a deficiency exposed so well by Willis, that could have far reaching and even dangerous implications.
I am sure that Willis’ tone in a private communication looked unprofessional, but the worst that should have happened should have been a few private words around the office about what an intromittent organ he is, followed by a properly diligent response to a substantive complaint. Willis’ is entitled to express himself as unprofessionally as he likes; the editors of Science don’t get a choice whether they do their job not. In fact they don’t get a choice whether they do their duty or not.
If all available data is given, presumably another scientist could reverse engineer the method for getting the same results, or do an independent study which generates other results. Apparently Pinsky et al. used the data selectively, and this raises the question whether their method of exclusion was actually right,as it may well have been. So if they don’t explain their method of selection, the reviewing scientist, in this case Willis, will not be in a good position to determine whether their results are correct.
“Shut Up”, Dr. McNutt explained.
My 2 cents is I would use a more polite tone. Describe *exactly* what you need and why you need it and thank them for their help in advance.
The “modern” traditions of science date back decades, and IMHO it shows. If a software engineer tried to get away with saying his code had been peer reviewed therefore it’s good, people would laugh in his face. But in science using peer review to support the claim of validity is par for the course.
Scientists work in a culture that facilitates bad results. Getting them to even be aware of that is an educational process. As Napolian said, there’s no need to invoke malice when ignorance will suffice. Modern scientists are, generally speaking, ignorant of the procedures needed to create testable, reproducible results.
It’s up to the public to teach them.
This is a funny video that can explain with some humour what Willis is asked to do with “raw” data. It’s about the invention of the train. Leonardo Da Vinci met two strange guys coming from the future. The two guys present themselves as scientists and colleagues and ask leonardo to build a train with just “raw” data… Enjoy
I am afraid this is but one example of what is happening, as The Team defend The Cause. The AR5 is proving to be a pack of scientific lies. This is being exposed on the blogsphere, but nowhere else. The Team are desperate to keep it that way, and not let the MSM and our politicians know that the warmists have been lying through their teeth for years. All we are going to get on this sort of issue, into the indefinite future, is stalling tactics.
James says:
October 22, 2013 at 4:06 am
They have made the raw data available that is all you should need.
===========
No, they have not made the data available. The missing data is the list showing what RACE data was specifically included/excluded. Without this data there is no way to replicate the results. Unless replicated, there can be no confidence the conclusions are valid.
Every published paper should come with a huge bold red stamp on the front “NOT REPLICATED”. Then, after the paper is replicated by multiple independent sources, the stamp can be replaced by a green “REPLICATED” stamp, along with the paper describing the replication.
Magazines like Science can then move into the industry of publishing replications, which in many ways tells us more about the validity of the science than does peer review.
Peer Review has been shown time and time again to be near worthless. It results in publication bias, because the reviewers are reinforcing their own beliefs, some of which are true and some of which are false.
For example, say you came up with a fantastic proof that quantum mechanics was dead wrong. You submitted the paper for publishing and it was rejected as nonsense, because it flies in the face of the current consensus in physics. So no one hears about your proof.
This consensus reinforcement holds back scientific progress, because no one knows for sure which beliefs are false. The proof that a scientific belief is false may well lie in a paper rejected by Peer Review because it was contrary to current beliefs.
Thus, we must rely on replication.
Correction:
Willis, your italics from
should be changed?
Thanks for your efforts Willis. Unfortunately the response of the journal ‘Science’ sounds like politics and some of the comments to your post sound like groupies acceptance of that type of politics. Keep trying to shine the light. Nasty things are done in the dark. The rotten buggers doing the nasty things tend to run for shadows when the light goes on. Keep after them.
The question that Science needs to honestly ask itself is this: Could we reproduce the Pinsky results based on where we pointed Willis to and nothing else? If they can’t then Pinsky is falsified OR it’s not science.
ferd berple says:
October 22, 2013 at 6:45 am
James says:
October 22, 2013 at 4:06 am
They have made the raw data available that is all you should need.
===========
No, they have not made the data available. The missing data is the list showing what RACE data was specifically included/excluded. Without this data there is no way to replicate the results. Unless replicated, there can be no confidence the conclusions are valid.
Every published paper should come with a huge bold red stamp on the front “NOT REPLICATED”. Then, after the paper is replicated by multiple independent sources, the stamp can be replaced by a green “REPLICATED” stamp, along with the paper describing the replication.
Magazines like Science can then move into the industry of publishing replications, which in many ways tells us more about the validity of the science than does peer review.
Peer Review has been shown time and time again to be near worthless. It results in publication bias, because the reviewers are reinforcing their own beliefs, some of which are true and some of which are false.
For example, say you came up with a fantastic proof that quantum mechanics was dead wrong. You submitted the paper for publishing and it was rejected as nonsense, because it flies in the face of the current consensus in physics. So no one hears about your proof.
This consensus reinforcement holds back scientific progress, because no one knows for sure which beliefs are false. The proof that a scientific belief is false may well lie in a paper rejected by Peer Review because it was contrary to current beliefs.
Thus, we must rely on replication.
Tim says:
ferd berple hit the nail on the head. What we have is a travesty representing itself as science or putting it another way: the journals are travestying science. That is very sad.
Willis,
Perhaps you should write a second data / methodology request letter to Dr. Marcia McNutt at Science Magazine, where you include a very sincere apology for any impoliteness / rudeness in your first data / methodology request letter. Ask again with a strictly professional demeanor.
John
PS – Willis, unfortunately your track record on unprofessional communications to Dr. Marcia McNutt goes back to the ill-advised wording and rather rambling context of your open letter to her on August 4, 2013 upon her assuming editor duty at Science Magazine. Note: I was always impressed with Steve McIntyre’s persistently professional and very polite communications over the many years in requesting data / methodology from the scientists whose research he was trying to audit / replicate. He is a great role model.
I’ve asked for data very, very nicely…. still didn’t get it…
(journal, authors & university, got as far as APS – ref Lewandwosky)
Got named in a paper as showing conspiracy ideation, (an example of punitive psychology) in response… by the same author – Lewandowsky.
I have nothing but contempt for the system now.
The same people who make their living playing fast and lose with the scientific method repeatedly call this forum “anti-science.” How they have learned from the politicians as the science has become politicized – accuse your opponent of exactly what you are doing.
As a graduate student in Chemistry, I had to document each reaction in a chemical synthesis that was published in ACS journals. If mistakes were made, other noted them and sent comments to the principal author. My advisor and I had to issue a correction to the paper even thought it was a simple misassignment of a peak in an analytical spectrum. But that is what was done in the 1970’s and ’80’s. When we found that researcher published irreproducible results, their reputations were tarnished and word spread quickly. not pretty, but necessary. I hope this is still the case in chemistry, but it is not in other scientific endevours.
Um… who is it who pressures the magazines to have data retention policies such as these?
Perhaps the learned societies? If so, how about a letter to them asking if this is the kind of thing that is acceptable, and would meet their guidelines?
I’m sure there is a way to embarrass Science. We just need to find it. Perhaps a tame politician…?
It is a ripoff.
“SCIENCE” is sold to subscribers who take the title at face value.
The content does not reflect the title–not even close.
Hence, those paying for it are being robbed and lied to.
“SCIENCE”–a den of thieves and liars.
(And many of you thought Willis was harsh…)
3 weeks is not enough time. I think typically it takes about 7 years to get data, and only if it is reused in another journal with stricter adherence to policies.
M. Schneider says:
October 22, 2013 at 3:58 am
It’s high time for database of “scientific” journals rated on an ABCDF scale. A link to said database should be top, front and center..
– Why aren’t we doing this?
———————
Perhaps another WUWT Reference Page is in order…
Interesting. SCIENCE’s reply:
” I can assure you that we require all data supporting the conclusions of Science papers to be in the public domain; the location of the data is usually specified in the Acknowledgements of each paper, as it was in the case of the Pinsky paper. Please can you double-check the Supplementary Material to the Pinsky et al paper and then specify the data to which you have been unable to gain access? At that point we can ask the authors to provide further details if necessary.”
seems to show that they took Willis’ query seriously.
(bold mine)
Willis’ now appears to have complied with the bolded section.
It is indeed disappointing that a reader of SCIENCE, any reader, should have to point out to them that “…data necessary to understand, assess, and extend the conclusions of the manuscript…” and “All computer codes involved in the creation or analysis of data” are not available in the manner required by the journal.
Should not that be one of the primary items “checked off” by the editorial staff?
Willis,
You are clearly a gifted talent in many areas. Getting people to do what you would like them to do does not appear to be one of those gifts. When you want people to do something like correct a deep mistake as described in your post, unless you have actual power over them, you treat them nicely…at least enough to draw them into a dialog. One of the great self-tricks that AGW prmoters develp is the ability to dismiss or ignore that which challenges their mission. You are an outsider, powerless from their perspective, and act more like a gadfly. that you are pointing out something that is correct and reasonable is completely irrelevant to these people. They have turf to protect and their faith to promote.
Losing your temper only gives them an excuse to dismiss you.
From one John to another:
John Whitman says:
October 22, 2013 at 7:16 am
Willis,
Perhaps you should write a second data / methodology request letter to Dr. Marcia McNutt at Science Magazine, where you include a very sincere apology for any impoliteness / rudeness in your first data / methodology request letter. Ask again with a strictly professional demeanor.
John
Perhaps, instead of attacking Willis, YOU could write a data/methodology request letter (written using a strictly professional demeanor) to Science Magazine (address it to Dr. Marcia McNutt if you wish) requesting the same information based on the same problem.
Then, in the interest of science (and “Science” magazine), you can report back the results.
Just a suggestion.
“…researchers have been inundated over the past few years with what feel like unreasonable and malicious demands for their raw data. They fear the hacking of their emails is the culmination of a concerted attack by data terrorists.”
https://web.archive.org/web/20091127140130/http://www.newscientist.com/article/mg20427363.200-researchers-must-stay-on-the-moral-high-ground.html
I think Willis has made a mistake here in how he has dealt with this. When asking someone for information that you know they don’t want to give, or that may paint them in a bad light, you must always, always remain the most calm and polite person in the room. Chase them up twice a day, but always do so in respectful terms. You know from the outset that the fight will be long and hard anyway, the secret is not to give them a single excuse to refuse or delay your request. Not one.
People are people and no-one ever thinks they are the one being unreasonable. Being rude, abrupt or sarcastic only reinforces their view that your request is malicious, and something they can legitimately ignore for as long as possible. Stand your ground by all means, but state the facts, and only the facts, clearly and simply no matter how many times you have to do it. Refer back to previous delays and refusals, but betray no emotion at all. Tell them that their delays appear unreasonable, or unjustifiable, but always with (if necessary) icy calm.
I have learnt this the hard way, working with clients, suppliers and industry regulators.
Good luck though Willis and I hope you get there in the end.
data as used. code as run.
what the hell do these people not understand.
I’ve had similar problems with scaffetta and vaughan Pratt.
This is not that hard. Turn in the data as you used it and the code as you ran it.
That way I can
A. Check the work quickly
B. Build on your science if it good
C. help correct it if its not.
I would suugest that people at WUWT start filling email boxes at Science. with polite requests for the data and the code. Grass roots effort. people rely too much on guys like willis and steve Mc and me to make these requests. The more people who ask or complain in a nice way the better chance we have of changing things. We might not agree about the climate but we can agree about the importance of supplying code and data. On this there should be no sides to the debate. both sides, all sides. Show your data and your code or stop wasting our time.
Wmasaw failed to see the tongue-in-cheek nature of Andrew Harding’s comment. Some people don’t get irony, but in this case perhaps Wmaswa does not know the history behind this.
The question “Why should they let you see their data when all you want to do is find something wrong with it?” is adapted almost word for word from a response by the hockey stick team (can’t remember who exactly) to requests for data. It was the standard response before requests backed by the FOIA started to be sent and before Climategate. Looks like some people have not learned anything.
– – – – – – – – –
JohnWho,
Appreciate getting the comment from you. Thanks.
Can you please explain why you think my comment was ‘attacking’ Willis? That is a sincere question on my part; I am not trying to be rhetorical; I really would like to know what you think constitutes ‘attacking’ on my part.
As to your point suggesting that I request data/methodology from Science Magazine, no. I have my own priorities wrt the climate science situation, as I am sure you do, and Science Magazine is not one of mine at this time.
John
“”Or are you going to do the Pontius Pilate?”
As usual Willis, you’re going to have to decide whether your mission is to actually get people to do what you want (in this case provide data), or be satisfied with showing them up. I understand and share the impulse to do the latter, but it’s not the way to go if you’re interested in results..
“Things fall apart; the centre cannot hold… ”
That means different things in different contexts, but what I mean by it here is that “scientific” journals no longer are doing the job they were created to do, advancing science. Now they are a place for the “ins” to meet requirements for tenure, or research contracts, or whatever. Real science is carefully excluded, because it is usually done by someone outside the circle. And I’m not just talking about “Science” but as far as I know, it’s true of all of them.
Those doing real science need a place to publish their findings, but “science” journals are not that place. The “place” will end up being on-line, With all the problems that will have, it’s still the only way. It might be on a web-site maintained by a university, but they are going to go away too, because even more than the science journals are no longer about science, the universities are no longer about learning, and also not needed for it. Probably each scientist will have to set up his own web-site, publish his results there, and try to get the attention of others in his field, probably by sending them emails.
Anyway trying to reform Science is a waste of time, because it’s really already dead.
What scientists mean when they write… ☺
It seems to me that many posters are missing the point that Willis was making. That is the magazine (sorry “Journal”) has a stated policy …..
“All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. …” Notice “All data necessary to understand, assess and extend the conclusions……any reader of Science. All computer codes involved in the creation or analysis………..”
The Journal has not followed that policy. Whether he could have used incomplete data or a different computer code of his own is irrelevant. That would not be meeting the policy.
The magazine (for that is what it is) is LIE ing!!!!!!!!!
You could of course ask other esteemed establishments if they have a study of climate and its effects on fish. If they point you at this study, just use your ammunition to prove that is not a valid study.
If you paper arrives at the “correct” conclusion, why would you need to produce data? “Everybody knows” the conclusion is “correct”, right?
@ John Whitman
Whether “attacking” is the right term or not is not something I wish to discuss further. To me, your tone was certainly not praising and appeared, again, to me, to go beyond simple constructive criticism.
Things “to me” and things “to you” may often be different – such is what makes the world go round.
You said: “I have my own priorities wrt the climate science situation, …”
and for you efforts in those endeavors, based on what I’ve seen from you, I am entirely supportive and appreciative.
Regards,
another John
Steven Mosher says:
October 22, 2013 at 8:06 am
data as used. code as run.
what the hell do these people not understand.
Agreed.
Moreover, it is not like they are trying to advance science and serve society worldwide or something.
Oh, wait, from the sciencemag.org website:
“The Science family of sites is published by the nonprofit AAAS, whose mission is to advance science and serve society worldwide.”
Never mind.
🙂
Seems to me that Willis is onto something big here, and that Science is playing with fire. Some commenters have suggested an escalation of letter writing. Might I suggest an alternative: an escalation of evaluation of studies for compliance with journal policy? This could be crowdsourced and would be a great service to science and humanity. Expand it to other journals as well so that Science doesn’t feel (unduly) picked-on. Do it well, publish the results, data and methodology, call a press conference or Stossel or 60 Minutes and open the can for all to see and smell. Would this not put serious fear into the journals’ hearts? Better they should start playing nice with Willis, but too late; there are bigger fish to fry here.
dbstealey says:October 22, 2013 at 8:36 am
What scientists mean when they write… Thanks, hilarious.
fred berple: “The missing data is the list showing what RACE data was specifically included/excluded. ”
Ok, wasn’t missing anything pre-coffee. Thing is, what you select from the dataset, why, and how you deal with outliers are not minor bits of statistical arcanum. That’s all part of methodology in statistics.
And if that is not there, and apparently it is not, then the issue isn’t that the journal Science isn’t upholding it’s data policy. It is that it is publishing papers that are *facially invalid* as they cannot be replicated even in principle. Nor can it be properly peer reviewed as a discussion of the peer reviewers cannot validate how *completely normal* considerations of statistics were dealt with.
The idea that they’re not producing they’re source dataset is simply wrong and misguided.
Willis was quite correct to write:
Once again, it appears you have published a study without requiring archiving of the data, as is specifically required by your policies. I cannot find a public archive of their data anywhere.
If anything, Willis was overly polite; businesslike, and to the point. No name calling, just facts.
Science is clearly in the wrong. But Dr. McNutt’s response is to circle the wagons. She will not easily give in, because her pride and ego are now in play — and we see how that shakes out when someone lacks character. The truth takes a back seat to the line drawn by McNutt and Science [to which I subscribed for more than twenty years, as I watched it become more and more political].
You could not find another case where there is such a clear distinction between right and wrong; between honesty and propaganda. Between ethical conduct, and an unethical refusal to abide by a clear, unambiguous, written policy.
That corrupt attitude comes straight from the top. We can see it in our national affairs, where the guy at the top deliberately disregards the law. It is an indication that we are entering another Dark Age. The current cycle must go to its inevitable extreme, before we have another Enlightenment and Renaissance.
Even if Science gives in and eventually provides the requested data, and methodology, and code, getting it will be harder than pulling teeth. And it will only apply to this particular paper. Because it is clear that Science has an unwritten policy of deliberately ignoring its written policy. They do this all the time, because they are pushing an agenda, not engaging in the search for knowledge. This paper slyly purports to blame AGW for the movement of fish. That is what is important to Science, not the scientific method, and certainly not the truth.
The unethical scoundrels who have managed to get the reins of power have an agenda. They do not want the truth, they only want to advance their agenda. McNutt has demonstrated her lack of character in the past, and this is simply another example. She disregards written policy because she has ulterior motives.
McNutt has sold her soul to the devil, and there is no getting it back. She is bought and paid for. If she suddenly began to enforce the policy requiring the public archiving of all data, code and methods, Science would simply replace her with someone else who would push their agenda.
It is the system itself that is rotten. This is just another example. Honesty and ethical conduct are the first casualties in this descent into mass ignorance. The truth is now the enemy.
Hi Willis,
I’ve not glanced at the comments yet, and so this comment may repeat something said upstream. It is counterproductive to lecture the editors or whoever is in charge at Science as to the nature of science needing to be falsifiable in order to be ‘science’. You should instead address the problem simply in terms of Science having a responsibility of making data available so that other researchers can ,replicate the data analysis. This approach would appear more compatible with what I think most working ‘professional’ scientists would request.
Keep up the good work!
The tar baby that is big climate science marches on, tainting all who embrace it.
For those who think W.E.’s tone too harsh, it is wise to remember that this is not the first time something like this has happened. There is a certain level of duplicity going back for years and exposed clearly in the climategate revelations.
Imagine that you have been shortchanged at a cash register. No big deal right? You point out the error calmly and the error is rectified. Now suppose that every time you shop at a certain store the same short-change ‘mistake’ is made. At this point you watch the cashiers very carefully, and you are not at all shy about showing what you are doing and what you think of the process.
That’s what I read in WE’s tone.
Of course the real solution is to take the business to a reputable establishment.
oldspanky says:
October 22, 2013 at 6:09 am
“about what an intromittent organ he is”
Perhaps I should adopt that nickname rather than my current one.
AAAS Membership Department
PO Box 96177
WASHINGTON DC 20077-7054
USA
Dear Sirs,
Thank you for the invitation to become a member. I was considering this very seriously, if only because I was exasperated with Scientific American, to which I have subscribed since I was a boy. However, their reliance on journalists rather than scientists had become so much the norm that I finally gave in and started to look elsewhere.
Membership of your Association was quite high on my list. Most of my research showed that, via Science, I might be better informed of broad developments in science than either Scientific American or Nature. I was on the point of subscribing when I read a blog by Willis Eschenbach in Watts Up With That. Suffice it to say that I was horrified that scientific standards have so slipped that you felt able to ignore a reasoned complaint regarding data availability and the associated ability to test the hypothesis independently.
I shall not be taking up membership until such time as you can demonstrate a greater level of adherence to your written policies than you have shown in this matter. I value honesty very highly. Compromise for the sake of political expediency is a personal anathema.
Yours faithfully
ferd berple said @ October 22, 2013 at 6:45 am
Sadly, Ferd, very little replication happens in science these days. There’s no funding for it. And even if funding were available, the journals won’t publish a paper unless it has something original to say (or at least that’s policy) and a replication is by definition not original research.
Where published research gets found out is when someone attempts to use the results as input for their own research and anomalous results become clear. It’s in the nature of the industry I’m afraid. And yes, Willis is correct to challenge Science/McNutt’s intransigence in not applying Science stated policy. Perhaps they need to change their policy to one of “We refuse to require any of our published scientists to provide any data, or methodology needed to replicate their work”.
A question, if anyone knows: Is Science equally lax in applying their written policy when it comes to studies in areas other than climate change?
I think you were very polite in all your emails. Good work Willis, I can’t imagine the time and work going into this trying to “extract water from a stone” but I and many others are grateful for your persistence in this and many other of your pursuits in the true name of science.
Kudos to you,
J.Philip
Willis, you could have dripped words of purest honey , sung to an anthem plucked from Orpheus’s lyre, with a backing chorus of sirens and seraphim, and I’d still bet the usual 5 Quatloos that you are not seeing any data or code as long as there’s an aperture in your fundament. These are
little bureaucratic leprechauns defending their pot of gold. The only thing that will bring them round is a metaphorical smack in the mouth hard enough to put them on the floor.
– – – – – – – – –
JohnWho,
Thank you for answering my question and your interaction. Appreciate the quickness too.
Yes, sometimes I am profoundly critical of Willis; constructive or not, it is the case. And sometimes I am profoundly in agreement.
In this matter of his communication to Science Magazine, I think his very important and appropriate and reasonable request appears, due to his non-professional approach, as a harangue at a professional person (Dr. Marcia McNutt). Especially so given some of the issues with his August open letter to Dr. Marcia McNutt.
Could it be taken as Willis having an ongoing pattern of initiating anti-professional behavior toward Dr. Marcia McNutt? I shrug in answer to my own question.
John
PS – JohnWho, and thanks for your kind words in your last paragraph. I should not deserve them. As you may notice my path sometimes has only a few or no intellectual companions. : )
I have always found one tends to get better results from another if you don’t piss them off from the beginning. These people don’t owe anything to anyone without a court order at this point.
Just sayin, how would you feel if a similar communication was directed to you for some reason?
“You” being everyone who read my comment……
OssQss said @ October 22, 2013 at 12:50 pm
The Git shamefacedly admits to spending the last six years of his life as a public servant. In that role, he was frequently required to supply information on request. Such requests varied from the polite through to far, far ruder than Willis’s relatively mild tone on this occasion.
I suspect that I would have been considered ill-mannered, or insane had I refused to do what I was being paid to do. It appears that McNutt, for whatever reason, is not required to implement AAAS policy. This in itself is telling. Willis’s lack of politesse has nothing whatsoever to do with this.
Reproducible means that someone can reproduce the results of an experiment.
First to verify that the methodology is fully described and therefore CAN be reproduced.
Second to verify the ORIGINAL experimenters did not make any mistakes (intentional or accidental).
As a side effect it also means that other researchers may look more closely at how the experiment was performed and have more or less confidence in the actual result.
In the early 80’s, I begin to think that I needed to pay serious attention to Climate Change. Towards this end I look around for Journals which I thought were paying serious attention to the question. The best looked to be Science and I subscribed. At that time in Science because a majority of the articles were promoters of the CO2 theory, however many of the best were undecided or plainly skeptical. Now I would say not more than 1/3 were of the second variety. My opinion at the time was that science was still debating the issue and I would watch over the coming months and years as the issues were clarified and investigated. I read those articles religiously for almost 3 years, then all of the sudden all the skeptical or semi skeptical articles totally disappeared. I was curious so I did what research one could in those days. I found an interview with Philip Abelson who had just left the position as Editor of the Journal. He stated that one of the reasons he was ushered out was his willingness to publish skeptical articles. This interview is very hard to find and is vociferously denied by groups like Wikipedia. He also published an editorial opinion in Science which was reported in the LA times as following:
‘Philip H. Abelson, editor of Science magazine and one of the most respected men in the fields of engineering and applied sciences, wrote an editorial in the March 30, 1990, issue titled: “Uncertainties About Global Warming.”
He reported that 14 groups are now computer modeling the atmosphere and when they examine the doubling of “greenhouse gases,” they see an effect on their computer screens. However, none of these computer program models works well enough to predict anything beyond sunrise; not even one rainstorm.’
The article itself can be found on Science’s list of publications, but there is no easy way to buy it. http://www.sciencemag.org/content/247/4950/1529
Anyway that was one of my turning points from sympathetic to skeptic and I think indicative of the Journal’s stance
It’s time to start a campaign for separation of Science and The State. It will eliminate most of the rent seekers, and focus scientists on real problems. It’s funny how the need to have real results makes better scientists.
Greg: Well that is one argument but it’s not the written policy of Science, which requires code.
This is correct. The written policy of Science is unambiguous. They require that all data and code be made available. Not to be too sharp about it, but “code” includes the lines in the programs that read the raw data and select some while rejecting others; sorting and merging different files, and all of that stuff. You can’t tell whether a minor procedural difference accounts for an apparently different result unless you can reproduce the original study exactly.
As recounted by Willis, you can not tell from the Pinsky paper which subsets of the full data file they downloaded they analyzed and reported. Thus, you can not tell whether their result is a result of a selection bias, either a bias in selecting data, or a bias in selecting which results to write up for publication. Those are both well-documented and non-negligible sources of error.
– – – – – – – –
wsbriggs,
I agree.
To undertake the campaign I consider that an open discussion in the open marketplace of ideas is needed on a basic plan and strategy development. By open I mean a broad public forum based process.
I am in if the effort is transparent.
John
“Data and materials availability. All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data and materials must be fulfilled. Any restrictions on the availability of data, codes, or materials, including fees and original data obtained from other sources (Materials Transfer Agreements), must be disclosed to the editors upon submission.”
From Science web pages.
Also,
“As a condition of publication, authors must agree to make available all data necessary to understand and assess the conclusions of the manuscript to any reader of Science. Data must be included in the body of the paper or in the supplementary materials, where they can be viewed free of charge by all visitors to the site. Certain types of data must be deposited in an approved online database, including DNA and protein sequences, microarray data, crystal structures, and climate records.”
Now, that does not state that computer codes should be available as part of the article or supplementary data , unlike the raw data. I think they could argue that it would be enough for the authors to supply any codes to anyone who asked.
My thinking on all this is very simple. If your work is good and your science strong you would leap at the chance to not only give Willis what he wants but even more! It’s easy in today’s digital world with large servers and email attachments.
If I wrote a paper that I thought was robust and Willis asked to see my workings and code I would give him EVERYTHING – my workings, data, code the whole lot. Why not? Instead Science would rather have an air of doubt hanging over this paper. Imagine if a sceptic behaved in this way. I wouldn’t support such behaviour.
If you have nothing to hide you have nothing to fear. If your work is found to be bad then be thankful. I thought that was the point of science advancement!!
Just my 2 Cents.
I enjoy Willis’ posts and his vigorous pursuit of understanding and advancing science. Hurray for the citizen scientists who are absolutely needed in our society. For a good read on the contributions of a very important citizen scientist, read the book “Tuxedo Park : A Wall Street Tycoon and the Secret Palace of Science That Changed the Course of World War II.”
Jimmi_the_dalek says:
October 22, 2013 at 3:28 am
Jimmi, thanks for the comment, but I fear you’ve missed the point. if you have to ask the author for the data, it’s not science, because if you have to ask, then obviously the authors have not revealed enough to allow replication. That is the position I found myself in, unable to replicate their results.
The reason that Science and other journals have policies requiring the archiving of data and code is to allow anyone, not just friends of the author or people the author approves of but anyone at any time, to check the work. Even if you don’t speak the author’s language. Even if you’re too shy to contact them directly.
The other reason for public archiving is to maintain one clear, unaltered copy of exactly what they did. Otherwise, what if they have a computer crash and lose the data? Or they simply can’t find it five years down the line.
Journals require public archiving for a reason. We just need to hold them to it.
w.
James says:
October 22, 2013 at 4:06 am
Unless I know what data they used and what they (think they) did to it, we can’t tell if their results are correct.
What you say is an important second step, to redo the analysis using different data, different methods.
But the first step is to find out if they’ve made any errors. There are lots and lots of ways to end up with an erroneous result.
So they HAVE to give us what you call a “step by step walkthrough” in the form of the computer code that they actually used.
Only then can we see if they’ve done something wrong. Only then can we see if this is a path worth putting time and effort into, or if it’s all the result of a simple bug in the program … wouldn’t be the first time, or the thousandth time that’s happened.
Once the audit of their work is done, and we’re convinced that their data and code and results make sense, then we’re ready for your step two above—to replicate their work by doing the analysis differently, and see if it holds up.
All the best,
w.
John Whitman says:
October 22, 2013 at 7:16 am
Thanks for the ideas, John. However, let me restate what I said up above:
I write open letters to these folks in part to let them know that their actions in this regard would be watched. I don’t think they’ve quite realized that it’s 2013, and that the internet sees everything, so I wanted to warn them that the days when they could do this kind of thing in secret were over. Having told them that, they come back and do it again? Really?
You’d think that if they had half a brain, when I wrote to say hey, you’re not following your own policies, they’d get right back to me and either say “Sorry, not our job”, or “Right, that is our written policy, we’ll get on it.” Then they could write to Pinsky, get the list I requested and the code, and be done with it.
When I first wrote to them a month and a half ago, I was quite prepared for the frog to jump either way on this one. If they’d done the right thing, I was ready to write a very different post for WUWT, one saying “Hey, the folks atScience are actually doing the scientific thing!” I’m happy to report that kind of good news.
But nooo … rather than following their own policies, they’ve chosen to duck and run.
Is that being too krool to the editors of Science, saying that if they had half a brain they’d do the right thing? I claim it’s accurate. I mean, if you ran the magazine, and you got that request from me, after my writing not one but two previous open letters on the very question … would you advise “Let’s not answer him and hope he goes away”?
Or would you look at your policies, and realize that the Pinsky study is not auditable or replicable without the data and code? Since you have more than half a brain, I assume you’d take the obvious, most painless, most scientific course … and that’s rarely “run and hide” …
Like I said, whether I rub their tummies and blow in their ears isn’t the issue. It’s whether Science and the other journals follow their own policies, the policies that are at the heart of science—policies requiring their authors to transparently reveal all of the data and the code used in the study (including the rejected data as necessary).
w.
PS—And yes, you’re right, I keep trying to become more Canadian, and Steve McIntyre is my guru in that quest … obviously, it’s a work in progress …
lurker, passing through laughing says:
October 22, 2013 at 7:47 am
lurker, I fear that you mistake my intention in writing the emails.
It started because I wanted the Pinsky data. I still do, for that matter, and I don’t have it or the code.
In addition, however, I also wanted to make very public the nature of the commitment of Science editors to their stated policies.
As a result, I truly didn’t care which way they responded. I was going to report on it, good or bad. If they did the right thing, the scientific thing, I would report that … and if they didn’t I’d report it as well.
So before you say I’m no good at getting people to do what I want them to do … you should first consider that you don’t know what I wanted them to do.
Because you see, what I wanted was for them to expose their true nature. I wanted them to allow us to measure the depth of their commitment to transparent science.
And so quite the contrary to your claim, I say that I was extremely successful in getting them to do exactly what I wanted …
Losing my temper? My dear fellow, you’ll know when that happens. That wasn’t it.
As to whether I’m “powerless from their perspective”, I personally have little ability to change their practices.
However, I can speak to a big chunk of the major, minor, and peripheral players in climate science by way of WUWT … and among them are the ones with the power to effect the change.
Finally, the best disinfectant is sunshine, whether I am personally powerless or not.
w
pokerguy says:
October 22, 2013 at 8:26 am
Regarding getting people to “do what I want”, see my post to lurker above.
My “mission” in the case you quote was to make the two options perfectly clear to them. They could either provide data and code, or they could wash their hands of it. I fail to see why that clarity is a bad thing. At least that way, both of us know where we stand.
w.
I referred the other day to this article in The Scientist:
http://www.the-scientist.com//?articles.view/articleNo/37843/title/Mislabeled-Microbes-Cause-Two-Retractions/
Pamela Ronald always sets new recruits to her team replicating earlier research in her lab. When two new recruits found two errors in some work she had done in 1995, she immediately retratced the paper and told all and sundry at the conferences she attended. Faulty research is not just a problem for the researcher, but any who rely on that research.
As someone who promoted the concept of applying science to organic agriculture with some vigour to anyone who would listen back in the 80s and 90s, I am particularly pleased to note that this was in the very area that was then labelled “muck and mystery”.
What, Willis abrasive at times? Awww, he is just soft an’ cuddly like a hammerhead shark is all.
Kudos on the cuddly.
The Pompous Git: “Sadly, Ferd, very little replication happens in science these days. There’s no funding for it.”
Recently, and perhaps here or at Curry’s, on the point of replication was an argument that is loosely stated: “If the experiment is replicated, and comes to the same result, then what was the point of replication?” There seems to be an unwavering Faith that there are never any differences in what a man says he did, what he thinks he did, and what he actually did. Which rather reminds of a quote from Charles Babbage:
“On two occasions I have been asked, ‘Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.”
– – – – – – – –
Willis,
I was gently touched by your reply. That said, we are not in Gilead here on this thread and I have no balm to sooth your self-inflicted communication contusions caused by your lack of discipline when addressing Dr. Marcia McNutt.
Please Willis, a respectfully considerate kind of circumspection is reasonable and warranted in formal communications to reasonably well-established scientific professionals like Dr. Marcia McNutt.
John
@ Jquip
One of the major problems of replication is that of cost. Who can afford to put another satellite into space, assemble a team of 30 technical experts, purchase a new supercomputer, or build a Large Hadron Collider to perform a mere replication? Science is at a crossroads. Interesting times…
For those who do not know the history –
“On 18th February 2005 Professor P. D. Jones of CRU replied to my emailed requests for his land station data by including, “Why should I make the data available to you, when your aim is to try and find something wrong with it.” Author, Warwick Hughes, Australian geologist. This extract from
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc3502.htm
Warwick owns a blog named “Errors in IPCC climate science” (and others).
Warwick brought information on the work of Phil Jones, relating to estimation of UHI and adjustments to temperature data, to a think tank named Tasman Institute around 1992. We who were advisors to recommend funding for projects brought to Tasman failed to see the future implications of his visionary work. Despite our failure, it was Warwick’s work that catalysed me to become involved in the matter, starting with emails to Phil Jones about year 2005.
Some of the issues current at the time were difficult, until it was realised that a way to fix inconvenient data was to change it and call it an ‘adjustment’.
Git, I don’t see it that way after following science over the decades.
You say: “One of the major problems of replication is that of cost. Who can afford to put another satellite into space, assemble a team of 30 technical experts, purchase a new supercomputer, or build a Large Hadron Collider to perform a mere replication? Science is at a crossroads. Interesting times…”
Doesn’t really work that way as I have seen it. New machines and satellites are usually justified to answer new questions but the fact that they then have available platforms to also perform replication of older yet-to-be-verified science this is piggy-backed onto the new instruments payload and costs. And that includes CERN and the LHC. Replication is a large part of particle physics and since each machine extends the capabilities of the former generations with more precision the cost is quite relatively low per verification or refutation. You do want these discoveries and concepts to be verified by replication don’t you? I do, but this cycle of advancement/verification has been going on for decades, that is how it is done. (and please exclude climate science from the context used, that is the problem with that branch)
Willis, you wrote at 2:47 PM:
Maybe the authors performed their data selection from the raw data in a spreadsheet, by manually deleting records, so without computer code. Still, they would be obliged to show exactly what records were in their selection, as the policy says:
To repeat a posting of mine half a year ago on ClimateAudit:
Latimer Alder then responded the following:
wayne: “You do want these discoveries and concepts to be verified by replication don’t you? I do,”
I’m pretty ambivalent. If I can pick it up at Walmart, I could hardly care less whether it was based on theory A, which posits a multiverse of denied realities, or theory B, that describes quantum tunneling from a conceptual framework of 15th century goat herding practices. No matter the case, someone engineered something on some basis.
And if it becomes a policy debate question of the Capitol One sort (“What’s in your wallet?”) then I remain completely ambivalent until people can show results. If they can and have, then it’s interesting for discussion. If they have not, then they can go pound sand about their pet hysterics until they can come back to the table with at least what’s required to manufacture shower curtains for Bed, Bath, and Beyond.
Because until there’s an ability to engineer with it, nothing has been ‘discovered.’ Only hyperventilated.
It seems to me that they have an explicit policy that is intended to suggest we should trust the articles because the data and code are available.
They charge money for their publication.
The data and code are not available.
This seems like a simple case of fraud.
Maybe someone should point this out to Science, and if they can’t rectify the situation, forward this to the appropriate authorities.
Willis, you were way too aggressive and frustrated while writing your e-mails. I can understand it well but it’s way more productive to write in a polite and calm way. Sure, Science doesn’t follow its archiving policy and you probably knew that they wouldn’t while writing, but don’t let that affect the way you communicate. It gives your opponents an easy way of just pointing to a few sentences and accusing you of being aggressive and unreasonable. And people will buy it as they are too lazy to read through the whole matter.
When you write, do it always like Stephen McIntyre does it.
@ Wayne
It’s important that replications be performed with identical equipment and by a different person/team. Satellites used to estimate cloud cover for example have consistently shown more clouds than their predecessors. The importance of independence is illustrated by this email from my now deceased friend, Bo Leuf:
LOL!
I respectfully disagree with those who criticize Willis for being blunt. Scientists, science writers, and science journal editors are essentially the same as the vast majority of people who live in the real world.
In commercial settings, well outside of the scientific arena, I have gotten good results, beginning my letters of complaint with:
Can’t you idiots do anything right?
Nevertheless results are sometimes overrated. It”s the process that’s most important. Like virtue, sarcasm can be its own reward. 🙂
not replicable or auditable is a basic fail . as any undergraduate taking a science course should be told .
So why is it that the ‘professional ‘ cannot meet the standards they would demand of their own students ?
Larry Fields says: October 22, 2013 at 11:46 pm re Salutations
Did you ever have the pleasure of reading the Henry Root letters?
Typically, he (an Englishman) would start with “Here’s a quid” and enclose the money.
It was said that many people felt obligated to reply, if only to return the money with an explanatory letter about receipt of funds policy.
Indeed, pro forma approaches to publishers as mapped by Henry Root and other useful letter styles to address a number of situations described on WUWT, can be found at –
http://www.thehenryroot.com/
However, I recommend you read the books. They are hilarious.
One would have thought that one of the benefits of having a group of skeptics against you is that you are able to consider what they say and do something about it: dot the “i’s” and cross the “t’s” until you have narrowed the base on which the skeptics stand to something small and insignificant.
It seems complaints levelled at the CAGW scientists go completely unheeded and they allow us to attack them across a broad front. Perhaps they prefer us to attack such matters as archiving of data rather than wehn we focus on central issues. Shame for them that we do both.
Spot on. There is an explosive article in the Economist this week about the fact that many papers claiming breakthroughs etc are never replicated by anyone as replication is not a career enhancing occupation. (It is the week’s Economist Briefing, entitled “Unreliable Research” dated 19 October 2013). Also, where replication is undertaken it often fails to replicate in a significant number of instances. This article is a must as much of what it covers sums up the climate science charade very nicely, although nowhere does the author mention it. S/he is anonymous as all Economist writers are.
Maybe more to the point is that if Science depended on publishing sound work fully documented they wouldn’t be able to fill their pages.
M. Schneider says:
October 22, 2013 at 3:58 am
It’s high time for database of “scientific” journals rated on an ABCDF scale. A link to said database should be top, front and center..
– Why aren’t we doing this?
==============================
A very good idea.
Something a group of people with access to a number of journals could do.
Review articles, checking availability of data and producing a league table of journals, each month, here on WUWT!
@Willis:
These days we are used to seeing ‘RoHS Compliant’ tags on things. We are used to seeing ‘CSA Approved’ and ISO 9000. All these relate to the quality of the product and are obtained by following and implementing the protocols needed to properly use the ‘qualification’.
Why should there not be a similar compliance tag for scientific papers that have (already) met the required needs of making the materials and methods available? I don’t have a snappy name for it, but Sources, Methods and Code Provided (SMCP) would be a good start.
If the sources, methods and code used to create the paper are not already available, the links provided to the paper would not be allowed to place the tag on the name. Clicking on the SNCP tag would take one not to the paper, but to the sources. If the sources do not exist, no need for the tag.
Journals could keep publishing the papers but with that tag missing, it should not be considered ‘real science’. The trick used by the Team of pointing to a heap of crap data and saying ‘it is somewhere in there’ and hiding the code would not necessarily end, but their use of the tag would be disallowed. Publicity (think: name and shame campaign) would follow any document that proclaimed something unproven or unprovable.
It is time to put an end to this quasi-scientific shite.
It is possible to have an independent body issue a certificate in the way computer communications are arranged. If a paper claims to SMCP status, the certificate is created and it is attached to the paper’s link. If there is objection that the provision is not met, the certificate is cancelled breaking the link to the data until requirements are met. A publisher (journal or not) would be responsible for monitoring compliance in the same way Science is now plus one over-seeing group. If they do not police their own publication, their right to issue certificates could be suspended (if giving out false certificates).
When you click on the link, it would first check that there is a valid certificate standing, then go to the sources, rather like TinyURL. A groundswell of support by real scientists doing real science should sweep away the junk science that infests the pages of journals. We might be surprised by what drops away, and in which fields.
“Journals could keep publishing the papers but with that tag missing, it should not be considered ‘real science’. ” – indeed. Such papers should not count for the h-index; there should be an additional j-index (j=junk) for such papers; a combined hj-index would indicate the quantity of references between real science papers and junk papers.
@André
Heh heh! Very good. I don’t think that would fly in the hallowed halls of academia, but a tag showing ‘this is real science’ would. The whole point is to get those who live in the Kingdom of Names to aspire to an additional Name.
CV’s are digital these days with live links (i.e. a PDF version). Imagine a CV with a long list of papers none of which have a compliance tag. Compare that with one that has 100% compliance.
Crispin, that seems an excellent idea. Maybe the professional societies could perform this service. It might be interesting to have a member propose this and then see if it goes anywhere.
Do we really need an “SMCP” compliancy statement? If the journal would enforce it’s own rules, published articles would mean sources, methods and code are provided.
A journal can not really be a science journal otherwise, can it?
Want to get the attention of journals like Science Magazine? I suggest all one would need to do is start a serious grass roots effort in the part of the public who are highly focused on publically funded science. The grass root movement suggested is ‘a more verifiable science process in the electronic era without the generically unaccountable journals’. A central theme of the movement could be unrestricted peer review in the social media. Bang, no more journals. Bang, open access of non-anonymous peers to review a paper would indeed raise the demand for data and method and code availability prior to acceptance. It would be open and transparent, no secrecy.
The journals, by the grassroots effort, would be shown to be an archaic arbitrary convention blocking rigorous verifiability.
John
John Whitman says:
October 23, 2013 at 7:07 am
___________________
Good points, John. A massive political roadblock stands in your way. A website is needed…
The “policies” of journals like Science appear to be more for show than for reality, similar to the “policies” that I worked under for a large multinational company. “We follow all laws. We are ethical. We don’t lie.” The policies were written by lawyers, for lawyers and regulators, and they made stockholders and investors feel safe. We even had a “Compliance Department” that you could anonymously report policy infractions to. The only thing was that the “Compliance Department” was more like the data police, destroying any and all evidence of wrong doing, and not really stopping the wrong doing. When confronted with accusations that the company was involved in wrong doing, the official response was, “No, that is not accurate. We have a policy against that.” Having a policy and actually following it are two very separate issues. My observation is that the purpose of having the policy is to convince everyone on the outside that everything is being done by the rules, but actually following the rules on the inside of the organization is a far different matter. THAT is an inconvenient truth.
It seems that scientists are worried about the quality of science and some are doing something about it:
Source: http://www.the-scientist.com//?articles.view/articleNo/37969/title/Post-Publication-Peer-Review-Mainstreamed/
– – – – – – – – –
Alan Robertson,
Appreciate your comment.
It would.
I think the interest in the effort is broad enough. A spark is needed . . . .
John
: James says:
October 22, 2013 at 4:06 am
I disagree.
They have made the raw data available that is all you should need. An important part of science is reproducibility of results. However, in order to make the reproductions useful they shouldn’t just be a rework of the original methodology. It is the result that is important. If they were to give you a step by step walkthrough. It would very likely bias the person trying to reproduce the result to apply the same methodology and that isn’t as useful as someone who thinks for themselves about how they’d analysis the data.
Methodology has to be reproducible as well. One of the problems with many studies is that Type 1 errors can occur regardless of the statistical likelihood of the event. So, repeated analyses using the identical methodology and new data sets are the most desirable approach to reproducibility. Simply analyzing the same data with the same methods should reveal potential errors in procedures or calculation – for instance, I’ve found errors the way two big-name stat packages calculated Fisher’s Exact Test (since fixed).
Also if you are just looking in detail at what someone else has done. The likely result is “I wouldn’t have done it that way I think you are wrong”. This is not useful. The most useful thing Willis could do is take the raw data do his own analysis and present the result. If it’s different then this prompts the debate about why and which way is better. If Willis does indeed have the better solution then we end up with a new piece of science that is an improvement on the previous version and this is what we want in the end.
Much of what you suggest is a good idea here. What you need to consider to broaden its validity is that both “audits” – reworking the the same analysis using the same data and the same methods (possibly through a different software system for instance) helps to screen the validity of the original analysis. That is, no apparent errors in how the original data was processed or in how the resultant numbers were used, should free up the decks for a discussion of science rather than methodology. Once the debate turns to science, that is regular patterned or “lawful” behavior in nature, then collecting new data from the same source areas becomes important. This is actual reproduced research (not merely an analysis). Ideally new tree rings from Yamal should be collected to compare with the older sets for example.
Gandrud gives a fairly detailed argument and explanation of the importance of what Willis asks for:
Gandrud, Christopher (2013) Reproducible Research with R and RStudio. (Chapman & Hall/CRC The R Series)
Git;
Who would “qualify” the commenting scientists with access to the site?
Absolutely no idea. Perhaps you could ask Prof Tibshirani. I’m just reporting what I read in The Scientist.
As Dale Carnegie said when talking about how to get someone to do what you want: “Arouse an eager want”. Get the person to want to help you. Willis’ letter doesn’t do that IMO. Certainly the journal should enforce their requirements and Willis shouldn’t have to ask for the data and code. But if they haven’t then the question is “How do you get the person you are communicating with to want to help you get the data and code archived?”
The two recommendations I have are be polite and be concise. Using language such as “Once again you have…” immediately puts the person on the defensive and that is unlikely to make them want to be helpful. I doubt they will want to read several paragraphs when the entire request could easily fit in one or two paragraphs at most. Simple, concise, and polite.
ferd berple says:
October 22, 2013 at 6:45 am
Fred Berple is spot on…… If the results have not been replicated…… Then the paper and the Journal that prints it. Is worthless to the advancement of knowledge.
Bill Jamison says:
October 24, 2013 at 1:51 am
Bill, thanks for your comment. Like a number of the others you misapprehend what is going on.
I am a customer of Science magazine. I have been promised a level of quality. I’m not getting it. So I am an irate customer. However, I started out a relatively chill irate customer.
So in my first email, I did like you said, polite, concerned, all that. What did I get?
Blown off. Not even the courtesy of an answer. So, I wrapped up my initial letter in bows and ribbons and circulated it. This time I got an answer in one day.
Great, thinks I, someone is going to do something … but then I find it is just handwaving. The next day, I write back to explain why what they’ve written is handwaving. What do I get for a response?
Nothing. So I write again, and by this time I’m becoming a very irate customer. My money is paying these jerks’ salaries and not only can I not get a straight answer out of them … I can’t get any answer at all.
Now, I suppose Dale Carnegie would say something very profound at that point … but I’m like most customers. I expect the service I’ve been promised, I’m pissed off when I don’t get it, and I’m unwilling to pretend not to be upset.
Finally, I don’t have much hope that my own umbrage will have much effect. It certainly hasn’t moved them. That’s why I wrote this up, so other people will not be fooled by their lies and evasions.
Now, if you think all of that could have been avoided if I’d just used the right tone of voice or been nice to them or been “simple, concise, and polite”, you’re not following the bouncing ball. These are people willing to break rules, flout scientific norms, disobey their own stated policies, and throw unwelcome emails into the trash.
As a result, your claim that they would have done it differently if I’d only gone all Dale Carnegie on them seems terribly naive to me … but perhaps you are right. If you think so, then as in the head post, I invite you to dress up in your best Cargegie outfit and write to them.
Let us know how that comes out, Bill. Because although your wondrous Dale Carnegie model says that would work perfectly, you haven’t tried your model out on the Science editors, and many a lovely theory has foundered on the reefs of experiment … try the experiment and let us know the results.
I find that there are lots and lots of folks out there like you, Bill. You’re willing to tell me how to do it right using your miracle method… but you’re not willing to show me how well your claimed method works by actually testing it out in the real world. Talk is cheap, my friend … come back when you have exchanged some emails with the Science editors, and have some actual experience to back up your fantasies of how to do it …
w.
Willis, you could have left off “bozo” and so forth to the guy who actually took the time to write back to you.
I get your point. We all do. But why start off being a jackass about it? Why not write them, initially, in a calm and respectful manner and then, if that doesn’t work, escalate?
You could have made the same points without that. Then you could have included the biting language in your post after demonstrating they didn’t reply to your respectfully- and professionally-put letters. You’d look far stronger here.
Those.
Anthony, you really find this sort of thing acceptable or advisable from the probably the most prolific contributor you have on your website?
WUWT has done the world an enormous service in being a major player in getting the public behind re-analysing the science and also both in presenting a forum for scientific ideas to be presented and discussed, and for engaging consensus climate scientists in productive debate or exposing their foibles.
How does this sort of ill-advised writing help in these efforts? Can you not, at a minimum, tell Willis to tone that back? He isn’t some commenter who gets hot under the collar (been there). He’s a headlining writer! He forms a major part of your brand.
[Reply: You advise censorship? ~ mod.]
Larry Fields says:
I respectfully disagree with those who criticize Willis for being blunt.
I also disagree with the kissy-face approach. These are not business people, who must worry about the competition. They are pushing an agenda, not pursuing science.
As michaelozanne says:
These are little bureaucratic leprechauns defending their pot of gold. The only thing that will bring them round is a metaphorical smack in the mouth hard enough to put them on the floor.
Exactly. They have made it very clear that they are not interested in complying with their own Policy. Whether Willis is hat-in-hand groveling, or in-their-face demanding, it makes no difference at all. McNutt has an agenda, which would be undermined by Science providing what is being requested. Science is pushing the catastrophic global warming narrative. Honest science and the Scientific Method conflicts with that, so Willis will be blown off again and again. Anything he might eventually get will certainly be inadequate to falsify the paper.
For example, Michael Mann has never provided the full data, methodology, metadata and code for his hockey stick chart, despite fifteen years of requests by Mc&Mc. Welcome to the new Dark Age, where truth and transparency are the enemies of the establishment.
Christoph Dollis says:
October 25, 2013 at 10:52 am
For those who, like me, were in mystery about what you were referring to, I find this. Next time, quote it yourself. I’d said:
And you find this worthy of attack?
Christoph, I fear you misunderstand what’s going on here.
I think we both agree that Science not enforcing their own policies is a bad thing.
Let me repeat my invitation from the head post.
Rather than criticizing my efforts, how about lets see you do it? I’m tired of people whining about and criticizing my efforts to promote scientific transparency.
If you’re so damn smart, if you’re the professional in this arena, then where’s your email to set this straight? Instead of coming up with endless trivial nit-picking objections to my words, how about you get up off your dead asterisks and write a letter to them yourself, and post it here?
Then we can see how the professionals do it … and we can see what kind of response the professionals get.
My point remains. Whether I am a saint or a sinner, Science should follow its own simple tenets and policies for scientific transparency that it said it would follow.
So let’s assume you’ve proven beyond a shadow of a doubt that I’m a sinner, that I should have played nicey-nicey, that I shouldn’t have called the scientific requirement for transparency “bozo-simple” (despite the fact that it is, I mean, “show your work” is not rocket science). You’ve asserted endlessly that I did it wrong … so what?
Your turn, Cristoph. Assuming that you’re not all talk and no walk, how about we see how well you do it, and whether you succeed, before you start lecturing us all.
w.
Christoph Dollis says:
October 25, 2013 at 11:07 am
Mmm … there was a bit more to the paragraph, Christoph. You’ve left off the first part, which contains the reason I wrote the sentence you quote:
I truly don’t understand your objection to this paragraph, Christoph. The data and code should be available to anyone anywhere at any time.
In particular, as I was trying to say, it needs to be available to people in developing countries who often are greatly handicapped in the scientific arena. First, they may not be able to express themselves in the language of the authors. Next, they may be good scientifically, but they may also be young or inexperienced, and hesitant to write an email to a published author. Finally, their time on the internet may be limited. Rather than explain all that, I tried (unsuccessfully in your case) to describe this person with these barriers to writing to the authors as a “teenaged researcher in Ghana”. Mea culpa. I still don’t understand your objection, though … surely there are scientifically minded teenagers in Ghana striving to be a part of modern research, so what is your problem with what I said?
My point remains. The reason for archiving the data and code is so that anyone at any time, including people in developing countries like perhaps some teenaged researcher in Ghana, can verify or falsify or even expand upon the author’s work without having to write someone for permission.
w.
Phase 2 completed.
Willis’ POV is clearly what both the journal’s policy and scientific honesty and standards require. It’s not even inconvenient, assuming the article was properly researched and put together. Jones would have no traction with his ‘messy office’ excuses.