Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
Very good post in times where: politics + (bad-mediocre)science= religion.
To misquote Churchill
“Anonymous peer review is the worst method for evaluating journal articles,
except for all the rest!”.
The article was published, Steig and/or coauthor was given room to
write a 70 paper critique with which he (i.e. Stieig) essentially hanged
himself. Basically, the system worked.
It is not uncommon for authors being criticized to have input into a review.
What should not happen is that their opinion becomes the sole determining
factor. This did not occur in the case of the O’Donnell paper.
“Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.”
I disagree with this statement. Didn’t Steve McIntyre and Ross McKitrick have a paper, rejected by Nature, only to be accepted by another journal. If the paper was published on the web no other journal would accept it.
I fully agree with the rest of Willis’s ideas. Peer review in “Climate Science” is a joke.
What I can’t understand is why Anthony Broccoli gave a paper consisting entirely of statistics to reviewer A, who describes himself as NOT being a statistician and also reviewer B who wanted who wanted to refer the paper to Steig and/or Mann. Obviously reviewer B was not worried about the conflict of views either.
Dear Willis,
I think you should distinguisch two kinds of conflict of interests; adverserial and non-adverserial.
The former is in most cases not appropriate, as you stated, but the latter is worse. I feel that the second type is is the one referred to in most COI policies (and mostly practised by the Team).
rgds
I think the post is very good but I think the issue with peer review is slightly wider than what is being discussed. If we take a paper like Steig et al, this now forms part of the peer reviewed literature. There are half a dozen authors on the paper and it was probably reviewed by perhaps 3 reviewers and the editor. So maybe 10 people looked at Steig et al in all. Reiewers are generally checking for gross errors in method or logic and simply accept a lot of stuff without checking, or only check in detail something which is their pet subject. In my experience, many reviewers and editors give a pass to the heavy weight maths stuff – it is difficult to wade through and many are not competent to check it. Reviewers B and C for O’Donnell did just that. Steig later admitted he is not a statistician.
The problem with peer review is that it is ok as a basic check on papers before appearing in a journal but it is no basis for public policy. The problem with the AGW argument and the IPCC is that “peer reviewed” paper has been used to pretend this means tested and verified science. This is exactly what happened with MBH98 and the Hockey Stick. A peer reviewed paper is no such thing, it is just a first pass. The second problem with peer review in AGW science is that pointed out by Wegman in the investigation into the Hockey Stick – the number of people publishing in the field is very tiny. For example 3 of the authors of Steig et al are in the Wegman network diagram for the “Hockey Stick” (including Mann).
To me, O’Donnell highlighted the problem with Steig et al far more effectively in the blog response at ClimateAudit than in the O’Donnell et al paper. When a perturbation is added to the stations, the Steig et al model gives absurd results in the mapping. Eg adding +0.20 trend to the peninsula gives alarming warming in West Antarctica whereas adding +0.40 trend to West Antartica gives almost no warming…in West Antarctica. If the Steig paper had been any good these tests should have already been in the paper. Did Steig et al perform basic tests like this? If they did, the paper would never have been published. If they didn’t, why did the reviewers of Steig et al not ask for them? That is the failure of peer review and why it should not be used to inform public policy. A peer reviewed paper has been used as a substitute for validated science in a subject area where relatively little is known and verification is not possible and so arm waving at inconvenient facts can get you a very long way.
I suppose we could do a radical thing and actually test Willis’ idea.
BTW, good to see you tonight Willis and thx for the mention
richard telford
The best reviewers for the ODonnell paper were probably the same reviewers who passed Steig’s effort through.
A further point to make about peer review, credible science and public policy is that one reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram.
It seems to me the “Pal Review” system in climate science has replaced “Peer Review” gradually over a period of 20 years or more. “Pal Review” seems to be sufficiently well embedded that editors can’t even recognise the significance of what they and the chosen reviewers are doing to science. Many of the current editors began publishing thier own work under the current “Pal” system and even actively encourage it. So the significance of climategate just slips right on by. They just don’t seem to have gotten the alert.
For some reason skeptics, luke-warmers and climate realists seem to struggle to understand how, in a post-climategate world, it has been possible for the O’Donnell – Steig affair to have developed. Why are we so surprised by the antics of Broccoli and Steig? The alarmists still think they hold the moral high ground. To date, with one or two notable exceptions (e.g. Judith Curry), none of the alarmist set has voiced any concern. Personally I don’t see this changing any time soon. Realistically I see the current practices dying a very slow death, if at all. We can expect more of the same, though it may be done more subtly from now on in one particular journal.
Richard Telford says:
February 17, 2011 at 4:52 pm
Steig has some other attributes you seem to have neglected, Richard. In addition to those listed above, he would also be highly motivated to see if he could keep the paper from being published. And if he could not get it thrown out, he would be highly motivated to see if he could push the results in the direction of his results. Both of these are visible in his response.
In addition, Steig himself said that he is not a statistician … and the debate is about statistics. Under what rubric are you calling him an expert on the question? My dear Richard, as the O’Donnell paper clearly showed Steig is the one who didn’t understand the procedure he was using … and you want to pick him to be the expert on that procedure? Yeah, that’s the ticket … I don’t think I’ll let you pick a doctor if I have trouble …
Are there many people out there so naive as to think that Steig is anything other than horribly conflicted? Richard, why do you think that the journals have policies against conflicts of interest? Just for fun?
It is because they recognize that scientists are humans who can be counted upon, if there is a conflict, to push for their own side. Even with the best of motives and intentions.
w.
DocMartyn says:
February 17, 2011 at 4:55 pm
So Broccoli comes out great for breaking journal policies by employing a conflicted reviewer, by siding with Steig and rejecting the Chadni patterns, and finally by throwing out Steig’s work and ” ignored his final recommendations, bringing in a forth unbiased referee”.
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
How about if he just started with an unbiased referee, and avoided all of the nonsense? Then I’d give him some stars. But as it is? I find that he is the one who acted most unethically in the whole thing.
For you to think that he acted ethically is a huge surprise to me, and is contrary to the Journal policy … but hey, that’s all OK, we’re in the bizarro world of climate science …
w.
The answer to this problem is simple enough.
Reviewers remain anonymous until the MS is accepted or rejected.
If accepted, the published version contains, at its foot: “Reviewed by Groucho, Harpo & Chico”.
If rejected, the lead author is notified in similar vein: “Your work was reviewed by Groucho, Harpo & Chico”.
In neither case is any reviewer identified with any of the three reviews – we just know each of the three is due to one of the named people.
Appending reviewers’ identities to the end of a paper would discourage powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be questioned as to their diligence in waving it by.
Telling the lead author who is responsible for reviewing their (rejected) paper would probably avoid a lot of conspiracy theories as authors speculate as to who has bounced their work. And would make editors try hard to find neutral reviewers.
Finally, I would also append to the published paper the name of the sub-editor of the journal who was responsible for editing that particular MS. Obviously the authors already know this but the community at large do not. Editors can have conflicts of interest or allegiances too.
Jens says:
February 17, 2011 at 6:08 pm
I don’t see that posting reviews on the internet increases the noise. You don’t have to read the reviews, just read the Journal article if you wish.
Wrong, wrong, wrong. We have proven quite clearly that in climate science, we cannot depend on the integrity of the Editors. We need a system where we can see what the Editor is doing, because at this point, I don’t trust them one iota.
That’s like saying that papers shouldn’t include online supplements because it makes the paper too big. If you don’t want to read the reviews online, then don’t read them. How is the noise increased?
w.
Leif Svalgaard says: February 17, 2011 at 5:01 pm
…. that often is not enough to deter the author or others to give up their silly ideas – we have seen many examples of that on this very blog.
Here is an example that the good doc S. has in mind:
http://www.vukcevic.talktalk.net/LFC2.htm
Even if it was accepted by a respectable paper it would have been long forgotten. However, thanks to WUWT it has been seen more than 2500 times (out of total of 53000 views of my graphs since Jan 2009). Among those were many from NASA, NOAA and many universities.
Peer review process is ‘Stalinist cabal’ .
All power to WUWT !
Mike says:
February 17, 2011 at 6:15 pm
Mike, although you and I don’t consider it a fail-safe process, the IPCC does. If a paper is not reviewed, it can’t come in.
And if you don’t think climate claims can hurt, people are talking about taking trillions of dollars out of taxpayer’s pockets based on those very papers. Your money. My money. Since that is the case, the current peer-review process is a joke, wildly inappropriate for the job it is currently being called on to do.
w.
Mike says:
February 17, 2011 at 6:15 pm
Oh, Mike, that’s so charming, you probably actually believe that.
Stick around climate science for a while, my friend. You will rapidly lose such childish illusions as the idea that the NAS is a neutral player in the game.
w.
(I have had to write this again because my previous try vanished when I clicked on ‘post’. Hopefully two versions won’t appear!)
The answer to this problem is simple. Reviewers remain anonymous until the paper is published or rejected, at which point their identities are made public:
If the paper is published, the identities of reviewers are appended thus: “Reviewed by Groucho, Harpo and Chico.”
If the paper is rejected, the lead author is notified of the reviewers’ identities in a similar way.
In no case is a particular reviewer identified with a particular review. As authors we just know that *one* of the three reviewers was responsible for *one* of the three reviews.
The first case, with the paper published, mitigates the problem of powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be called to account: “Why did you wave by this paper, with all its obvious flaws?” (It would be interesting to know, for the cynic, who reviewed S09).
The second case would be beneficial for authors worrying about who they think is responsible for blocking their work: chances are it isn’t their ‘arch-enemy’ and just a neutral. It also means that reviewers will feel more inclined to be fair in their reviews, and not say things in print they would not say in person. And that editors will try hard to find neutral voices.
Final point, the sub-editor should also be identified at the end of the paper once it is in print. Sub-editors are scientists too, with the same potential conflicts of interest as reviewers.
4 says:
February 17, 2011 at 6:21 pm
Apologies for the lack of clarity. The reviews would be published on the web. The cost and work in that is trivial.
w.
Excellent, Willis. I posted my view of the matter on Lucia’s site. I suggested that any potential reviewer who felt a conflict of interest existed should recuse themselves in the interests of natural justice.
My only experience with Peer Review was as Chair of a committee overseeing a civic facility set up under the aegis of the NZ Environmental Protection Act; The facility, which impacted upon the regional environment, had to be checked on a quarterly basis by a a Peer Review team of three professionally qualified and currently licenced and practicing civil engineers. The Peer Review team checked inital data, sampling methods and all calcs, then forwarded a report to the facility’s management and to the overseeing committee. All reports were signed by each reviewer and there was nothing hidden from view. Where they thought any practice or method could be improved they said so. The process had tense moments, but with everything on the table, best practice and truth won out every time.
Reynold Stone says:
February 17, 2011 at 6:43 pm
Dr. Broccoli may indeed be a good man as regards your case.
But in the Steig case, he a) appointed a reviewer with not just a conflict, but the largest conflict imaginable, in direct opposition to journal policy, and b) refused to reconsider the choice of such a conflicted reviewer despite being directly asked by the authors to reconsider his unethical actions.
Now, you say his actions were “not prudent”. I say his actions were in direct violation of both policy and common sense, and that his actions suck.
I’m sorry, Reynold, but next to that the fact that he was straightforward with you means nothing to me. He probably likes kittens too. But he used his position of power unethically.
Perhaps you could look at that, and not at the fact that he acted professionally in your instance. That’s a long way from being “not prudent”.
w.
http://www.scientificexploration.org/journal/jse_03_2_gold.pdf
Andrew Guenthner says:
February 17, 2011 at 6:58 pm
Excellent exposition. I’ve added it to the head post with the comment:
My greatest thanks,
w.
A further point to make about peer review, credible science and public policy is that a good reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram).
A further point to make about peer review, credible science and public policy is that one reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram).
Its obvious that Steig was the right choice for a reviewer. He has already worked on the problem, and he will be hard on someone trying to overturn his work. So he is ideal. The editor can certainly override what Steig said, no problem, and that is what happened in the end.
The system worked, and worked well.