Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
“Jit says:
February 18, 2011 at 1:27 am
(I have had to write this again because my previous try vanished when I clicked on ‘post’. Hopefully two versions won’t appear!)
The answer to this problem is simple. Reviewers remain anonymous until the paper is published or rejected, at which point their identities are made public:
If the paper is published, the identities of reviewers are appended thus: “Reviewed by Groucho, Harpo and Chico.”
The first case, with the paper published, mitigates the problem of powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be called to account: “Why did you wave by this paper, with all its obvious flaws?” (It would be interesting to know, for the cynic, who reviewed S09).”
Again, I refereed a paper, and found only minor problems with it. Made a few suggestions and ticked the accept with minor revision box.
Two months later I get a new version of the paper, two completely new figures, two figures missing, and different methods section. I get the comments of another referee who spotted something I missed, this reviewer demanded that two datasets be repeated, using different methodology.
This second reviewer was right.
Me, I missed it completely. Publish my name and my initial review and I look a right pillock.
Want me to keep doing 8 hours unpaid work, whilst leaving myself open to looking like a complete prat?
Suggestion:
The distinction between “informative” review and “gatekeeping” (in the good sense) review should be made explicit for all science journals.
Steig should be a reviewer in this case, but explicitly as an information source without controlling authority. He has the the strongest motivation to find fault with the critical paper—-it is okay to put his motivation to work. But his clear conflict of interest should keep him out of the gatekeeping function, and this should be explicit. Non-conflicted referees can then benefit from the Socratic back and forth between Steig and the paper critical of Steig’s work and perform their essential filtering or gatekeeping role with improved information.
Matt: But in DC they still re-elect the politician! People don’t like to question their beliefs. It’s too scary/painful. They’d rather live with cognitive dissonance.
At 8:330 PM on 17 February, in response to the proposal that “Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them,” Clay Ross observes that:
.
To this Mr. Ross adds the entirely un-facetious thought that:
.
This addresses the point that a manuscript not accepted for publication – even after the authors had gone through the laborious process of responding to editors’ and reviewers’ comments and amending their manuscript – becomes without further condition the “property” of the journal to which it had been submitted with the intention of getting it published therein.
The intention had not been for the work to be publicly rejected and thereby denied all further value.
A paper submitted to The New England Journal of Medicine and declined by the officers of that periodical might be perfectly suitable for publication in Gastroenterology, and would be welcomed by the editors and reviewers of that journal – but not if that article goes onto the NEJM Web site on its publicly accessible “reject” pile before the editors of Gastroenterology can get a look at it.
If a condition is sought in which authors accept the punishing liability that their work – both in research and in the preparation of manuscripts reporting that research – can be summarily denied even the chance of publication in a peer-reviewed scientific periodical which can be legitimately cited because that work had been thrown onto an Internet “reject” pile by editorial fiat upon its first submission, then what the hell inducement is there for any researcher to submit anything to any journal in the first place?
Might as well just post it online and leave it to the search engines to put it in front of readers’ eyes.
Re: “Note that both of these papers were in Physical Review journals and had nothing whatsoever to do with climate science. So, the claim that people here seem to be making about climate science having unique issues such as the use of reviewers who might have a conflict of interest is utterly without foundation.”
I would argue that the problems which pervade our scientific institutions are indeed foundational problems. And they can be analyzed either within a philosophical manner through a discussion of the weaknesses of the Socratic-dialectic method which has been deployed to select the dominant framework; or, they can be discussed within the context of the role of the “professional” which PhD’s are expected to abide by, as Jeff Schmidt has superbly accomplished in “Disciplined Minds”.
A professional is somebody who is expected to adopt the framework, and problem-solve entirely within this unquestioned framework. A professional NEVER questions the framework itself, because that is outside of the scope of his professional duties.
This code of conduct is instituted in our physics graduate programs: Students who do not adopt the framework are literally dropped out. But, most physics grad students just plain forget all of those curious ideas they had in their undergrad and high school years, because they are forced to memorize stacks of problem sets in preparation for their PhD qualifying exams. These problem sets include fun little mathematical tricks which must also be memorized in order to complete the exams on time (and to be clear, these mathematical tricks are never used again, after the exam). This process of memorization tends to generate ideologically docile students.
Then, making things worse, PhD physics students are not even guaranteed a PhD if they pass the exam. At that point, the professors come together and use the test score as just a means of evaluating whether or not they view this person as a professional representative of the views of the institution of physics. Can they be trusted to espouse conventional wisdom?
Any discussion of the problems of science which glosses over these insane problems will lead to superficial fixes. Jeff Schmidt:
“For understanding the professional, the concept of “ideology” will emerge as much more useful than that of “skill.” But what is ideology, exactly? Ideology is thought that justifies action, including routine day-to-day activity … your ideology justifies your own actions to yourself. Economics may bring you back to your employer day after day, but it is ideology that makes that activity feel like a reasonable or unreasonable way to spend your life … Work in general is becoming more and more ideological, and so is the workforce that does it.” (Jeff Schmidt, Disciplined Minds, p15)
“Employers have always scrutinized the attitudes and values of the people they hire, to protect themselves from unionists, radicals and others whose “bad attitude” would undermine workplace discipline …
My thesis is that the criteria by which individuals are deemed qualified or unqualified to become professionals involve not just technical knowledge as is generally assumed, but also attitude – in particular, attitude toward working within an assigned political and ideological framework …
The qualifying attitude, I find, is an uncritical subordinate one, which allows professionals to take their ideological lead from their employers, and appropriately fine-tune the outlook that they bring to their work. The resulting professional is an obedient thinker, an intellectual property whom employers can trust to experiment, theorize, innovate and create safely within the confines of an assigned ideology. The political and intellectual timidity of today’s most highly educated employees is no accident …
Furthermore, professionals are the role models of the society toward which we are heading, a society in which ideology trumps gender, race and class origin as the biggest factor underlying the individual’s success or failure.” (Disciplined Minds, Jeff Schmidt, p15 … 19)
“This book’s analysis finds the supposed political neutrality of the process of professional qualification a myth: Neither weeding out nor adjustment to the training institution’s values are politically neutral processes. Even the qualifying examination — its cold, tough, technical questions supposedly testimony to the objectivity and integrity of the system of professional qualification and to the purity of the moment of personal triumph in every professional’s training — does not act neutrally. The ideological obedience that the qualification system required for success turns out to be identical to the ideological obedience that characterizes the work of the salaried professional.” (Jeff Schmidt, Disciplined Minds, p26)
“No matter what the product is, employers divide the work into many parts and assign each employee to one type of activity. Narrowly focused individuals can work in a more machine-like way and get more work done per hour. Moreover, people who exercise fewer skills or simpler skills can be paid less. Hence, employers label the division of labor “efficient.” But it is efficient only if one ignores the social cost of organizing production in a way in which jobs tend to be monotonous and unsatisfying. Such jobs, instead of allowing individuals to develop their mental and physical faculties by exercising them freely and fully (that is, instead of being fun), numb the mind and the body and retard the personal development of those employed to do them. A system of production that works efficiently toward the goals of employers does not necessarily work efficiently toward the goals of employees or toward the goals of society in general.
More important to employers than the economic benefits, however, are the political benefits of the division of labor — benefits that help management maintain its authority in he workplace. Confined to a range of activity that is limited both horizontally and vertically, employees do not gain firsthand knowledge of the overall organization, strategy or goals of the institution that employs them. Those who work within this division of labor see the consequent ignorance in themselves and in their coworkers and feel a need to be directed by people who comprehend the whole operation. Management has the broadest view of what is going on, and this helps make its supreme authority in the workplace seem natural and justified.” (Jeff Schmidt, Disciplined Minds, p91)
Jeff Schmidt was an editor of Physics Today for 19 years. He has a PhD in physics from the University of CA at Irvine. He was fired for writing this book, and then won the resulting lawsuit.
To this day, people tend to pretend as though this book was never even written. It’s all business-as-usual in the world of science.
Philip Shehan,
“O’Donnell also sttes, concerning the review process:
“I am quite satisfied that the review process was fair and equitable”
Well of course O’Donnell is going to try and calm troubled waters. I would find it very strange if he voiced any acrimony over this. He would certainly want to retain as much goodwill from both the editor and the publisher as possible.
As for your other comment that O’Donnell effectively agrees with the Steig paper that the Antarctic continent is warming, again, he is stepping back from confrontation. But the data speaks for itself – you only have to look at and compare the two temperature anomaly maps. They are different, and in the end, isn’t that all that counts?
Kudos to Dr. B. The peer-review system _sort_of_ worked, despite apparent attempts to game it. Perhaps it would have been better if Dr. S’s contribution had been in the form of a _signed_ comment distributed to the authors and reviewers at the beginning of the review process. One might also hope that Dr. S. would have been allowed space for a signed reply to the final paper in the same issue. However, allowing him to sit in judgment as an anonymous reviewer was not a wise decision. The public is left with the impression of “tacky” behavior by scientists and scientific journals. This is bad PR and we all may end up paying for the loss of public trust.
Chris Reeve says:
February 18, 2011 at 9:02 am
A professional is somebody who is expected to adopt the framework, and problem-solve entirely within this unquestioned framework. A professional NEVER questions the framework itself, because that is outside of the scope of his professional duties.
Apart from the dubious allusion to ‘professionals’, scientists are NOT professionals in your sense. Their job is to overthrow [if they can] any frameworks, this is called advancing knowledge. It is every physicist’s dream to prove Einstein wrong. You are confusing your ‘framework’ with the ‘scientific method’. The latter is indeed unquestioned as it should be.
A side effect of publishing reviews might be to lessen the number of incomplete papers submitted, if people could see how much the paper needed to be developed based on reviewer comments. I once observed a prolific and respected researcher screaming, after having a major flaw found in one of his papers after publication, blaming the anonymous reviewers for not having caught the mistake. Not the slightest hint of embarrassment at having made the mistake in the first place.
Here are the arXiv rules for endorsement:
http://arxiv.org/help/endorsement
Chris Reeve says:
Sorry, but your description of the process of getting a physics PhD bares essentially zero relation to how I experienced it. Can you tell us how you came up with this view? Is it based on personal experience, people you’ve talked to, or what?
Not all scientists would agree on disqualifying authors with conflict of interest as defined by Eschenbach. Here is Roy Spencer commenting on peer review:
http://web.archive.org/web/20040531091046/http://www.techcentralstation.com/050504H.html
“This kind of mistake would not get published with adequate peer review of manuscripts submitted for publication. But in recent years, a curious thing has happened. The popular science magazines, Science and Nature, have seemingly stopped sending John Christy and me papers whose conclusions differ from our satellite data analysis. This is in spite of the fact that we are (arguably) the most qualified people in the field to review them. This is the second time in nine months that these journals have let papers be published in the satellite temperature monitoring field that had easily identifiable errors in their methodology. ”
Actually, it often makes a lot of sense to send a paper for review to an author of the paper it claims to contradict. Science is a specialized world, and that author is likely to know more about the field and have more comments than just about anyone else the editor is likely to find. The responsibility then lies with the editor to interpret the response and see what is substantial comments and what may be just a knee jerk reaction to block an inconvenient paper. In this case it seems the comments from Stieg made the paper by O’Donnell considerably better.
Crusty the Clown says:
February 18, 2011 at 9:17 am
Kudos to Dr. B. The peer-review system _sort_of_ worked, despite apparent attempts to game it. Perhaps it would have been better if Dr. S’s contribution had been in the form of a _signed_ comment distributed to the authors and reviewers at the beginning of the review process. One might also hope that Dr. S. would have been allowed space for a signed reply to the final paper in the same issue. However, allowing him to sit in judgment as an anonymous reviewer was not a wise decision.
He did not sit in judgement, the editor did, he was asked for his opinion as an expert witness and the editor made the decision. The inexperienced authors were offended by the 1st review and the editor added another reviewer for them (but he agreed with A anyway on several matters as the editor pointed out).
Joel Shore.
The case you site is nowhere near similar.
“The first paper that I ever was asked to referee for Physical Review was when I was a grad student and some authors sent in a manuscript and essentially said, “Shore and [co-author] are wrong” in the abstract of the paper (which was talking about the work that formed my graduate thesis). ”
As your self all the relevant questions about this situation and steigs. They are only remotely similar
Willis Eschenbach says:
February 18, 2011 at 1:03 am
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
No. As a scientist who has been on both sides of the referee process that’s not how it works.
Calling in another referee when there has been an impasse with one does not mean the referee is biased.
We have all had to deal with “difficult” (or conscientious, depending on your point of view) referees. Often there can be much back and forth commentary and requests for multiple revisions before they agree to publication. Or not. It’s all part of the game. Too glowing a report from an odd man out when others are harsher can mark the referee out as a soft touch, sloppy or too obviously friendly to the authors. That can get you struck off the journals list of referees.
An 88 page referees report does seem somewhat over the top, but for all I know the points raised were entirely legitmate and raising points for clarification or amendment is not the same as rejection.
Broccoli did not grant Steig power of veto over publication. It’s the editors job to make judgement calls on how to deal with impasses. Most would not be happy with two out of three votes in favour. Calling in another opinion is one method.
And again O’donnell commended Broccoli for his handling of the matter.
Sorry if this is a repeat but it seems not to have gone through the first time:
Vince: We can take O’Donnell at his printed word or not. Any interpretation other than that is pure speculation. There is no need for O’Donnell to “voice acrimony” about the editorial process if he feels hard done by. There are plenty of sections and commentators on this blog (including this section) who are doing that for him. He can simply maintain a dignified silence while his supporters go to bat for him. There is apparently enough good will toward him on the part of the editor and publisher, even after all the alleged argy bargy, for him to have his paper published. There is no need for him to suck up to them.
Similarly whether he and Steig are at daggers drawn or simply agreeing to disagree, having had his paper published with Steig as referee means that he does not have to step back from confrontation at this stage. There is nothing to be gained from promoting confrontation, but again all that requires is dignified silence. There is nothing dignified about hypocritically sucking up to Steig.
On the other hand, perhaps O’Donnell is a person of honesty and integrity and means what he writes. Call me foolish but that is my default position with people unless there is a good reason to think otherwise, and I have argued that there is no good reason.
Yes, I have looked at the maps and there certainly are differences. And again I have no reason to doubt what the authors write in the abstract when they point to those differences. To suggest that they are agreeing with the Steig paper that the Antarctic continent is warming simply to avoid confrontation, is taking reticence to upset people into the realms of a total abandonment of scientific integrity and into the territory of scientific fraud.
But my main complaint is with Mr Watts, who represents the paper as saying” the whole of the continent of Antarctica has been shown not to have any statistically significant warming…” when it says quite the opposite.
I take O’Donnell at his word that his paper is not a repudiation of Steig and his comments on the strengths and weaknesses are an honest appraisal. For Watts to decided that O’Donnell is being dishonest and insisting the paper is a rebuttal is very careless y at best and casting aspersions on O’Donnell’s integrity at worst.
DocMartyn says:
February 18, 2011 at 5:02 am
So the journal and other policies about not using reviewers with conflicts of interest, I guess those are just for the little people, editors and people like you don’t have to pay any attention to ethics policies …
Impeccably? Only if you ignore common sense and more importantly, journal policy. Broccoli had a duty to the authors as you point out, and there are a variety of ways he could have honestly discharged that duty. But he also has a duty to us that includes not putting the fox in charge of the henhouse.
Doc, your plan works beautifully as long as the players are by and large honest. But that’s not the situation here, we are dealing with crooks. Rules for dealing with crooks (like journal ethics policies) outweigh rules for dealing with honest people. This is the constitutional question again, where the US Constitution works because it assumes that the participants are humans who will lie, cheat and steal if given the chance. There definitely was a day when your system generally worked, Doc, just like there was a day when I could leave my house unlocked.
But that ain’t today, as the horrendous mess of climate science clearly demonstrates.
w.
Richard Telford says:
February 18, 2011 at 5:19 am
The latter is much better sport … but seriously, you and others are arguing for a system that (in the climate science field) has failed us horribly. Now mine may not be the best plan for getting out of that situation.
But arguing for the failed current plan isn’t going to do it, been there, failed that, got the Climategate scars to prove it.
w.
Mike says:
February 18, 2011 at 6:25 am
BWAHAHAHA, oh, Mike, you really do believe their stuff.
The IPCC has recently changed its rules for the next Assessment report to let in gray literature. For the previous reports that was not supposed to be the case. Here’s a Reuters News interview with Pachauri, the head of the IPCC (emphasis mine), from 2009:
See that part in there that says “using only articles approved by other scientists, called peer review”? Perhaps you could explain what that means to the rest of us. Or not.
In addition, yes, the reviews were published. And Steve McIntyre and others had to fight like hell to get them published. He would laugh to think that fools could use the results of his work in prying open the IPCC against strong opposition, use that very work to prate about how open the IPCC is …
And you want to tell me that I don’t understand the IPCC? Dude, you come in here with enough misinformation about the IPCC to fill a wheelbarrow, and you want to lecture us on what’s what? You’re talking to guys who actually fought the battles, Mike. I was the first man to put in a Freedom of Information request to the CRU, and you want to explain the IPCC to me? Funny, I didn’t see you on the front lines, how was your knowledge of this war acquired? Because we got our knowledge the hard way, through personal interactions with the players involved.
BZZZZZT. Next contestant, please …
Thomas says:
February 18, 2011 at 10:14 am
No, no, and no. They should be asked to comment, either informally before publication or in a rebuttal in the same same issue or in some other manner.
But they should never be reviewers. I keep asking this, but if you guys claim is true that authors are the best reviewers, not the best to comment on a paper but the best reviewers, then why are there journal policies against using reviewers with even the APPEARANCE of a conflict??? Seems like if what you claim were true, that would be in the policies … but where are those policies that say “It’s ok to use a reviewer with a huge conflict of interest if they’re one of the paper’s authors”?
If a guy stood to lose a lot of money from a scientific paper being published, would you let him review it? I wouldn’t, because of the conflict of interest. If he knew a lot about the subject I might, as editor, ask his opinion (although that wasn’t the case in this instance with Steig). But put him in as a reviewer? Everyone would just say “Yeah, of course your journal wouldn’t publish an anti-McDonalds study, Ray Kroc was one of the referees that shot the paper down”. Might not be true, Roy might have actually supported the paper, but it destroys credibility and faith in the institution.
So if a guy stands to lose a lot of scientific prestige from a paper being published, would you let him review it?
w.
Philip Shehan says:
February 18, 2011 at 10:58 am
I appreciate your thoughts, Philip. Again, like others, you assume that we are dealing with honest people of good will. The founders of the US Constitution were much wiser. They assumed that e.g. if a man was obviously too friendly to the authors, rather than striking him off the list of referees, some editors would simply use him when they are in need of a puffball easy review for that class of authors. Duh.
Philip, the system we have, as you point out, assumes that the actors will act honorably and with good will. And for many years that seems to have been generally true enough to keep the system working. As Climategate showed, however, in climate science currently that is not true. Because of the false importance placed by the IPCC on peer-review, people of bad will have gained the ascendancy and gamed the system to act as gatekeepers for their particular version of science.
That’s why I say we need a new system. Despite the spirited defense of the status quo by yourself and others here, that system is clearly not working for climate science. You can have a Star Chamber only as long as the judges are honest … and as the good folk of England found out, when you allow people to judge other people secretly and without responsibility for their actions, even judges who might start out honest soon fall from grace in that situation …
w.
“Theo Goodwin says:
After sharing some of his experiences as journal author and reviewer, none of which involve any reference to moral judgement whatsoever, DocMartyn writes that the journal editor, Broccoli, had a duty to Steig and coauthors and he satisfied this duty by enlisting Steig as a reviewer. What duty is that, DocMartyn? Would you please explain? I believe that you are incapable of writing one sentence about duty that is coherent. So, here is your chance.”
Steig et al., was published and published in Science, one of the highest ranked journals.
O’Donnell et al., wrote a paper that challenged the methodology and conclusions of Steig et al.,, not the data from which the conclusions were drawn. The whole of Steig et al., which was being critiqued by O’Donnell et al., was the authors analysis.
First author generally does the most work and last author generally has the most money/seniority. Steig was the front man and the Editor had a duty towards the papers authors, he had to make sure that O’Donnell et al., critiques were fair. The experts, for the defense, were Steig and co-workers. The editor asked the front man to examine the manuscript. Twice Steig reviewed, after which the editor ignored his input. He gave the Steig team a heads up of incoming criticism and allowed them to find any major faults. They found none. So Broccoli gave the Steig team first bite of the cherry and Broccoli gave O’Donnell et al., a baptism in fire and it survived.
Theo, I suspect that you do not realize the emotional investment there is in generating a paper. Each paper has about the same level of pampering and the emotional investment as a pedigree show dog. Papers are to scientists what ships are to sailors or land is to farmers. They are part of you and you are part of them. When someone attacks your work, in review or post publication, it hurts, it really does hurt.
Each paper, sometimes subtly, affects your pay, promotion and job chances. When you go for an interview for a job, your can be sure that the people behind the desk have mined for publications for citations whereby they have been creamed. I have 48 peer reviewed papers and two chapters. Only one has been directly panned, unfairly in my view, and that single citation, out of 79, still pisses me off.
Excellent post, Willis. To the extent that science is real science, and not religion, it should implement policies like these. That should go, incidentally, for humanities as well.
Willis Eschenbach says:
What you actually quoted from talks about the responsibility of the reviewers, not of the editor. The point is to not have a situation where a reviewer might have biases that the editor is not aware of. I don’t see anything in the policy that states that editors must only use reviewers who he/she thinks have no such biases. (The only reference to conflicts of interest that I see under responsibilities of the editor is that the editor him or herself not have a conflict of interest.)
And, as I have noted, in Physical Review, from what I can tell, it is considered good form for the editor of a paper that is largely a critique of another paper to get a review from the authors of that other paper. In fact, I wasn’t even on Physical Review’s radar screen as a potential reviewer until they received a paper that was largely a critique of my work. (Once you get on their radar screen though, you never get off!) In this case, the editor understands that the reviewer will have his/her biases; however, that reviewer also has the most knowledge of the work being critiqued and possible flaws with the critiquing work and this knowledge is invaluable.
In this case, the procedure that the editor used from what I understand of it (and I admit that I haven’t followed the gory details to the degree that some have) seems very reasonable. He solicited the opinion of Steig as a reviewer and took his review seriously but ultimately did not give Steig veto power over the manuscript or unlimited authority to demand changes in the manuscript. I guess I am having a hard time understanding what exactly the complaint is.
@ur momisugly Willis Eschenbach,February 18, 2011 at 11:36 am
“The use of non-peer-reviewed sources, which is sometimes called gray literature, also has been controversial and blamed for some of the errors. However, we found that such material, which can include technical reports, conference proceedings, observational data or model results, often is relevant and appropriate for inclusion in the assessment reports. IPCC has guidelines for the use of such sources, but these guidelines are vague and have not always been followed. We recommend that these guidelines be made more specific – including noting what types of sources are unacceptable — and that they be more strictly enforced to ensure that unpublished and non-peer-reviewed literature is adequately evaluated and appropriately flagged in the reports.”
See:
http://reviewipcc.interacademycouncil.net/OpeningStatement.html
You can look through the IPCC 4 references like here.
As for the Interview with Reuters, the first line you quote is a paraphrase. Maybe the reporter got it wrong or P misspoke. The direct quotes do not say gray literature was never used. But all sources are reviewed by the IPCC scientists – they are reviewing the literature gray and peer reviewed. BTW: I disagree with the last line from P, as nothing 100% fail-safe.
Folks: It is painfully obvious that despite whatever good intentions Willis may have, he does not know beans about how science is done. He is not is any position to engage in meaningful discussions about climate change. There are lots of places where you can read about climate science. This is not one of them.
“BWAHAHAHA” yourself Willis. Grow up.