Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
@ur momisugly John Brookes
Worked in this case, eventually.
But this might very well be the exception that proves the rule. 🙂
vukcevic says:
February 18, 2011 at 1:22 am
However, thanks to WUWT it has been seen more than 2500 times (out of total of 53000 views of my graphs since Jan 2009).
It is misuse of WUWT to use it to promote junk ‘papers’.
Lucid prose Willis, well done per usual. Provocative.
But I’m not sure that publishing the reviews will be all that interesting however. If a reviewer has comments on egregious errors (say, a math error or bust in logic or otherwise obvious flaw)- they SHOULD first be communicated to the Authors to avoid an embarrassing, bone-head mistake. Those type – along with ‘mechanics’ of the paper (improving prose (English as second language issue), figures, tables, etc.) – are what makes up 90% of a reviewers comments in my experience.
But I think we need to remember that Peer Review is (IMHO) NOT meant to take be the “First and Only Crack” at scientific discussion of the paper ala implementation by The Team. The “Peers” are the Journal’s READERSHIP – NOT the Reviewers.
Substantive comments that turn on a point of art (ala Steig 88 page Reply) more properly belong in a Reply or Discussion of the paper in the Journal.
Thanks, Doc. Your ideas are always interesting and welcome.
DocMartyn says:
February 17, 2011 at 7:28 pm
The Journal acted unethically, in my opinion. They should have given you a copy of the proposed paper, and offered you space in the same issue to publish your response. You could have demolished them with the fourth panel, it would have been whacking great scientific theatre, and journal sales would have gone up.
I see several issues here. First, the anonymous review system is obviously not working in this instance, because as you say the guy is getting papers published. Picking an example where the anonymous peer review system isn’t working seems like an odd choice of example.
Second, I couldn’t do that. I couldn’t anonymously put the knife into someone’s work and then smile and make polite dinner table conversation with them. I am serious when I say that I am sorry that you are in that position. I simply could not do it. If I decided to do the review, I would go to the man myself first and privately discuss the question with him, show him the problems, offer him a graceful way out. If he said no, I’d put in the review, and he’d know who wrote it. But no, I couldn’t bear the other way, I’d turn it down, my soul’s not engineered for that kind of tensile strain. I grew up on a cattle ranch, and now I’m a reformed cowboy, I gave up most of my rough ways.
But somewhere inside I’m still eighteen and on horseback in chaps, helping push a dusty, cranky herd of cattle to the upper range, with my hat shading a blinding bright summer view over the winter pastures far below and on past verdant hills to an unknown future … and a cowboy shoots a man in the front. It’s in the contract.
Third, while anonymity protects reviewers from powerful authors, sunlight protects science itself from powerful reviewers. The IPCC (a poisonous body which does not exist in any other field) said it would only consider peer-reviewed science. This put huge pressure on the field. In response, editors picked reviewers who could be counted on to do the right thing. This has been lethal for climate science. Things like l’affair Steig are destroying the trust of the people in science itself.
Given the choice, and since it would relieve people such as yourself from the unpleasant burden of being two-faced and secretive with your colleagues, I’d choose to protect science itself over protecting reviewers.
But hey, that’s just me.
w.
PS – you say:
I couldn’t disagree more. The infiltration and diversion of climate science by a combination of noble cause corruption, politics, money and the IPCC, including the corruption of the climate science peer review system, has led to billions of dollars being spent on an idea which, to date, has been unable to falsify the null hypothesis.
I think that future generations will view the whole episode as one of the most laughably misguided examples of pseudo-scientific fear-mongering in history.
And for me that means that no, the system’s not working well at all.
Good work Willis.
But we should remember that peer review is only part of the problem.
I am currently preparing a proposal for a root and branch overhaul of the whole publishing process for climate research.
I can see that I had better hurry and put ot up for consideration as things are developing fast.
Ali Baba says:
February 17, 2011 at 8:27 pm
You mean like the collegiality of Newton and Leibnitz? Or how Wegener was treated? By the “adversarial system” I meant only that science is a battle of ideas, where scientists try to show that their idea is correct, and to point out the flaws in the other scientist’s reasoning. They are not partners, Wegener and the “immovable continent” folks. They are adversaries.
And if the penalty for oh, say, advising people to delete incriminating emails, or hiding adverse results in a CENSORED folder were “professionally fatal”, Michael Mann would be pushing up daisies right now … instead, he’s still feted and invited to conferences and the like. I’d love it if fraud were scientific suicide, but he’s made it into a brilliant career choice.
Finally, science based on trust? You are seriously talking about climate science based on trust? After Climategate, after the Jesus Paper, after the multiple exaggerations in the IPCC report, after the famous “Nature trick”, after the farcical “investigations”, after all of that and more you can sit there with a straight face and talk about climate science and trust in the same sentence?
I can’t do that. I start cracking up laughing. Trust a climate scientist? Get real. You can trust them all you want, my friend … I’ll pass.
w.
Off topic, but some what on topic. I have posted on Tamino’s site “an open mind” However not one of my posts have been posted. One was even complimentary that the ice ages were being looked at after I posted a please look at the ice ages as the science should be able to explain these first easily before the understanding of the small changes occurring lately. But disappeared I have, no open mind there.
While I agree with Willis’ proposals (double-blind review process; publication of reviews on the web after publication), may I make two suggestions, which could be implemented straightaway:
1) no author of a paper being criticised should be appointed as reviewer.
Once the critical paper has been accepted, an editor can write to the author(s) whose paper has been criticised, and tell them that such a paper will be published in this or that issue, and they are given so many lines to answer the criticism in that issue.
This would immediately remedy the issue of CoI.
2) addressing what actually is meant by ‘peer’.
I have the impression that far too many people, especially in The Team, seem to think a peer, like a Lord, has a status higher than the others, such as critical authors or lesser-known ones, and are therefore the only ones who should be allowed to ‘judge’.
Well, it ain’t so: peer to peer originally said that peasants cannot judge lords, only other lords can.
Therefore, and keeping in mind that the review process is meant to sort the wheat from the unscientific chaff, I find it very odd indeed that, as was the case with Steig09/O’Donnel10, that at least two reviewers said they are not statisticians.
Why were no statisticians asked to review this paper?
Therefore editors must select at least one reviewer who is knowledgeable in the specific area a paper addresses: statisticians for papers critical of statistical methods; biologists for papers using biological proxies, etc …
So editors should look carefully at what the paper is really about.
And they should always be mindful that a paper must advance the science, not just fiddle around with one or another parameter of what is already known.
We’re all aware of the many fluffy papers being published with great fanfare, showing nothing much at all, which Anthony posts from time to time to the general delight of the WUWT audience. Would be nice to know which reviewer(s) gave them a pass …
And that brings us back to Willis’ proposals!
I’m sorry, from what I’ve seen, climate “science” peer review is about as corrupt as you can possibly get.
And what’s more THEY KNOW it is corrupt, because I’ve been in a quite a few “discussions” where the warmists make little jokes and quips about peer review – which I could only take to be an acknowledgement by them that they control the peer review system.
And … the appropriate response is not to get involved in their corrupt system. IGNORE IT … DON’T GO ALONG WITH IT … CREATE YOUR OWN JOURNALS , BUT PLEASE PLEASE DON’T ADD CREDIBILITY TO CORRUPT ANTI-SCIENCE JOURNALS BY TRYING TO GET PUBLISHED IN THEM!
What you guys seem to think is some exceptional practice is in fact common practice…at least in physics. And it is wrong to pretend otherwise. Here is what I wrote in one of the previous threads on this topic about my personal experience:
As has been pointed out here by others, the reviewers likely to be most knowledgeable are ones who, by your standards, would tend to have some conflict of interest. Furthermore, an editor often especially wants to hear the response of those whose work is being criticized by the paper under review, even if he will weigh it with the knowledge in mind.
The first paper that I ever was asked to referee for Physical Review was when I was a grad student and some authors sent in a manuscript and essentially said, “Shore and [co-author] are wrong” in the abstract of the paper (which was talking about the work that formed my graduate thesis). And, I was able to explain very clearly why the authors of the manuscript were largely mistaken. The final version of their paper essentially said, “Shore and [co-author] were largely correct although there is one aspect where they were incorrect” and included an argument from me supporting and amplifying the discussion of the point where they had disagreed with what we had originally said. The result was a better science than either their original manuscript or our paper alone and less of embarrassment for them than publishing their original manuscript would have been.
In another case, several years later, a paper was written in which they showed that something that I and a colleague had found in one model and had conjectured was true quite generally did not in fact appear to be true for a particular alteration of the original model that we looked at. Again, their paper was sent to me to review and I basically wrote a long review saying that the paper should be accepted but in which I also made several comments, queries, and optional suggestions for revision. I even wrote in the review essentially that I did not find the evidence that the authors presented to be completely convincing but that, since I was not an objective reviewer, my standard on that matter was particularly high and I thought objectively they had in fact presented strong enough evidence to warrant publication. The authors responded with a statement thanking the reviewer for his very thorough and comprehensive review and then a comment that still makes me chuckle to this day, in which they said something to the effect of, “…and in fact we are quite sure that we have never seen such a long positive review.”
Note that both of these papers were in Physical Review journals and had nothing whatsoever to do with climate science. So, the claim that people here seem to be making about climate science having unique issues such as the use of reviewers who might have a conflict of interest is utterly without foundation.
Willis Eschenbach says,
“Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”??? ”
Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.
After two iterations from the reviewer, he ignored reviewer A’s additional comments.
Broccoli behaved impeccably.
Willis Eschenbach says:
February 18, 2011 at 3:12 am
“They should have given you a copy of the proposed paper, and offered you space in the same issue to publish your response. You could have demolished them with the fourth panel, it would have been whacking great scientific theatre, and journal sales would have gone up.”
Is it better for a bad paper to be published then demolished or never to be published?
Some papers are probably wrong but make interesting challenges to the status quo, stimulating research and new ways of thinking. Ruddiman’s early anthropocene might be an example of this. These are certainly worth publishing.
Some papers are bad, with trivial or obvious errors such as McLean et al 2009. Should these be published to? If they are published then someone ought to put time aside to write a reply – I have done this twice, it is a lot of effort with scant reward. Do you think it advances science to see such poor papers published? Or does it advance the agenda of those who would wish to convince the public that the scientific literature is riddled with doubt, uncertainty and controversy, and use this an excuse to avoid action?
DocMartyn says:
February 18, 2011 at 5:02 am
Willis Eschenbach says,
“Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”??? ”
Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee. After two iterations from the reviewer, he ignored reviewer A’s additional comments. Broccoli behaved impeccably.”
Please explain to me how Broccoli acted impeccably when journal policy is:
“A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.”
The logic of this statement of journal policy is that there should not be an appearance of conflict of interest. How exactly do you think Broccoli could have thought this policy would be met by appointing Steig as a reviewer? It would be clear even to someone with the IQ of a chimp that such a conflict existed.
Clearly your idea of acting impeccably is very different from mine. You’d best head over The Carbon Brief. They’ll probably welcome you with open arms, with them being such champions of independent review and all.
Willis: “Mike, although you and I don’t consider it a fail-safe process, the IPCC does. If a paper is not reviewed, it can’t come in.”
Both statements are false. IPCC does make use of non peer reviewed studies – the so called gray literature. Even if the second statement were true, it does not imply the first. Since you don’t understand the IPCC process, maybe you are not in a position to critique it. IPCC is a a review body were the reviews are published and all names reveled. Exactly what you want!
Good post Willis.
I would say though that If I were Broccoli and harbored deep distrust of the pal review system that has reared it’s ugly head, I might put Steig on just to give him the tempting target that they had clearly “gone to town” on before as evidenced by the climategate e-mails. In short, if I were Broccoli, I might have done this with the intent that the hostile reviewer would out himself and this kind of furor would result. If I were trying to expose Climategate as less than some kind of out-of-context invasion-of-privacy to the world, I can’t think of a better way to do this than to create a situation where someone from the Team felt threatened enough to expose themselves.
But, I’m a sneaky b@stard…
You wonder if the good Herr Dr Professor Brocoli is related to anyone from the cabbage patch.
Hello Willis,
We all recognize the problems with peer review. The issues raised with the O’Donnell paper are unfortunately not new. I’ve done a bit of perusing on the subject and have found some interesting tidbits. The basis for peer review is apparently quite old. It seems to be directly related to “imprimatur”. The term is defined as:
“1 a : a license to print or publish especially by Roman Catholic Episcopal authority
b : approval of a publication under circumstances of official censorship”
Mario Biagioli wrote a book titled: “From Book Censorship to Academic Peer Review” in which he ties the early book printing licensing by the state and the creation of the editorial peer review process, with the state funding of academic societies of the time, and both for the same purpose: censorship of ideas that could potentially foment heresy and subversion. The Royal Society of London is quoted as saying in 1663 that “No book be printed by order of the council, which hath not been perused and considered by two of the council, who shall report, that such book contains nothing but what is suitable to the design and work of the society.” The Society of course, was funded by the state that granted its imprimatur.
The divorce of the original peer review doctrine between the state, journals and societies came about later. This allowed the “self-policing” of ideas put into print by the societies which in theory became more about the validity of the idea and not protectionism by censorship. Nevertheless, unknowingly or not, these “liberated” reviews were inherently still censored, and protected the “establishment” of any particular academic discipline by definition, because it was this same establishment that reviewed the submitted works.
Even if nowadays the whole process is purported to be more about accuracy rather than legality, the fact remains that it still is, and will be until revised, a system for the establishment to grant imprimatur.
There might be some light at the end of the tunnel though. The same country that might have invented peer review has now given us what just could be the beginnings of a more open model to follow – the BMJ – British Medical Journal: http://resources.bmj.com/bmj/about-bmj . More is needed but theirs are small steps in the right direction. A BMJ paper on the subject of peer review can be found on the second link below.
Best,
Jose
Sources:
http://mediacommons.futureofthebook.org/mcpress/plannedobsolescence/one/the-history-of-peer-review/
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2127543/pdf/9345164.pdf – Page 3
Andrew Bolt has carried this in his blog today referencing Watts Up With That. I commented on Mr Bolts blog:
I agree that Steig should not have been a referee for that paper. [On checking I note the paper was specifically flagged as a critique of Stieg being titled: “Improved methods for PCA-based reconstructions: case study using the Steig et al. (2009) Antarctic temperature reconstruction.” In this case I think it may be appropriate that Steig be given a chance to comment on a paper that is entirely a critique of his work. ]
But Steigs work is a long way from having fallen apart. [Mr Bolts assertion]
My source for this opinion is none other than the authors of the paper in question.
[I then reprinted my critique of earlier of earlier WUWT items on these papers]:
[Watts] discusses a new paper by his “friends” O’Donnell and Condon which says that Antarctic warming is not as severe as that proposed by Steig et al. The paper itself is not available for perusal but here is the Watts Spin:
http://wattsupwiththat.com/2011/01/13/another-overhyped-global-warming-claim-bites-the-dust/
“Oh and let’s not forget the fact that the whole of the continent of Antarctica has been shown not to have any statistically significant warming…”
But then Watts provides a link to the earlier posting on this where the abstract for the paper is given:
http://wattsupwiththat.com/2010/12/01/skeptic-paper-accepted-on-antarctica-rebuts-steig-et-al/
“Rather than finding warming concentrated in West Antarctica, we find warming over the period of 1957-2006 to be concentrated in the Peninsula (≈0.35oC decade-1). We also show average trends for the continent, East Antarctica, and West Antarctica that are half or less than that found using the unimproved method. Notably, though we find warming in West Antarctica to be smaller in magnitude, we find that statistically significant warming extends at least as far as Marie Byrd Land. “…
Watts headline to the Condon and O’Donell paper is boldly headed :
Skeptic paper on Antarctica accepted – rebuts Steig et al
Lead author Ryan O’Donell writes (in line with the title of their paper, Improved methods for PCA-based reconstructions: case study using the Steig et al. (2009) Antarctic temperature reconstruction
“Overall, we find that the Steig reconstruction overestimated the continental trends and underestimated the Peninsula – though our analysis found that the trend in West Antarctica was, indeed, statistically significant. I would hope that our paper is not seen as a repudiation of Steig’s results, but rather as an improvement.
In my opinion, the Steig reconstruction was quite clever, and the general concept was sound. A few of the choices made during implementation were incorrect; a few were suboptimal. Importantly, if those are corrected, some of the results change. Also importantly, some do not. Hopefully some of the cautions outlined in our paper are incorporated into other, future work. Time will tell!”
O’Donnell also sttes, concerning the review process:
“I am quite satisfied that the review process was fair and equitable, although I do believe excessive deference was paid to this one particular reviewer at the beginning of the process. While the other two reviews were positive (and contained many good suggestions for improvement of the manuscript), the other review was quite negative. As the situation progressed, however, the editor at Journal of Climate – Dr. Anthony Broccoli – added a fourth reviewer to obtain another opinion, which was also positive. My feeling is that Dr. Broccoli did a commendable job of sorting through a series of lengthy reviews and replies in order to ensure that the decision made was the correct one.”
DocMartyn,
“Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.”
Is it your assertion then, that an editor must, by all that is ethical and just in the world of journal publishing, bring in as reviewer, the author of the paper that is being attacked?
If this is correct, then this whole article is redundant. We don’t even need to argue whether or not a conflict of interest exists, or whether the editor broke journal policy, because you have incontrovertible proof that this is the way editors must act. Why don’t you just show us this policy that says the author of the critiqued paper must be the reviewer? Or is this just your own opinion?
John Brookes,
” John Brookes says:
February 18, 2011 at 1:58 am
Its obvious that Steig was the right choice for a reviewer. He has already worked on the problem, and he will be hard on someone trying to overturn his work. So he is ideal. The editor can certainly override what Steig said, no problem, and that is what happened in the end.”
Your logic is a little flawed. If the editor was knowledgeable in statistical techniques, then he would indeed be able to override Steig. But the evidence is that he reacted to what was becoming a vexatious challenge by Steig to O’Donnell’s paper. Eventually, having ‘had it up to here,’ with Steig’s behaviour, he was forced to dump him for an unbiased reviewer.
So the eventual correct outcome happened in spite of the decision to appoint Steig, not because of it. If anything can be said in Broccoli’s favour, it is that he had enough common sense to see, in the end, that he was being jerked around and booted him out.
First comment says it well, “Spot on as usual Willis.”
DocMartyn says:
February 18, 2011 at 5:02 am
“Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.After two iterations from the reviewer, he ignored reviewer A’s additional comments.Broccoli behaved impeccably.”
After sharing some of his experiences as journal author and reviewer, none of which involve any reference to moral judgement whatsoever, DocMartyn writes that the journal editor, Broccoli, had a duty to Steig and coauthors and he satisfied this duty by enlisting Steig as a reviewer. What duty is that, DocMartyn? Would you please explain? I believe that you are incapable of writing one sentence about duty that is coherent. So, here is your chance.
You are going to need some help with this, DocMartyn, so let me get you started. As journal editor, I have a duty to be impartial to authors who submit papers for review. As part of that duty, I must ensure that reviewers WHO SERVE ME are impartial. My journal publishes a policy damning conflict of interest. So, I select as reviewer the one person most likely to have a conflict of interest, Steig, and, thereby, violate my journal’s policy against conflict of interest and my duty to be impartial. What do you not understand about this, DocMartyn. I am eager to explain it for you.
Finally, DocMartyn, I know that this is going to be difficult for you because the topic is moral judgement and not the usual mud slinging and sausage making. I feel for you. However, you just have to face up to it. The topic is moral judgement.
Leif Svalgaard says:
February 17, 2011 at 5:01 pm
There are boreholes, and there are boreholes. The problem is whose work is being dumped into the borehole and why, versus whose work IS NOT being dumped into the borehole and WHY NOT? The pal reviews by the so-called TEAM could by some objective measures deserve to be consigned to an infinitely deep borehole for the simple fact that they refuse to provide the open scientific community with the customary data and methods required to replicate the claimed scientific experiments.
I would suggest using the e-print arXiv as a starting point for developing an objective and merit based approach for the scientific community and the broader general public to promote and demote the papers by initial and ongoing reviews of the paperss and their subject matter. In particular, when a given paper fails to meet certain basic criteria such as proper documentation, different communities can choose to apply the objective criteria to demote the paper to the rank of those failing to meet fundamental requirements for documeentation of mthods and data, while the pal reviewers can can choose if they so wish to risk their reputations by refusing to apply those basic scientific standards. In any eveent, outright censorship and secretive political ceensorship can be avoided or at least mitigated.
D. Patterson says:
February 18, 2011 at 8:13 am
I would suggest using the e-print arXiv as a starting point for developing an objective and merit based approach for the scientific community and the broader general public to promote and demote the papers
arXiv is very useful and serves an even more important function: to get results out quickly. But [and this is proper] there is still a filtering performed: you cannot post to arXiv unless you have been endorsed by another scientist with sufficient credentials to be an ‘endorser’. In this way obvious cranks are kept out as they should.
I must say that I am shocked and dismayed at the willingness of the Warmista to create sheer blather as a kind of smokescreen to prevent criticism of poor or nonexistent moral judgement. The situation is as follows:
Officer: Sir, I saw you drive through the stop sign and run down the pedestrian who now lies bleeding beneath your car.
Driver: Officer, you say that as if it is a bad thing.