Peer Review, Pal Review, and Broccoli

Guest Post by Willis Eschenbach

The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.

Image Source

For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.

The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.

It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:

A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.

Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:

We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …

This request was ignored by the Editor.

Steven Mosher had an interesting comment on this issue:

What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one

Imagine, for example, that you are a grad student with zero publications.

Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”

If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.

Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:

Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.

While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.

Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.

But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …

I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.

The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?

And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.

Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?

And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.

This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.

And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.

My suggestions to make peer review a better system are:

• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.

• All reviews get published with the paper, with each one signed by the responsible reviewer.

This has a number of advantages over the current system:

1.  Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.

2.  Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.

3.  While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.

4.  As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.

5.  People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.

6.  There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.

People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.

First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?

Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.

That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.

[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.

Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.

In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.

These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.

Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.

It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?

Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.

Regards to everyone,

w.

[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:

I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.

Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …

Advertisements

  Subscribe  
newest oldest most voted
Notify of
David C

Spot on as usual Willis.

Rhoda R

I agree that sunshine is badly needed in the whole review process – the more the better. It’s the dirty work at the crossroads that’s done in the dark and that’s more than half the problem in that the ‘team’ has been allowed to maneuver without any kind of accoountability.

chris y

Willis said- “People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer.”
This potential issue is easy to resolve. Simply make each review a referenced publication that partially counts towards tenure and promotion. After all, it is published. It may not be peer-reviewed, but many/most conference presentations are not peer-reviewed either.
Faculty will line up in droves to review papers, and the quality of reviews will dramatically improve.

All reviews get published with the paper, with each one signed by the responsible reviewer
I agree. There is small problem not addressed: what if the paper is rejected by a conflicted reviewer?

Rhoda R

I need to add to my previous comment. Sunshine will only h ighlight problems that we see with published works, but what about good studies that are NOT published due to undue influence. For instance, if Stein had managed to get the article rejected would we even have known about it other than a plaintive or inflamed blog? How do we keep the gatekeepers honest?

Latitude

Willis, the problem with peer review is that it is what it is….
It’s a method of getting other people to agree with you….
..Most of the time, it’s almost impossible

Vorlath

Brilliant post. I said the same thing at ClimateAudit. This is what stuck out the most for me. Serious conflict of interest. Basically judge, jury and one of the parties all rolled into one. I like your suggestion to publish the reviews. I think it’d be cool to make available the reviews online or something, but to publish only the reviewers final thoughts. This happens in the legal system where there is more than one judge. Both dissenting and supporting opinions are available.

Willis Eschenbach’s points are totally on target. They’ve been made before by others, but the current peer-review system has become a kind of locked-in-cement paradigm that will take time to fix. Fortunately, the web will speed up the inevitable, and it’s likely we’ll eventually arrive at a review system that will closely resemble what Willis and others have proposed. For the record, I’ve reviewed a fair number of papers and books, and I usually tell the editor to leave my name on the review.

Luther Wu

listening to: Talking Heads- “Stop Making Sense”

Long ago in my publishing days, Nature and Science were not considered true peer review journals. How things have changed.

Archonix

“This one we call Crossing the Desert.” *whackwhackwhackwhackwhack* “And this is the Unblinking Eye” *whackwhackwhackwhackwhackwhack*
“Hey did anyone notice that unblinking eye was just like crossing the desert?”
“And now for the paddling of the swollen ass. With paddles!” *whackwhackwhackwhackwhack*

Benjamin P.

Uh, the paper got published.

Peter Milford

Very good Willis, except that haven’t you (inadvertently) got the last sentence backwards, when you say :-
“But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process hidden. Keeping it secret just encourages the current abusive system of pal review.” ?

Willis Eschenbach

Leif Svalgaard says:
February 17, 2011 at 4:13 pm

All reviews get published with the paper, with each one signed by the responsible reviewer
I agree. There is small problem not addressed: what if the paper is rejected by a conflicted reviewer?

Thanks for pointing that out, Leif and others, I had forgotten to address that.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … where’s the fun in that.
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
I’ll add this to the head post.
w.

Richard Telford

Steig was the best possible choice for a reviewer. He knows the material well, he would be motivated to do a thorough review, and his potential conflict of interest is well known, so his review could easily be discounted if necessary.
Most journals use only two reviewers. AMS used four.
If you want an easy review, stick to E&E.
” First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?”
Because the reviewer is not anonymous to the editor who is making the decision!

DocMartyn

“The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. ”
No. The editor was absolutely correct in picking Steig as a referee. He then had the paper go through two complete rewrites, which Steig attacked and ignored his final recommendations, bringing in a forth unbiased referee.
The one person who comes out really well in the whole saga is Dr. Anthony Broccoli.
I only wish I had such a clear-sighted and ethical editor to handle my papers.
Hats off to Anthony Broccoli, a true professional.

But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process hidden.

Typo? Isn’t that the opposite of what you meant to say? Apart from that, I think you’ve got it right.
[Fixed, thanx. ~dbs]

Green Sand

As an engineer I am indoctrinated with the desire of failsafe systems. Life hazard is my mantra. Every single calc I ever carried out was third party checked, and at least one of the checks was “longhand”.
This “peer review” (academic theatrical performance) does not give me a feeling of well being. As a “peer view” suggesting that there might be something in this paper, then OK. But as an “absolute review” upon which to base the recommendation that the whole of mankind change the way they live? Scary, very, very scary and quite frankly, if you give it just a little thought it is absolutely daft!

Mark Twang

Silly people! Well-meaning Gaian scientists really have no peers. They are nonpareils!

Willis Eschenbach says:
February 17, 2011 at 4:47 pm
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
Some journals reject more than 90% of papers submitted, so that ‘borehole’ will be enormous and probably have a lot of genuine junk in it. Some journals reject ‘obvious’ junk directly by the editor [or his assistant] before the paper even goes to review. There are the issues of proper formatting and procedures and civility that must be met. So, I’m not so sure your idea is workable unless one steps on the slippery slope of defining categories of rejection.
If a reviewer spends a lot of effort to debunk a junky ‘paper’ [and there are some that ‘are not even wrong’ and can’t be meaningfully debunked] that often is not enough to deter the author or others to give up their silly ideas – we have seen many examples of that on this very blog. The authors might even take solace and even pride from the fact that a serious scientist spent time showing that the paper is junk: “there must be something to my idea since you so vehemently rejects it”. This very blog is replete with such examples too. Whole subculture might grow up around the boreholes.

MattN

If this entire incident does not open some eyes to WTF is REALLY going on, then frankly, nothing will. This is beyond “hand in the cookie jar”. This is “elected public official caught naked with a hooker in a closet with a pile of coke on the coffee table….”
It’s THAT obvious….

I suspect that the O’Donnell paper was published in spite of Steig, not because of him as some have alleged.
For a similar example of someone trying to publish a correction to a bigwig professor, see Prof Rick Trebino’s endless hassles here.

Jack Maloney

But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process hidden. Keeping it secret just encourages the current abusive system of pal review.
Assume you meant keeping the peer review process open rather than hidden.

Caleb

“The price of Liberty is eternal vigilance.”
How are we to be vigilant if we don’t know the facts, don’t hear all the arguments, and allow things to occur “behind the scenes?”
Yes, even the logic behind not-publishing-a-paper should be part of the public record. And this is especially true if even a penny of public, taxpayer-provided money is involved.

Latitude

Willis Eschenbach says:
February 17, 2011 at 4:47 pm
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science
======================================================
Lat said: “It’s a method of getting other people to agree with you….”
You only see the papers that the peers agree with, or papers that agree with the peers.
Not only makes it real hard to get anything published that they don’t agree with, but also stops any contrary ideas and new ideas.
But it does make it easier to get funding and keep control over the ‘science’…………

Ian W

Can anyone give one area where climate science can be said to be really ‘quality science’?
It seems here that peer review is continually being ‘used’ to block dissent.
As the subject of this paper showed and as shown repeatedly against others, climate ‘scientists’ are poor to undergraduate level at statistics.
The use of botanical proxies seems to show a lack of knowledge in botany – taking width of tree rings to equate to only temperature for example.
The loss of original data and unwillingness to publish data and methods.
Continual alteration of original data without any record keeping or publication of _why_ records decades old need to be altered and no authoritative sign off of the changes.
Fudge factors often repeatedly added to observations without any record or reasoning why and why that amount of change.
Climate historical monitoring carried out at weather observation sites that are laughably poor quality with no record keeping of the status of the site.
Invention of data based on the laughably poor quality to sites 1500km distant in a way that even amateur meteorologists would balk at.
Is there not one area of science or scientific method where climate science exhibits professionalism and quality standards? One area where people can say I am proud of the quality of climate science?
And yet the world economy is being driven into the ground by the prognostications of these ungifted amateurs?

DSW

This may be rehashing what you and others have already said, but in politics the appearance of impropriety is as bad as actual impropriety and is to be avoided at all costs. That Broccoli thought he (and science in general for that matter) was immune to this axiom indicates he thinks that not enough people would catch on to what was being done. People like this (in all spheres) count on that very thing to continue being a successful repeat offender. The sad truth is it works without watchdogs keeping a vigilant eye on the perpetrators and sounding the alarm – precisely why sites like this are essential today, in an age when the scoundrels are bolder than ever.

jorgekafkazar

“2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent…”
I don’t think Dr. Broccoli ever intended to allow Steig to act as a gatekeeper, merely as an informed commenter. Perhaps Dr. B even hoped for a meeting of the minds to some degree. When it became clear that Strig was pushing way too hard for O2010 to agree with Stieg 09, Dr. B may have decided that further delay would constitute gatekeeping, and pulled the rug from under Stig.
Given the outcome, Dr. B may have done everyone a huge favor by making it abundantly clear that Climatology is no place for anachronistic peer-review, any more than treating gangrene with a physic.

Mike

Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
Not a bad idea. Often an author is building on their previous work so you can tell who they are. If a paper contradicts the reviewer’s own work s/he is bound to notice this even if they don’t know who the authors are. But, it is still not a bad idea.
All reviews get published with the paper, with each one signed by the responsible reviewer.
This totally conflicts with your first reform! And it makes little sense. Most of the reviews I’ve written consist of corrections of minor errors. Who would want to read that? If the author wants to post the reviews somewhere s/he can do do.
Perhaps journals could have a formal appeal process where a rejected author could ask for a different editor for a second opinion. But, there are many journals, so it is easy to just submit the paper elsewhere. In some reviews I have recommended rejection but also suggested another journal where the paper would be a better fit.
I don’t see a problem with Steig being one of the reviewers since there were others. It is up to the editor to assess the reviews. That was done in this case and an improved version of the paper was published.
The biggest problem with peer review is that too many reviewers spend little time on the paper and write a brief positive recommendation. This does not help the editor. The other problem is reviewers who take forever to get a round to doing the review. Also very little is done to teach new scientists how to write a review.

Konrad

I agree with most of this post, however I have a small concern with the issue of the anonymity of reviewers. The knowledge that names will be put to review comments could lead to some reviewers limiting their criticism of papers from those seen as powerful or influential in a given scientific field. It also gives the media an opening to fan conflict and profit from it.
The human primate is a social animal, and our basic social structure is tribal and hierarchical. To overcome the limitations of this in research and analysis we need systems that circumvent normal primate behaviour. The anonymous review has some advantages in this area. Even publishing review comments without the authors name would remove anonymity in scientific circles.
Do the positives of preventing conflict of interest and gate keeping out weigh the risk of social pressure on the named reviewers?

Jens

I agree with Leif Svalgaard’s latest comment. We must not lose sight of the main function of the refereeing system, to filter the signal from noise. Even with this filtering the literature is enormous. And when the system works, the refereeing is a constructive process producing better papers. The only defense against gatekeeping from established scientists is an appeal to the editors, so the integrity of editors is important. It is usually not difficult to spot a hostile review. It seems that the editor Broccoli acted reasonably in the end but the initial choice of referees does not strike me as very wise. The open discussions on the internet of controversial areas of scince is a great supplement to the peer reviewed literature and often very entertaining to follow but it does not provide a good record of progress. For this reason it is important that valuable contributions like the O’Donnell paper be published. But to publish also the discussion with the referees would add far too much ‘noise’.

Theo Goodwin

Willis’ diagnosis of the problem is excellent. I cannot imagine someone disagreeing with that diagnosis. It is also a first-rate indictment of Steig, the editor, and the journal. I cannot imagine anyone disagreeing with that indictment. Clearly, the peer review process has been hijacked and serious action must be taken by parties who control the purse strings. The result of all this should be a Congressional investigation of this affair and the Climategate affair. Short of a serious investigation undertaken by critics of this kind of behavior, there is no way to restore science to climate science.

Steve Fitzpatrick

Editor Broccoli understood the conflict of interest in this case. I hope (and believe) that he took that conflict into account when considering Eric Steig’s reviews. I do not personally see anything wrong with having a potentially conflicted reviewer… so long as journal rules on that are clear. Perhaps specific guidelines like “Editors will always be mindful of potential conflicts of interest, and will discount any review by a conflicted reviewer if there is any indication that reviewer is not acting 100% in good faith.”

Mike

@Green Sand:
Peer review is not and was never considered a fail-safe process. It is a paper, not a product that could hurt someone, that we are talking about. Peer review is a minor step at quality control so that papers that are poorly done or irrelevant get weeded out or that at least some errors get fixed. It is not a test of truth. Even a well run experiment could produce data that points the wrong way just by chance. We want odd ball ideas and uncharacteristic results to get published. The next step is for other scientists (and now some lay people) to go over the paper during the course of their work. Some will publish review articles comparing, contrasting and synthesizing several papers. Eventually review panels review the literature in the field and, if possible, form a consensus. The consensus might be “we do not know” or “the uncertainties are very large,” as was the case with climate studies in the 70’s and 80’s. But in many cases the truth slowly emerges. This happened with climate science in the 90’s. Yet we still have done little in response.
When popular science publications report on what the latest peer reviewed findings suggest you should never take it as gospel. They use the word “suggests” for a reason. When the National Academy of Sciences says they have come to a consensus that human caused climate change is real and very dangerous you should take it very seriously.
As an example, several credible papers are suggesting that some extreme weather events are linked to AGW. But many researchers are not yet convinced. Review panels have not yet gone over this area. That will probably take a few more years. So, I am concerned when I read about possible connections between extreme weather and AGW, but I will wait and see what comes out of it before I form a solid opinion on this.

John Whitman

Willis,
Thoughtful post. Thanks. You stimulated some thoughts, as you are often wont to
do. : )
First, you mention “. . . the ancient tradition of adversarial science”. I suggest that a more comprehensive way to characterize the ancient scientific tradition would be to call it open and unrepressed (free) argumentation. The perception of science as a process primarily involving adversaries does not encompass the broader spirit of the scientific process. Sure, some scientists may be primarily adversarial, but that is not even close to the broader perspective involving all the interactions/relationships of scientists.
Also, another thought is that scientific journals do not enjoy a monopoly in the area of scientific communication. They have just a share in the scientific communication market. Like laissez faire processes in economics, so laissez faire in the open market of scientific ideas allows for a continuous balancing of various competing communication processes for keeping open the free market place of scientific ideas. If the entire climate journal field was to become 100% biased toward a politically correct scientific theory, still, scientific progress is not stopped. For example this post of yours is evidence of what I am saying; what you are doing is just one example of many alternate communication processes. Science can self-correct independent of scientific journals. Indeed, climate science has been self-correcting widely; without the help of science journals (rather in spite of science journals).
Still another thought, if science is essentially applied reason then it is only the reasoning and the evidence supporting it that is the body of science. The scientific body is not essentially tied to the comings and goings of science journals or any of their arbitrarily constructed and constantly shifting (politically correct fashions of the moment) processes. It may be practical and efficient to have the science journals be more logical and have more intellectual integrity, but science journals have no essential bearing on the science itself.
John

Chris Reeve

Re: “That’s what I see as broken about the system, and how I would fix it”
Unfortunately, this is a fix to just one tiny aspect of a much larger set of very profound problems in our system of science today.
This system of peer review and consensus which dominates the sciences today largely originated with Socrates:
1: “… In the deductive method, one starts with a presumed law of nature – an obviously correct (accepted) generalization about the way things work – and deduces (works out, derives) its logical consequences.” (Don Scott, The Electric Sky)
2: “A hypothesis arrived at via this deductive method is promoted to the status of being a theory when and if a large enough body of experts accepts it. This is an application of the Socratic method, also sometimes called the `dialectic method.’ Socrates (469–399 B.C.) believed that truth was discovered through intense conversations with other informed people. In this method, a vote of the experts determines when and if a theory is correct.” (Don Scott, TES)
3: “Once such a theory has been accepted, it is not easily rejected in light of conflicting evidence.” (Don Scott, TES)
4: “It is, however, often modified – made more complicated. When over time a theory becomes officially accepted, the essence of the matter has been settled and fixed. Modifications to the fine points of the theory can then be proposed and debated, but the backbone structure of the theory is set. That framework has already been firmly established.” (Don Scott, TES)
This Socratic-dialectic methodology (SDM) to investigating Nature exhibits numerous weaknesses:
1. If the obviously correct basic starting-point presumption is in fact wrong, then we’ve possibly assumed the wrong framework.
2. The SDM method tends to generate cosmologies which lack any physical support, as the process of deduction favors those which can extend from the conjecture’s consequences. This is fundamentally a sequential process of reasoning.
3. The SDM prioritizes consensus over skepticism. This is a major problem because the framework depends upon theories which span the scales of existence from the subatomic to the super-galactic. At these ends of the spectrum, observation and experimentation become hampered, and uncertainty rises. Skepticism of the framework should be the rule, due to this uncertainty. At these two ends of the scales, inferences are essentially weak analogies to that which we are more confident of on our more immediate scales of existence down here on Earth — even when complex math is involved.
4. The SDM is most susceptible to error when theorists insist upon deducing from just one set of assumptions. Theorists should instead hedge their bets if they are concerned about philosophy of science.
5. With nothing to compare the dominant framework against, widespread ignorance of workable alternatives will naturally lead to the possibly misinformed belief that the framework needs no competitors. People will simply prefer that which they know, and dislike that which they do not. The only way to fix it is for theorists to be forced to investigate alternative frameworks.
6. The SDM does not obligate theorists to culture or incubate fresh frameworks. This process of formulating a new framework is a lengthy process which possesses a creative component to it. And this creative process is sensitive to disruption by vocal conventional theorists.
7. In a scientific world with just one dominant framework, the perceived need to abide by a philosophy in science tends to be inversely proportionate to a person’s belief that the dominant theory is true.
Welcome to the problematic world of the single scientific framework. This one single problem you’ve identified is in fact no panacea for these very heavy philosophical issues.

4

Good thoughts, but I disagree with this one:
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This would significantly increase the already high cost of publishing. As in this case the reviewer wrote 80 some pages. I have had a reviewer write almost 20 pages. Also, it would only be appropriate for the authors response to reviews…which is also written…be published. If you think reading an academic paper is boring and tedious, try reading all the reviews and responses. Sorry, this is just not a good idea.
Just because something is peer reviewed doesn’t mean it is gospel. Reviewers do not QC the data or rerun your analyses or models. Peer review is supposed to catch the real crap. When you read a paper, you still have to decide if the methods, observations, and interpretations were valid.

Theo Goodwin

DocMartyn says:
February 17, 2011 at 4:55 pm
“No. The editor was absolutely correct in picking Steig as a referee. He then had the paper go through two complete rewrites, which Steig attacked and ignored his final recommendations, bringing in a forth unbiased referee.”
Then you will have a standard for conflict of interest that cannot be applied by any observer but only by the journal editor. That is no standard at all. That simply says give the power to the editor.

Don Horne

No one has addressed the retaliation that is might be directed against the reviewer who has rejected a paper after the reviewer’s identity has been divulged. This fear will result in timid reviews if one would even review a paper under such rules. That is why anonymous reviews were used during my 30 year career in biochemical research.

Caleb

The more I think about the failures of this “pal review” system, the more I pity the fellows who, sometimes blithely and naively, went against the flow.
The hurricane expert Bill Gray springs to mind. Back when Gore was Vice President and Hansen was relishing his first experiences of power, Gray again and again asked for funding to research thermohaline circulation, suggesting it might explain the warming side of natural cycles. Again and again funding was denied, and he was told, “Stick with hurricanes, Bill.” Instead of gaining data on real reality, the funding went for Hansen’s virtual reality.
Or think of Zibgniew Jarorowski, who dared suggest the ice core records might not be as “pristine” as everyone suggested. Talk about a fellow who has been pummeled! He has done a lot of hard work, but barely gets mentioned in Wikipedia, (and mostly as being incorrect.) But I guess that’s what hard work gets you, when you dare question the gospel, which is what the ice core records are to some people.

Reynold Stone

Dear Willis,
Thanks for this interesting and valuable article. The points you raise and your suggestions are, of course, worthy of consideration. I agree that Dr. Broccoli’s decision to use Dr. Steig as a reviewer may not have been prudent.
Having said that, however, I wish to state that my own recent experience with Dr. Broccoli over the past 12 months has generally been reasonable. Earlier this week (Feb. 14), I received a letter of acceptance for my critique of a paper used in the IPCC AR4 report titled: “Observed trends in indices of daily temperataure extremes in South America 1960-2000” by Vincent, L.A., T.C. Peterson and 23 other co-authors.
During the lengthy review process, I am convinced that Dr. Broccoli tried to be fair and objective. I had only one concern which I drew to his attention. In the final step of the review process, he allowed the lead author (Dr. Lucie A. Vincent) to enlist the assistance of two co-authors in her Reply. These two co-authors were not among the 25 co-authors of the original paper but, more importantly, I wasn’t given the opportunity to revise my manuscript to take into account any new points they would have introduced in the Reply.
Nevertheless, my experience with Dr. Broccoli was much better than that with an editor at the Journal of Geophysical Research, Atmospheres. The editor rejected my critique of the Peterson et al. Caribbean study (also used in the IPCC AR4 report) in spite of admitting that the points I raised were not unreasonable. My critique did not even make it to the starting line of the peer-review process!
So maybe, we shouldn’t be too hard on Dr. Broccoli who may have now recognized his lapse in judgement by appointing Dr. Steig as a reviewer.
Here is the link to my critique of the Caribbean study by Dr. Tom Peterson and my exchange with a biased editor at the Journal of Geophysical Research, Atmospheres:
http://pielkeclimatesci.wordpress.com/2009/08/05/comment-on-%E2%80%9Crecent-changes-in-climate-extremes-in-the-caribbean-region-by-peterson-et-al-2002%E2%80%9D-by-rj-stone/
Reynold

Andrew Guenthner

I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.

The Man

I have a quibble with the following proposal:
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
I don’t see how this could possibly work. For example, how would such a paper count towards a tenure application? Who would put a paper that was only published as “REJECTED but published for the sake of completeness” on his or her CV? Who would submit to a journal where that might be the result. These are, perhaps, minor issues of academic science but unless you have a comprehensive plan for the complete overhaul of academic appointment, promotion and tenure then this is not solution to the problem of biased and self-serving peer review. No one would use it.

Chris Reeve says:
February 17, 2011 at 6:17 pm
dominates the sciences today largely originated with Socrates:
1: “… In the deductive method, one starts with a presumed law of nature

Unfortunately for Don Scott, this is not how science works. He couldn’t be more wrong.

The Man says:
February 17, 2011 at 7:04 pm
Who would put a paper that was only published as “REJECTED but published for the sake of completeness” on his or her CV?
Perhaps not on the CV, but in own case, I put both accepted and rejected papers on my website http://www.leif.org/research/ together with the review history [at least I’ve started doing this recently].

old44

The whole affair merely serves to highlight how incestuous the AGW community is.

FrankK

This sort of thing is not new you know. The Royal Society in the UK during Newtons time, appointed him as the President (1703-1727). As some may recall there was a dispute about who, Newton or Leibniz in Germany, was the inventor of the Calculus method or as Newton called it Fluxions. It was a very heated dispute with Newton claiming plagiarism on the part of the German’s contribution.
Well the Royal Society decided to do a (peer) review of the dispute to finally determine who could claim the method.. So who do you think was allotted this task, yes Newton himself.! So there you have it of course -Leibniz’s had the last laugh (if he had been alive) because his notation is essentially the one we use today.
History often repeats and repeats ………

wayne

A conflicting reviewer should only be able to comment and inform the other actual reviewers, not become a front-line reviewer himself. He can say, inform, and object all he wants to the real reviewers. Why should a conflict of interest ever be allowed within the peer review system?

BioBob

The corruption of climate science proceeds apace.
Defund it all or prosecute the crooked, or nothing will change.

DocMartyn

|”Jens says:
It seems that the editor Broccoli acted reasonably in the end but the initial choice of referees does not strike me as very wise”
Oh come on here. An editor gets a paper which basically states “This paper, that made the front cover of Science, is complete crap and we show in great detail its crap”
No editor on the planet would not pick the first or last author a review. If you want to go toe to toe with the establishment, expect some bruises. Steig and the team now look like assholes and the story is entering the greater public; don’t throw the baby out with the bathwater. Remember, the system works.
Two true life stories for you to ponder.
Someone wrote a paper attacking a paper I published; they did an experiment in three panels, which appeared to show I was wrong. If I had been a referee, I would have made them perform and show the missing forth panel; the positive control. What they would have shown was that they were observing an interesting, but unimportant artifact. I alas, was not chosen, and the flawed data is running free.
There is a giant in my field, who I know professionally and socially. Nice guy, very smart, hard working and very, very good in his area. Outside his area, he is a bit klusty, when he gets into chemistry, he is a disaster. He gets things published,that should not be, as many referees can’t believe that such a great scientist can be so wrong.
I had to review a paper he submitted, in my area. I was picked as the original referee told the editor that this guy was a personal friend suggested that since I was the first and joint author with her on the most recent papers in this area.
I destroyed the paper, I cited each of my points and sent back a 6 page damnation.
It would be very difficult for me socially, and very dangerously professionally, for the author to know who shot his baby.
He had spent a lot of time on his grand unified theory, but the flaws were obvious to anyone with a strong chemistry/bioenergtics background.
Anonymity protects reviewers form powerful authors.