Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
Spot on as usual Willis.
I agree that sunshine is badly needed in the whole review process – the more the better. It’s the dirty work at the crossroads that’s done in the dark and that’s more than half the problem in that the ‘team’ has been allowed to maneuver without any kind of accoountability.
Willis said- “People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer.”
This potential issue is easy to resolve. Simply make each review a referenced publication that partially counts towards tenure and promotion. After all, it is published. It may not be peer-reviewed, but many/most conference presentations are not peer-reviewed either.
Faculty will line up in droves to review papers, and the quality of reviews will dramatically improve.
All reviews get published with the paper, with each one signed by the responsible reviewer
I agree. There is small problem not addressed: what if the paper is rejected by a conflicted reviewer?
I need to add to my previous comment. Sunshine will only h ighlight problems that we see with published works, but what about good studies that are NOT published due to undue influence. For instance, if Stein had managed to get the article rejected would we even have known about it other than a plaintive or inflamed blog? How do we keep the gatekeepers honest?
Willis, the problem with peer review is that it is what it is….
It’s a method of getting other people to agree with you….
..Most of the time, it’s almost impossible
Brilliant post. I said the same thing at ClimateAudit. This is what stuck out the most for me. Serious conflict of interest. Basically judge, jury and one of the parties all rolled into one. I like your suggestion to publish the reviews. I think it’d be cool to make available the reviews online or something, but to publish only the reviewers final thoughts. This happens in the legal system where there is more than one judge. Both dissenting and supporting opinions are available.
Willis Eschenbach’s points are totally on target. They’ve been made before by others, but the current peer-review system has become a kind of locked-in-cement paradigm that will take time to fix. Fortunately, the web will speed up the inevitable, and it’s likely we’ll eventually arrive at a review system that will closely resemble what Willis and others have proposed. For the record, I’ve reviewed a fair number of papers and books, and I usually tell the editor to leave my name on the review.
listening to: Talking Heads- “Stop Making Sense”
Long ago in my publishing days, Nature and Science were not considered true peer review journals. How things have changed.
“This one we call Crossing the Desert.” *whackwhackwhackwhackwhack* “And this is the Unblinking Eye” *whackwhackwhackwhackwhackwhack*
“Hey did anyone notice that unblinking eye was just like crossing the desert?”
“And now for the paddling of the swollen ass. With paddles!” *whackwhackwhackwhackwhack*
Uh, the paper got published.
Very good Willis, except that haven’t you (inadvertently) got the last sentence backwards, when you say :-
“But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process hidden. Keeping it secret just encourages the current abusive system of pal review.” ?
Leif Svalgaard says:
February 17, 2011 at 4:13 pm
Thanks for pointing that out, Leif and others, I had forgotten to address that.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … where’s the fun in that.
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
I’ll add this to the head post.
w.
Steig was the best possible choice for a reviewer. He knows the material well, he would be motivated to do a thorough review, and his potential conflict of interest is well known, so his review could easily be discounted if necessary.
Most journals use only two reviewers. AMS used four.
If you want an easy review, stick to E&E.
” First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?”
Because the reviewer is not anonymous to the editor who is making the decision!
“The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. ”
No. The editor was absolutely correct in picking Steig as a referee. He then had the paper go through two complete rewrites, which Steig attacked and ignored his final recommendations, bringing in a forth unbiased referee.
The one person who comes out really well in the whole saga is Dr. Anthony Broccoli.
I only wish I had such a clear-sighted and ethical editor to handle my papers.
Hats off to Anthony Broccoli, a true professional.
Typo? Isn’t that the opposite of what you meant to say? Apart from that, I think you’ve got it right.
[Fixed, thanx. ~dbs]
As an engineer I am indoctrinated with the desire of failsafe systems. Life hazard is my mantra. Every single calc I ever carried out was third party checked, and at least one of the checks was “longhand”.
This “peer review” (academic theatrical performance) does not give me a feeling of well being. As a “peer view” suggesting that there might be something in this paper, then OK. But as an “absolute review” upon which to base the recommendation that the whole of mankind change the way they live? Scary, very, very scary and quite frankly, if you give it just a little thought it is absolutely daft!
Silly people! Well-meaning Gaian scientists really have no peers. They are nonpareils!
Willis Eschenbach says:
February 17, 2011 at 4:47 pm
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
Some journals reject more than 90% of papers submitted, so that ‘borehole’ will be enormous and probably have a lot of genuine junk in it. Some journals reject ‘obvious’ junk directly by the editor [or his assistant] before the paper even goes to review. There are the issues of proper formatting and procedures and civility that must be met. So, I’m not so sure your idea is workable unless one steps on the slippery slope of defining categories of rejection.
If a reviewer spends a lot of effort to debunk a junky ‘paper’ [and there are some that ‘are not even wrong’ and can’t be meaningfully debunked] that often is not enough to deter the author or others to give up their silly ideas – we have seen many examples of that on this very blog. The authors might even take solace and even pride from the fact that a serious scientist spent time showing that the paper is junk: “there must be something to my idea since you so vehemently rejects it”. This very blog is replete with such examples too. Whole subculture might grow up around the boreholes.
If this entire incident does not open some eyes to WTF is REALLY going on, then frankly, nothing will. This is beyond “hand in the cookie jar”. This is “elected public official caught naked with a hooker in a closet with a pile of coke on the coffee table….”
It’s THAT obvious….
I suspect that the O’Donnell paper was published in spite of Steig, not because of him as some have alleged.
For a similar example of someone trying to publish a correction to a bigwig professor, see Prof Rick Trebino’s endless hassles here.
But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process hidden. Keeping it secret just encourages the current abusive system of pal review.
Assume you meant keeping the peer review process open rather than hidden.
“The price of Liberty is eternal vigilance.”
How are we to be vigilant if we don’t know the facts, don’t hear all the arguments, and allow things to occur “behind the scenes?”
Yes, even the logic behind not-publishing-a-paper should be part of the public record. And this is especially true if even a penny of public, taxpayer-provided money is involved.
Willis Eschenbach says:
February 17, 2011 at 4:47 pm
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science
======================================================
Lat said: “It’s a method of getting other people to agree with you….”
You only see the papers that the peers agree with, or papers that agree with the peers.
Not only makes it real hard to get anything published that they don’t agree with, but also stops any contrary ideas and new ideas.
But it does make it easier to get funding and keep control over the ‘science’…………
Can anyone give one area where climate science can be said to be really ‘quality science’?
It seems here that peer review is continually being ‘used’ to block dissent.
As the subject of this paper showed and as shown repeatedly against others, climate ‘scientists’ are poor to undergraduate level at statistics.
The use of botanical proxies seems to show a lack of knowledge in botany – taking width of tree rings to equate to only temperature for example.
The loss of original data and unwillingness to publish data and methods.
Continual alteration of original data without any record keeping or publication of _why_ records decades old need to be altered and no authoritative sign off of the changes.
Fudge factors often repeatedly added to observations without any record or reasoning why and why that amount of change.
Climate historical monitoring carried out at weather observation sites that are laughably poor quality with no record keeping of the status of the site.
Invention of data based on the laughably poor quality to sites 1500km distant in a way that even amateur meteorologists would balk at.
Is there not one area of science or scientific method where climate science exhibits professionalism and quality standards? One area where people can say I am proud of the quality of climate science?
And yet the world economy is being driven into the ground by the prognostications of these ungifted amateurs?
This may be rehashing what you and others have already said, but in politics the appearance of impropriety is as bad as actual impropriety and is to be avoided at all costs. That Broccoli thought he (and science in general for that matter) was immune to this axiom indicates he thinks that not enough people would catch on to what was being done. People like this (in all spheres) count on that very thing to continue being a successful repeat offender. The sad truth is it works without watchdogs keeping a vigilant eye on the perpetrators and sounding the alarm – precisely why sites like this are essential today, in an age when the scoundrels are bolder than ever.
“2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent…”
I don’t think Dr. Broccoli ever intended to allow Steig to act as a gatekeeper, merely as an informed commenter. Perhaps Dr. B even hoped for a meeting of the minds to some degree. When it became clear that Strig was pushing way too hard for O2010 to agree with Stieg 09, Dr. B may have decided that further delay would constitute gatekeeping, and pulled the rug from under Stig.
Given the outcome, Dr. B may have done everyone a huge favor by making it abundantly clear that Climatology is no place for anachronistic peer-review, any more than treating gangrene with a physic.
Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
Not a bad idea. Often an author is building on their previous work so you can tell who they are. If a paper contradicts the reviewer’s own work s/he is bound to notice this even if they don’t know who the authors are. But, it is still not a bad idea.
All reviews get published with the paper, with each one signed by the responsible reviewer.
This totally conflicts with your first reform! And it makes little sense. Most of the reviews I’ve written consist of corrections of minor errors. Who would want to read that? If the author wants to post the reviews somewhere s/he can do do.
Perhaps journals could have a formal appeal process where a rejected author could ask for a different editor for a second opinion. But, there are many journals, so it is easy to just submit the paper elsewhere. In some reviews I have recommended rejection but also suggested another journal where the paper would be a better fit.
I don’t see a problem with Steig being one of the reviewers since there were others. It is up to the editor to assess the reviews. That was done in this case and an improved version of the paper was published.
The biggest problem with peer review is that too many reviewers spend little time on the paper and write a brief positive recommendation. This does not help the editor. The other problem is reviewers who take forever to get a round to doing the review. Also very little is done to teach new scientists how to write a review.
I agree with most of this post, however I have a small concern with the issue of the anonymity of reviewers. The knowledge that names will be put to review comments could lead to some reviewers limiting their criticism of papers from those seen as powerful or influential in a given scientific field. It also gives the media an opening to fan conflict and profit from it.
The human primate is a social animal, and our basic social structure is tribal and hierarchical. To overcome the limitations of this in research and analysis we need systems that circumvent normal primate behaviour. The anonymous review has some advantages in this area. Even publishing review comments without the authors name would remove anonymity in scientific circles.
Do the positives of preventing conflict of interest and gate keeping out weigh the risk of social pressure on the named reviewers?
I agree with Leif Svalgaard’s latest comment. We must not lose sight of the main function of the refereeing system, to filter the signal from noise. Even with this filtering the literature is enormous. And when the system works, the refereeing is a constructive process producing better papers. The only defense against gatekeeping from established scientists is an appeal to the editors, so the integrity of editors is important. It is usually not difficult to spot a hostile review. It seems that the editor Broccoli acted reasonably in the end but the initial choice of referees does not strike me as very wise. The open discussions on the internet of controversial areas of scince is a great supplement to the peer reviewed literature and often very entertaining to follow but it does not provide a good record of progress. For this reason it is important that valuable contributions like the O’Donnell paper be published. But to publish also the discussion with the referees would add far too much ‘noise’.
Willis’ diagnosis of the problem is excellent. I cannot imagine someone disagreeing with that diagnosis. It is also a first-rate indictment of Steig, the editor, and the journal. I cannot imagine anyone disagreeing with that indictment. Clearly, the peer review process has been hijacked and serious action must be taken by parties who control the purse strings. The result of all this should be a Congressional investigation of this affair and the Climategate affair. Short of a serious investigation undertaken by critics of this kind of behavior, there is no way to restore science to climate science.
Editor Broccoli understood the conflict of interest in this case. I hope (and believe) that he took that conflict into account when considering Eric Steig’s reviews. I do not personally see anything wrong with having a potentially conflicted reviewer… so long as journal rules on that are clear. Perhaps specific guidelines like “Editors will always be mindful of potential conflicts of interest, and will discount any review by a conflicted reviewer if there is any indication that reviewer is not acting 100% in good faith.”
@Green Sand:
Peer review is not and was never considered a fail-safe process. It is a paper, not a product that could hurt someone, that we are talking about. Peer review is a minor step at quality control so that papers that are poorly done or irrelevant get weeded out or that at least some errors get fixed. It is not a test of truth. Even a well run experiment could produce data that points the wrong way just by chance. We want odd ball ideas and uncharacteristic results to get published. The next step is for other scientists (and now some lay people) to go over the paper during the course of their work. Some will publish review articles comparing, contrasting and synthesizing several papers. Eventually review panels review the literature in the field and, if possible, form a consensus. The consensus might be “we do not know” or “the uncertainties are very large,” as was the case with climate studies in the 70’s and 80’s. But in many cases the truth slowly emerges. This happened with climate science in the 90’s. Yet we still have done little in response.
When popular science publications report on what the latest peer reviewed findings suggest you should never take it as gospel. They use the word “suggests” for a reason. When the National Academy of Sciences says they have come to a consensus that human caused climate change is real and very dangerous you should take it very seriously.
As an example, several credible papers are suggesting that some extreme weather events are linked to AGW. But many researchers are not yet convinced. Review panels have not yet gone over this area. That will probably take a few more years. So, I am concerned when I read about possible connections between extreme weather and AGW, but I will wait and see what comes out of it before I form a solid opinion on this.
Willis,
Thoughtful post. Thanks. You stimulated some thoughts, as you are often wont to
do. : )
First, you mention “. . . the ancient tradition of adversarial science”. I suggest that a more comprehensive way to characterize the ancient scientific tradition would be to call it open and unrepressed (free) argumentation. The perception of science as a process primarily involving adversaries does not encompass the broader spirit of the scientific process. Sure, some scientists may be primarily adversarial, but that is not even close to the broader perspective involving all the interactions/relationships of scientists.
Also, another thought is that scientific journals do not enjoy a monopoly in the area of scientific communication. They have just a share in the scientific communication market. Like laissez faire processes in economics, so laissez faire in the open market of scientific ideas allows for a continuous balancing of various competing communication processes for keeping open the free market place of scientific ideas. If the entire climate journal field was to become 100% biased toward a politically correct scientific theory, still, scientific progress is not stopped. For example this post of yours is evidence of what I am saying; what you are doing is just one example of many alternate communication processes. Science can self-correct independent of scientific journals. Indeed, climate science has been self-correcting widely; without the help of science journals (rather in spite of science journals).
Still another thought, if science is essentially applied reason then it is only the reasoning and the evidence supporting it that is the body of science. The scientific body is not essentially tied to the comings and goings of science journals or any of their arbitrarily constructed and constantly shifting (politically correct fashions of the moment) processes. It may be practical and efficient to have the science journals be more logical and have more intellectual integrity, but science journals have no essential bearing on the science itself.
John
Re: “That’s what I see as broken about the system, and how I would fix it”
Unfortunately, this is a fix to just one tiny aspect of a much larger set of very profound problems in our system of science today.
This system of peer review and consensus which dominates the sciences today largely originated with Socrates:
1: “… In the deductive method, one starts with a presumed law of nature – an obviously correct (accepted) generalization about the way things work – and deduces (works out, derives) its logical consequences.” (Don Scott, The Electric Sky)
2: “A hypothesis arrived at via this deductive method is promoted to the status of being a theory when and if a large enough body of experts accepts it. This is an application of the Socratic method, also sometimes called the `dialectic method.’ Socrates (469–399 B.C.) believed that truth was discovered through intense conversations with other informed people. In this method, a vote of the experts determines when and if a theory is correct.” (Don Scott, TES)
3: “Once such a theory has been accepted, it is not easily rejected in light of conflicting evidence.” (Don Scott, TES)
4: “It is, however, often modified – made more complicated. When over time a theory becomes officially accepted, the essence of the matter has been settled and fixed. Modifications to the fine points of the theory can then be proposed and debated, but the backbone structure of the theory is set. That framework has already been firmly established.” (Don Scott, TES)
This Socratic-dialectic methodology (SDM) to investigating Nature exhibits numerous weaknesses:
1. If the obviously correct basic starting-point presumption is in fact wrong, then we’ve possibly assumed the wrong framework.
2. The SDM method tends to generate cosmologies which lack any physical support, as the process of deduction favors those which can extend from the conjecture’s consequences. This is fundamentally a sequential process of reasoning.
3. The SDM prioritizes consensus over skepticism. This is a major problem because the framework depends upon theories which span the scales of existence from the subatomic to the super-galactic. At these ends of the spectrum, observation and experimentation become hampered, and uncertainty rises. Skepticism of the framework should be the rule, due to this uncertainty. At these two ends of the scales, inferences are essentially weak analogies to that which we are more confident of on our more immediate scales of existence down here on Earth — even when complex math is involved.
4. The SDM is most susceptible to error when theorists insist upon deducing from just one set of assumptions. Theorists should instead hedge their bets if they are concerned about philosophy of science.
5. With nothing to compare the dominant framework against, widespread ignorance of workable alternatives will naturally lead to the possibly misinformed belief that the framework needs no competitors. People will simply prefer that which they know, and dislike that which they do not. The only way to fix it is for theorists to be forced to investigate alternative frameworks.
6. The SDM does not obligate theorists to culture or incubate fresh frameworks. This process of formulating a new framework is a lengthy process which possesses a creative component to it. And this creative process is sensitive to disruption by vocal conventional theorists.
7. In a scientific world with just one dominant framework, the perceived need to abide by a philosophy in science tends to be inversely proportionate to a person’s belief that the dominant theory is true.
Welcome to the problematic world of the single scientific framework. This one single problem you’ve identified is in fact no panacea for these very heavy philosophical issues.
Good thoughts, but I disagree with this one:
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This would significantly increase the already high cost of publishing. As in this case the reviewer wrote 80 some pages. I have had a reviewer write almost 20 pages. Also, it would only be appropriate for the authors response to reviews…which is also written…be published. If you think reading an academic paper is boring and tedious, try reading all the reviews and responses. Sorry, this is just not a good idea.
Just because something is peer reviewed doesn’t mean it is gospel. Reviewers do not QC the data or rerun your analyses or models. Peer review is supposed to catch the real crap. When you read a paper, you still have to decide if the methods, observations, and interpretations were valid.
DocMartyn says:
February 17, 2011 at 4:55 pm
“No. The editor was absolutely correct in picking Steig as a referee. He then had the paper go through two complete rewrites, which Steig attacked and ignored his final recommendations, bringing in a forth unbiased referee.”
Then you will have a standard for conflict of interest that cannot be applied by any observer but only by the journal editor. That is no standard at all. That simply says give the power to the editor.
No one has addressed the retaliation that is might be directed against the reviewer who has rejected a paper after the reviewer’s identity has been divulged. This fear will result in timid reviews if one would even review a paper under such rules. That is why anonymous reviews were used during my 30 year career in biochemical research.
The more I think about the failures of this “pal review” system, the more I pity the fellows who, sometimes blithely and naively, went against the flow.
The hurricane expert Bill Gray springs to mind. Back when Gore was Vice President and Hansen was relishing his first experiences of power, Gray again and again asked for funding to research thermohaline circulation, suggesting it might explain the warming side of natural cycles. Again and again funding was denied, and he was told, “Stick with hurricanes, Bill.” Instead of gaining data on real reality, the funding went for Hansen’s virtual reality.
Or think of Zibgniew Jarorowski, who dared suggest the ice core records might not be as “pristine” as everyone suggested. Talk about a fellow who has been pummeled! He has done a lot of hard work, but barely gets mentioned in Wikipedia, (and mostly as being incorrect.) But I guess that’s what hard work gets you, when you dare question the gospel, which is what the ice core records are to some people.
Dear Willis,
Thanks for this interesting and valuable article. The points you raise and your suggestions are, of course, worthy of consideration. I agree that Dr. Broccoli’s decision to use Dr. Steig as a reviewer may not have been prudent.
Having said that, however, I wish to state that my own recent experience with Dr. Broccoli over the past 12 months has generally been reasonable. Earlier this week (Feb. 14), I received a letter of acceptance for my critique of a paper used in the IPCC AR4 report titled: “Observed trends in indices of daily temperataure extremes in South America 1960-2000” by Vincent, L.A., T.C. Peterson and 23 other co-authors.
During the lengthy review process, I am convinced that Dr. Broccoli tried to be fair and objective. I had only one concern which I drew to his attention. In the final step of the review process, he allowed the lead author (Dr. Lucie A. Vincent) to enlist the assistance of two co-authors in her Reply. These two co-authors were not among the 25 co-authors of the original paper but, more importantly, I wasn’t given the opportunity to revise my manuscript to take into account any new points they would have introduced in the Reply.
Nevertheless, my experience with Dr. Broccoli was much better than that with an editor at the Journal of Geophysical Research, Atmospheres. The editor rejected my critique of the Peterson et al. Caribbean study (also used in the IPCC AR4 report) in spite of admitting that the points I raised were not unreasonable. My critique did not even make it to the starting line of the peer-review process!
So maybe, we shouldn’t be too hard on Dr. Broccoli who may have now recognized his lapse in judgement by appointing Dr. Steig as a reviewer.
Here is the link to my critique of the Caribbean study by Dr. Tom Peterson and my exchange with a biased editor at the Journal of Geophysical Research, Atmospheres:
http://pielkeclimatesci.wordpress.com/2009/08/05/comment-on-%E2%80%9Crecent-changes-in-climate-extremes-in-the-caribbean-region-by-peterson-et-al-2002%E2%80%9D-by-rj-stone/
Reynold
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
I have a quibble with the following proposal:
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
I don’t see how this could possibly work. For example, how would such a paper count towards a tenure application? Who would put a paper that was only published as “REJECTED but published for the sake of completeness” on his or her CV? Who would submit to a journal where that might be the result. These are, perhaps, minor issues of academic science but unless you have a comprehensive plan for the complete overhaul of academic appointment, promotion and tenure then this is not solution to the problem of biased and self-serving peer review. No one would use it.
Chris Reeve says:
February 17, 2011 at 6:17 pm
dominates the sciences today largely originated with Socrates:
1: “… In the deductive method, one starts with a presumed law of nature
Unfortunately for Don Scott, this is not how science works. He couldn’t be more wrong.
The Man says:
February 17, 2011 at 7:04 pm
Who would put a paper that was only published as “REJECTED but published for the sake of completeness” on his or her CV?
Perhaps not on the CV, but in own case, I put both accepted and rejected papers on my website http://www.leif.org/research/ together with the review history [at least I’ve started doing this recently].
The whole affair merely serves to highlight how incestuous the AGW community is.
This sort of thing is not new you know. The Royal Society in the UK during Newtons time, appointed him as the President (1703-1727). As some may recall there was a dispute about who, Newton or Leibniz in Germany, was the inventor of the Calculus method or as Newton called it Fluxions. It was a very heated dispute with Newton claiming plagiarism on the part of the German’s contribution.
Well the Royal Society decided to do a (peer) review of the dispute to finally determine who could claim the method.. So who do you think was allotted this task, yes Newton himself.! So there you have it of course -Leibniz’s had the last laugh (if he had been alive) because his notation is essentially the one we use today.
History often repeats and repeats ………
A conflicting reviewer should only be able to comment and inform the other actual reviewers, not become a front-line reviewer himself. He can say, inform, and object all he wants to the real reviewers. Why should a conflict of interest ever be allowed within the peer review system?
The corruption of climate science proceeds apace.
Defund it all or prosecute the crooked, or nothing will change.
|”Jens says:
It seems that the editor Broccoli acted reasonably in the end but the initial choice of referees does not strike me as very wise”
Oh come on here. An editor gets a paper which basically states “This paper, that made the front cover of Science, is complete crap and we show in great detail its crap”
No editor on the planet would not pick the first or last author a review. If you want to go toe to toe with the establishment, expect some bruises. Steig and the team now look like assholes and the story is entering the greater public; don’t throw the baby out with the bathwater. Remember, the system works.
Two true life stories for you to ponder.
Someone wrote a paper attacking a paper I published; they did an experiment in three panels, which appeared to show I was wrong. If I had been a referee, I would have made them perform and show the missing forth panel; the positive control. What they would have shown was that they were observing an interesting, but unimportant artifact. I alas, was not chosen, and the flawed data is running free.
There is a giant in my field, who I know professionally and socially. Nice guy, very smart, hard working and very, very good in his area. Outside his area, he is a bit klusty, when he gets into chemistry, he is a disaster. He gets things published,that should not be, as many referees can’t believe that such a great scientist can be so wrong.
I had to review a paper he submitted, in my area. I was picked as the original referee told the editor that this guy was a personal friend suggested that since I was the first and joint author with her on the most recent papers in this area.
I destroyed the paper, I cited each of my points and sent back a 6 page damnation.
It would be very difficult for me socially, and very dangerously professionally, for the author to know who shot his baby.
He had spent a lot of time on his grand unified theory, but the flaws were obvious to anyone with a strong chemistry/bioenergtics background.
Anonymity protects reviewers form powerful authors.
Did it again Willis – Nailed it!
Too bad the JoC (joke?) doesn’t get it.
[snip . . off topic]
I have to say I’m with Theo Goodwin on this. I’d cut the good Dr. more slack if it weren’t for the fact that his jounal’s own standards insist “…A reviewer should be sensitive even to the appearance of a conflict of interest…” Dr. Broccoli was rather more like the boy who delights in placing ants from different colonies in the same container and observing the ensuing mayhem.
For the benefit of the public record, how many hours of work did O’Donnell spend on replies, modifications, and rebuttals to Steig’s written objections? How many of those hours contributed to making a better paper?
In the age of the internet, all review documents with the editor in the distribution list should be published on line. They need to be part of the historical record. Of course, once on line, the identities of the reviewers must be published, too.
A fast index of papers, abstracts, 100-word review summaries, maybe even a grade A through F attached to the Paper AND cross linked to the reviewer would quickly be benefical.
Wouldn’t have been interesting to see who might have recommended rejection of Einsteins three short papers in 1905? I heard one story several years ago that later in his life, Einstein refereed a paper that went into higher dimensionality than his. models; he delayed by years asking for more work and changes. Einstein, through delay, ruined the best productive years of that little known mathematician by keeping him and his ideas unknown to most in the field. Lengthy delays to publication just are not right for it robs people of time, a resource most difficult to recover.
Great thread Willis.
I believe technology is already driving a change in publishing and peer review. Some online journals are already publishing reviewers comments with their names as well as the authors responses to these reviews. I do not agree with comments that people are less likely to review papers if their names are published — if someone cannot stand by his or her comments they should not be a reviewer.
I can see it all going abit further in the near future in that because of the ease of online searches it does not matter to the author in which journal the paper gets published . So the prestige will not be from the Journals name or brand but it will from the calibre of reviewer it can attract to comment on the paper.
I also agree with Leif’s comment that publication of rejected papers does not serve any useful purpose to anyone.
>> Mike says:
February 17, 2011 at 6:15 pm
When the National Academy of Sciences says they have come to a consensus that human caused climate change is real and very dangerous you should take it very seriously. <<
Could you please post a link to when a vote of the membership was taken, and the results of that vote?
I disagree that reviewers should be named, for the simple reason that then it will be much more difficult to get reviewers. Of course the editor should not have used Steig as a reviewer, and the journal should acknowledge this and review & fix the process.
“in the ancient tradition of adversarial science”
No, there is no such tradition. You are misapprehending the spirit of inquiry and projecting blog science or “auditing” or what have you onto science, which has always been based on trust and collegiality. That’s why the penalty for fraud is professionally fatal.
I too have a problem with the following proposal:
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In order for this to succeed, there would have to be a mechanism for a reasonable period of time to pass before a rejected paper would be “published” online. An otherwise excellent paper might be a poor fit with one journal, yet be an excellent fit for another. To kill off such a paper by “publishing” it too early could be a real loss of an important result.
Facetious thought: maybe all papers should include a list of the journals that rejected them. Of course this might create a game of seeing just how many quality journals could be coerced into rejecting a given paper before it is accepted.
There is always someone wanting to game the system.
CCR
Willis Eschenbach says:
February 17, 2011 at 4:47 pm
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
and Leif replies to it that most of the papers are rejected etc. I agree with him.
In addition most people whose paper is rejected by one publication will apply to another one, so there is no meaning in having a reject blog on each publication, for obvious reasons: each publication keeps a copyright if it decides to publish.
I think slowly the internet with the archive sites is evolving towards having a record of all works sent to publication, so this problem will be solved eventually.
The suggestion of publishing the reviews in parallel is good.
Willis, you said:
“Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.”
—-
Absolutely brilliant, sir! The Internet is proving to be an incredibly valuable form of peer-review, as we see on this blog. If this process can be worked into a consistent process, science will benefit tremendously.
DocMartyn says:
Anonymity protects reviewers form powerful authors.
I had not thought of this. I can also now see a pecking order created if reviewers are unmasked. Maybe just the reviews should be published, with the identity known only to the editors. Maybe it would be enough that malicious reviewing would be public and the public would respond with letters to the editor castigating biased or wrong reviews.
Too bad scientists no longer need a reputation.
Ehrlic and Holdren, the science Czar, included.
Apparently a pHd can still say anything, hide the data, and fudge the findings.
Wow
Great posting… and great comments…
The term Peer Review seems to be intrinsically linked to the concepts of Learned Journals of Record, Group Think, Self Interest and CYA – Cover Your Arse… while you suggestions would significantly improve the situation they would not eradicate the problem… it is the age old problem of who polices the police force… just take a look at Wikipedia to see what happens to the truth when the thought police are given control.
However, there is room to be optimistic in the internet age… the review and comment process is alive and well on the internet – WUWT is a classic example… the open forum is alive and well on the internet – the Electric Universe web sites are a classic example… and the other positive development is that the Learned Journals that hide behind paywalls are no longer Journals of Record be definition because they are inaccessible and are withering on the vine.
So the internet is the embodiment of the phase Publish and be damned provided that we have an open internet, ethical search engines and ethical web sites… and it is in these areas that we need to be especially vigilant… the concept of an open internet is increasingly becoming an illusion… there are very few ethical web sites that are open for all comments and many websites have a habit of rewriting history… the concept of an ethical search engine is already commercially and politically compromised… so be vigilant and at times… always look for multiples sources of information… and always check the stated facts.
PS
I am pleased to say WUWT is on my personal ethical web site list… its not perfect… but very good… Thank you Mr Watts.
The best science is an art, in the sense that any great, breakthrough achievements are not properly understood by the contemporary button sorters and bottle washers professionally occupied in the same field.
This is the reason why Grigoriy Perelman rejected two most prestigious (and lucrative) awards in mathematics. “These people are not qualified to give me awards,” — he said to the Russian reporter who managed to make Grigoriy talk a little, because he joined him in the forest for mushroom hunting.
Consider: while there was no “peer review” in music, we had Beethoven or Chopin every 15 years or so. As classical music was quickly becoming a government-sponsored, institutionalized national sport in the beginning of the 20th century, real talent moved to unregulated and non-sponsored fields of jazz and pop music. Now these genres are also being institutionalized — and, therefore, killed off.
Same happened to all arts in the 20th century. Same happens now to the science.
A true genius has no peers. Posterity is his judge.
Given the closeness of the Pal review system and the team, I have doubts as to whether the double blind system would be truly blind.
One question that arises out of this affair is how the original Steig paper got through the review process? Why didn’t someone spot the obvious flaws in that paper? The failure to spot these flaws shows that the review system does not ensure quality or robustness.
Does anyone know who reviewed the original Steig paper. Was this an example of review by the Team and pal review? Can anyone find out who the reviewers were?
The hurricane expert Bill Gray springs to mind. Back when Gore was Vice President and Hansen was relishing his first experiences of power, Gray again and again asked for funding to research thermohaline circulation, suggesting it might explain the warming side of natural cycles. Again and again funding was denied, and he was told, “Stick with hurricanes, Bill.” Instead of gaining data on real reality, the funding went for Hansen’s virtual reality.
When a military contractor holds a contract review with military personnel present and the contractor provides lunch, the contractor must put out a”basket” to collect contributions from the military personnel. The military personnel must “pay” for the lunch in order to avoid the “apperance of conflict”. Failure to comply with this requirement might lead to legal action against the contractor, the military officer, or both! As a former military officer, I find the arguments that Dr. Steig had no conflict of interest in being an anonymous reviewer of the O’Donnell paper simply foolish. That the editor of the journal is defended by others indicates how far the scientific community has drifted from the pursuit of truth. The editor essentially allowed Dr. Steig to make a rebuttal IN PRIVATE, leaving open the possibility that we, the public, would never see either argument. This is the antithesis of science.
??? moderators??? Was there something offensive in my previous post? I note that others on the thread have voiced similar concerns. Was it the use of the term “Human primates”? I understand that this is an American site, and that some may find this controversial. I would not be overly concerned if this were the reason for moderation, however a [snip] with a state reason would be helpful 🙂
[Reply: Your comment was in the spam folder. It’s been rescued and posted. ~dbs, mod.]
Beyond excellent, Willis, this is central to the Steig/O’Donnell controversy. AMI editors have a DUTY to avoid conflict of interests as stated in AMI policy. (John Nielsan Gallimon failed to address this issue in the discussion,17/02/11, on Climate Audit.) Tear down the walls !
Peer Review, Pal Review, and Broccoli Posted on February 17, 2011 by Willis Eschenbach
This was a most excellent analysis and summary including the posters, thank you Willis. And how slow the responses were. Perhaps so many scientists have moved on to other areas and no longer contribute- they are either dead or silenced, vowed to never return to their work and the broader ethic as they understood of being a scientist.
While I think the cartoon prescient, I am however reminded of Solzhenitzen, The Gulag Archipelago in the Preface
‘”…but by an unexpected turn of history, a bit of the truth, an insignificant part of the whole, was allowed out in the open. But those same hands which once screwed tight our handcuffs now hold out their hands in reconciliation.”
“No, don’t! Don’t dig up the past. Dwell on the past and you’ll lose an eye.”
But the proverb goes on to say ‘Forget the past and you’ll lose both eyes.’
Peer review accounts for a biased sample: of those having consensus; that lied/manipulated the truth [MattN says: February 17, 2011 at 5:10 pm]; or hopped on the band wagon at latter stages, to contribute their well (or not) -founded knowledge. This is not science. It is peer review and marketing at its pinnacle.
Science and Freedom has no place there.
Dr. Broccoli is a climate modeller…
‘Nuff said.
Konrad said :
Even publishing review comments without the authors name would remove anonymity in scientific circles.
Do the positives of preventing conflict of interest and gate keeping out weigh the risk of social pressure on the named reviewers?
I expressed a similar worry above, if the reviewers names are published. But if only the format “reviewer A,B,C” is kept the recognition of the identity by the cognoscenti is not bad: it will keep the heads of the field from being too arrogant and it will shield the less well known ones from vindictiveness. If the field is too narrow in any case the reviewers will be guessed. Publication will keep them on the straight and narrow, certainly in scientific content..
I forgot – another damn fine post, Willis!
I’m really fed up by the way both sides of this debate have thrown around the term “peer review” as if it were in any way meaningful. So I’ll recount a brief personal anecdote.
A few decades ago I was the lead author of an interesting if not exactly earth-shaking paper on the molecular pathology of mammalian tumours. That study involved the administration of substance LMN to tumour-bearing rats and rabbits. LMN had originally been prepared by an obscure enzymology genius – ABC – working in the top secret UK Porton Down military laboratories, by a tedious and difficult biosynthetic process involving fractionation and purification of rabbit urine. His method was published in ABC1960 (or thereabouts).
LMN is an interesting molecule with potential anti-tumour properties and – praise the Lord – ABC subsequently published an enormously improved and totally synthetic catalytic preparation in ABC1970 (or thereabouts).
We prepared LMN by the ABC1970 method with excellent high purity yields and got very interesting results from our poor bunnies and rats. So we submitted what we thought was a pretty good paper to one of the foremost experimental pathology journals of the day, in 1978.
It was rejected out of hand. The (sole) peer reviewer opined that our preparation of LMN (done by the method of ABC1970) was rubbish since “it is well known the only way to produce LMN is by the method of ABC1960”. There was apparently no way of appealing this judgement by our anonymous, lazy, ignorant and possibly senile “peer”. Let’s call him “FKWT”
Anyway, in disgust I absconded from academia to private practice and have published nothing since. A decision, I might add, I have never regretted.
So, please don’t use “Peer Review” as a “gotcha”. It’s definitely not.
Cheers.
Very good post in times where: politics + (bad-mediocre)science= religion.
To misquote Churchill
“Anonymous peer review is the worst method for evaluating journal articles,
except for all the rest!”.
The article was published, Steig and/or coauthor was given room to
write a 70 paper critique with which he (i.e. Stieig) essentially hanged
himself. Basically, the system worked.
It is not uncommon for authors being criticized to have input into a review.
What should not happen is that their opinion becomes the sole determining
factor. This did not occur in the case of the O’Donnell paper.
“Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.”
I disagree with this statement. Didn’t Steve McIntyre and Ross McKitrick have a paper, rejected by Nature, only to be accepted by another journal. If the paper was published on the web no other journal would accept it.
I fully agree with the rest of Willis’s ideas. Peer review in “Climate Science” is a joke.
What I can’t understand is why Anthony Broccoli gave a paper consisting entirely of statistics to reviewer A, who describes himself as NOT being a statistician and also reviewer B who wanted who wanted to refer the paper to Steig and/or Mann. Obviously reviewer B was not worried about the conflict of views either.
Dear Willis,
I think you should distinguisch two kinds of conflict of interests; adverserial and non-adverserial.
The former is in most cases not appropriate, as you stated, but the latter is worse. I feel that the second type is is the one referred to in most COI policies (and mostly practised by the Team).
rgds
I think the post is very good but I think the issue with peer review is slightly wider than what is being discussed. If we take a paper like Steig et al, this now forms part of the peer reviewed literature. There are half a dozen authors on the paper and it was probably reviewed by perhaps 3 reviewers and the editor. So maybe 10 people looked at Steig et al in all. Reiewers are generally checking for gross errors in method or logic and simply accept a lot of stuff without checking, or only check in detail something which is their pet subject. In my experience, many reviewers and editors give a pass to the heavy weight maths stuff – it is difficult to wade through and many are not competent to check it. Reviewers B and C for O’Donnell did just that. Steig later admitted he is not a statistician.
The problem with peer review is that it is ok as a basic check on papers before appearing in a journal but it is no basis for public policy. The problem with the AGW argument and the IPCC is that “peer reviewed” paper has been used to pretend this means tested and verified science. This is exactly what happened with MBH98 and the Hockey Stick. A peer reviewed paper is no such thing, it is just a first pass. The second problem with peer review in AGW science is that pointed out by Wegman in the investigation into the Hockey Stick – the number of people publishing in the field is very tiny. For example 3 of the authors of Steig et al are in the Wegman network diagram for the “Hockey Stick” (including Mann).
To me, O’Donnell highlighted the problem with Steig et al far more effectively in the blog response at ClimateAudit than in the O’Donnell et al paper. When a perturbation is added to the stations, the Steig et al model gives absurd results in the mapping. Eg adding +0.20 trend to the peninsula gives alarming warming in West Antarctica whereas adding +0.40 trend to West Antartica gives almost no warming…in West Antarctica. If the Steig paper had been any good these tests should have already been in the paper. Did Steig et al perform basic tests like this? If they did, the paper would never have been published. If they didn’t, why did the reviewers of Steig et al not ask for them? That is the failure of peer review and why it should not be used to inform public policy. A peer reviewed paper has been used as a substitute for validated science in a subject area where relatively little is known and verification is not possible and so arm waving at inconvenient facts can get you a very long way.
I suppose we could do a radical thing and actually test Willis’ idea.
BTW, good to see you tonight Willis and thx for the mention
richard telford
The best reviewers for the ODonnell paper were probably the same reviewers who passed Steig’s effort through.
A further point to make about peer review, credible science and public policy is that one reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram.
It seems to me the “Pal Review” system in climate science has replaced “Peer Review” gradually over a period of 20 years or more. “Pal Review” seems to be sufficiently well embedded that editors can’t even recognise the significance of what they and the chosen reviewers are doing to science. Many of the current editors began publishing thier own work under the current “Pal” system and even actively encourage it. So the significance of climategate just slips right on by. They just don’t seem to have gotten the alert.
For some reason skeptics, luke-warmers and climate realists seem to struggle to understand how, in a post-climategate world, it has been possible for the O’Donnell – Steig affair to have developed. Why are we so surprised by the antics of Broccoli and Steig? The alarmists still think they hold the moral high ground. To date, with one or two notable exceptions (e.g. Judith Curry), none of the alarmist set has voiced any concern. Personally I don’t see this changing any time soon. Realistically I see the current practices dying a very slow death, if at all. We can expect more of the same, though it may be done more subtly from now on in one particular journal.
Richard Telford says:
February 17, 2011 at 4:52 pm
Steig has some other attributes you seem to have neglected, Richard. In addition to those listed above, he would also be highly motivated to see if he could keep the paper from being published. And if he could not get it thrown out, he would be highly motivated to see if he could push the results in the direction of his results. Both of these are visible in his response.
In addition, Steig himself said that he is not a statistician … and the debate is about statistics. Under what rubric are you calling him an expert on the question? My dear Richard, as the O’Donnell paper clearly showed Steig is the one who didn’t understand the procedure he was using … and you want to pick him to be the expert on that procedure? Yeah, that’s the ticket … I don’t think I’ll let you pick a doctor if I have trouble …
Are there many people out there so naive as to think that Steig is anything other than horribly conflicted? Richard, why do you think that the journals have policies against conflicts of interest? Just for fun?
It is because they recognize that scientists are humans who can be counted upon, if there is a conflict, to push for their own side. Even with the best of motives and intentions.
w.
DocMartyn says:
February 17, 2011 at 4:55 pm
So Broccoli comes out great for breaking journal policies by employing a conflicted reviewer, by siding with Steig and rejecting the Chadni patterns, and finally by throwing out Steig’s work and ” ignored his final recommendations, bringing in a forth unbiased referee”.
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
How about if he just started with an unbiased referee, and avoided all of the nonsense? Then I’d give him some stars. But as it is? I find that he is the one who acted most unethically in the whole thing.
For you to think that he acted ethically is a huge surprise to me, and is contrary to the Journal policy … but hey, that’s all OK, we’re in the bizarro world of climate science …
w.
The answer to this problem is simple enough.
Reviewers remain anonymous until the MS is accepted or rejected.
If accepted, the published version contains, at its foot: “Reviewed by Groucho, Harpo & Chico”.
If rejected, the lead author is notified in similar vein: “Your work was reviewed by Groucho, Harpo & Chico”.
In neither case is any reviewer identified with any of the three reviews – we just know each of the three is due to one of the named people.
Appending reviewers’ identities to the end of a paper would discourage powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be questioned as to their diligence in waving it by.
Telling the lead author who is responsible for reviewing their (rejected) paper would probably avoid a lot of conspiracy theories as authors speculate as to who has bounced their work. And would make editors try hard to find neutral reviewers.
Finally, I would also append to the published paper the name of the sub-editor of the journal who was responsible for editing that particular MS. Obviously the authors already know this but the community at large do not. Editors can have conflicts of interest or allegiances too.
Jens says:
February 17, 2011 at 6:08 pm
I don’t see that posting reviews on the internet increases the noise. You don’t have to read the reviews, just read the Journal article if you wish.
Wrong, wrong, wrong. We have proven quite clearly that in climate science, we cannot depend on the integrity of the Editors. We need a system where we can see what the Editor is doing, because at this point, I don’t trust them one iota.
That’s like saying that papers shouldn’t include online supplements because it makes the paper too big. If you don’t want to read the reviews online, then don’t read them. How is the noise increased?
w.
Leif Svalgaard says: February 17, 2011 at 5:01 pm
…. that often is not enough to deter the author or others to give up their silly ideas – we have seen many examples of that on this very blog.
Here is an example that the good doc S. has in mind:
http://www.vukcevic.talktalk.net/LFC2.htm
Even if it was accepted by a respectable paper it would have been long forgotten. However, thanks to WUWT it has been seen more than 2500 times (out of total of 53000 views of my graphs since Jan 2009). Among those were many from NASA, NOAA and many universities.
Peer review process is ‘Stalinist cabal’ .
All power to WUWT !
Mike says:
February 17, 2011 at 6:15 pm
Mike, although you and I don’t consider it a fail-safe process, the IPCC does. If a paper is not reviewed, it can’t come in.
And if you don’t think climate claims can hurt, people are talking about taking trillions of dollars out of taxpayer’s pockets based on those very papers. Your money. My money. Since that is the case, the current peer-review process is a joke, wildly inappropriate for the job it is currently being called on to do.
w.
Mike says:
February 17, 2011 at 6:15 pm
Oh, Mike, that’s so charming, you probably actually believe that.
Stick around climate science for a while, my friend. You will rapidly lose such childish illusions as the idea that the NAS is a neutral player in the game.
w.
(I have had to write this again because my previous try vanished when I clicked on ‘post’. Hopefully two versions won’t appear!)
The answer to this problem is simple. Reviewers remain anonymous until the paper is published or rejected, at which point their identities are made public:
If the paper is published, the identities of reviewers are appended thus: “Reviewed by Groucho, Harpo and Chico.”
If the paper is rejected, the lead author is notified of the reviewers’ identities in a similar way.
In no case is a particular reviewer identified with a particular review. As authors we just know that *one* of the three reviewers was responsible for *one* of the three reviews.
The first case, with the paper published, mitigates the problem of powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be called to account: “Why did you wave by this paper, with all its obvious flaws?” (It would be interesting to know, for the cynic, who reviewed S09).
The second case would be beneficial for authors worrying about who they think is responsible for blocking their work: chances are it isn’t their ‘arch-enemy’ and just a neutral. It also means that reviewers will feel more inclined to be fair in their reviews, and not say things in print they would not say in person. And that editors will try hard to find neutral voices.
Final point, the sub-editor should also be identified at the end of the paper once it is in print. Sub-editors are scientists too, with the same potential conflicts of interest as reviewers.
4 says:
February 17, 2011 at 6:21 pm
Apologies for the lack of clarity. The reviews would be published on the web. The cost and work in that is trivial.
w.
Excellent, Willis. I posted my view of the matter on Lucia’s site. I suggested that any potential reviewer who felt a conflict of interest existed should recuse themselves in the interests of natural justice.
My only experience with Peer Review was as Chair of a committee overseeing a civic facility set up under the aegis of the NZ Environmental Protection Act; The facility, which impacted upon the regional environment, had to be checked on a quarterly basis by a a Peer Review team of three professionally qualified and currently licenced and practicing civil engineers. The Peer Review team checked inital data, sampling methods and all calcs, then forwarded a report to the facility’s management and to the overseeing committee. All reports were signed by each reviewer and there was nothing hidden from view. Where they thought any practice or method could be improved they said so. The process had tense moments, but with everything on the table, best practice and truth won out every time.
Reynold Stone says:
February 17, 2011 at 6:43 pm
Dr. Broccoli may indeed be a good man as regards your case.
But in the Steig case, he a) appointed a reviewer with not just a conflict, but the largest conflict imaginable, in direct opposition to journal policy, and b) refused to reconsider the choice of such a conflicted reviewer despite being directly asked by the authors to reconsider his unethical actions.
Now, you say his actions were “not prudent”. I say his actions were in direct violation of both policy and common sense, and that his actions suck.
I’m sorry, Reynold, but next to that the fact that he was straightforward with you means nothing to me. He probably likes kittens too. But he used his position of power unethically.
Perhaps you could look at that, and not at the fact that he acted professionally in your instance. That’s a long way from being “not prudent”.
w.
http://www.scientificexploration.org/journal/jse_03_2_gold.pdf
Andrew Guenthner says:
February 17, 2011 at 6:58 pm
Excellent exposition. I’ve added it to the head post with the comment:
My greatest thanks,
w.
A further point to make about peer review, credible science and public policy is that a good reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram).
A further point to make about peer review, credible science and public policy is that one reason why a peer reviewed paper is completely unsuitable for forming public policy is because the reviewers are anonymous.
Eg I would love to see the Wegman network chart for the MBH Hockey stick case with the reviewers of the papers included as well. Similarily for Steig et el (although it would probably be pretty much the same diagram).
Its obvious that Steig was the right choice for a reviewer. He has already worked on the problem, and he will be hard on someone trying to overturn his work. So he is ideal. The editor can certainly override what Steig said, no problem, and that is what happened in the end.
The system worked, and worked well.
@ John Brookes
Worked in this case, eventually.
But this might very well be the exception that proves the rule. 🙂
vukcevic says:
February 18, 2011 at 1:22 am
However, thanks to WUWT it has been seen more than 2500 times (out of total of 53000 views of my graphs since Jan 2009).
It is misuse of WUWT to use it to promote junk ‘papers’.
Lucid prose Willis, well done per usual. Provocative.
But I’m not sure that publishing the reviews will be all that interesting however. If a reviewer has comments on egregious errors (say, a math error or bust in logic or otherwise obvious flaw)- they SHOULD first be communicated to the Authors to avoid an embarrassing, bone-head mistake. Those type – along with ‘mechanics’ of the paper (improving prose (English as second language issue), figures, tables, etc.) – are what makes up 90% of a reviewers comments in my experience.
But I think we need to remember that Peer Review is (IMHO) NOT meant to take be the “First and Only Crack” at scientific discussion of the paper ala implementation by The Team. The “Peers” are the Journal’s READERSHIP – NOT the Reviewers.
Substantive comments that turn on a point of art (ala Steig 88 page Reply) more properly belong in a Reply or Discussion of the paper in the Journal.
Thanks, Doc. Your ideas are always interesting and welcome.
DocMartyn says:
February 17, 2011 at 7:28 pm
The Journal acted unethically, in my opinion. They should have given you a copy of the proposed paper, and offered you space in the same issue to publish your response. You could have demolished them with the fourth panel, it would have been whacking great scientific theatre, and journal sales would have gone up.
I see several issues here. First, the anonymous review system is obviously not working in this instance, because as you say the guy is getting papers published. Picking an example where the anonymous peer review system isn’t working seems like an odd choice of example.
Second, I couldn’t do that. I couldn’t anonymously put the knife into someone’s work and then smile and make polite dinner table conversation with them. I am serious when I say that I am sorry that you are in that position. I simply could not do it. If I decided to do the review, I would go to the man myself first and privately discuss the question with him, show him the problems, offer him a graceful way out. If he said no, I’d put in the review, and he’d know who wrote it. But no, I couldn’t bear the other way, I’d turn it down, my soul’s not engineered for that kind of tensile strain. I grew up on a cattle ranch, and now I’m a reformed cowboy, I gave up most of my rough ways.
But somewhere inside I’m still eighteen and on horseback in chaps, helping push a dusty, cranky herd of cattle to the upper range, with my hat shading a blinding bright summer view over the winter pastures far below and on past verdant hills to an unknown future … and a cowboy shoots a man in the front. It’s in the contract.
Third, while anonymity protects reviewers from powerful authors, sunlight protects science itself from powerful reviewers. The IPCC (a poisonous body which does not exist in any other field) said it would only consider peer-reviewed science. This put huge pressure on the field. In response, editors picked reviewers who could be counted on to do the right thing. This has been lethal for climate science. Things like l’affair Steig are destroying the trust of the people in science itself.
Given the choice, and since it would relieve people such as yourself from the unpleasant burden of being two-faced and secretive with your colleagues, I’d choose to protect science itself over protecting reviewers.
But hey, that’s just me.
w.
PS – you say:
I couldn’t disagree more. The infiltration and diversion of climate science by a combination of noble cause corruption, politics, money and the IPCC, including the corruption of the climate science peer review system, has led to billions of dollars being spent on an idea which, to date, has been unable to falsify the null hypothesis.
I think that future generations will view the whole episode as one of the most laughably misguided examples of pseudo-scientific fear-mongering in history.
And for me that means that no, the system’s not working well at all.
Good work Willis.
But we should remember that peer review is only part of the problem.
I am currently preparing a proposal for a root and branch overhaul of the whole publishing process for climate research.
I can see that I had better hurry and put ot up for consideration as things are developing fast.
Ali Baba says:
February 17, 2011 at 8:27 pm
You mean like the collegiality of Newton and Leibnitz? Or how Wegener was treated? By the “adversarial system” I meant only that science is a battle of ideas, where scientists try to show that their idea is correct, and to point out the flaws in the other scientist’s reasoning. They are not partners, Wegener and the “immovable continent” folks. They are adversaries.
And if the penalty for oh, say, advising people to delete incriminating emails, or hiding adverse results in a CENSORED folder were “professionally fatal”, Michael Mann would be pushing up daisies right now … instead, he’s still feted and invited to conferences and the like. I’d love it if fraud were scientific suicide, but he’s made it into a brilliant career choice.
Finally, science based on trust? You are seriously talking about climate science based on trust? After Climategate, after the Jesus Paper, after the multiple exaggerations in the IPCC report, after the famous “Nature trick”, after the farcical “investigations”, after all of that and more you can sit there with a straight face and talk about climate science and trust in the same sentence?
I can’t do that. I start cracking up laughing. Trust a climate scientist? Get real. You can trust them all you want, my friend … I’ll pass.
w.
Off topic, but some what on topic. I have posted on Tamino’s site “an open mind” However not one of my posts have been posted. One was even complimentary that the ice ages were being looked at after I posted a please look at the ice ages as the science should be able to explain these first easily before the understanding of the small changes occurring lately. But disappeared I have, no open mind there.
While I agree with Willis’ proposals (double-blind review process; publication of reviews on the web after publication), may I make two suggestions, which could be implemented straightaway:
1) no author of a paper being criticised should be appointed as reviewer.
Once the critical paper has been accepted, an editor can write to the author(s) whose paper has been criticised, and tell them that such a paper will be published in this or that issue, and they are given so many lines to answer the criticism in that issue.
This would immediately remedy the issue of CoI.
2) addressing what actually is meant by ‘peer’.
I have the impression that far too many people, especially in The Team, seem to think a peer, like a Lord, has a status higher than the others, such as critical authors or lesser-known ones, and are therefore the only ones who should be allowed to ‘judge’.
Well, it ain’t so: peer to peer originally said that peasants cannot judge lords, only other lords can.
Therefore, and keeping in mind that the review process is meant to sort the wheat from the unscientific chaff, I find it very odd indeed that, as was the case with Steig09/O’Donnel10, that at least two reviewers said they are not statisticians.
Why were no statisticians asked to review this paper?
Therefore editors must select at least one reviewer who is knowledgeable in the specific area a paper addresses: statisticians for papers critical of statistical methods; biologists for papers using biological proxies, etc …
So editors should look carefully at what the paper is really about.
And they should always be mindful that a paper must advance the science, not just fiddle around with one or another parameter of what is already known.
We’re all aware of the many fluffy papers being published with great fanfare, showing nothing much at all, which Anthony posts from time to time to the general delight of the WUWT audience. Would be nice to know which reviewer(s) gave them a pass …
And that brings us back to Willis’ proposals!
I’m sorry, from what I’ve seen, climate “science” peer review is about as corrupt as you can possibly get.
And what’s more THEY KNOW it is corrupt, because I’ve been in a quite a few “discussions” where the warmists make little jokes and quips about peer review – which I could only take to be an acknowledgement by them that they control the peer review system.
And … the appropriate response is not to get involved in their corrupt system. IGNORE IT … DON’T GO ALONG WITH IT … CREATE YOUR OWN JOURNALS , BUT PLEASE PLEASE DON’T ADD CREDIBILITY TO CORRUPT ANTI-SCIENCE JOURNALS BY TRYING TO GET PUBLISHED IN THEM!
What you guys seem to think is some exceptional practice is in fact common practice…at least in physics. And it is wrong to pretend otherwise. Here is what I wrote in one of the previous threads on this topic about my personal experience:
As has been pointed out here by others, the reviewers likely to be most knowledgeable are ones who, by your standards, would tend to have some conflict of interest. Furthermore, an editor often especially wants to hear the response of those whose work is being criticized by the paper under review, even if he will weigh it with the knowledge in mind.
The first paper that I ever was asked to referee for Physical Review was when I was a grad student and some authors sent in a manuscript and essentially said, “Shore and [co-author] are wrong” in the abstract of the paper (which was talking about the work that formed my graduate thesis). And, I was able to explain very clearly why the authors of the manuscript were largely mistaken. The final version of their paper essentially said, “Shore and [co-author] were largely correct although there is one aspect where they were incorrect” and included an argument from me supporting and amplifying the discussion of the point where they had disagreed with what we had originally said. The result was a better science than either their original manuscript or our paper alone and less of embarrassment for them than publishing their original manuscript would have been.
In another case, several years later, a paper was written in which they showed that something that I and a colleague had found in one model and had conjectured was true quite generally did not in fact appear to be true for a particular alteration of the original model that we looked at. Again, their paper was sent to me to review and I basically wrote a long review saying that the paper should be accepted but in which I also made several comments, queries, and optional suggestions for revision. I even wrote in the review essentially that I did not find the evidence that the authors presented to be completely convincing but that, since I was not an objective reviewer, my standard on that matter was particularly high and I thought objectively they had in fact presented strong enough evidence to warrant publication. The authors responded with a statement thanking the reviewer for his very thorough and comprehensive review and then a comment that still makes me chuckle to this day, in which they said something to the effect of, “…and in fact we are quite sure that we have never seen such a long positive review.”
Note that both of these papers were in Physical Review journals and had nothing whatsoever to do with climate science. So, the claim that people here seem to be making about climate science having unique issues such as the use of reviewers who might have a conflict of interest is utterly without foundation.
Willis Eschenbach says,
“Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”??? ”
Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.
After two iterations from the reviewer, he ignored reviewer A’s additional comments.
Broccoli behaved impeccably.
Willis Eschenbach says:
February 18, 2011 at 3:12 am
“They should have given you a copy of the proposed paper, and offered you space in the same issue to publish your response. You could have demolished them with the fourth panel, it would have been whacking great scientific theatre, and journal sales would have gone up.”
Is it better for a bad paper to be published then demolished or never to be published?
Some papers are probably wrong but make interesting challenges to the status quo, stimulating research and new ways of thinking. Ruddiman’s early anthropocene might be an example of this. These are certainly worth publishing.
Some papers are bad, with trivial or obvious errors such as McLean et al 2009. Should these be published to? If they are published then someone ought to put time aside to write a reply – I have done this twice, it is a lot of effort with scant reward. Do you think it advances science to see such poor papers published? Or does it advance the agenda of those who would wish to convince the public that the scientific literature is riddled with doubt, uncertainty and controversy, and use this an excuse to avoid action?
DocMartyn says:
February 18, 2011 at 5:02 am
Willis Eschenbach says,
“Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”??? ”
Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee. After two iterations from the reviewer, he ignored reviewer A’s additional comments. Broccoli behaved impeccably.”
Please explain to me how Broccoli acted impeccably when journal policy is:
“A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.”
The logic of this statement of journal policy is that there should not be an appearance of conflict of interest. How exactly do you think Broccoli could have thought this policy would be met by appointing Steig as a reviewer? It would be clear even to someone with the IQ of a chimp that such a conflict existed.
Clearly your idea of acting impeccably is very different from mine. You’d best head over The Carbon Brief. They’ll probably welcome you with open arms, with them being such champions of independent review and all.
Willis: “Mike, although you and I don’t consider it a fail-safe process, the IPCC does. If a paper is not reviewed, it can’t come in.”
Both statements are false. IPCC does make use of non peer reviewed studies – the so called gray literature. Even if the second statement were true, it does not imply the first. Since you don’t understand the IPCC process, maybe you are not in a position to critique it. IPCC is a a review body were the reviews are published and all names reveled. Exactly what you want!
Good post Willis.
I would say though that If I were Broccoli and harbored deep distrust of the pal review system that has reared it’s ugly head, I might put Steig on just to give him the tempting target that they had clearly “gone to town” on before as evidenced by the climategate e-mails. In short, if I were Broccoli, I might have done this with the intent that the hostile reviewer would out himself and this kind of furor would result. If I were trying to expose Climategate as less than some kind of out-of-context invasion-of-privacy to the world, I can’t think of a better way to do this than to create a situation where someone from the Team felt threatened enough to expose themselves.
But, I’m a sneaky b@stard…
You wonder if the good Herr Dr Professor Brocoli is related to anyone from the cabbage patch.
Hello Willis,
We all recognize the problems with peer review. The issues raised with the O’Donnell paper are unfortunately not new. I’ve done a bit of perusing on the subject and have found some interesting tidbits. The basis for peer review is apparently quite old. It seems to be directly related to “imprimatur”. The term is defined as:
“1 a : a license to print or publish especially by Roman Catholic Episcopal authority
b : approval of a publication under circumstances of official censorship”
Mario Biagioli wrote a book titled: “From Book Censorship to Academic Peer Review” in which he ties the early book printing licensing by the state and the creation of the editorial peer review process, with the state funding of academic societies of the time, and both for the same purpose: censorship of ideas that could potentially foment heresy and subversion. The Royal Society of London is quoted as saying in 1663 that “No book be printed by order of the council, which hath not been perused and considered by two of the council, who shall report, that such book contains nothing but what is suitable to the design and work of the society.” The Society of course, was funded by the state that granted its imprimatur.
The divorce of the original peer review doctrine between the state, journals and societies came about later. This allowed the “self-policing” of ideas put into print by the societies which in theory became more about the validity of the idea and not protectionism by censorship. Nevertheless, unknowingly or not, these “liberated” reviews were inherently still censored, and protected the “establishment” of any particular academic discipline by definition, because it was this same establishment that reviewed the submitted works.
Even if nowadays the whole process is purported to be more about accuracy rather than legality, the fact remains that it still is, and will be until revised, a system for the establishment to grant imprimatur.
There might be some light at the end of the tunnel though. The same country that might have invented peer review has now given us what just could be the beginnings of a more open model to follow – the BMJ – British Medical Journal: http://resources.bmj.com/bmj/about-bmj . More is needed but theirs are small steps in the right direction. A BMJ paper on the subject of peer review can be found on the second link below.
Best,
Jose
Sources:
http://mediacommons.futureofthebook.org/mcpress/plannedobsolescence/one/the-history-of-peer-review/
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2127543/pdf/9345164.pdf – Page 3
Andrew Bolt has carried this in his blog today referencing Watts Up With That. I commented on Mr Bolts blog:
I agree that Steig should not have been a referee for that paper. [On checking I note the paper was specifically flagged as a critique of Stieg being titled: “Improved methods for PCA-based reconstructions: case study using the Steig et al. (2009) Antarctic temperature reconstruction.” In this case I think it may be appropriate that Steig be given a chance to comment on a paper that is entirely a critique of his work. ]
But Steigs work is a long way from having fallen apart. [Mr Bolts assertion]
My source for this opinion is none other than the authors of the paper in question.
[I then reprinted my critique of earlier of earlier WUWT items on these papers]:
[Watts] discusses a new paper by his “friends” O’Donnell and Condon which says that Antarctic warming is not as severe as that proposed by Steig et al. The paper itself is not available for perusal but here is the Watts Spin:
http://wattsupwiththat.com/2011/01/13/another-overhyped-global-warming-claim-bites-the-dust/
“Oh and let’s not forget the fact that the whole of the continent of Antarctica has been shown not to have any statistically significant warming…”
But then Watts provides a link to the earlier posting on this where the abstract for the paper is given:
http://wattsupwiththat.com/2010/12/01/skeptic-paper-accepted-on-antarctica-rebuts-steig-et-al/
“Rather than finding warming concentrated in West Antarctica, we find warming over the period of 1957-2006 to be concentrated in the Peninsula (≈0.35oC decade-1). We also show average trends for the continent, East Antarctica, and West Antarctica that are half or less than that found using the unimproved method. Notably, though we find warming in West Antarctica to be smaller in magnitude, we find that statistically significant warming extends at least as far as Marie Byrd Land. “…
Watts headline to the Condon and O’Donell paper is boldly headed :
Skeptic paper on Antarctica accepted – rebuts Steig et al
Lead author Ryan O’Donell writes (in line with the title of their paper, Improved methods for PCA-based reconstructions: case study using the Steig et al. (2009) Antarctic temperature reconstruction
“Overall, we find that the Steig reconstruction overestimated the continental trends and underestimated the Peninsula – though our analysis found that the trend in West Antarctica was, indeed, statistically significant. I would hope that our paper is not seen as a repudiation of Steig’s results, but rather as an improvement.
In my opinion, the Steig reconstruction was quite clever, and the general concept was sound. A few of the choices made during implementation were incorrect; a few were suboptimal. Importantly, if those are corrected, some of the results change. Also importantly, some do not. Hopefully some of the cautions outlined in our paper are incorporated into other, future work. Time will tell!”
O’Donnell also sttes, concerning the review process:
“I am quite satisfied that the review process was fair and equitable, although I do believe excessive deference was paid to this one particular reviewer at the beginning of the process. While the other two reviews were positive (and contained many good suggestions for improvement of the manuscript), the other review was quite negative. As the situation progressed, however, the editor at Journal of Climate – Dr. Anthony Broccoli – added a fourth reviewer to obtain another opinion, which was also positive. My feeling is that Dr. Broccoli did a commendable job of sorting through a series of lengthy reviews and replies in order to ensure that the decision made was the correct one.”
DocMartyn,
“Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.”
Is it your assertion then, that an editor must, by all that is ethical and just in the world of journal publishing, bring in as reviewer, the author of the paper that is being attacked?
If this is correct, then this whole article is redundant. We don’t even need to argue whether or not a conflict of interest exists, or whether the editor broke journal policy, because you have incontrovertible proof that this is the way editors must act. Why don’t you just show us this policy that says the author of the critiqued paper must be the reviewer? Or is this just your own opinion?
John Brookes,
” John Brookes says:
February 18, 2011 at 1:58 am
Its obvious that Steig was the right choice for a reviewer. He has already worked on the problem, and he will be hard on someone trying to overturn his work. So he is ideal. The editor can certainly override what Steig said, no problem, and that is what happened in the end.”
Your logic is a little flawed. If the editor was knowledgeable in statistical techniques, then he would indeed be able to override Steig. But the evidence is that he reacted to what was becoming a vexatious challenge by Steig to O’Donnell’s paper. Eventually, having ‘had it up to here,’ with Steig’s behaviour, he was forced to dump him for an unbiased reviewer.
So the eventual correct outcome happened in spite of the decision to appoint Steig, not because of it. If anything can be said in Broccoli’s favour, it is that he had enough common sense to see, in the end, that he was being jerked around and booted him out.
First comment says it well, “Spot on as usual Willis.”
DocMartyn says:
February 18, 2011 at 5:02 am
“Broccoli had a duty to the authors of the Science paper being attacked, so he asked the first author to be a referee.After two iterations from the reviewer, he ignored reviewer A’s additional comments.Broccoli behaved impeccably.”
After sharing some of his experiences as journal author and reviewer, none of which involve any reference to moral judgement whatsoever, DocMartyn writes that the journal editor, Broccoli, had a duty to Steig and coauthors and he satisfied this duty by enlisting Steig as a reviewer. What duty is that, DocMartyn? Would you please explain? I believe that you are incapable of writing one sentence about duty that is coherent. So, here is your chance.
You are going to need some help with this, DocMartyn, so let me get you started. As journal editor, I have a duty to be impartial to authors who submit papers for review. As part of that duty, I must ensure that reviewers WHO SERVE ME are impartial. My journal publishes a policy damning conflict of interest. So, I select as reviewer the one person most likely to have a conflict of interest, Steig, and, thereby, violate my journal’s policy against conflict of interest and my duty to be impartial. What do you not understand about this, DocMartyn. I am eager to explain it for you.
Finally, DocMartyn, I know that this is going to be difficult for you because the topic is moral judgement and not the usual mud slinging and sausage making. I feel for you. However, you just have to face up to it. The topic is moral judgement.
Leif Svalgaard says:
February 17, 2011 at 5:01 pm
There are boreholes, and there are boreholes. The problem is whose work is being dumped into the borehole and why, versus whose work IS NOT being dumped into the borehole and WHY NOT? The pal reviews by the so-called TEAM could by some objective measures deserve to be consigned to an infinitely deep borehole for the simple fact that they refuse to provide the open scientific community with the customary data and methods required to replicate the claimed scientific experiments.
I would suggest using the e-print arXiv as a starting point for developing an objective and merit based approach for the scientific community and the broader general public to promote and demote the papers by initial and ongoing reviews of the paperss and their subject matter. In particular, when a given paper fails to meet certain basic criteria such as proper documentation, different communities can choose to apply the objective criteria to demote the paper to the rank of those failing to meet fundamental requirements for documeentation of mthods and data, while the pal reviewers can can choose if they so wish to risk their reputations by refusing to apply those basic scientific standards. In any eveent, outright censorship and secretive political ceensorship can be avoided or at least mitigated.
D. Patterson says:
February 18, 2011 at 8:13 am
I would suggest using the e-print arXiv as a starting point for developing an objective and merit based approach for the scientific community and the broader general public to promote and demote the papers
arXiv is very useful and serves an even more important function: to get results out quickly. But [and this is proper] there is still a filtering performed: you cannot post to arXiv unless you have been endorsed by another scientist with sufficient credentials to be an ‘endorser’. In this way obvious cranks are kept out as they should.
I must say that I am shocked and dismayed at the willingness of the Warmista to create sheer blather as a kind of smokescreen to prevent criticism of poor or nonexistent moral judgement. The situation is as follows:
Officer: Sir, I saw you drive through the stop sign and run down the pedestrian who now lies bleeding beneath your car.
Driver: Officer, you say that as if it is a bad thing.
“Jit says:
February 18, 2011 at 1:27 am
(I have had to write this again because my previous try vanished when I clicked on ‘post’. Hopefully two versions won’t appear!)
The answer to this problem is simple. Reviewers remain anonymous until the paper is published or rejected, at which point their identities are made public:
If the paper is published, the identities of reviewers are appended thus: “Reviewed by Groucho, Harpo and Chico.”
The first case, with the paper published, mitigates the problem of powder-puff reviews because if the paper is later shown to be so much slurry, the reviewers can be called to account: “Why did you wave by this paper, with all its obvious flaws?” (It would be interesting to know, for the cynic, who reviewed S09).”
Again, I refereed a paper, and found only minor problems with it. Made a few suggestions and ticked the accept with minor revision box.
Two months later I get a new version of the paper, two completely new figures, two figures missing, and different methods section. I get the comments of another referee who spotted something I missed, this reviewer demanded that two datasets be repeated, using different methodology.
This second reviewer was right.
Me, I missed it completely. Publish my name and my initial review and I look a right pillock.
Want me to keep doing 8 hours unpaid work, whilst leaving myself open to looking like a complete prat?
Suggestion:
The distinction between “informative” review and “gatekeeping” (in the good sense) review should be made explicit for all science journals.
Steig should be a reviewer in this case, but explicitly as an information source without controlling authority. He has the the strongest motivation to find fault with the critical paper—-it is okay to put his motivation to work. But his clear conflict of interest should keep him out of the gatekeeping function, and this should be explicit. Non-conflicted referees can then benefit from the Socratic back and forth between Steig and the paper critical of Steig’s work and perform their essential filtering or gatekeeping role with improved information.
Matt: But in DC they still re-elect the politician! People don’t like to question their beliefs. It’s too scary/painful. They’d rather live with cognitive dissonance.
At 8:330 PM on 17 February, in response to the proposal that “Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them,” Clay Ross observes that:
.
To this Mr. Ross adds the entirely un-facetious thought that:
.
This addresses the point that a manuscript not accepted for publication – even after the authors had gone through the laborious process of responding to editors’ and reviewers’ comments and amending their manuscript – becomes without further condition the “property” of the journal to which it had been submitted with the intention of getting it published therein.
The intention had not been for the work to be publicly rejected and thereby denied all further value.
A paper submitted to The New England Journal of Medicine and declined by the officers of that periodical might be perfectly suitable for publication in Gastroenterology, and would be welcomed by the editors and reviewers of that journal – but not if that article goes onto the NEJM Web site on its publicly accessible “reject” pile before the editors of Gastroenterology can get a look at it.
If a condition is sought in which authors accept the punishing liability that their work – both in research and in the preparation of manuscripts reporting that research – can be summarily denied even the chance of publication in a peer-reviewed scientific periodical which can be legitimately cited because that work had been thrown onto an Internet “reject” pile by editorial fiat upon its first submission, then what the hell inducement is there for any researcher to submit anything to any journal in the first place?
Might as well just post it online and leave it to the search engines to put it in front of readers’ eyes.
Re: “Note that both of these papers were in Physical Review journals and had nothing whatsoever to do with climate science. So, the claim that people here seem to be making about climate science having unique issues such as the use of reviewers who might have a conflict of interest is utterly without foundation.”
I would argue that the problems which pervade our scientific institutions are indeed foundational problems. And they can be analyzed either within a philosophical manner through a discussion of the weaknesses of the Socratic-dialectic method which has been deployed to select the dominant framework; or, they can be discussed within the context of the role of the “professional” which PhD’s are expected to abide by, as Jeff Schmidt has superbly accomplished in “Disciplined Minds”.
A professional is somebody who is expected to adopt the framework, and problem-solve entirely within this unquestioned framework. A professional NEVER questions the framework itself, because that is outside of the scope of his professional duties.
This code of conduct is instituted in our physics graduate programs: Students who do not adopt the framework are literally dropped out. But, most physics grad students just plain forget all of those curious ideas they had in their undergrad and high school years, because they are forced to memorize stacks of problem sets in preparation for their PhD qualifying exams. These problem sets include fun little mathematical tricks which must also be memorized in order to complete the exams on time (and to be clear, these mathematical tricks are never used again, after the exam). This process of memorization tends to generate ideologically docile students.
Then, making things worse, PhD physics students are not even guaranteed a PhD if they pass the exam. At that point, the professors come together and use the test score as just a means of evaluating whether or not they view this person as a professional representative of the views of the institution of physics. Can they be trusted to espouse conventional wisdom?
Any discussion of the problems of science which glosses over these insane problems will lead to superficial fixes. Jeff Schmidt:
“For understanding the professional, the concept of “ideology” will emerge as much more useful than that of “skill.” But what is ideology, exactly? Ideology is thought that justifies action, including routine day-to-day activity … your ideology justifies your own actions to yourself. Economics may bring you back to your employer day after day, but it is ideology that makes that activity feel like a reasonable or unreasonable way to spend your life … Work in general is becoming more and more ideological, and so is the workforce that does it.” (Jeff Schmidt, Disciplined Minds, p15)
“Employers have always scrutinized the attitudes and values of the people they hire, to protect themselves from unionists, radicals and others whose “bad attitude” would undermine workplace discipline …
My thesis is that the criteria by which individuals are deemed qualified or unqualified to become professionals involve not just technical knowledge as is generally assumed, but also attitude – in particular, attitude toward working within an assigned political and ideological framework …
The qualifying attitude, I find, is an uncritical subordinate one, which allows professionals to take their ideological lead from their employers, and appropriately fine-tune the outlook that they bring to their work. The resulting professional is an obedient thinker, an intellectual property whom employers can trust to experiment, theorize, innovate and create safely within the confines of an assigned ideology. The political and intellectual timidity of today’s most highly educated employees is no accident …
Furthermore, professionals are the role models of the society toward which we are heading, a society in which ideology trumps gender, race and class origin as the biggest factor underlying the individual’s success or failure.” (Disciplined Minds, Jeff Schmidt, p15 … 19)
“This book’s analysis finds the supposed political neutrality of the process of professional qualification a myth: Neither weeding out nor adjustment to the training institution’s values are politically neutral processes. Even the qualifying examination — its cold, tough, technical questions supposedly testimony to the objectivity and integrity of the system of professional qualification and to the purity of the moment of personal triumph in every professional’s training — does not act neutrally. The ideological obedience that the qualification system required for success turns out to be identical to the ideological obedience that characterizes the work of the salaried professional.” (Jeff Schmidt, Disciplined Minds, p26)
“No matter what the product is, employers divide the work into many parts and assign each employee to one type of activity. Narrowly focused individuals can work in a more machine-like way and get more work done per hour. Moreover, people who exercise fewer skills or simpler skills can be paid less. Hence, employers label the division of labor “efficient.” But it is efficient only if one ignores the social cost of organizing production in a way in which jobs tend to be monotonous and unsatisfying. Such jobs, instead of allowing individuals to develop their mental and physical faculties by exercising them freely and fully (that is, instead of being fun), numb the mind and the body and retard the personal development of those employed to do them. A system of production that works efficiently toward the goals of employers does not necessarily work efficiently toward the goals of employees or toward the goals of society in general.
More important to employers than the economic benefits, however, are the political benefits of the division of labor — benefits that help management maintain its authority in he workplace. Confined to a range of activity that is limited both horizontally and vertically, employees do not gain firsthand knowledge of the overall organization, strategy or goals of the institution that employs them. Those who work within this division of labor see the consequent ignorance in themselves and in their coworkers and feel a need to be directed by people who comprehend the whole operation. Management has the broadest view of what is going on, and this helps make its supreme authority in the workplace seem natural and justified.” (Jeff Schmidt, Disciplined Minds, p91)
Jeff Schmidt was an editor of Physics Today for 19 years. He has a PhD in physics from the University of CA at Irvine. He was fired for writing this book, and then won the resulting lawsuit.
To this day, people tend to pretend as though this book was never even written. It’s all business-as-usual in the world of science.
Philip Shehan,
“O’Donnell also sttes, concerning the review process:
“I am quite satisfied that the review process was fair and equitable”
Well of course O’Donnell is going to try and calm troubled waters. I would find it very strange if he voiced any acrimony over this. He would certainly want to retain as much goodwill from both the editor and the publisher as possible.
As for your other comment that O’Donnell effectively agrees with the Steig paper that the Antarctic continent is warming, again, he is stepping back from confrontation. But the data speaks for itself – you only have to look at and compare the two temperature anomaly maps. They are different, and in the end, isn’t that all that counts?
Kudos to Dr. B. The peer-review system _sort_of_ worked, despite apparent attempts to game it. Perhaps it would have been better if Dr. S’s contribution had been in the form of a _signed_ comment distributed to the authors and reviewers at the beginning of the review process. One might also hope that Dr. S. would have been allowed space for a signed reply to the final paper in the same issue. However, allowing him to sit in judgment as an anonymous reviewer was not a wise decision. The public is left with the impression of “tacky” behavior by scientists and scientific journals. This is bad PR and we all may end up paying for the loss of public trust.
Chris Reeve says:
February 18, 2011 at 9:02 am
A professional is somebody who is expected to adopt the framework, and problem-solve entirely within this unquestioned framework. A professional NEVER questions the framework itself, because that is outside of the scope of his professional duties.
Apart from the dubious allusion to ‘professionals’, scientists are NOT professionals in your sense. Their job is to overthrow [if they can] any frameworks, this is called advancing knowledge. It is every physicist’s dream to prove Einstein wrong. You are confusing your ‘framework’ with the ‘scientific method’. The latter is indeed unquestioned as it should be.
A side effect of publishing reviews might be to lessen the number of incomplete papers submitted, if people could see how much the paper needed to be developed based on reviewer comments. I once observed a prolific and respected researcher screaming, after having a major flaw found in one of his papers after publication, blaming the anonymous reviewers for not having caught the mistake. Not the slightest hint of embarrassment at having made the mistake in the first place.
Here are the arXiv rules for endorsement:
http://arxiv.org/help/endorsement
Chris Reeve says:
Sorry, but your description of the process of getting a physics PhD bares essentially zero relation to how I experienced it. Can you tell us how you came up with this view? Is it based on personal experience, people you’ve talked to, or what?
Not all scientists would agree on disqualifying authors with conflict of interest as defined by Eschenbach. Here is Roy Spencer commenting on peer review:
http://web.archive.org/web/20040531091046/http://www.techcentralstation.com/050504H.html
“This kind of mistake would not get published with adequate peer review of manuscripts submitted for publication. But in recent years, a curious thing has happened. The popular science magazines, Science and Nature, have seemingly stopped sending John Christy and me papers whose conclusions differ from our satellite data analysis. This is in spite of the fact that we are (arguably) the most qualified people in the field to review them. This is the second time in nine months that these journals have let papers be published in the satellite temperature monitoring field that had easily identifiable errors in their methodology. ”
Actually, it often makes a lot of sense to send a paper for review to an author of the paper it claims to contradict. Science is a specialized world, and that author is likely to know more about the field and have more comments than just about anyone else the editor is likely to find. The responsibility then lies with the editor to interpret the response and see what is substantial comments and what may be just a knee jerk reaction to block an inconvenient paper. In this case it seems the comments from Stieg made the paper by O’Donnell considerably better.
Crusty the Clown says:
February 18, 2011 at 9:17 am
Kudos to Dr. B. The peer-review system _sort_of_ worked, despite apparent attempts to game it. Perhaps it would have been better if Dr. S’s contribution had been in the form of a _signed_ comment distributed to the authors and reviewers at the beginning of the review process. One might also hope that Dr. S. would have been allowed space for a signed reply to the final paper in the same issue. However, allowing him to sit in judgment as an anonymous reviewer was not a wise decision.
He did not sit in judgement, the editor did, he was asked for his opinion as an expert witness and the editor made the decision. The inexperienced authors were offended by the 1st review and the editor added another reviewer for them (but he agreed with A anyway on several matters as the editor pointed out).
Joel Shore.
The case you site is nowhere near similar.
“The first paper that I ever was asked to referee for Physical Review was when I was a grad student and some authors sent in a manuscript and essentially said, “Shore and [co-author] are wrong” in the abstract of the paper (which was talking about the work that formed my graduate thesis). ”
As your self all the relevant questions about this situation and steigs. They are only remotely similar
Willis Eschenbach says:
February 18, 2011 at 1:03 am
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
No. As a scientist who has been on both sides of the referee process that’s not how it works.
Calling in another referee when there has been an impasse with one does not mean the referee is biased.
We have all had to deal with “difficult” (or conscientious, depending on your point of view) referees. Often there can be much back and forth commentary and requests for multiple revisions before they agree to publication. Or not. It’s all part of the game. Too glowing a report from an odd man out when others are harsher can mark the referee out as a soft touch, sloppy or too obviously friendly to the authors. That can get you struck off the journals list of referees.
An 88 page referees report does seem somewhat over the top, but for all I know the points raised were entirely legitmate and raising points for clarification or amendment is not the same as rejection.
Broccoli did not grant Steig power of veto over publication. It’s the editors job to make judgement calls on how to deal with impasses. Most would not be happy with two out of three votes in favour. Calling in another opinion is one method.
And again O’donnell commended Broccoli for his handling of the matter.
Sorry if this is a repeat but it seems not to have gone through the first time:
Vince: We can take O’Donnell at his printed word or not. Any interpretation other than that is pure speculation. There is no need for O’Donnell to “voice acrimony” about the editorial process if he feels hard done by. There are plenty of sections and commentators on this blog (including this section) who are doing that for him. He can simply maintain a dignified silence while his supporters go to bat for him. There is apparently enough good will toward him on the part of the editor and publisher, even after all the alleged argy bargy, for him to have his paper published. There is no need for him to suck up to them.
Similarly whether he and Steig are at daggers drawn or simply agreeing to disagree, having had his paper published with Steig as referee means that he does not have to step back from confrontation at this stage. There is nothing to be gained from promoting confrontation, but again all that requires is dignified silence. There is nothing dignified about hypocritically sucking up to Steig.
On the other hand, perhaps O’Donnell is a person of honesty and integrity and means what he writes. Call me foolish but that is my default position with people unless there is a good reason to think otherwise, and I have argued that there is no good reason.
Yes, I have looked at the maps and there certainly are differences. And again I have no reason to doubt what the authors write in the abstract when they point to those differences. To suggest that they are agreeing with the Steig paper that the Antarctic continent is warming simply to avoid confrontation, is taking reticence to upset people into the realms of a total abandonment of scientific integrity and into the territory of scientific fraud.
But my main complaint is with Mr Watts, who represents the paper as saying” the whole of the continent of Antarctica has been shown not to have any statistically significant warming…” when it says quite the opposite.
I take O’Donnell at his word that his paper is not a repudiation of Steig and his comments on the strengths and weaknesses are an honest appraisal. For Watts to decided that O’Donnell is being dishonest and insisting the paper is a rebuttal is very careless y at best and casting aspersions on O’Donnell’s integrity at worst.
DocMartyn says:
February 18, 2011 at 5:02 am
So the journal and other policies about not using reviewers with conflicts of interest, I guess those are just for the little people, editors and people like you don’t have to pay any attention to ethics policies …
Impeccably? Only if you ignore common sense and more importantly, journal policy. Broccoli had a duty to the authors as you point out, and there are a variety of ways he could have honestly discharged that duty. But he also has a duty to us that includes not putting the fox in charge of the henhouse.
Doc, your plan works beautifully as long as the players are by and large honest. But that’s not the situation here, we are dealing with crooks. Rules for dealing with crooks (like journal ethics policies) outweigh rules for dealing with honest people. This is the constitutional question again, where the US Constitution works because it assumes that the participants are humans who will lie, cheat and steal if given the chance. There definitely was a day when your system generally worked, Doc, just like there was a day when I could leave my house unlocked.
But that ain’t today, as the horrendous mess of climate science clearly demonstrates.
w.
Richard Telford says:
February 18, 2011 at 5:19 am
The latter is much better sport … but seriously, you and others are arguing for a system that (in the climate science field) has failed us horribly. Now mine may not be the best plan for getting out of that situation.
But arguing for the failed current plan isn’t going to do it, been there, failed that, got the Climategate scars to prove it.
w.
Mike says:
February 18, 2011 at 6:25 am
BWAHAHAHA, oh, Mike, you really do believe their stuff.
The IPCC has recently changed its rules for the next Assessment report to let in gray literature. For the previous reports that was not supposed to be the case. Here’s a Reuters News interview with Pachauri, the head of the IPCC (emphasis mine), from 2009:
See that part in there that says “using only articles approved by other scientists, called peer review”? Perhaps you could explain what that means to the rest of us. Or not.
In addition, yes, the reviews were published. And Steve McIntyre and others had to fight like hell to get them published. He would laugh to think that fools could use the results of his work in prying open the IPCC against strong opposition, use that very work to prate about how open the IPCC is …
And you want to tell me that I don’t understand the IPCC? Dude, you come in here with enough misinformation about the IPCC to fill a wheelbarrow, and you want to lecture us on what’s what? You’re talking to guys who actually fought the battles, Mike. I was the first man to put in a Freedom of Information request to the CRU, and you want to explain the IPCC to me? Funny, I didn’t see you on the front lines, how was your knowledge of this war acquired? Because we got our knowledge the hard way, through personal interactions with the players involved.
BZZZZZT. Next contestant, please …
Thomas says:
February 18, 2011 at 10:14 am
No, no, and no. They should be asked to comment, either informally before publication or in a rebuttal in the same same issue or in some other manner.
But they should never be reviewers. I keep asking this, but if you guys claim is true that authors are the best reviewers, not the best to comment on a paper but the best reviewers, then why are there journal policies against using reviewers with even the APPEARANCE of a conflict??? Seems like if what you claim were true, that would be in the policies … but where are those policies that say “It’s ok to use a reviewer with a huge conflict of interest if they’re one of the paper’s authors”?
If a guy stood to lose a lot of money from a scientific paper being published, would you let him review it? I wouldn’t, because of the conflict of interest. If he knew a lot about the subject I might, as editor, ask his opinion (although that wasn’t the case in this instance with Steig). But put him in as a reviewer? Everyone would just say “Yeah, of course your journal wouldn’t publish an anti-McDonalds study, Ray Kroc was one of the referees that shot the paper down”. Might not be true, Roy might have actually supported the paper, but it destroys credibility and faith in the institution.
So if a guy stands to lose a lot of scientific prestige from a paper being published, would you let him review it?
w.
Philip Shehan says:
February 18, 2011 at 10:58 am
I appreciate your thoughts, Philip. Again, like others, you assume that we are dealing with honest people of good will. The founders of the US Constitution were much wiser. They assumed that e.g. if a man was obviously too friendly to the authors, rather than striking him off the list of referees, some editors would simply use him when they are in need of a puffball easy review for that class of authors. Duh.
Philip, the system we have, as you point out, assumes that the actors will act honorably and with good will. And for many years that seems to have been generally true enough to keep the system working. As Climategate showed, however, in climate science currently that is not true. Because of the false importance placed by the IPCC on peer-review, people of bad will have gained the ascendancy and gamed the system to act as gatekeepers for their particular version of science.
That’s why I say we need a new system. Despite the spirited defense of the status quo by yourself and others here, that system is clearly not working for climate science. You can have a Star Chamber only as long as the judges are honest … and as the good folk of England found out, when you allow people to judge other people secretly and without responsibility for their actions, even judges who might start out honest soon fall from grace in that situation …
w.
“Theo Goodwin says:
After sharing some of his experiences as journal author and reviewer, none of which involve any reference to moral judgement whatsoever, DocMartyn writes that the journal editor, Broccoli, had a duty to Steig and coauthors and he satisfied this duty by enlisting Steig as a reviewer. What duty is that, DocMartyn? Would you please explain? I believe that you are incapable of writing one sentence about duty that is coherent. So, here is your chance.”
Steig et al., was published and published in Science, one of the highest ranked journals.
O’Donnell et al., wrote a paper that challenged the methodology and conclusions of Steig et al.,, not the data from which the conclusions were drawn. The whole of Steig et al., which was being critiqued by O’Donnell et al., was the authors analysis.
First author generally does the most work and last author generally has the most money/seniority. Steig was the front man and the Editor had a duty towards the papers authors, he had to make sure that O’Donnell et al., critiques were fair. The experts, for the defense, were Steig and co-workers. The editor asked the front man to examine the manuscript. Twice Steig reviewed, after which the editor ignored his input. He gave the Steig team a heads up of incoming criticism and allowed them to find any major faults. They found none. So Broccoli gave the Steig team first bite of the cherry and Broccoli gave O’Donnell et al., a baptism in fire and it survived.
Theo, I suspect that you do not realize the emotional investment there is in generating a paper. Each paper has about the same level of pampering and the emotional investment as a pedigree show dog. Papers are to scientists what ships are to sailors or land is to farmers. They are part of you and you are part of them. When someone attacks your work, in review or post publication, it hurts, it really does hurt.
Each paper, sometimes subtly, affects your pay, promotion and job chances. When you go for an interview for a job, your can be sure that the people behind the desk have mined for publications for citations whereby they have been creamed. I have 48 peer reviewed papers and two chapters. Only one has been directly panned, unfairly in my view, and that single citation, out of 79, still pisses me off.
Excellent post, Willis. To the extent that science is real science, and not religion, it should implement policies like these. That should go, incidentally, for humanities as well.
Willis Eschenbach says:
What you actually quoted from talks about the responsibility of the reviewers, not of the editor. The point is to not have a situation where a reviewer might have biases that the editor is not aware of. I don’t see anything in the policy that states that editors must only use reviewers who he/she thinks have no such biases. (The only reference to conflicts of interest that I see under responsibilities of the editor is that the editor him or herself not have a conflict of interest.)
And, as I have noted, in Physical Review, from what I can tell, it is considered good form for the editor of a paper that is largely a critique of another paper to get a review from the authors of that other paper. In fact, I wasn’t even on Physical Review’s radar screen as a potential reviewer until they received a paper that was largely a critique of my work. (Once you get on their radar screen though, you never get off!) In this case, the editor understands that the reviewer will have his/her biases; however, that reviewer also has the most knowledge of the work being critiqued and possible flaws with the critiquing work and this knowledge is invaluable.
In this case, the procedure that the editor used from what I understand of it (and I admit that I haven’t followed the gory details to the degree that some have) seems very reasonable. He solicited the opinion of Steig as a reviewer and took his review seriously but ultimately did not give Steig veto power over the manuscript or unlimited authority to demand changes in the manuscript. I guess I am having a hard time understanding what exactly the complaint is.
@ Willis Eschenbach,February 18, 2011 at 11:36 am
“The use of non-peer-reviewed sources, which is sometimes called gray literature, also has been controversial and blamed for some of the errors. However, we found that such material, which can include technical reports, conference proceedings, observational data or model results, often is relevant and appropriate for inclusion in the assessment reports. IPCC has guidelines for the use of such sources, but these guidelines are vague and have not always been followed. We recommend that these guidelines be made more specific – including noting what types of sources are unacceptable — and that they be more strictly enforced to ensure that unpublished and non-peer-reviewed literature is adequately evaluated and appropriately flagged in the reports.”
See:
http://reviewipcc.interacademycouncil.net/OpeningStatement.html
You can look through the IPCC 4 references like here.
As for the Interview with Reuters, the first line you quote is a paraphrase. Maybe the reporter got it wrong or P misspoke. The direct quotes do not say gray literature was never used. But all sources are reviewed by the IPCC scientists – they are reviewing the literature gray and peer reviewed. BTW: I disagree with the last line from P, as nothing 100% fail-safe.
Folks: It is painfully obvious that despite whatever good intentions Willis may have, he does not know beans about how science is done. He is not is any position to engage in meaningful discussions about climate change. There are lots of places where you can read about climate science. This is not one of them.
“BWAHAHAHA” yourself Willis. Grow up.
IPCC scientists? Don’t make us laugh.
DocMartyn says:
February 18, 2011 at 12:10 pm
“Steig et al., was published and published in Science, one of the highest ranked journals.”
Utterly, totally, abominably irrelevant.
“O’Donnell et al., wrote a paper that challenged the methodology and conclusions of Steig et al.,, not the data from which the conclusions were drawn.”
A distinction without a difference. Data can be criticized. Methodology can be criticized.
“The whole of Steig et al., which was being critiqued by O’Donnell et al., was the authors analysis.”
Utterly, totally, abominably irrelevant.
“First author generally does the most work and last author generally has the most money/seniority. ”
Utterly, totally, abominably irrelevant.
“Steig was the front man and the Editor had a duty towards the papers authors, he had to make sure that O’Donnell et al., critiques were fair.”
I guess this is it. There is no such duty. The editor has the duty to be impartial. That means, has always meant, and will always mean staying out of the argument. The editor has a duty not to attempt to balance the scales between author and reviewer or to influence in any way the relationship between author and reviewer. To do anything it to violate impartiality. Your problem is that you have this bizarre idea that the science is supposed to get done among author, editor, and reviewers. That is manifestly not the case. The editor is supposed to guarantee impartiality among reviewers whose job is to (1) find blunders before they become embarrassing to the journal, (2) ensure that the essay has some appeal to some scientists in the field, and (3) ensure that the article is reasonably well written, by the low standards used for scientists.
“The experts, for the defense, were Steig and co-workers.”
Listen at you! Are you insane, sir? This is not a court proceeding. This is not even an arena in which science should be done. This is one more among gazillions of boring, ordinary reviews for a journal. It must be carried out by boring, ordinary reviewers who have no stake in the outcome. And it must be presented to an editor who is going about his daily grind of boring, ordinary activities, and who has no stake in the outcome.
“The editor asked the front man to examine the manuscript. Twice Steig reviewed, after which the editor ignored his input.”
No, no, no! You neglect the time wasted and the wear and tear on the nerves of the O’Donnell folks. This was punishment. Also, it delayed Steig’s reckoning.
“He gave the Steig team a heads up of incoming criticism and allowed them to find any major faults.”
What the hell does that mean? Do you even know? It sounds as if the editor gave Steig a WARNING! Do you not know how wrong it would be for an editor to do such a thing?
“They found none. So Broccoli gave the Steig team first bite of the cherry and Broccoli gave O’Donnell et al., a baptism in fire and it survived.”
Yes, sir, you are insane or nearly so. You are so full of the hubris and grandiosity of “The Team” that you believe that it is reasonable to say that an editor gave an author a “baptism of fire.” You have confirmed in your own words, that the whole world can read, all of my suspicions of you and my accusations against you. Every editor knows that he has the duty to treat authors impartially; that is, treat them all the same. Sir, you reveal that you do not know the meaning of the word ‘impartiality’. So what are you doing here writing about it?
Any editor who takes it upon himself to conduct a “baptism of fire” is acting beyond any and all standards of responsibility, drunk on his grandiosity, a danger to society, and certainly should occupy no position of responsibility including that of dogcatcher. Sir, you are a symptom of what most ails climate science at this time, namely, a degree of hubris and grandiosity yet unknown to mankind.
In sum, the only reading of your words that makes sense is that the editor manipulated the review process for his own ends, including placing arbitrary burdens on O’Donnell and coauthors, and gave free reign to Steig to raise hell with O’Donnell. When he was finished playing cat and mouse, the editor stopped the nonsense and published the paper that should have been published months earlier.
Philip Shehan says:
February 18, 2011 at 10:58 am
Willis Eschenbach says:
February 18, 2011 at 1:03 am
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
Well that’s a mischaracterization of what happened. After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.
An 88 page referees report does seem somewhat over the top, but for all I know the points raised were entirely legitmate and raising points for clarification or amendment is not the same as rejection.
Of course that didn’t happen, it was a 14 page review!
Phil. says:
“After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.”
Oh. Well, I guess that makes it A-OK then.
Smokey says:
February 18, 2011 at 4:32 pm
Phil. says:
“After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.”
Oh. Well, I guess that makes it A-OK then.
Yes it sure does, nothing to complain about just some neophyte authors moaning.
Note the following from Broccoli:
“To allay some of the concerns you have expressed about Rev. A, I have sought the advice of an additional reviewer (Rev. D). Please note that several of Rev. D’s comments are similar to points made by Rev. A and thus warrant especially close attention.”
Theo Godwin: Since you are presenting yourself such an expert in what the proper role of editors and reviewers in the peer-review process, it would be nice to know where your expertise in this matter is derived from.
In my case, it is from having been a referee on over 80 articles in the last 19 years submitted to physics journals (primarily the Physical Review journals)…and having been a published author of about 30 papers in peer-reviewed journals. I find it a little strange for people to be pontificating on what this without explaining how they have arrived at their expertise in this area and how they know, for example, that somehow what happens in climate science is dramatically different than what occurs in other physical science fields.
Theo, take a pill man
“Your problem is that you have this bizarre idea that the science is supposed to get done among author, editor, and reviewers. That is manifestly not the case. ”
This is not science.
Science is the modeling of the universe around us to develop a coherent description of the underlying rules that govern matter.
What you are talking about is the propagation of scientific discourse.
You have the idea that editors are impartial and that all corresponding authors should be treated in the same manner.
Editors cannot be impartial, they cannot be because they must know the track record of the referees they chose, normally knowing them personally. They must weigh up the fact that someone has a dog in the fight with the fact that information at the cutting edge science is asymmetric.
If an editor wants to have someone review a paper looking at brain mitochondria, nitric oxide, Fenton/Haber–Weiss you have only about six people to chose from, that’s just the way it is.
The only people who fully understood Steig et al., were Steig et al., and O’Donnell et al. That is the way it is.
Yawn.
Gentlemen, you are arguing about particular details of the process that is fundamentally flawed in principle.
In a Maoist society, any IPCC-style or Wikipedia-style selection process will result in Maoist-only articles selection. Substitute “Maoist” with any prevailing ideology, and there you have it.
Or would you insist that there is no prevailing ideology in IPCC or Wikipedia? Then all you deserve would be a silent smile of reason, the one that silly children see sometimes on the faces of their grandparents.
EJ says:
February 17, 2011 at 9:25 pm
Too bad scientists no longer need a reputation.
Ehrlic and Holdren, the science Czar, included.
Apparently a pHd can still say anything, hide the data, and fudge the findings.
Wow
————
REPLY Hah! They’ll have to dig into their own pockets if they want to say anything!
This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research. The original cuts to the budget outlined yesterday would have cut $379 million from NASA’s budget. These members want climate out of NASA’s purview entirely, however. Funding climate research, said Adams in a statement, “undercuts one of NASA’s primary and most important objectives of human spaceflight.”
http://motherjones.com/blue-marble/2011/02/republican-climate-nasa-budget
Mike says: February 18, 2011 at 2:12 pm
Here is a site that discusses the use of grey literature in the 2007 IPCC report and how some came to be developed and/or promulgated
http://nofrakkingconsensus.blogspot.com/2010/01/greenpeace-and-nobel-winning-climate_28.html
Read more: “Dr Pachuri……..IPCC studies only peer-review science. Let someone publish data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it in a dustbin.”
http://timesofindia.indiatimes.com/india/No-proof-of-Himalayan-ice-melting-due-to-climate-change/articleshow/5213045.cms
I’ve posted a link on Tips and Notes that might be of interest to you Mike, if you are still reading this site. About 56 mins viewing, well worth the objective analysis presented of government interference with public policy. Particularly in regard to the trillion $ meltdown.
As Willis and others have written on public policy and the manner in which it is informed by ‘science’ it is the taxes we pay that support this.
It is disheartening that WUWT, Antony, guest writers and the articles are not to YOUR standards of reading climate science. This website has provided many people who are not scientists (or are), can not afford pay-articles, did not previously have an interest in science these luxuries. And with the hyperlinks provided there is a choice!
Phil,
You have such a moral blind spot. What was done to O’Donnell was underhanded. It was deliberate conniving deception, no more and no less. And O’Donnell was published despite the sneaky Steig, not because of him. Publishing was simply damage control by the spineless Broccoli once the cat was out of the bag.
As Willis says, climate peer review “…assumes that the actors will act honorably and with good will. And for many years that seems to have been generally true enough to keep the system working. As Climategate showed, however, in climate science currently that is not true. Because of the false importance placed by the IPCC on peer-review, people of bad will have gained the ascendancy and gamed the system to act as gatekeepers for their particular version of science.”
We have seen the depth of dishonesty among the Climategate claque. They would easily give Bernie Madoff a run for his money, and that is no exaggeration. Lies, deception, trickery and gaming the system are their stock in trade. The wretched Eric Steig is no different from scoundrels like that lying little worm Michael Mann or the odious censor Gavin Schmidt. They should be sentenced to a long stay the penitentiary for deliberately lying to the taxpaying public about their global warming fraud. The truth is simply not in them, and you would do well defending honest folks instead of pseudo-scientific hucksters and charlatans.
DocMartyn says:
February 18, 2011 at 5:37 pm
“You have the idea that editors are impartial and that all corresponding authors should be treated in the same manner.”
I am so very happy that you concede my main point. Shame that you haven’t learned anything by doing so. Editors have the duty to be impartial and all authors and reviewers should be treated the same, yes.
“Editors cannot be impartial, they cannot be because they must know the track record of the referees they chose, normally knowing them personally.”
Track record at what? Publishing good science? Writing good reviews? Politics? Knowing them personally has nothing to do with impartiality.
“They must weigh up the fact that someone has a dog in the fight with the fact that information at the cutting edge science is asymmetric.”
There you go again, sir, with “cutting edge science.” Editors of scientific journals discover submissions containing cutting edge science maybe two or three times in a career. They might publish such more often, say five or six times, but they solicit it after learning about it over the grapevine. You are a hopeless romantic. You believe that the editor of Nature is dealing with cutting edge science daily. Fail. Worse, you believe that Steig’s work qualifies as cutting edge science. BALONEY! Get a life, son. Every honest journal editor will tell you that the vast majority of what they publish is hardly worth reading. A subscription to Nature will prove that fact to an intelligent person. What Steig’s work qualifies as is propaganda. Academic journals exist to help advance academic careers. That is a good thing because they have nothing approaching the means necessary to evaluate for truth the submissions they receive. Some good science gets published in academic journals, but that is a by-product of their reason for being.
“If an editor wants to have someone review a paper looking at brain mitochondria, nitric oxide, Fenton/Haber–Weiss you have only about six people to chose from, that’s just the way it is.”
Someone has been lying to you. Since about 1974, the number of high quality doctorates produced has greatly exceeded the number of jobs in academia or research that they might occupy. Most journals have dozens of those folks hanging around trying to network through the journal. Aside from the wealth of talent available, any journal editor is incredibly well networked. Attend a conference on science and the journal editors are the individuals followed by crowds.
“The only people who fully understood Steig et al., were Steig et al., and O’Donnell et al. That is the way it is.”
Sir, your point is relevant only if the goal of the editor is to evaluate the science. It is most definitely not. As explained in my earlier post above, the job of the editor is to determine whether the submission is worthy of publication, not whether its claims are true. If editors had the duty of policing truth, then there would be a bazillion journals and each of them would be read by about six scientists. Your ideas of how academic publishing works are all but non-existent.
Editors who evaluate the science for truth are gate keepers and they corrupt the peer review process. If scientists were to learn that an editor considered himself a gate keeper for science then that editor would be out of his job in a matter of days. Sir, the main complaint that sceptics have against climate scientists is that they do consider themselves gate keepers for science. That is what we call corrupting the peer review process. If you have inside knowledge of some editor’s practices then you are doing a good job of proving that he is the worst nightmare of climate sceptics.
In sum, you are now defending the corruption of the peer review process. You believe, wrongly, that editors do so because they have to deal with cutting edge science daily, but that is a fantasy. You believe that Steig’s work was cutting edge science, but that is roundly laughable. You are digging yourself an ever deeper hole and you cannot understand that you are embracing exactly the faults that sceptics are criticizing.
Honesty, like O’Donnell who promised that he wouldn’t reveal Steig’s identity, however:
“I did not explicitly tell Eric that I would keep the information about him being a reviewer confidential. However, he did request that I do so, and I fully intended to do so. Additionally, I explicitly told him that I would keep the reviews confidential (which are now online), and I meant that to include his identity as a reviewer.”
Or his two co-authors who were told in advance that it was unethical to publish the reviews and identify the reviewer, but they did so anyway. One of them continues in his usual disingenuous manner to weasel about it.
There was nothing underhand about the review of O’Donnell et al., they got a very thorough review, which certainly improved the paper.
Nice kickoff post by Willis and excellent energy in the comments. Willis, you do know how to throw a great blog party. : )
But I would like to move on to what I consider the question that is more fundamental to process of science journals and their processes for reviewing, approving and publishing papers.
FUNDAMENTAL QUESTION: In essence, do the current range of science journals and their processes prevent freedom of ideas/speech in the scientific communication area?
Strictly speaking, my answer is that freedom of ideas/speech in scientific communication is not blocked by the science journals. I would add that breakthrough science needs to work harder to get through the processes of science journals, but that is natural given the hierarchical nature of scientific knowledge.
John
“This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research.”
Yes, it is on the news wires. The House has voted on several bills, passing all of them by roughly 240-170, that will shut down various climate related items at NASA, the EPA, and other government agencies. No doubt Hansen is planning to unionize climate scientists and march on DC.
Phil. says:
February 18, 2011 at 6:54 pm
“There was nothing underhand about the review of O’Donnell et al., they got a very thorough review, which certainly improved the paper.”
You really should read what others have posted. Then you would not embarrass yourself.
I propose a new system of Certified Peer-to-Peer Review for Professional Climate Journals.
For some time, there has been a growing unease that submissions to Climate Journals are handled either with velvet or iron fisted gloves, depending on the attitude of the editor towards the climate debate. A root and branch reformation of the whole professional journal publishing and review process is now required.
In contrast, papers published on the web are open to comment and criticism by all. The web is international and has a vast following by experts from every field and calling. This process often gives a submission a more thorough examination, than is possible by the small group of reviewers used in the traditional publishing process. This new happening has become known as Peer-to-Peer Review.
The following draft code for climate journals, attempts to incorporate the benefits of this new process, while preserving the best traditions of the existing system. The whole procedure to be called Certified Peer-to-Peer Review.
A draft Code of Conduct, incorporating many of these concepts is now submitted for general discussion. It is not intended that this is a finished document, but rather a sketch which may eventually form the basis of a useable set of guidelines. That will require the input of a number of experienced practitioners.
Comments and suggestions for improvement are welcome at http://ausiedan.com/
The Steig paper had to heve been reviewed right? So if these reviewers were deemed “good enough” to review the Steig paper, why wouldn’t they be deemed “good enough” to review the O’Donnell rebuttal paper, why would Steig be necessary as a reviewer at all?
In rare cases, say in very new fields where not enough “experts” existed, I can understand the need to engage authors of the original paper in the review of the rebuttal paper, but why would the authors of the original paper need to be anonymous in that case?
Some commentors are suggesting annonymity protects reviewers (being asked to review a paper was an honour I would have thought) A self-damning suggestion in itself.
Steig did what he did on the blogs BECAUSE HE THOUGHT HE WAS ANONYMOUS.
Steig was caught out trying to stifle a paper that demolished his. No amount of backtracking, smoke screening or handwaving will change that.
Willis – you said about regected publications:
QUOTE
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
UNQUOTE
Now that ASSUMES that the rejection came from a “high-powered statistician” and not from a highly engaged member of some strange cult or congregation.
[trimmed.]
Sorry all,
As usual my typing and spelling are terrible.
And what the last, half paragraph meant, I really cannot imagine.
Re: “You are confusing your ‘framework’ with the ‘scientific method’. The latter is indeed unquestioned as it should be.”
Where people stop asking questions is exactly where the investigators should look. The gravitational framework is giving way to the electrical. Within a few years, we’ll be asking completely different questions. For a teaser of what’s to come …
Some of us can see where all of this is heading. The rest of you will continue to resist to learn the competing framework until the technologies are developed by others.
Mike says:
February 18, 2011 at 2:12 pm
Mike, Nature Magazine thought enough of my meaningful discussion about climate change to publish it as a peer reviewed “Brief Communications Arising” … and how many opinions of yours about climate science have been published in Nature Magazine? (I’d look it up just out of curiosity, but unfortunately I can’t, you are concealing your identity …) So while you may have a low opinion of my scientific ability, Nature Magazine disagrees. I am one of the very few amateur scientists to get anything published in Nature Magazine in the last quarter-century.
Folks: Mike is trying to use anonymous character assassination as a substitute for science … curious given the subject of the thread is anonymous review. This is a perfect example of the type of things that people do when they are anonymous, and it is one of the reasons I oppose anonymous review. I write under my own name. I am always very up front about my science, my methods, my data, and my sources. If I think something is wrong, I will say it directly to the person or people involved. I openly invite people to check and criticize my work, and I admit when I am wrong.
At the end of the day, however, whether I have published in Nature Magazine or not is immaterial. The question is, are my claims true? If you think they are not true, cheap attacks on my character or publication record are not the way to get any traction here …
w.
PS – you have not provided any evidence of the IPCC approval of the use of “gray literature” prior to the publication of AR4, which is what I had said. The cites you provided were to a recent IPCC review post from Aug. 30, 2010, and a citation to what seems like a random page of IPCC reference citations on a totally different subject …
In addition, Pachauri has said a number of times that the IPCC used only peer-reviewed literature. Here he is in 2008:
Take a look here for a bunch more Pachauri quotes on the subject. He has bragged for years, as have other high IPCC personalities, that there was nothing in the Assessment Reports but pure peer-reviewed science.
Willis: Sorry if I hurt your feelings. You are not exactly Mr. Politeness you know.
Congrats on your Nature letter. I’ll read when I get to work, and the authors’ reply.
You said: “PS – you have not provided any evidence of the IPCC approval of the use of “gray literature” prior to the publication of AR4, which is what I had said.”
[snip] There is no mention of AR4 in your comment. You said: “The IPCC has recently changed its rules for the next Assessment report to let in gray literature. For the previous reports that was not supposed to be the case. “ My link was to the a reference page in AR4.
[You are on thin ice. ~dbs]
This reminds me of when Tim Searchinger, whom the NRDC refers to as a “leading acedemic” but is merely a special interest eco-nut lawyer, was appointed to a peer review board to review… you guessed correctly, his scam hypothesis of indirect land use change. The same guy that claimed corn-for-ethanol caused an increase in illegal soybean farming in Brazil.. Remember, those “missing soybeans” from American farms weren’t really missing and the amount of exported protein increased thanks to all of those bushels of distiller’s grains enabled by the technology of grain ethanol.
http://www.princeton.edu/~tsearchi/
http://switchboard.nrdc.org/blogs/ngreene/in_face_of_hunger_corn_ethanol.html
http://www.facebook.com/topic.php?uid=26096368676&topic=8085 (original behind paywall)
Chris Reeve says:
February 18, 2011 at 10:27 pm
Where people stop asking questions is exactly where the investigators should look. The gravitational framework is giving way to the electrical.
So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.
Willis Eschenbach says:
February 19, 2011 at 12:26 am
Willis’ understanding of scientific method is rock bottom solid, stellar at its apogee, and as wide as the horizon. Willis’ explanations of scientific method are essential to the extremely intelligent and well balanced discourse found on WUWT. Aside from trolls, no one has offered a criticism of Willis’ views on scientific method. Keep up the good work, Willis. And don’t let the trolls get you down.
Re: “So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.”
I highly recommend that you watch the video.
Chris Reeve says:
February 19, 2011 at 9:49 am
I highly recommend that you watch the video.
Very entertaining, but the guy is a total nut. Now what does that make you?
[snip]
Mike says:
February 19, 2011 at 6:55 am
Mike, you accused me of knowing nothing about climate, and tried to rally others to your view. I noted that my ideas must be scientifically important, since they were published in Nature Magazine, and that yours was a superfluous personal attack … in other words, that you were acting like a jerk, and an uninformed one, to attack me on bogus grounds rather than discuss the science.
Your response to my pointing out that your actions were uninformed, unpleasant, and unscientific was to say you were sorry “if you hurt my feelings” …
Do I look like a hothouse flower to you? Do you really believe that you acting like a jerk hurts my feelings?
Well, no, Mike, acting like that is your Constitutional right, and it doesn’t affect me at all. It hurts your reputation and credibility, particularly when you think that the issue is whether you hurt my feelings. Nor is your reputation or credibility enhanced when you don’t apologize for your actions, but instead seem to think that the issue is my feelings.
w.
PS: on a more substantive note:
Seems like there’s a misunderstanding here, I’m not sure. Let me restate my point.
For the first four IPCC Assessment reports (FAR, SAR, TAR, AR4) the claim was made, over and over, by Pachauri and others, that the IPCC reports contained only and solely peer-reviewed science. This unusual situation, unknown in other sciences, led to a huge press by AGW supporters to deny peer-review to their opponents’ science.
This claim of 100% peer-review turned out to be an IPCC lie. Nothing that you have provided has contradicted either of those facts. It was, for example, the inclusion of an NGO’s puff piece on the Amazon (in the guise of peer-reviewed science) that led to “Amazongate”.
PPS – For future reference, a proper apology would be on the order of “Gosh, Willis, I guess you must know something about climate science after all. Sorry for my ad hominem attack, I was out of line.” Then I say “That’s OK, Mike, I’ve gone off the rails myself more than once”, and we move forwards.
Saying you’re sorry that you hurt my feelings and adding that you’ll read my Nature article and the response later (what … you gonna grade me on my work?), on the other hand, means you still don’t see that the issue is not my feelings or your judgement of the quality of my Nature piece. The issue is, you acted improperly when you accused me of being clueless about climate science and tried to use your foolish claim to get people to ignore my work … not nice, Mike, not nice at all.
Chris Reeve says:
February 18, 2011 at 10:27 pm
Chris, this is so far off topic that I don’t know how to characterize it. Please take it to a thread that has something to do with whatever it is that the guy you like is on about. This thread is discussing peer review.
Many thanks,
w.
Re: “Very entertaining, but the guy is a total nut. Now what does that make you?”
Classic response. You’ve previously disagreed that the van der waals force can explain the solid and liquid states of matter as well. These experimental results are destroying the conventional assumption (which is absolutely necessary to the gravitational framework) that charge separation is rare in the universe. We teach astrophysicists right now assumptions which directly contradict this experimentation.
Re: “Chris, this is so far off topic that I don’t know how to characterize it. Please take it to a thread that has something to do with whatever it is that the guy you like is on about. This thread is discussing peer review.”
No problem, I’ll wait until the next time that you guys talk about water.
Re: “So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.”
You’ve certainly summed up the longstanding approach inherent to our current scientific methodology: Assume that electricity is always contained within a box, wherever it is witnessed.
And where you see magnetic fields permeating space on the largest observable scales, do NOT infer an electrical current cause. In fact, “new physics” causes are preferred by gravitationalists over electrical current causes, as an explanation for the cosmic variety of magnetic fields. They’d rather propose unverified — oftentimes unverifiable — hypothetical processes for maintaining cosmic magnetic fields, than even consider the electrical cause which would be taken for granted in the laboratory.
There is nothing philosophical about this methodology. It’s designed specifically to suit the framework itself.
This unphysical preference for an inference is a direct result of the deductive Socratic-dialectic methodology which is taught to scientists everywhere, as otherwise, there would exist no heresy within the peer review system in proposing a far more physical basis (electric currents) for the magnetic fields which we observe to permeate both galaxies and even intergalactic space.
Making matters worse, after more than enough time, money and effort invested into Einstein’s “thought experiment,” we now know that this line of reasoning leads us to a dead end, with a universe which is 95% invisible, hypothetical matter.
Deduction is possibly necessary to scientific investigation, but at what point do scientists yearn for a more physical model, which they can use to make actual predictions with? If people want to pretend as though climate theory is unaffected by all of this, then they are in denial. Climate theory assumes the framework, just like every other single scientific discipline.
Fixing some minor aspect of peer review is not going to have this huge impact on science that the author of this original article proposes. The real problems in science can be plainly seen in the philosophical approach itself — which currently prioritizes those theories which can be arrived at through deduction from the original conjecture over those theories which can claim to be more physical than the consensus view.
I think that this video helps to provide a very concrete example for why our scientific methodology is seriously broken: Clearly, we are asking the wrong questions, if charge separation occurs in a cup of water exposed to light. The astrophysicists never saw this coming, and they will surely now try to ignore it as best they can.
Okay, I’ll now defer to those who prefer to imagine that this conversation is unrelated to peer review.
Chris Reeve says:
February 19, 2011 at 12:44 pm
These experimental results are destroying the conventional assumption (which is absolutely necessary to the gravitational framework) that charge separation is rare in the universe. We teach astrophysicists right now assumptions which directly contradict this experimentation.
On the contrary, charge separation happens a lot as is evidenced in all the electric currents that we see cause explosions all over the place. What you miss is that that charge separation is caused by plasma moving across magnetic fields, and that the separation doesn’t last long as it blows up all the time [in this case X=flares].
Steig had published a ground-breaking paper in a prestiguous that claimed GW could now be detected in significant parts of Antarctica. Along comes a paper from O’Donnell et al claiming that a more appropriate analysis of the data changes Steig’s conclusion. Would comments from Steig be useful to the editor, Broccoli, and other reviewers in their evaluation of O’Donnell’s work? Absolutely. Does Steig deserve a vote on whether O’Donnell’s paper should be published and what changes were necessary. Absolutely not.
Willis, yet again a great article. Sorry to arrive so late. Not just your article but your responses and your evidence of shifting in response to some other peoples’ responses. And not just your responses but everyone’s responses; as someone said, you throw great blog parties.
Having said which, I still, surprisingly, have a measure of disagreement with your judgement of Broccoli and Nielsen-Gammon. I think that here we have typical situations where people who have risen to power in the orthodox hierarchy are governed to some degree by the Stockholm Syndrome: without which, they would simply not even be editors in positions from which they can get O10 published. Is Gorbachev that different? I think that publication of O10 is a great step forward, albeit at cost, and I think most people on both sides would agree with that.
Of course, I agree passionately with your own point of view, as well. Strong, reasonably courteous, free of Stockholm Syndrome, and basically righteous, it is also doing one of the things most needed: opening up the debate.
Chris Reeves: your material is too important for you to let yourself get off topic’d. Work more carefully from the topic subject matter to your perceived consequences. And pay attention carefully to Scott: as an electrical engineer I trust him but I do not think he is right about Socrates. IMHO. Watch the nuances.
Chris Reeve says:
February 19, 2011 at 1:21 pm
Clearly, we are asking the wrong questions, if charge separation occurs in a cup of water exposed to light. The astrophysicists never saw this coming, and they will surely now try to ignore it as best they can.
Einstein got his Nobel Prize for his explanation of the photoelectric effect where charge separation occurs when a metal surface is exposed to ultraviolet light. Solar panels work in a similar [but slightly different] way. I think you better study up a bit, before you partake in further philosophical OT-discussion.
Re: “On the contrary, charge separation happens a lot as is evidenced in all the electric currents that we see cause explosions all over the place. What you miss is that that charge separation is caused by plasma moving across magnetic fields, and that the separation doesn’t last long as it blows up all the time [in this case X=flares].”
Lief, it’s apparently now being observed to occur in a cup of water hit by light. I refer you to the experimental validations for this *observation* which have been presented by the speaker in the video, using several different types of imaging. The photographs Pollack presented of the exclusion zone using pH-sensitive dye are more than sufficient to demonstrate that the hydrogen ions are not equally distributed in water. But, he went even several steps further than that in validating that the molecules in question were not spinning, and that they also exhibited reduced infrared emissions in the “exclusion zone” (ie, they are less “thermal”).
The totality of the evidence presented in that video supports the inference of the van der waals force as the primary actor. What remains to be explained is why the van der waals force which creates these “exclusion zones” on hydrophobic surfaces offers such an efficient mechanism for converting photons into electrons. It could possibly have to do with the geometry of the electron drift through that liquid crystal array. There might just be fewer collisions … ?
It’s also worth mentioning that Wal’s theory for gravity involves a process which is fundamentally akin to the van der waals (whereby the electrons themselves become dipolar). This same force is also implicated in the solid and liquid states of matter, such as …
“Collectively known as van der Waals forces, dispersion forces, dipole-dipole forces, and dipole–induced dipole forces affect such physical properties as melting points and boiling points.”
Van der waals is, most simply, an electromagnetic resonance which is disrupted by thermal activity. It is especially prominent in water because water is especially dipolar.
But, of course, in your conventional view, none of these questions and lines of reasoning are worth our time, and we must skip over this amazing claim for the underlying process for photosynthesis, as it is a diversion from our attempts to validate the gravitational framework. In my view, this has stopped being a search for truth; it is now solely a quest to prove the conventional theories, to the detriment of any good idea which stands in the way of conventional wisdom.
Our approach to science entirely determines the questions that we ask. That we are now talking about a universe where charge separation is presumably happening in every single drop of water in it should clearly have very profound impact upon how we view the universe. And it should similarly cause us to seriously reflect upon how we are approaching science.
A curious person would wonder what it is that is preventing the charge recombination near the edge of water? But, this is arguably no less of a mystery than water tension itself! And so, we must ask ourselves: Why have we not spent more time investigating water’s inherent tension? In a gravity-centric universe, water’s tension is simply not important. And this is the exact problem of choosing just a single conjecture to deduce from.
And I’d like to point out that people have been having this conversation over electricity in space, in various manifestations, since the time of Hannes Alfven more than a half-century ago. That should indicate the risk inherent to assuming one side of the debate as our starting conjecture for formulating the framework which underpins the *entirety* of scientific discourse.
Talk about a risky gamble! The assumptions which serve as our starting point for all our scientific investigation should *never* have a half-century of baggage associated with them!
Re: “Einstein got his Nobel Prize for his explanation of the photoelectric effect where charge separation occurs when a metal surface is exposed to ultraviolet light. Solar panels work in a similar [but slightly different] way. I think you better study up a bit, before you partake in further philosophical OT-discussion.”
Why does the charge not recombine in the water, Lief?
Chris Reeve says:
February 19, 2011 at 3:40 pm
Why does the charge not recombine in the water, Lief?
You tell me, but better in some other forum, where it is on topic.
Lucy Skywalker says:
February 19, 2011 at 1:44 pm
“I think that here we have typical situations where people who have risen to power in the orthodox hierarchy are governed to some degree by the Stockholm Syndrome: without which, they would simply not even be editors in positions from which they can get O10 published. Is Gorbachev that different?”
Very interesting thought. I have tended to think that individual editors have chosen the dark side for their own reasons. Maybe I was wrong. Your view is rather more frightening than what I have been thinking.
“I think that publication of O10 is a great step forward, albeit at cost, and I think most people on both sides would agree with that.”
Yes, except on your view why it happened is inexplicable on rational grounds. That is not a criticism. I will be thinking about what you have said.
Frank says:
February 19, 2011 at 1:35 pm
“Steig had published a ground-breaking paper in a prestiguous that claimed GW could now be detected in significant parts of Antarctica.”
Could we please stop repeating this “ground-breaking paper” absurdity? If the editor had consulted a competent statistician, the paper would have been rejected on that basis alone.
Truly, Chris and Leif, discussion on the Van Der Waals force and the like are most definitely off topic. I say again, please discuss this on an appropriate thread.
Thanks again,
w.
******
REPLY Hah! They’ll have to dig into their own pockets if they want to say anything!
This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research. The original cuts to the budget outlined yesterday would have cut $379 million from NASA’s budget. These members want climate out of NASA’s purview entirely, however. Funding climate research, said Adams in a statement, “undercuts one of NASA’s primary and most important objectives of human spaceflight.”
******
NASA has virtually no business sucking from the climate-change trough. Cut everything concerning climate-change, and redirect that to developing unmanned probes to Titan, Europa, Enceladus, Mars, Venus, nearby comets & asteroids, etc. To me, these types of missions have by far the most bang-for-the-buck.
I got Wired magazine for 1 year (about 3 or 4 years ago), and in one of the issues was the idea of having all scientific studies published. To me it made sense that you wanted the data and the methodology out there for everyone to see even if it wasn’t good enough to be in a 1st tier magazine. Basically because the studies aren’t published, we loose knowledge and sometimes rediscover it and sometimes don’t.
(yes I no longer get wired in part because their cool tech items are not sufficiently interesting to out weigh their crappy leftist bent)
As to NASA and Climate Research, I’m torn on this, on the one hand if their mission is getting manned space flight, then putting statalites in orbit helps them practice rocket shots. On the other hand, should we leave something as important as getting off planet to the Fed gov’t?
very interesting write up.
AC says:
February 20, 2011 at 7:50 am
I got Wired magazine for 1 year (about 3 or 4 years ago), and in one of the issues was the idea of having all scientific studies published.
The problem is: what is a scientific study? Uncle Al’s crazy scheme for unlimited energy from oxygenated water? Or Roy Spencer’s paper on negative feedback? or…