Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
IPCC scientists? Don’t make us laugh.
DocMartyn says:
February 18, 2011 at 12:10 pm
“Steig et al., was published and published in Science, one of the highest ranked journals.”
Utterly, totally, abominably irrelevant.
“O’Donnell et al., wrote a paper that challenged the methodology and conclusions of Steig et al.,, not the data from which the conclusions were drawn.”
A distinction without a difference. Data can be criticized. Methodology can be criticized.
“The whole of Steig et al., which was being critiqued by O’Donnell et al., was the authors analysis.”
Utterly, totally, abominably irrelevant.
“First author generally does the most work and last author generally has the most money/seniority. ”
Utterly, totally, abominably irrelevant.
“Steig was the front man and the Editor had a duty towards the papers authors, he had to make sure that O’Donnell et al., critiques were fair.”
I guess this is it. There is no such duty. The editor has the duty to be impartial. That means, has always meant, and will always mean staying out of the argument. The editor has a duty not to attempt to balance the scales between author and reviewer or to influence in any way the relationship between author and reviewer. To do anything it to violate impartiality. Your problem is that you have this bizarre idea that the science is supposed to get done among author, editor, and reviewers. That is manifestly not the case. The editor is supposed to guarantee impartiality among reviewers whose job is to (1) find blunders before they become embarrassing to the journal, (2) ensure that the essay has some appeal to some scientists in the field, and (3) ensure that the article is reasonably well written, by the low standards used for scientists.
“The experts, for the defense, were Steig and co-workers.”
Listen at you! Are you insane, sir? This is not a court proceeding. This is not even an arena in which science should be done. This is one more among gazillions of boring, ordinary reviews for a journal. It must be carried out by boring, ordinary reviewers who have no stake in the outcome. And it must be presented to an editor who is going about his daily grind of boring, ordinary activities, and who has no stake in the outcome.
“The editor asked the front man to examine the manuscript. Twice Steig reviewed, after which the editor ignored his input.”
No, no, no! You neglect the time wasted and the wear and tear on the nerves of the O’Donnell folks. This was punishment. Also, it delayed Steig’s reckoning.
“He gave the Steig team a heads up of incoming criticism and allowed them to find any major faults.”
What the hell does that mean? Do you even know? It sounds as if the editor gave Steig a WARNING! Do you not know how wrong it would be for an editor to do such a thing?
“They found none. So Broccoli gave the Steig team first bite of the cherry and Broccoli gave O’Donnell et al., a baptism in fire and it survived.”
Yes, sir, you are insane or nearly so. You are so full of the hubris and grandiosity of “The Team” that you believe that it is reasonable to say that an editor gave an author a “baptism of fire.” You have confirmed in your own words, that the whole world can read, all of my suspicions of you and my accusations against you. Every editor knows that he has the duty to treat authors impartially; that is, treat them all the same. Sir, you reveal that you do not know the meaning of the word ‘impartiality’. So what are you doing here writing about it?
Any editor who takes it upon himself to conduct a “baptism of fire” is acting beyond any and all standards of responsibility, drunk on his grandiosity, a danger to society, and certainly should occupy no position of responsibility including that of dogcatcher. Sir, you are a symptom of what most ails climate science at this time, namely, a degree of hubris and grandiosity yet unknown to mankind.
In sum, the only reading of your words that makes sense is that the editor manipulated the review process for his own ends, including placing arbitrary burdens on O’Donnell and coauthors, and gave free reign to Steig to raise hell with O’Donnell. When he was finished playing cat and mouse, the editor stopped the nonsense and published the paper that should have been published months earlier.
Philip Shehan says:
February 18, 2011 at 10:58 am
Willis Eschenbach says:
February 18, 2011 at 1:03 am
Umm … Doc, you do understand why Broccoli had to bring in an unbiased referee? Because the previous referee was biased, duh. So Broccoli brought in a biased referee, finally had enough of him, and brought in an unbiased referee … and you want to give him the “best of class”???
Well that’s a mischaracterization of what happened. After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.
An 88 page referees report does seem somewhat over the top, but for all I know the points raised were entirely legitmate and raising points for clarification or amendment is not the same as rejection.
Of course that didn’t happen, it was a 14 page review!
Phil. says:
“After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.”
Oh. Well, I guess that makes it A-OK then.
Smokey says:
February 18, 2011 at 4:32 pm
Phil. says:
“After the first review the authors complained that reviewer A was one of the authors of Steig et al. so Broccoli added reviewer D but still kept reviewer A.”
Oh. Well, I guess that makes it A-OK then.
Yes it sure does, nothing to complain about just some neophyte authors moaning.
Note the following from Broccoli:
“To allay some of the concerns you have expressed about Rev. A, I have sought the advice of an additional reviewer (Rev. D). Please note that several of Rev. D’s comments are similar to points made by Rev. A and thus warrant especially close attention.”
Theo Godwin: Since you are presenting yourself such an expert in what the proper role of editors and reviewers in the peer-review process, it would be nice to know where your expertise in this matter is derived from.
In my case, it is from having been a referee on over 80 articles in the last 19 years submitted to physics journals (primarily the Physical Review journals)…and having been a published author of about 30 papers in peer-reviewed journals. I find it a little strange for people to be pontificating on what this without explaining how they have arrived at their expertise in this area and how they know, for example, that somehow what happens in climate science is dramatically different than what occurs in other physical science fields.
Theo, take a pill man
“Your problem is that you have this bizarre idea that the science is supposed to get done among author, editor, and reviewers. That is manifestly not the case. ”
This is not science.
Science is the modeling of the universe around us to develop a coherent description of the underlying rules that govern matter.
What you are talking about is the propagation of scientific discourse.
You have the idea that editors are impartial and that all corresponding authors should be treated in the same manner.
Editors cannot be impartial, they cannot be because they must know the track record of the referees they chose, normally knowing them personally. They must weigh up the fact that someone has a dog in the fight with the fact that information at the cutting edge science is asymmetric.
If an editor wants to have someone review a paper looking at brain mitochondria, nitric oxide, Fenton/Haber–Weiss you have only about six people to chose from, that’s just the way it is.
The only people who fully understood Steig et al., were Steig et al., and O’Donnell et al. That is the way it is.
Yawn.
Gentlemen, you are arguing about particular details of the process that is fundamentally flawed in principle.
In a Maoist society, any IPCC-style or Wikipedia-style selection process will result in Maoist-only articles selection. Substitute “Maoist” with any prevailing ideology, and there you have it.
Or would you insist that there is no prevailing ideology in IPCC or Wikipedia? Then all you deserve would be a silent smile of reason, the one that silly children see sometimes on the faces of their grandparents.
EJ says:
February 17, 2011 at 9:25 pm
Too bad scientists no longer need a reputation.
Ehrlic and Holdren, the science Czar, included.
Apparently a pHd can still say anything, hide the data, and fudge the findings.
Wow
————
REPLY Hah! They’ll have to dig into their own pockets if they want to say anything!
This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research. The original cuts to the budget outlined yesterday would have cut $379 million from NASA’s budget. These members want climate out of NASA’s purview entirely, however. Funding climate research, said Adams in a statement, “undercuts one of NASA’s primary and most important objectives of human spaceflight.”
http://motherjones.com/blue-marble/2011/02/republican-climate-nasa-budget
Mike says: February 18, 2011 at 2:12 pm
Here is a site that discusses the use of grey literature in the 2007 IPCC report and how some came to be developed and/or promulgated
http://nofrakkingconsensus.blogspot.com/2010/01/greenpeace-and-nobel-winning-climate_28.html
Read more: “Dr Pachuri……..IPCC studies only peer-review science. Let someone publish data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it in a dustbin.”
http://timesofindia.indiatimes.com/india/No-proof-of-Himalayan-ice-melting-due-to-climate-change/articleshow/5213045.cms
I’ve posted a link on Tips and Notes that might be of interest to you Mike, if you are still reading this site. About 56 mins viewing, well worth the objective analysis presented of government interference with public policy. Particularly in regard to the trillion $ meltdown.
As Willis and others have written on public policy and the manner in which it is informed by ‘science’ it is the taxes we pay that support this.
It is disheartening that WUWT, Antony, guest writers and the articles are not to YOUR standards of reading climate science. This website has provided many people who are not scientists (or are), can not afford pay-articles, did not previously have an interest in science these luxuries. And with the hyperlinks provided there is a choice!
Phil,
You have such a moral blind spot. What was done to O’Donnell was underhanded. It was deliberate conniving deception, no more and no less. And O’Donnell was published despite the sneaky Steig, not because of him. Publishing was simply damage control by the spineless Broccoli once the cat was out of the bag.
As Willis says, climate peer review “…assumes that the actors will act honorably and with good will. And for many years that seems to have been generally true enough to keep the system working. As Climategate showed, however, in climate science currently that is not true. Because of the false importance placed by the IPCC on peer-review, people of bad will have gained the ascendancy and gamed the system to act as gatekeepers for their particular version of science.”
We have seen the depth of dishonesty among the Climategate claque. They would easily give Bernie Madoff a run for his money, and that is no exaggeration. Lies, deception, trickery and gaming the system are their stock in trade. The wretched Eric Steig is no different from scoundrels like that lying little worm Michael Mann or the odious censor Gavin Schmidt. They should be sentenced to a long stay the penitentiary for deliberately lying to the taxpaying public about their global warming fraud. The truth is simply not in them, and you would do well defending honest folks instead of pseudo-scientific hucksters and charlatans.
DocMartyn says:
February 18, 2011 at 5:37 pm
“You have the idea that editors are impartial and that all corresponding authors should be treated in the same manner.”
I am so very happy that you concede my main point. Shame that you haven’t learned anything by doing so. Editors have the duty to be impartial and all authors and reviewers should be treated the same, yes.
“Editors cannot be impartial, they cannot be because they must know the track record of the referees they chose, normally knowing them personally.”
Track record at what? Publishing good science? Writing good reviews? Politics? Knowing them personally has nothing to do with impartiality.
“They must weigh up the fact that someone has a dog in the fight with the fact that information at the cutting edge science is asymmetric.”
There you go again, sir, with “cutting edge science.” Editors of scientific journals discover submissions containing cutting edge science maybe two or three times in a career. They might publish such more often, say five or six times, but they solicit it after learning about it over the grapevine. You are a hopeless romantic. You believe that the editor of Nature is dealing with cutting edge science daily. Fail. Worse, you believe that Steig’s work qualifies as cutting edge science. BALONEY! Get a life, son. Every honest journal editor will tell you that the vast majority of what they publish is hardly worth reading. A subscription to Nature will prove that fact to an intelligent person. What Steig’s work qualifies as is propaganda. Academic journals exist to help advance academic careers. That is a good thing because they have nothing approaching the means necessary to evaluate for truth the submissions they receive. Some good science gets published in academic journals, but that is a by-product of their reason for being.
“If an editor wants to have someone review a paper looking at brain mitochondria, nitric oxide, Fenton/Haber–Weiss you have only about six people to chose from, that’s just the way it is.”
Someone has been lying to you. Since about 1974, the number of high quality doctorates produced has greatly exceeded the number of jobs in academia or research that they might occupy. Most journals have dozens of those folks hanging around trying to network through the journal. Aside from the wealth of talent available, any journal editor is incredibly well networked. Attend a conference on science and the journal editors are the individuals followed by crowds.
“The only people who fully understood Steig et al., were Steig et al., and O’Donnell et al. That is the way it is.”
Sir, your point is relevant only if the goal of the editor is to evaluate the science. It is most definitely not. As explained in my earlier post above, the job of the editor is to determine whether the submission is worthy of publication, not whether its claims are true. If editors had the duty of policing truth, then there would be a bazillion journals and each of them would be read by about six scientists. Your ideas of how academic publishing works are all but non-existent.
Editors who evaluate the science for truth are gate keepers and they corrupt the peer review process. If scientists were to learn that an editor considered himself a gate keeper for science then that editor would be out of his job in a matter of days. Sir, the main complaint that sceptics have against climate scientists is that they do consider themselves gate keepers for science. That is what we call corrupting the peer review process. If you have inside knowledge of some editor’s practices then you are doing a good job of proving that he is the worst nightmare of climate sceptics.
In sum, you are now defending the corruption of the peer review process. You believe, wrongly, that editors do so because they have to deal with cutting edge science daily, but that is a fantasy. You believe that Steig’s work was cutting edge science, but that is roundly laughable. You are digging yourself an ever deeper hole and you cannot understand that you are embracing exactly the faults that sceptics are criticizing.
Honesty, like O’Donnell who promised that he wouldn’t reveal Steig’s identity, however:
“I did not explicitly tell Eric that I would keep the information about him being a reviewer confidential. However, he did request that I do so, and I fully intended to do so. Additionally, I explicitly told him that I would keep the reviews confidential (which are now online), and I meant that to include his identity as a reviewer.”
Or his two co-authors who were told in advance that it was unethical to publish the reviews and identify the reviewer, but they did so anyway. One of them continues in his usual disingenuous manner to weasel about it.
There was nothing underhand about the review of O’Donnell et al., they got a very thorough review, which certainly improved the paper.
Nice kickoff post by Willis and excellent energy in the comments. Willis, you do know how to throw a great blog party. : )
But I would like to move on to what I consider the question that is more fundamental to process of science journals and their processes for reviewing, approving and publishing papers.
FUNDAMENTAL QUESTION: In essence, do the current range of science journals and their processes prevent freedom of ideas/speech in the scientific communication area?
Strictly speaking, my answer is that freedom of ideas/speech in scientific communication is not blocked by the science journals. I would add that breakthrough science needs to work harder to get through the processes of science journals, but that is natural given the hierarchical nature of scientific knowledge.
John
“This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research.”
Yes, it is on the news wires. The House has voted on several bills, passing all of them by roughly 240-170, that will shut down various climate related items at NASA, the EPA, and other government agencies. No doubt Hansen is planning to unionize climate scientists and march on DC.
Phil. says:
February 18, 2011 at 6:54 pm
“There was nothing underhand about the review of O’Donnell et al., they got a very thorough review, which certainly improved the paper.”
You really should read what others have posted. Then you would not embarrass yourself.
I propose a new system of Certified Peer-to-Peer Review for Professional Climate Journals.
For some time, there has been a growing unease that submissions to Climate Journals are handled either with velvet or iron fisted gloves, depending on the attitude of the editor towards the climate debate. A root and branch reformation of the whole professional journal publishing and review process is now required.
In contrast, papers published on the web are open to comment and criticism by all. The web is international and has a vast following by experts from every field and calling. This process often gives a submission a more thorough examination, than is possible by the small group of reviewers used in the traditional publishing process. This new happening has become known as Peer-to-Peer Review.
The following draft code for climate journals, attempts to incorporate the benefits of this new process, while preserving the best traditions of the existing system. The whole procedure to be called Certified Peer-to-Peer Review.
A draft Code of Conduct, incorporating many of these concepts is now submitted for general discussion. It is not intended that this is a finished document, but rather a sketch which may eventually form the basis of a useable set of guidelines. That will require the input of a number of experienced practitioners.
Comments and suggestions for improvement are welcome at http://ausiedan.com/
The Steig paper had to heve been reviewed right? So if these reviewers were deemed “good enough” to review the Steig paper, why wouldn’t they be deemed “good enough” to review the O’Donnell rebuttal paper, why would Steig be necessary as a reviewer at all?
In rare cases, say in very new fields where not enough “experts” existed, I can understand the need to engage authors of the original paper in the review of the rebuttal paper, but why would the authors of the original paper need to be anonymous in that case?
Some commentors are suggesting annonymity protects reviewers (being asked to review a paper was an honour I would have thought) A self-damning suggestion in itself.
Steig did what he did on the blogs BECAUSE HE THOUGHT HE WAS ANONYMOUS.
Steig was caught out trying to stifle a paper that demolished his. No amount of backtracking, smoke screening or handwaving will change that.
Willis – you said about regected publications:
QUOTE
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
UNQUOTE
Now that ASSUMES that the rejection came from a “high-powered statistician” and not from a highly engaged member of some strange cult or congregation.
[trimmed.]
Sorry all,
As usual my typing and spelling are terrible.
And what the last, half paragraph meant, I really cannot imagine.
Re: “You are confusing your ‘framework’ with the ‘scientific method’. The latter is indeed unquestioned as it should be.”
Where people stop asking questions is exactly where the investigators should look. The gravitational framework is giving way to the electrical. Within a few years, we’ll be asking completely different questions. For a teaser of what’s to come …
Some of us can see where all of this is heading. The rest of you will continue to resist to learn the competing framework until the technologies are developed by others.
Mike says:
February 18, 2011 at 2:12 pm
Mike, Nature Magazine thought enough of my meaningful discussion about climate change to publish it as a peer reviewed “Brief Communications Arising” … and how many opinions of yours about climate science have been published in Nature Magazine? (I’d look it up just out of curiosity, but unfortunately I can’t, you are concealing your identity …) So while you may have a low opinion of my scientific ability, Nature Magazine disagrees. I am one of the very few amateur scientists to get anything published in Nature Magazine in the last quarter-century.
Folks: Mike is trying to use anonymous character assassination as a substitute for science … curious given the subject of the thread is anonymous review. This is a perfect example of the type of things that people do when they are anonymous, and it is one of the reasons I oppose anonymous review. I write under my own name. I am always very up front about my science, my methods, my data, and my sources. If I think something is wrong, I will say it directly to the person or people involved. I openly invite people to check and criticize my work, and I admit when I am wrong.
At the end of the day, however, whether I have published in Nature Magazine or not is immaterial. The question is, are my claims true? If you think they are not true, cheap attacks on my character or publication record are not the way to get any traction here …
w.
PS – you have not provided any evidence of the IPCC approval of the use of “gray literature” prior to the publication of AR4, which is what I had said. The cites you provided were to a recent IPCC review post from Aug. 30, 2010, and a citation to what seems like a random page of IPCC reference citations on a totally different subject …
In addition, Pachauri has said a number of times that the IPCC used only peer-reviewed literature. Here he is in 2008:
Take a look here for a bunch more Pachauri quotes on the subject. He has bragged for years, as have other high IPCC personalities, that there was nothing in the Assessment Reports but pure peer-reviewed science.
Willis: Sorry if I hurt your feelings. You are not exactly Mr. Politeness you know.
Congrats on your Nature letter. I’ll read when I get to work, and the authors’ reply.
You said: “PS – you have not provided any evidence of the IPCC approval of the use of “gray literature” prior to the publication of AR4, which is what I had said.”
[snip] There is no mention of AR4 in your comment. You said: “The IPCC has recently changed its rules for the next Assessment report to let in gray literature. For the previous reports that was not supposed to be the case. “ My link was to the a reference page in AR4.
[You are on thin ice. ~dbs]
This reminds me of when Tim Searchinger, whom the NRDC refers to as a “leading acedemic” but is merely a special interest eco-nut lawyer, was appointed to a peer review board to review… you guessed correctly, his scam hypothesis of indirect land use change. The same guy that claimed corn-for-ethanol caused an increase in illegal soybean farming in Brazil.. Remember, those “missing soybeans” from American farms weren’t really missing and the amount of exported protein increased thanks to all of those bushels of distiller’s grains enabled by the technology of grain ethanol.
http://www.princeton.edu/~tsearchi/
http://switchboard.nrdc.org/blogs/ngreene/in_face_of_hunger_corn_ethanol.html
http://www.facebook.com/topic.php?uid=26096368676&topic=8085 (original behind paywall)
Chris Reeve says:
February 18, 2011 at 10:27 pm
Where people stop asking questions is exactly where the investigators should look. The gravitational framework is giving way to the electrical.
So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.