Guest Post by Willis Eschenbach
The recent problems with the publication of the O’Donnell et al. response to the Steig et al. paper on Antarctica have focused attention on continuing problems with the current system of peer review, problems initially highlighted by the CRU emails. In addition to significant questions revealed in this particular case, I’d like to look at other general issues with peer review.
For me, the most inexplicable and interesting part of the Steig/O’Donnell affray has nothing to do with the scientific questions. It also has nothing to do with the actions of Steig or O’Donnell, actions which have much exercised discussion of scientific and personal ethics on the blogosphere. It also has nothing to do with Antarctica, or with statistics.
The inexplicable part to me was that Dr. Steig was named as a reviewer of the O’Donnell paper by the Journal Editor, Dr. Anthony Broccoli.
It was inexplicable because in the ancient tradition of adversarial science, the O’Donnell paper claimed that there were serious issues with the Steig methods. That being the case, the very last person to be given any say as to whether the paper should be published is Steig. If it were my Journal, I would have immediately called Dr. Broccoli, the Editor, on the carpet to explain such an egregious breach of both the journal policy and more importantly, common sense. Appointing Steig as a reviewer is contrary to the stated policies of the journal, which say:
A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
Having Steig as a reviewer was done even though the authors of the O’Donnell paper wrote directly to the Editor (Broccoli) wrote to ask that Steig “be treated as a conflicted reviewer or that his review, at least, be sent to unconflicted reviewers for consideration before requiring us to make more major revisions.” The exact wording of the request was:
We have several concerns that we feel do not belong in the response and are more appropriately expressed in a letter. With this in mind, we would like to take a few moments of your time to discuss them. First, it is quite clear that Reviewer A is one (or more) of the authors of S09. This results in a conflict of interest for the reviewer when examining a paper that is critical of their own. This conflict of interest is apparent in the numerous misstatements of fact in the review. The most important of these were: …
This request was ignored by the Editor.
Steven Mosher had an interesting comment on this issue:
What makes this case different from any other “conflicted” reviewer case I’ve seen is this: Steig had made a public challenge to meet the author on the battlefield of peer reviewed literature. And in the case of Ryan [O’Donnell] this is an author who has no track record. That kind of challenge has no analogue that I’ve ever seen. Let’s see if I can make one
Imagine, for example, that you are a grad student with zero publications.
Imagine you make a pointed criticism or two of Judith Curry at a public forum, say an AGU Keynote. Imagine that Judy responds to you by saying, “go ahead try to get that published kid”
If you were that kid would you feel it was appropriate to have Judith review the paper? Would you have any reason to wonder if she was doing more than defending the science if as reviewer she gave you a hard time? Heck, even taking the reviewer assignment would be a sign to you that she intended to defend two things: her published paper and her public challenge/reputation.
Even beyond the special issues in this particular case highlighted by Steven Mosher, using a reviewer with such a glaring conflict of interest is also contrary to more general policies on conflicts of interest, such as the policy of the International Committee of Medical Journal Editors:
Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest–for example, those who work in the same department or institution as any of the authors.
While this seems clear to me, and likely to you, Dr. Broccoli seems not to have gotten the memo.
Please be clear that I am not saying that Steig should not be offered every opportunity to respond to the issues raised by O’Donnell et al. He should indeed be offered that. The normal way that this is done would be for the Journal Editor to give space (usually in the issue where the new paper is published) for Steig to respond to the issues.
But giving Steig a say of any kind in whether the paper should be published? Where is the common sense in that? Does anyone seriously believe that in that position, some scientists would not try to prevent the publication of the new paper? Human nature roolz last time I looked …
I have seen Dr. Broccoli’s actions defended in the blogosphere, usually by saying that the Editor will use their expert judgement to determine if a reviewer is engaged in gatekeeping behavior. They also say that the most knowledgeable person about a paper is likely the author, so the Editor needs their specialized knowledge.
The problem that I have with that idea is, if the Editor is so knowledgeable about the statistical issues in question that he can distinguish Steig’s gatekeeping from true claims, then why does he need Steig as a reviewer?
And if the Editor is not knowledgeable in the statistical questions involved (Dr. Broccoli is a climate modeller, not a statistician … nor is Steig a statistician for that matter), then he won’t have the knowledge to see whether Steig is gatekeeping or not.
Also, if the Editor is that good and knowledgeable, then why do scientific journals (including Dr. Broccoli’s journal) have policies strongly discouraging reviewers with conflicts of interest?
And even if the Editor is that knowledgeable (which Dr. Broccoli seems not to be), remember that the goal is to avoid even the “appearance of a conflict of interest” … just how did Dr. Broccoli decide that having Steig as a goalkeeper does not present the “appearance of a conflict of interest”? My grandma could see that conflict of interest from her current residence … and she’s been dead for fifty years.
This farrago shows once again, just as was shown in the CRU emails revealed by Climategate, that peer review for AGW scientists is far too often “pal review” – just a gatekeeping fiction to keep any kind of opposing views from seeing the light of day, and to give puffball reviews to AGW supporting papers. Yes, as a number of people have said, at the end of the day the system kinda sorta worked, with a crippled paper (e.g. no Chladni patterns) emerging from the process. But I can say from my own experience that sometimes it ends up with a paper going in the trash can, purely because of gatekeeping from AGW pal review.
And in any case, is that all that scientists are asking for? A system that kinda sorta works some of the time? Because that’s certainly not what the public either wants or expects.
My suggestions to make peer review a better system are:
• Double blind reviews, where neither the reviewers nor the author are aware of each others’ identities. At present this is true in some journals but not others.
• All reviews get published with the paper, with each one signed by the responsible reviewer.
This has a number of advantages over the current system:
1. Reviewers comments become part of the record. This is very important, as for example a minority review which is outvoted to get the paper published may contain interesting objections and other ideas. Or a favorable review can immediately be seen to be based on false logic.
2. Gatekeeping and conflicts of interest of the kind favored by Dr. Broccoli will be immediately apparent.
3. While it is sometimes possible for authors or reviewers to guess each others’ identities, at least it will only be a guess.
4. As the experience of the internet shows, anonymity does not encourage honesty or collegialty … it is easy to say anything you want if you know that you will never have to take responsibility for your words.
5. People could start to get a sense about the editorial judgement of the editors of the journals. If an editor frequently uses conflicted reviewers, for example, people should be aware of that.
6. There will be a permanent record of the process, so even years later we can see how bad paper slipped through or what logical mistakes led to unnecessary changes in the paper. This can only lead to improvements in the science.
People have said that if we publish reviews and reviewers’ names, people will be less willing to be reviewers, so the quality of reviews will suffer. I don’t think that’s true, for two reasons.
First, if someone wants to be an anonymous reviewer but is unwilling to sign their name to their opinion … why on earth would we pay any more attention to their opinion than that of a random anonymous blogger?
Second, if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.
That’s what I see as broken about the system, and how I would fix it … with sunshine, the universal disinfectant. Yes, it is important during the review for the reviewers and the authors to be anonymous and the proceedings secret. But once the procedure is complete, there is nothing to lose and everything to gain by keeping the peer review process open. Keeping it secret just encourages the current abusive system of pal review.
[Addendum] A couple of posters noted that I had not addressed rejected papers, my thanks for the feedback.
Each journal should publish papers that have been rejected, in electronic form only, and allow free public access to them.
In a way, this is more important than publishing the accepted papers. Science proceeds by falsification. But we have hidden away the most important falsification in the entire process, the falsification done by the reviewers.
These provisionally falsified claims are very important. If the reviewers’ rejections hold up, it will provide the ideas and logic needed to assess future repetitions of the same claim. If an eminent statistician has convincingly refuted my argument, THAT SHOULD BE IN THE PUBLIC RECORD.
Then the next time the argument comes up, someone could just say nope, someone tried that, here’s why it doesn’t work.
It would also encourage people to be reviewers, since their eminently scientific work of falsification would not be hidden away forever … and where’s the fun in that?
Now that I think about it, the current Journal practice of hiding scientific falsifications of proposed ideas is greatly hindering the progress of science. We’re throwing good scientific data and logic and argument in the trash can, folks. And by not showing the world that some idea has been judged and found wanting (and why), the same ideas keep coming up over and over again. As George Santayana didn’t say, “Those who cannot remember the falsification are condemned to repeat it”.
Regards to everyone,
w.
[Addendum 2] Gotta love the instant feedback of the web. Andrew Guenthner says in the comments below:
I would agree with Leif that requiring journals to publish rejected papers is a bad idea, for many reasons. For one thing, getting science published is not difficult. Sure, getting it published in a top-tier journal can be tough, but there are plenty of places where the level of competition is low. In reality, most rejected papers with good science do not end up in the “trash can”; they end up in more specialized publications where there is less competition. And in most places, it is easy (and getting easier) to self-publish. The real issues in most cases involve prestige and attention, not actual publication, and putting rejected papers online won’t make people pay attention to them, especially if (as would be likely) most scientific indexing services ignore them. Even now, a lot of technical papers get self-published online and appear in Google searches, and the purpose of many search tools is changing from simply finding out about work to filtering out the bad or irrelevant work. Making journals publish rejected papers just shifts part of the burdens and costs from the authors to the journals. Besides all this, journals generally require authors to give them the copyrights to work that they publish, and many journals will not publish material if it has appeared in some form already. As a scientific author, you are much better off retaining control of the distribution of your rejected paper, trying to improve its quality before it gets in front of a large audience, and looking for a more suitable venue than simply forcing someone to put it “out there” for you.
Good points all, Andrew, I can’t gainsay any of that. I stand corrected. I’d still like to find a system whereby when a high-powered statistician shows that my idea is 100% wrong, it is in the public record so we don’t have to do it again and again. I’m taking ideas on this one …
Willis Eschenbach says:
February 19, 2011 at 12:26 am
Willis’ understanding of scientific method is rock bottom solid, stellar at its apogee, and as wide as the horizon. Willis’ explanations of scientific method are essential to the extremely intelligent and well balanced discourse found on WUWT. Aside from trolls, no one has offered a criticism of Willis’ views on scientific method. Keep up the good work, Willis. And don’t let the trolls get you down.
Re: “So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.”
I highly recommend that you watch the video.
Chris Reeve says:
February 19, 2011 at 9:49 am
I highly recommend that you watch the video.
Very entertaining, but the guy is a total nut. Now what does that make you?
[snip]
Mike says:
February 19, 2011 at 6:55 am
Mike, you accused me of knowing nothing about climate, and tried to rally others to your view. I noted that my ideas must be scientifically important, since they were published in Nature Magazine, and that yours was a superfluous personal attack … in other words, that you were acting like a jerk, and an uninformed one, to attack me on bogus grounds rather than discuss the science.
Your response to my pointing out that your actions were uninformed, unpleasant, and unscientific was to say you were sorry “if you hurt my feelings” …
Do I look like a hothouse flower to you? Do you really believe that you acting like a jerk hurts my feelings?
Well, no, Mike, acting like that is your Constitutional right, and it doesn’t affect me at all. It hurts your reputation and credibility, particularly when you think that the issue is whether you hurt my feelings. Nor is your reputation or credibility enhanced when you don’t apologize for your actions, but instead seem to think that the issue is my feelings.
w.
PS: on a more substantive note:
Seems like there’s a misunderstanding here, I’m not sure. Let me restate my point.
For the first four IPCC Assessment reports (FAR, SAR, TAR, AR4) the claim was made, over and over, by Pachauri and others, that the IPCC reports contained only and solely peer-reviewed science. This unusual situation, unknown in other sciences, led to a huge press by AGW supporters to deny peer-review to their opponents’ science.
This claim of 100% peer-review turned out to be an IPCC lie. Nothing that you have provided has contradicted either of those facts. It was, for example, the inclusion of an NGO’s puff piece on the Amazon (in the guise of peer-reviewed science) that led to “Amazongate”.
PPS – For future reference, a proper apology would be on the order of “Gosh, Willis, I guess you must know something about climate science after all. Sorry for my ad hominem attack, I was out of line.” Then I say “That’s OK, Mike, I’ve gone off the rails myself more than once”, and we move forwards.
Saying you’re sorry that you hurt my feelings and adding that you’ll read my Nature article and the response later (what … you gonna grade me on my work?), on the other hand, means you still don’t see that the issue is not my feelings or your judgement of the quality of my Nature piece. The issue is, you acted improperly when you accused me of being clueless about climate science and tried to use your foolish claim to get people to ignore my work … not nice, Mike, not nice at all.
Chris Reeve says:
February 18, 2011 at 10:27 pm
Chris, this is so far off topic that I don’t know how to characterize it. Please take it to a thread that has something to do with whatever it is that the guy you like is on about. This thread is discussing peer review.
Many thanks,
w.
Re: “Very entertaining, but the guy is a total nut. Now what does that make you?”
Classic response. You’ve previously disagreed that the van der waals force can explain the solid and liquid states of matter as well. These experimental results are destroying the conventional assumption (which is absolutely necessary to the gravitational framework) that charge separation is rare in the universe. We teach astrophysicists right now assumptions which directly contradict this experimentation.
Re: “Chris, this is so far off topic that I don’t know how to characterize it. Please take it to a thread that has something to do with whatever it is that the guy you like is on about. This thread is discussing peer review.”
No problem, I’ll wait until the next time that you guys talk about water.
Re: “So you are stuck in the unquestioned [science is settled] electrical framework. Gravity is the root cause of everything that happens, even electrical phenomena.”
You’ve certainly summed up the longstanding approach inherent to our current scientific methodology: Assume that electricity is always contained within a box, wherever it is witnessed.
And where you see magnetic fields permeating space on the largest observable scales, do NOT infer an electrical current cause. In fact, “new physics” causes are preferred by gravitationalists over electrical current causes, as an explanation for the cosmic variety of magnetic fields. They’d rather propose unverified — oftentimes unverifiable — hypothetical processes for maintaining cosmic magnetic fields, than even consider the electrical cause which would be taken for granted in the laboratory.
There is nothing philosophical about this methodology. It’s designed specifically to suit the framework itself.
This unphysical preference for an inference is a direct result of the deductive Socratic-dialectic methodology which is taught to scientists everywhere, as otherwise, there would exist no heresy within the peer review system in proposing a far more physical basis (electric currents) for the magnetic fields which we observe to permeate both galaxies and even intergalactic space.
Making matters worse, after more than enough time, money and effort invested into Einstein’s “thought experiment,” we now know that this line of reasoning leads us to a dead end, with a universe which is 95% invisible, hypothetical matter.
Deduction is possibly necessary to scientific investigation, but at what point do scientists yearn for a more physical model, which they can use to make actual predictions with? If people want to pretend as though climate theory is unaffected by all of this, then they are in denial. Climate theory assumes the framework, just like every other single scientific discipline.
Fixing some minor aspect of peer review is not going to have this huge impact on science that the author of this original article proposes. The real problems in science can be plainly seen in the philosophical approach itself — which currently prioritizes those theories which can be arrived at through deduction from the original conjecture over those theories which can claim to be more physical than the consensus view.
I think that this video helps to provide a very concrete example for why our scientific methodology is seriously broken: Clearly, we are asking the wrong questions, if charge separation occurs in a cup of water exposed to light. The astrophysicists never saw this coming, and they will surely now try to ignore it as best they can.
Okay, I’ll now defer to those who prefer to imagine that this conversation is unrelated to peer review.
Chris Reeve says:
February 19, 2011 at 12:44 pm
These experimental results are destroying the conventional assumption (which is absolutely necessary to the gravitational framework) that charge separation is rare in the universe. We teach astrophysicists right now assumptions which directly contradict this experimentation.
On the contrary, charge separation happens a lot as is evidenced in all the electric currents that we see cause explosions all over the place. What you miss is that that charge separation is caused by plasma moving across magnetic fields, and that the separation doesn’t last long as it blows up all the time [in this case X=flares].
Steig had published a ground-breaking paper in a prestiguous that claimed GW could now be detected in significant parts of Antarctica. Along comes a paper from O’Donnell et al claiming that a more appropriate analysis of the data changes Steig’s conclusion. Would comments from Steig be useful to the editor, Broccoli, and other reviewers in their evaluation of O’Donnell’s work? Absolutely. Does Steig deserve a vote on whether O’Donnell’s paper should be published and what changes were necessary. Absolutely not.
Willis, yet again a great article. Sorry to arrive so late. Not just your article but your responses and your evidence of shifting in response to some other peoples’ responses. And not just your responses but everyone’s responses; as someone said, you throw great blog parties.
Having said which, I still, surprisingly, have a measure of disagreement with your judgement of Broccoli and Nielsen-Gammon. I think that here we have typical situations where people who have risen to power in the orthodox hierarchy are governed to some degree by the Stockholm Syndrome: without which, they would simply not even be editors in positions from which they can get O10 published. Is Gorbachev that different? I think that publication of O10 is a great step forward, albeit at cost, and I think most people on both sides would agree with that.
Of course, I agree passionately with your own point of view, as well. Strong, reasonably courteous, free of Stockholm Syndrome, and basically righteous, it is also doing one of the things most needed: opening up the debate.
Chris Reeves: your material is too important for you to let yourself get off topic’d. Work more carefully from the topic subject matter to your perceived consequences. And pay attention carefully to Scott: as an electrical engineer I trust him but I do not think he is right about Socrates. IMHO. Watch the nuances.
Chris Reeve says:
February 19, 2011 at 1:21 pm
Clearly, we are asking the wrong questions, if charge separation occurs in a cup of water exposed to light. The astrophysicists never saw this coming, and they will surely now try to ignore it as best they can.
Einstein got his Nobel Prize for his explanation of the photoelectric effect where charge separation occurs when a metal surface is exposed to ultraviolet light. Solar panels work in a similar [but slightly different] way. I think you better study up a bit, before you partake in further philosophical OT-discussion.
Re: “On the contrary, charge separation happens a lot as is evidenced in all the electric currents that we see cause explosions all over the place. What you miss is that that charge separation is caused by plasma moving across magnetic fields, and that the separation doesn’t last long as it blows up all the time [in this case X=flares].”
Lief, it’s apparently now being observed to occur in a cup of water hit by light. I refer you to the experimental validations for this *observation* which have been presented by the speaker in the video, using several different types of imaging. The photographs Pollack presented of the exclusion zone using pH-sensitive dye are more than sufficient to demonstrate that the hydrogen ions are not equally distributed in water. But, he went even several steps further than that in validating that the molecules in question were not spinning, and that they also exhibited reduced infrared emissions in the “exclusion zone” (ie, they are less “thermal”).
The totality of the evidence presented in that video supports the inference of the van der waals force as the primary actor. What remains to be explained is why the van der waals force which creates these “exclusion zones” on hydrophobic surfaces offers such an efficient mechanism for converting photons into electrons. It could possibly have to do with the geometry of the electron drift through that liquid crystal array. There might just be fewer collisions … ?
It’s also worth mentioning that Wal’s theory for gravity involves a process which is fundamentally akin to the van der waals (whereby the electrons themselves become dipolar). This same force is also implicated in the solid and liquid states of matter, such as …
“Collectively known as van der Waals forces, dispersion forces, dipole-dipole forces, and dipole–induced dipole forces affect such physical properties as melting points and boiling points.”
Van der waals is, most simply, an electromagnetic resonance which is disrupted by thermal activity. It is especially prominent in water because water is especially dipolar.
But, of course, in your conventional view, none of these questions and lines of reasoning are worth our time, and we must skip over this amazing claim for the underlying process for photosynthesis, as it is a diversion from our attempts to validate the gravitational framework. In my view, this has stopped being a search for truth; it is now solely a quest to prove the conventional theories, to the detriment of any good idea which stands in the way of conventional wisdom.
Our approach to science entirely determines the questions that we ask. That we are now talking about a universe where charge separation is presumably happening in every single drop of water in it should clearly have very profound impact upon how we view the universe. And it should similarly cause us to seriously reflect upon how we are approaching science.
A curious person would wonder what it is that is preventing the charge recombination near the edge of water? But, this is arguably no less of a mystery than water tension itself! And so, we must ask ourselves: Why have we not spent more time investigating water’s inherent tension? In a gravity-centric universe, water’s tension is simply not important. And this is the exact problem of choosing just a single conjecture to deduce from.
And I’d like to point out that people have been having this conversation over electricity in space, in various manifestations, since the time of Hannes Alfven more than a half-century ago. That should indicate the risk inherent to assuming one side of the debate as our starting conjecture for formulating the framework which underpins the *entirety* of scientific discourse.
Talk about a risky gamble! The assumptions which serve as our starting point for all our scientific investigation should *never* have a half-century of baggage associated with them!
Re: “Einstein got his Nobel Prize for his explanation of the photoelectric effect where charge separation occurs when a metal surface is exposed to ultraviolet light. Solar panels work in a similar [but slightly different] way. I think you better study up a bit, before you partake in further philosophical OT-discussion.”
Why does the charge not recombine in the water, Lief?
Chris Reeve says:
February 19, 2011 at 3:40 pm
Why does the charge not recombine in the water, Lief?
You tell me, but better in some other forum, where it is on topic.
Lucy Skywalker says:
February 19, 2011 at 1:44 pm
“I think that here we have typical situations where people who have risen to power in the orthodox hierarchy are governed to some degree by the Stockholm Syndrome: without which, they would simply not even be editors in positions from which they can get O10 published. Is Gorbachev that different?”
Very interesting thought. I have tended to think that individual editors have chosen the dark side for their own reasons. Maybe I was wrong. Your view is rather more frightening than what I have been thinking.
“I think that publication of O10 is a great step forward, albeit at cost, and I think most people on both sides would agree with that.”
Yes, except on your view why it happened is inexplicable on rational grounds. That is not a criticism. I will be thinking about what you have said.
Frank says:
February 19, 2011 at 1:35 pm
“Steig had published a ground-breaking paper in a prestiguous that claimed GW could now be detected in significant parts of Antarctica.”
Could we please stop repeating this “ground-breaking paper” absurdity? If the editor had consulted a competent statistician, the paper would have been rejected on that basis alone.
Truly, Chris and Leif, discussion on the Van Der Waals force and the like are most definitely off topic. I say again, please discuss this on an appropriate thread.
Thanks again,
w.
******
REPLY Hah! They’ll have to dig into their own pockets if they want to say anything!
This week, Reps. Bill Posey (R-Fla.), Sandy Adams (R-Fla.) and Rob Bishop (R-Utah) called for a budget that would “reprioritize NASA” by axing the funding for climate change research. The original cuts to the budget outlined yesterday would have cut $379 million from NASA’s budget. These members want climate out of NASA’s purview entirely, however. Funding climate research, said Adams in a statement, “undercuts one of NASA’s primary and most important objectives of human spaceflight.”
******
NASA has virtually no business sucking from the climate-change trough. Cut everything concerning climate-change, and redirect that to developing unmanned probes to Titan, Europa, Enceladus, Mars, Venus, nearby comets & asteroids, etc. To me, these types of missions have by far the most bang-for-the-buck.
I got Wired magazine for 1 year (about 3 or 4 years ago), and in one of the issues was the idea of having all scientific studies published. To me it made sense that you wanted the data and the methodology out there for everyone to see even if it wasn’t good enough to be in a 1st tier magazine. Basically because the studies aren’t published, we loose knowledge and sometimes rediscover it and sometimes don’t.
(yes I no longer get wired in part because their cool tech items are not sufficiently interesting to out weigh their crappy leftist bent)
As to NASA and Climate Research, I’m torn on this, on the one hand if their mission is getting manned space flight, then putting statalites in orbit helps them practice rocket shots. On the other hand, should we leave something as important as getting off planet to the Fed gov’t?
very interesting write up.
AC says:
February 20, 2011 at 7:50 am
I got Wired magazine for 1 year (about 3 or 4 years ago), and in one of the issues was the idea of having all scientific studies published.
The problem is: what is a scientific study? Uncle Al’s crazy scheme for unlimited energy from oxygenated water? Or Roy Spencer’s paper on negative feedback? or…