Peer review is dead, long live blog review

By Marc Hendrickx writing in ABC’s The Drum

In January 2009, Nature splashed its front cover with the results of a new study titled ‘Warming of the Antarctic ice-sheet surface since the 1957 International Geophysical Year’.

The article was accompanied by a glowing editorial from Nature and was widely reported on in the media.

A very short time after the paper was published, a number of factual errors were found in the paper, along with significant issues with the methodology used to obtain the surprising results. The errors and the methodological problems were reported and discussed by climate change blogs Watts Up With That, The Air Vent, Climate Audit and Real Climate.

Imagine if at this stage Nature’s editor in chief looked at the reported blog commentary and decided the journal had published a paper, which while it had gone through the normal peer review processes, based on some of the blog commentary, was basically fundamentally flawed and should not have been published.

Furthermore, the original reviewers may have shared some of the climate alarm notions of the authors, bringing the veracity of the original review into question. Media coverage also sensationalised aspects of the results. The editor in chief is so embarrassed by the publication of the erroneous paper, he decides to resign.

Sounds farcical? In fact Nature’s editor did not resign. Indeed there was no need to resign, there was no expectation on the part of the scientific community that a resignation was called for, regardless of the issues with the paper.

Subsequently Nature published a correction by the authors that dealt with some of the factual errors. And later, the blog commentary dealing with the methodological problems, ended up being published as a peer reviewed paper, by Ryan O’Donnell, Nicolas Lewis, Steve McIntyre and Jeff Condon, in the Journal of Climate.

Unlike the original paper however, this received very little media attention. Perhaps the long time the paper spent in peer review (10 months) and the less sensational results dulled the media’s interest.

This is just one example of how the peer review system works. Papers are written, reviewed, rejected accepted, acclaimed, criticised, corrected, refuted and debunked. When they are significantly in error they may even be retracted. The process of science, and the reason why it works so well, is because it is one of continual correction and revision. Theory stands until a better theory comes along to replace it. Peer review acts as a general screening tool, but it is by no means perfect, and it is ridiculous to expect it to work perfectly every time.

============================

Read the full article at ABC’s The Drum

5 1 vote
Article Rating

Discover more from Watts Up With That?

Subscribe to get the latest posts sent to your email.

54 Comments
Inline Feedbacks
View all comments
Mike
September 21, 2011 7:09 am

The editor of RS resigned to make clear that the paper by S&B was fundamentally flawed and did not just require minor corrections.

Rob Potter
September 21, 2011 7:26 am

Thanks to medsvstherapy (September 21, 2011 at 6:42 am) for what I think is the money quote here:
“In an election, we have to pick the lesser of two evils, and someone has to “win,” and occupy the position, until a better candidate comes along. Science is not the same.”
For a while now I have bemoaned the application of legal theory to scientific problems: The presentation of supporting evidence for your theory instead of looking for refuting evidence. I think the point here is that a courtroom situation is like an election in the you have to ‘pick a side’, whereas science does not require you to do this and should not be used in that respect.
The corruption of science with this viewpoint (I don’t mean with money) can probably be traced back to non-scientists wanting to have a get-out clause for implementing a policy which will be unpopular. By saying that “scientists say we must….” they are abdicating their responsibility for the outcome, as well as being cowardly and lazy about making a proper case. This seems to have culminated in scientists themselves becoming activists/politicians and adopting this legal approach.
Although we are talking about this corruption in the case of contentious issues, such as climate change, my biggest worry is what is this doing to the rest of our scientific establishment. Too many scientists in other fields are seeing the rewards (in terms of fame if not fortune) in being an advocate and this bodes ill for the future.

Theo Goodwin
September 21, 2011 8:04 am

Mike says:
September 21, 2011 at 7:09 am
Such action has never been and will never be a legitimate part of editorial function in science. The purposes of a scientific journal or an editor of one do not include “rendering judgement as to what is true or false in genuine debate among scientists.” Warmista should learn this.

Theo Goodwin
September 21, 2011 8:09 am

I read the comments on Hendrickx’s paper. Once again, Warmista are unwilling to debate. Of course, it makes sense that they are unwilling to debate because they embrace principles that are completely indefensible. Principle One is that the output of computer models (simulations) count as empirical evidence. Principle Two is that empirical evidence obtained in the real world cannot be used to criticize simulations. Need I continue? You cannot debate scientific points when you have tossed scientific method to the winds.

Fred Bloggs
September 21, 2011 8:30 am

Multiscience is correct. As they promised in the climategate emails
“Kevin and I will keep them out somehow – even if we have to redefine what the peer-review literature is!” – Phil Jones 8/7/2004
By resigning from his post, Wagner has redefined what peer review is. He and Trenberth have intentionally and implicitly attempted to tell reviewers that they should only allow the publication of papers which fit with the current narrative. My description of Wagner for doing this would not be allowed on this blog.

Ed_B
September 21, 2011 9:18 am

Mike:
The editor of RS resigned to make clear that obedience to the Team trumps science every time.

September 21, 2011 9:20 am

jimmi_the_dalek says:
September 21, 2011 at 2:46 am
said “Come back in 2 or 3 years and it may have reached a conclusion.”
Now, that would be the normal course. But, here the Editor-in-Chief resigned, spouting all kinds of garbage about the paper, the authors, reviewers. That is the kind of tactics by the warmistas that make them science community’s equivalent of mafia hitmen. Disgrace. Kevin Trenberth, the suspected hoodlum here, is on govt payroll. a bigger disgrace

September 21, 2011 9:21 am

But nobody can hijack blog comment review. I think blog comment review is a significant leap forward.

James Sexton
September 21, 2011 9:35 am

Mike says:
September 21, 2011 at 7:09 am
The editor of RS resigned to make clear that the paper by S&B was fundamentally flawed and did not just require minor corrections.
===================================================
lol, Mike that’s funny, because I’ve got both Dessler’s and Trenberth’s responses and in neither do they show any fundamental flaws. Are you warmistas keeping it a secret? If you’ve got some proof of some fundamental flaws, bring them out and we can discuss them.

Septic Matthew
September 21, 2011 9:47 am

Mike says: The editor of RS resigned to make clear that the paper by S&B was fundamentally flawed and did not just require minor corrections.
Had the paper (SB11) been shown to be “fundamentally flawed”, the journal would have published a retraction. Instead, it was too important to stifle or ignore, it was debated (on line and in break rooms everywhere), and Dessler published a rebuttal (D11) in another journal. The SB11 and D11 papers will doubtless be elaborated and rebutted in the future. Dessler’s 2010 paper in Science, like SB11 and the Nature paper addressed here, presented a weak result that was hyped.
Hendrickx wrote an excellent article comparing and contrasting two cases. It deserves a wide readership.

Owen
September 21, 2011 12:10 pm

James Sexton,
I can not prove a theory with an experimental result, only support it. I can however refute it by showing one experimental case (to which the theory applies) in which the theory fails. We get sloppy in our language all the time in this regard. Much as the eclipse photos only supported general relativity, but had the measurements been different would have disproven it completely. We never really prove a theory, we only show an ever growing set of cases for which the theory works.
Other than that I agree wholeheartedly.

kadaka (KD Knoebel)
September 21, 2011 1:23 pm

From Septic Matthew on September 21, 2011 at 9:47 am:

Instead, it was too important to stifle or ignore, it was debated (on line and in break rooms everywhere), and Dessler published a rebuttal (D11) in another journal. The SB11 and D11 papers will doubtless be elaborated and rebutted in the future.

Stop the disinformation campaign, now. “Dessler 2011” does not exist. It was a comment published at GRL, not a paper, not peer-reviewed. Functionally it was just Dessler shooting off his mouth with GRL posting the scatter pattern. Even then, the earlier-appearing draft version was heavily blog-reviewed with Spencer et al garnering significant revisions before the posting of the final version.
===
Mark said on September 21, 2011 at 9:21 am:

But nobody can hijack blog comment review. I think blog comment review is a significant leap forward.

Perhaps on the whole, no hijacking. But it has long been hijacked, if it ever really existed, with the abusive censorship at (C)AGW-pushing sites like ReallyRealClimate(models) and SkepSci, with skeptical sites that are trying to do balanced reviews condemned as anti-science untrustworthy “denier” hangouts.
The complete hijacking of blog review hasn’t been accomplished, the internet is too big to accomplish such, but the fight to keep blog review viable and free is an ongoing never-ending battle. And history has shown it’s best to not forget there’s a war underway when you’re either a combatant or possible collateral damage. 😉

manicbeancounter
September 21, 2011 1:50 pm

The consensus folks there are missing the real significance of SB11. Even if Dessler 11 can overturns it (which it does not) it is at the expense undermining the evidence for strong positive feedbacks. Established science has strong verification. The favourable evidence is extremely weak.
Put it another way. Do you think a criminal conviction would be possible on such evidence?

September 21, 2011 4:06 pm

This may be a bit off topic, but I wonder if we might get as much or more traction by doing some promoting. It sure would be nice to see a groundswell campaign for Henrik Svensmark to get a Nobel Prize. And it seems to me that anyone who has been following this story for a reasonable length of time must by now realize how due such an honor must be. This is in no way to diminish the incredibly important work of other independent scientists, especially those who have worked tirelessly to combat the cliches of the majority, but I hope that most people would agree that Dr. Svensmark’s role in promoting a history changing re-evaluation of the assumptions of the current academic status quo has been and will doubtless continue to be of unique importance. I posted the idea on my facebook page — http://www.facebook.com/profile.php?sk=wall — I hope that link takes you there and you can look at how we might spread the campaign on social media. I would think that with a little energy such a thought might go wildfire on the internet. Well, big brushfire anyway….:)

Brian H
September 21, 2011 4:21 pm

Re: choices of conclusions re theories and hypotheses:
“We don’t know, and know that we don’t know” is the fundamental overarching H0.

September 21, 2011 4:47 pm

But here’s someone who got it DEAD RIGHT!
As poor Obama struggles with poor polling, persistent high unemployment, possibly a primary challenge within his own party and a stagnant economy saddled with massive deficits and debts, one area where he can claim success is his prediction he would slow sea level rise.
In similar fashion to baseball legend Babe Ruth calling his home run during the fifth inning of Game 3 of the 1932 World Series, Obama called it successfully on sea level rise! You betcha!
Obama declared on June 8, 2008 during his St. Paul victory speech for the Democratic Party nomination that his presidency will be “the moment when the rise of the oceans began to slow and our planet began to heal.”
It’s now official. Earlier this month, the European Space Agency’s Envisat monitoring, global sea level revealed a “two year long decline [in sea level] was continuing, at a rate of 5mm per year.”
In August 2011, NASA announced that global sea level was dropping and was “a quarter of an inch lower than last summer.” See: NASA: ‘Global sea level this summer is a quarter of an inch lower than last summer’
Surprisingly, despite Obama only saying he would “slow” the rise of the oceans, his presidency has presided over what some scientists are terming an “historic decline” (i.e. one that’s, uh, impossible to hide).
Obama appears to have underestimated his powers.
More impressive is the fact that just six months into his presidency, sea level started its historic reversal. In July 2009, sea level was already showing a “slowdown and was “still flattening.”
So now I’m waiting breathlessly (i.e. winding back my CO2 emissions), for this amazing President to explain to me how this latest two year decline in sea level, which can only result from a contraction of the world’s oceans (noting the ice caps and glaciers are still melting and polar bears are still , uh, declining) fits in with Kevin’s 10 year loss of heat into the deep ocean.
If he can pull that one off then I reckon it just has to be….President for Life!

mike g
September 21, 2011 5:35 pm

This article gives short shrift to any suggestion that the drop in sea level is due to cooling: http://climate.nasa.gov/news/index.cfm?FuseAction=ShowNews&NewsID=570&Print=Yes
I notice the article studiously avoids mention of any of the water accumulating on land being at high latitude and/or high altitude, meaning year/round snow/ice, and a long interval before it flows back into the sea.

Septic Matthew
September 21, 2011 6:01 pm

Kadaka says: “Dessler 2011″ does not exist. It was a comment published at GRL, not a paper, not peer-reviewed.
Not peer-reviewed? OK ( unless his friends helped him to write it).
Non-existent? Not OK. It exists, it will appear in print. It will be cited.
Most importantly, the main points raised by S&B and D will continue to be debated.

JimF
September 21, 2011 7:16 pm

That Drum article is interesting, especially the comments. In my reading (not all, but a long way into the commentary) it seems the Skeptics are pounding the Alarmists about the head and shoulders (and that lot don’t have much in their arsenal but snark, ad homs and True Belief). How refreshing.

September 21, 2011 8:10 pm

So here’s the global mean sea level data from December 2003:
ftp://ftp.aviso.oceanobs.com/pub/oceano/AVISO/indicators/msl/MSL_Serie_E
Very easy to plot in Excel etc. Looks pretty clear cut to me.
Sea level rose about 1 cm (only) from late 2003 to mid 2009 and now it has gone down about the same amount over the last 2 years We are now back where we were in late 2003 (in terms of global mean sea level). In theory we should now be 1.5 cm higher than where we were in late 2003.
So if Kevin Trenberth et al. is correct (in his modeling) and for the last 10 years the so-called ‘missing heat’ (which even he admits has ‘gone missing’) has been hiding down in the deep ocean (below 700 m) AND the polar ice caps and continental glaciers have also continued to melt, where is the evidence of thermal expansion of the deep ocean (much less all the melt water)?
To have your cake and eat it too, first you have to produce the cake. Sorry, no cake and no cigar for Kevin.
No cigars for his crappy ‘peer reviewers’ either.

phlogiston
September 21, 2011 9:24 pm

Mike says:
September 21, 2011 at 7:09 am
The editor of RS resigned to make clear that the paper by S&B was fundamentally flawed and did not just require minor corrections.
Wagner the editor of RS in his suicide note actually spelled out his agreement with the message of this post. What he said was effectively, journal peer review accepted the S&B paper but the warmist blogosphere review rejected it. The former was wrong, the latter was right.
So Mike do you agree with Wagner that blogosphere commentary has more authority than journal peer review?

AusieDan
September 21, 2011 9:47 pm

teve Short
Hi there!
This is an error onthe reference that you gave:
ftp://ftp.aviso.oceanobs.com/pub/oceano/AVISO/indicators/msl/MSL_Serie_E
It may be at th end where there are three full stops.
Is there something missing?

James Sexton
September 21, 2011 10:13 pm

Owen says:
September 21, 2011 at 12:10 pm
James Sexton,
…………. We never really prove a theory, we only show an ever growing set of cases for which the theory works.
Other than that I agree wholeheartedly.
====================================================
Yes, we can prove theories. We can and do prove theories. If an instructor states,
“If BD is a perpendicular bisector of AC, prove that ΔABC isosceles.”
and the response is…….
“To prove that ΔABC is isosceles, show that BA ! BC . We can
do this by showing that the two segments are corresponding
parts of congruent triangles.
Since BD is perpendicular to AC , m∠BDA = m∠BDC = 90°. Since BD bisects AC , AD ≅ CD . With BD≅ BD (reflexive property), ΔADB ≅ ΔCDB by SAS.
Finally, BA ≅ BC because corresponding parts of congruent triangles are congruent.
Therefore, ΔABC must be isosceles since two of the three sides are congruent.”
Then, a theory is proved. Bonus given to name the theory proved. We’ve proven theories often. The ones we can’t, is because they are flawed.
Related thoughts……. I don’t have to be a pitcher in order to be an umpire.
James

Jim Masterson
September 21, 2011 11:35 pm

>>
James Sexton says:
September 21, 2011 at 10:13 pm
Yes, we can prove theories. We can and do prove theories.
<<
You’re confusing mathematics with science. In mathematics, the universe of discourse is defined. Therefore, it’s possible to prove a conjecture correct for all cases. In science, the universe of discourse is unknown. Therefore, the best we can do is not find a case that disproves our hypothesis/theory/law.
Owen’s point is correct.
Jim

kadaka (KD Knoebel)
September 22, 2011 3:30 am

From James Sexton on September 21, 2011 at 10:13 pm:

Yes, we can prove theories. We can and do prove theories. If an instructor states,
“If BD is a perpendicular bisector of AC, prove that ΔABC isosceles.”
and the response is…….

Does D lie on AC? Seriously, that’s one of those trick questions you have to watch out for. Your proof assumes that it does. If what you presented was all I had to go on, I’d assign E as the intersection of BD and AC and go from there.