The following is from the blog Master Resource. Run by Robert Bradley. There are many interesting and well written essays there, and WUWT readers might benefit from a visit. Be sure to support them by checking out their books and leaving comments. – Anthony
Climategate: Is Peer-Review in Need of Change?
by Chip Knappenberger
December 1, 2009
In science, as in most disciplines, the process is as important as the product. The recent email/data release (aka Climategate) has exposed the process of scientific peer-review as failing. If the process is failing, it is reasonable to wonder what this implies about the product.
Several scientists have come forward to express their view on what light Climategate has shed on these issues. Judith Curry has some insightful views here and here, along with associated comments and replies. Roger Pielke Jr. has an opinion, as no doubt do many others.
Certainly a perfect process does not guarantee perfect results, and a flawed process does not guarantee flawed results, but the chances of a good result are much greater with the former than the latter. That’s why the process was developed in the first place.
Briefly, the peer-review process is this; before results are published in the scientific literature and documented for posterity, they are reviewed by one or more scientists who have some working knowledge of the topic but who are not directly associated with the work under consideration. The reviewers are typically anonymous and basically read the paper to determine if it generally seems like a reasonable addition to the scientific knowledge base, and that the results seem reproducible given the described data and methodology.
Generally, reviewers do not “audit” the results—that is, spend a lot of effort untangling the details of the data and or methodologies to see if they are appropriate, or to try to reproduce the results for themselves. How much time and effort is put into a peer review varies greatly from case to case and reviewer to reviewer. On most occasions, the reviewers try to include constructive criticism that will help the authors improve their work—that is, the reviewers serve as another set of eyes and minds to look over and consider the research, eyes that are more removed from the research than the co-authors and can perhaps offer different insights and suggestions.
Science most often moves forwards in small increments (with a few notable exceptions) and the peer-review process is designed to keep it moving efficiently, with as little back-sliding or veering off course as possible.
It is not a perfect system, nor, do I think, was it ever intended to be.
The guys over at RealClimate like to call peer-review a “necessary but not sufficient condition.”
Certainly is it not sufficient. But increasingly, there are indications that its necessity is slipping—and the contents of the released Climategate emails are hastening that slide.
Personally, I am not applauding this decline. I think that the scientific literature (as populated through peer-review) provides an unparalleled documentation of the advance of science and that it should not be abandoned lightly. Thus, I am distressed by the general picture of a broken system that is portrayed in the Climategate emails.
Certainly there are improvements that could make the current peer-review system better, but many of these would be difficult to impose on a purely voluntary system.
Full audits of the research would make for better published results, but such a requirement is too burdensome on the reviewers, who generally are involved in their own research (among other activities) and would frown upon having to spend a lot of time to delve too deeply into the nitty-gritty details of someone else’s research topic.
An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other. A few journals incorporate this double-blind review process, but the large majority does not. I am not sure why not. Such a process would go at least part of the way to avoiding pre-existing biases against some authors by some reviewers.
Another way around this would be to have a fully open review process, in which the reviewers and author responses were freely available and open for all to see, and perhaps contribute. A few journals in fact have instituted this type of system, but not the majority.
Nature magazine a few years ago hosted a web debate on the state of scientific peer-review and possible ways of improving it. It is worth looking at to see the wide range of views and reviews assembled there.
As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable.
Instead of improving the process, it has nearly destroyed it.
If the practitioners of peer-review begin to act like members of an exclusive club controlling who and what gets published, the risk is run that the true course of science gets sidetracked. Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.
Certainly as a scientist, you want to see your particular branch of science move forward as quickly as possible, but pushing it forward, rather than letting it move on its own accord, can oftentimes prove embarrassing.
As it was meant to be, peer-review is a necessary, but not sufficient condition. As it has become, however, the necessity has been eroded. And blogs have arisen to fill this need.
In my opinion, blogs should serve as discussion places where ideas get worked out. The final results of which, should then be submitted to the peer-reviewed literature. To me, blogs are a 21st-century post-seminar beer outing, lunch discussion, or maybe even scientific conference. But they should not be an alternative to the scientific literature—a permanent documentation of the development of scientific ideas.
But, the rise of blogs as repositories of scientific knowledge will continue if the scientific literature becomes guarded and exclusive. I can only anticipate this as throwing the state of science and the quest for scientific understanding into disarray as we struggle to figure out how to incorporate blog content into the tested scientific knowledgebase. This seems a messy endeavor.
Instead, I think that the current peer-review system either needs to be re-established or redefined.
The single-blind review system seems to be an outdated one. With today’s technology, a totally open process seems preferable and superior—as long as it can be constrained within reason. At the very least, double-blind reviews should be the default. Maybe even some type of an audit system could be considered by some journals or some organizations.
Perhaps some good will yet come out of this whole Climategate mess—a fairer system for the consideration of scientific contribution, one that could less easily be manipulated by a small group of influential, but perhaps misguided, individuals.
We can only hope.
===
Please check out Master Resource
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Good article.
NB Climategate raises that question, it doesn’t beg it.
Improving the review process would be good, but I think the key to the issue here is the fact that legislation is being based on hidden data and methods.
So, what we need to do is establish standards of openness that are required for any science that is being used as a foundation for legislation.
All data and methods being used to justify public policy must be open source.
From my experience, peer review is vastly over rated, but it is better than nothing.
It’s up to the editors to insure that politics does not hijack the process. Very difficult to keep politics out when research funds and power are the underlying issues.
“checking out there books”?
[Corrected.]
I have sent the following to my MP (UK):
Following the release of emails from UEA CRU, I would hope that you will support a thorough, independent enquiry into both the goings on at that institution, and, more importantly, a thorough re-evaluation of the science, as far as that is possible, given that certain original data sets have been destroyed. As it appears that the peer-review process in climate science has been compromised, such an enquiry should request a re-evaluation, by independent people, of sceptical papers that were rejected by this discredited process.
If the AGW theory that CO2 will cause dangerous climate change is proven and accepted by all, then let us get behind policies to reduce CO2. If on the other hand it is disproved, or unproven, then let us get on with saving the planet by focusing on reducing population growth (= eliminate poverty), habitat loss (especially rain forests), pollution (especially marine pollution), over fishing and the effects of by-catch and other issues of genuine importance.
The political class seems to have ignored this scandal, and the mainstream press, where they acknowledge it, have it on page 10.
Please don’t reply to me saying there is a “consensus”, or that the “science is settled” or that the IPCC has 2500 author’s who all agree – none of this is true.
CRU is just one example of the increasing politicization of science. And, not surprisingly, it runs both ways. Under the Bush administration, numerous scientists in the Forest Service and BLM chose to leave rather than sign off on scientifically unsound decisions that were pushed by resource users – users with connections in high places.
Most climate skeptics are conservative to very conservative. I happen to be one of the handful of left-sceptics on the planet. Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?
I think all science grads should have to study Karl Popper. The logic of scientific study and the open society and it’s enemies should be compulsory.
Review needs to be independent. From the e-mails, peer review in the CRU tape letters was manipulated and slanted. Corrupt is another word.
Peer review definitely has limitations. With Climategate I’m reminded of Galileo and the peer review he was subjected to and I see AGW having the same type of dogma as well as excommunications. Galileo used the scientific method while the peer reviewers engaged in dogmatic groupthink. It really troubles me – particularly when I read scientific magazines – how peer review (rather than adherence to the scientific method) is used to justify the standing of AGW. I felt rather sad when I read the latest New Scientist on this topic with how they blew off “hide the decline” on the basis of peer review. Science needs to be open sourced and the scientific method will win out. Open sourced science will discourage tribalism encourage scientific excellence of peer reviewers.
The Crime of the Century
November 30, 2009 by Doctor Zero
Few recent events have illustrated the ineptitude, and political agenda, of the mainstream media more dramatically than “Climagate.” The revelation of email correspondence from the Climate Research Unit of the University of East Anglia, documenting various attempts to suppress data and manipulate scientific “consensus” with thuggish tactics, confirms what critics of the global-warming movement have always maintained: it has a lot more to do with money and politics than science. In fact, the global-warming movement is essentially the opposite of science – the manipulation and destruction of empirical data to support a theory whose accuracy was decided in advance.
Those tempted to compare global warming to a religion should consider that religious faith does not require the willful suppression of knowledge. Calling the belief in global warming a “religion” is an insult to religion. It’s a mistake I have made in the past, and I owe the faithful of all religions an apology for doing so.
Global warming is a scam, pure and simple. By any objective measure, it’s the crime of the century, with a dollar value that dwarfs the sins of Bernie Madoff or Enron. People like Al Gore have become millionaires by selling books and “carbon credits” to their marks… many of whom knew perfectly well they were being taken for a ride, but felt political pressure to play along, or saw opportunities created by the exercise of raw government power. The economic damage from legislation passed in response to this hoax will run into trillions of dollars, if Barack Obama’s disastrous cap-and-trade legislation passes the Senate.
An objective media would respond to this blockbuster news story with front-page headlines and “special report” television treatment. By now, the authors of the incriminating Climate Research Unit emails would be infamous around the world. Top operators of the global warming racket, such as Al Gore, would be hiding in their mansions, afraid to face the mob of angry reporters gathered outside. Liberals love to accuse big corporations of manufacturing crises and taking advantage of consumers with false product information and deceptive advertising. Here is the paramount example of those offenses, on a scale that would widen the eyes of the greatest titans of industry. If a private corporation had conducted a scam as vast, and as destructive to the prosperity of nations – and the aspirations of the working poor…
… but no private corporation could do anything like this, could they? The global warming scam is the kind of crime that only Big Government can mastermind.
Continue reading at: http://hotair.com/greenroom/archives/2009/11/30/the-crime-of-the-century/?print=1
I believe that the future in Science has to be something like open-source in software…
Ecotretas
I don’t think the problem is with peer review as such, but with politicized science. Climate “science” as currently practiced blurs the distinction between researcher and political advocate. Mann, Jones, Hansen, et al. do not merely promote the alleged correctness of their research, but agitate for political change as a result of it. Fundamentally, the IPCC is a political, not a scientific, organization.
If a biologist publishes a paper asserting, say, that the Amazonian spotted frog is the same species as the Patagonian spotted frog, no politician or activist is going to care. And it is highly unlikely that peer review would be corrupted in this case.
The solution, then, is to separate the politics from the science. To this end, all genuine scientists should shun researchers who corrupt the scientific method for political ends and who attempt to leverage their prestige as “scientists” to agitate for social change.
I finally got a response from my MP, James Paice:
In other words, we have to trust the Scientists. I’m not sure he even knows about the CRU leak or if he does, he didn’t mention it even though that was the subject of my second email. I suppose it’s time to wheel out that Eisenhower quote about public policy being held captive by a scientific and technological elite.
List the peers. Let their reputations be at stake for poor or corrupt work. Footnote disagreements between the author and the peers. In other words, TRANSPARENCY and ACCOUNTABILITY.
“Most climate skeptics are conservative to very conservative. I happen to be one of the handful of left-sceptics on the planet. Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?”
I was for AGW legislation (in other words these were “my folks”) until I read RC’s explanation of “hide the decline,” which they turned me into a skeptic by their explanation that showed that the scientific method wasn’t practiced. What I want is good science and I hope then that good policy will come from good science. I want everyone held to high standards. Much of politics is about personal opinion based upon one’s individual values, but of the politics that is based on science, the science used for policy must be to high standards and beyond reproach. I’ve never been one to identify with political parties, but Climategate has re-enforced my belief that tribalism is harmful and will try even harder to be sure that I’m not engaging in tribalism where I put an agenda above objective facts.
I disagree. Blogs are very useful to discuss things and sometimes shed light on issues, but then there should be a way to move that to the scientific magazines. In this respect, blogs are as useful as any place where people can meet and talk. Fine for making up some drafts, but not anything that intends to contribute to the scientific knowledge just because of being put in a blog. Blogs are places for discussion, not for presentation of results.
The peer-review process has to improve. The author mentions double-blind reviews, which I agree would be a good thing. Also transparency of the process itself, which again I agree that it is needed. But most importantly, all of this scandal has to do with replicability of studies. Given that you cannot guarantee that the reviewer has done all the verifications needed, as it would be extremely time-consuming, you have to make sure that third parties can, if they want to. You have to force authors to make all data and methodology publicly accesible. As simple as that. Those who want to keep their data or methodology private, shall not be allowed to make their conclusions public, or pretend that they are scientific.
Science is not science without TRANSPARENCY.
Mike Hulme and Jerome Ravetz have written a rather excellent piece for BBC News:
http://news.bbc.co.uk/2/hi/science/nature/8388485.stm
Their vision for an “extended” peer review process that reaches out to the public is noble. It would certainly facilitate the trapping of errors overlooked by insiders blinded by “Groupthink.” Left unaddressed however, is the “gatekeeper” problem: how does someone outside the establishment overcome the bias of “peers” in order to publish paradigm-shattering ideas that are genuinely sound?
Ultimately, it’s up to the editor. If a reviewer makes unreasonable criticisms, then the editor has to make a judgment.
The double-blind idea is interesting, though I must note that sometimes one could still guess who wrote the paper – just as one can sometimes guess who wrote the anonymous review, based on the content.
I don’t share your enthusiasm that blogs will replace much of anything; rather, they will add something new. Blogs can be a place where ideas are refined, informal discussion takes place, and the laymen can learn or ask questions. Sometimes, a paper can have its roots in a blog post. That’s all useful and good. But face it, most blog comments are from non-specialists, who don’t “have some working knowledge of the topic”; their helpfulness in improving the work is thus very limited. A formal peer review process is still necessary, though not sufficient.
Open peer review is very interesting, but it would again come to the authors and ultimately the editor to decide which comments and criticisms are most helpful and worth heeding. Participation could also be sporadic; reviewing a paper takes time out of the day, and so you aren’t going to run around voluntarily reviewing everything that comes along.
As for what the emails show, keep in mind that publishing a single paper and writing the IPCC assessments are different processes, but ultimately, the papers in question did end up being mentioned in the IPCC report. As for anything else, if you think a paper makes basic errors which are apparent on a quick reading, it’s your job as a reviewer to point them out.
@John Egan
I’m also mostly liberal in my politics, especially relative to the environment.
I don’t think we’re so rare. I think there is a decent sized group:
Climate Skeptics for the Environment!
I think AGW has been distracting us from real provable environmental problems for too long.
Whilst we’d all desire a fairer system for peer review, I wonder that there is a broader issue.
Imagine for a second that there are no sceptics, and we all believe the experts on a particular topic. How does that become corrected?
Asking for fairer peer review is not unlike asking for a fair Guru; the Guru can always turn round and say to his devotees, “I am the Guru, I know more than you, and if you disagree, it is because of your own ignorance.”
Those who deeply disagree have no option but to leave. Perhaps they start their own alternative, and wait years for recognition. But say everyone is captivated by the authority of the Guru, and as far as anybody knows, everything he or she says makes sense. What then?
In trying to imagine this scenario, I come to think that the real problem with AGW theory, isn’t that peer review should have exposed flaws sooner—experts will argue whether even the flaws that have been discovered are of any significance in the end anyway—the problem is more subtle and wider. I suggest the problem is that we imagine it highly unlikely that the best scientific methods and data could, even when everyone agrees they are of high quality, turn out to be wrong.
In other words, even if 100% of all scientists (really all of them) came up with a watertight theory of global warming, and nobody had reason to be sceptical of it, we should still, for no other reason than simple, “survival”, have a plan B in case the world cooled instead of warmed. And we should not act as if we knew warming was going to happen.
We far overestimate our collective abilities (and I’m trying to say this respectfully to all scientists) whilst underestimating the complexity of the future world.
I realise this makes any debate much more complex, if we shift from arguing about what we think will happen, to what we don’t know might happen. But disasters are typically the nature of the unthinkable.
“We have to act” should never have been a collective response to the knowledge of the day. I’d prefer, “we need to be ready to adapt to a variety of scenarios”.
We have many technological marvels, but nonetheless, stuff doesn’t always work the way we expect, even when we really really expect it.
Dave
I am curious as to why you were for AGW legislation until the Real Climate response to the UAE data dump. Had you reviewed any of the skeptical arguements prior to this and were unconvinced or were you just taking Gavins position as valid. I think it would be enlightening to some of us skeptics to know as we try to have the issue debated honestly.
thanks
Help I’m in shock.
http://news.bbc.co.uk/1/hi/sci/tech/8388485.stm
‘Show Your Working’: What ‘ClimateGate’ means
Here’s a little fun with peer review literachur.
http://noconsensus.wordpress.com/2009/12/01/the-war-on-reason-continues/
I don’t know how this stuff gets through the review, well maybe I do.
From the CRU e-mails, I think it’s pretty clear that the peer review process in climate science is broken, at least with respect to the journals that the climate cabal (Mann, Jones, et al) were associated with. For example, here is an e-mail from Michael Mann that illustrates the problem:
Original Filename: 1047388489.txt
From: “Michael E. Mann”
To: Phil Jones ,rbradley@xxxxxxxxx.xxx, …
.
.
.
It is pretty clear that thee [sic] skeptics here have staged a bit of a coup, even in the presence of a number of reasonable folks on the editorial board (Whetton, Goodess, …). My guess is that Von Storch is actually with them (frankly, he’s an odd individual, and I’m not sure he isn’t himself somewhat of a skeptic himself), and without Von Storch on their side, they would have a very forceful personality promoting their new vision. There have been several papers by Pat Michaels, as well as the Soon & Baliunas paper, that couldn’t get published in a reputable journal. This was the danger of always criticising the skeptics for not publishing in the “peer-reviewed literature”. Obviously, they found a solution to that–take over a journal!
So what do we do about this? I think we have to stop considering “Climate Research” as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal. We would also need to consider what we tell or request of our more reasonable colleagues who currently sit on the editorial board…
What do others think?
mike
–
@Dave –
Your reasons for questioning the AGW line are quite similar to mine. When the argument depended upon ignoring the Medieval Warm Period, dismissing the Urban Heat Island Effect, and requiring an unproven CO2 feedback loop – I started to get suspicious. When people like James Hansen and Ellen Goodman started saying that any further debate was no longer permissible and that those who did were comparable to Nazi murderers – then I was convinced.
There is plenty of scientific evidence to suggest that global temperatures have increased and will continue to increase – albeit at a far more gradual rate than the IPCC suggests. Also, it is not prudent for humans to be fouling the only planet known to support life. Although I do not think that CO2 is the bugabear than some believe, I think it prudent to reduce emissions in a sane manner of the long term.
And, yes, science should play a crucial role in our future – a role that is, hopefully, as free as possible from political manipulation.