The following is from the blog Master Resource. Run by Robert Bradley. There are many interesting and well written essays there, and WUWT readers might benefit from a visit. Be sure to support them by checking out their books and leaving comments. – Anthony
Climategate: Is Peer-Review in Need of Change?
by Chip Knappenberger
December 1, 2009
In science, as in most disciplines, the process is as important as the product. The recent email/data release (aka Climategate) has exposed the process of scientific peer-review as failing. If the process is failing, it is reasonable to wonder what this implies about the product.
Several scientists have come forward to express their view on what light Climategate has shed on these issues. Judith Curry has some insightful views here and here, along with associated comments and replies. Roger Pielke Jr. has an opinion, as no doubt do many others.
Certainly a perfect process does not guarantee perfect results, and a flawed process does not guarantee flawed results, but the chances of a good result are much greater with the former than the latter. That’s why the process was developed in the first place.
Briefly, the peer-review process is this; before results are published in the scientific literature and documented for posterity, they are reviewed by one or more scientists who have some working knowledge of the topic but who are not directly associated with the work under consideration. The reviewers are typically anonymous and basically read the paper to determine if it generally seems like a reasonable addition to the scientific knowledge base, and that the results seem reproducible given the described data and methodology.
Generally, reviewers do not “audit” the results—that is, spend a lot of effort untangling the details of the data and or methodologies to see if they are appropriate, or to try to reproduce the results for themselves. How much time and effort is put into a peer review varies greatly from case to case and reviewer to reviewer. On most occasions, the reviewers try to include constructive criticism that will help the authors improve their work—that is, the reviewers serve as another set of eyes and minds to look over and consider the research, eyes that are more removed from the research than the co-authors and can perhaps offer different insights and suggestions.
Science most often moves forwards in small increments (with a few notable exceptions) and the peer-review process is designed to keep it moving efficiently, with as little back-sliding or veering off course as possible.
It is not a perfect system, nor, do I think, was it ever intended to be.
The guys over at RealClimate like to call peer-review a “necessary but not sufficient condition.”
Certainly is it not sufficient. But increasingly, there are indications that its necessity is slipping—and the contents of the released Climategate emails are hastening that slide.
Personally, I am not applauding this decline. I think that the scientific literature (as populated through peer-review) provides an unparalleled documentation of the advance of science and that it should not be abandoned lightly. Thus, I am distressed by the general picture of a broken system that is portrayed in the Climategate emails.
Certainly there are improvements that could make the current peer-review system better, but many of these would be difficult to impose on a purely voluntary system.
Full audits of the research would make for better published results, but such a requirement is too burdensome on the reviewers, who generally are involved in their own research (among other activities) and would frown upon having to spend a lot of time to delve too deeply into the nitty-gritty details of someone else’s research topic.
An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other. A few journals incorporate this double-blind review process, but the large majority does not. I am not sure why not. Such a process would go at least part of the way to avoiding pre-existing biases against some authors by some reviewers.
Another way around this would be to have a fully open review process, in which the reviewers and author responses were freely available and open for all to see, and perhaps contribute. A few journals in fact have instituted this type of system, but not the majority.
Nature magazine a few years ago hosted a web debate on the state of scientific peer-review and possible ways of improving it. It is worth looking at to see the wide range of views and reviews assembled there.
As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable.
Instead of improving the process, it has nearly destroyed it.
If the practitioners of peer-review begin to act like members of an exclusive club controlling who and what gets published, the risk is run that the true course of science gets sidetracked. Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.
Certainly as a scientist, you want to see your particular branch of science move forward as quickly as possible, but pushing it forward, rather than letting it move on its own accord, can oftentimes prove embarrassing.
As it was meant to be, peer-review is a necessary, but not sufficient condition. As it has become, however, the necessity has been eroded. And blogs have arisen to fill this need.
In my opinion, blogs should serve as discussion places where ideas get worked out. The final results of which, should then be submitted to the peer-reviewed literature. To me, blogs are a 21st-century post-seminar beer outing, lunch discussion, or maybe even scientific conference. But they should not be an alternative to the scientific literature—a permanent documentation of the development of scientific ideas.
But, the rise of blogs as repositories of scientific knowledge will continue if the scientific literature becomes guarded and exclusive. I can only anticipate this as throwing the state of science and the quest for scientific understanding into disarray as we struggle to figure out how to incorporate blog content into the tested scientific knowledgebase. This seems a messy endeavor.
Instead, I think that the current peer-review system either needs to be re-established or redefined.
The single-blind review system seems to be an outdated one. With today’s technology, a totally open process seems preferable and superior—as long as it can be constrained within reason. At the very least, double-blind reviews should be the default. Maybe even some type of an audit system could be considered by some journals or some organizations.
Perhaps some good will yet come out of this whole Climategate mess—a fairer system for the consideration of scientific contribution, one that could less easily be manipulated by a small group of influential, but perhaps misguided, individuals.
We can only hope.
===
Please check out Master Resource
As a clinical medical scientist, I spent many hours as the Associate Editor of a Journal allocating reviewers to manuscripts, reading the reviews and making decisions as to whether papers were accepted or not. Occasionally, an author would make a request that his paper not be sent to a particular reviewer. This would generally be honoured but we would extra careful to ensure that the reviewers we did choose were likely to give a fair review.
Reviewers were expected to declare any conflicts of interest, for example, the author was a former colleague or they had reviewed the paper previously.
With the “Climategate” story, it seems to me that there has been collusion between the authors, trying to limit the editors’ choice of reviewers, the editors in giving in to their demands, perhaps out of sympathy for the authors’ stance on AGW, and the reviewers for not declaring their relationship with the authors and other conflicts of interest. The reviewers, and the editors were at fault also for not insisting on seeing enough details of the methodology to be able to be sure that they were appropriate.
There is fundamentally little wrong with the peer review process as long as all concerned behave with integrity. The problem here seems to be that those involved regarded the ultimate message of AGW as more important than the scientific validity of the research.
I’ll add one more issue with blogs: you aren’t going to see people putting too many fresh new results out there, before submitting a paper. You don’t want to get scooped.
Robinson (08:54:39) :
Robinson, you need to explain to your MP that 50 billion tons of CO2 is….nothing compared to the total mass of the atmosphere, that it is not a poison but plant food.
50 billion tons is only 10ppmv/million.
He makes politics about nothing.
http://www.americanthinker.com/2009/11/politics_and_greenhouse_gasses.html
In climate science, “peer review” means the science has been approved by a peer of the British realm, most likely the Prince of Wales.
>>I don’t think the problem is with peer review as such, but with politicized science. Climate “science” as currently practiced blurs the distinction between researcher and political advocate. Mann, Jones, Hansen, et al. do not merely promote the alleged correctness of their research, but agitate for political change as a result of it. Fundamentally, the IPCC is a political, not a scientific, organization.
Winner, winner, chicken dinner.
One meme from the article that is a repetition is on how onerous it would be on the peer reviewers to actually check the work.
The onus should be on the authors to provide the material in such form as to be essentially turnkey.
It is, indeed, quite difficult to take a journal article from the cold (that is, not precisely your own area of interest) and try to replicate the math. Not even the results – just the slogging from the raw data to the tabulations presented. The details of precisely what the process actually is tends to spread out amongst a collection of papers. “Here, I’ve used the approach of Paper3…”
But if the journals held this standard – and bounced things for “Supplementary Material Insufficient,” the author is the one who would be doing the vast majority of the necessary extra work.
Marching through the references you’ve used and making a neat table of precisely what data you’ve used isn’t exceedingly difficult for the author. It is excessively voluminous for inclusion in the core text of the article. But this sort of “Appendix” for the article should be compiled by the author anyway to be able to answer questions at presentations and poster sessions.
“same standards applied when “their folks” are in the White House?”
Absolutely
The public is not that stupid, and the other choice is to let government take over our lives and make all decisions for us.
Most modern desktop computers can easily burn a CD with a large amount of data, using software such as Nero, for example. Thus, raw data from even extensive projects can be supplied to anyone with a legitimate request. The FOIA disputes should be ruled out of science in the first place — any data concealment should be considered an obvious breach of ethics and the perps excluded from research work. Moreover, this is not a ‘hard-line’, conservative, or political policy — transparency is a basic condition for any legitimate academic or scientific work.
The entire notion and focus on peer-reviewed papers published in journals as a basis for decisions that effect the health and safety of the public is simply wrong.
There is no other public-policy-decision arena, not a single one, in which peer-reviewed published papers form the primary basis of such decisions. The climate change issue remains the sole, and very significant singular, exception to this statement. And it is well-established that decisions in this area will impact the health and safety of everyone in the entire planet.
In all other public-policy-decision arenas, there are Independent organizations that have the responsibility to Review, Verify, and Validate the information prior to making decisions that effect the health and safety of the public.
Generally, some part of the information presented for review to these Independent organizations might be based on peer-reviewed published papers, but that is not a necessary condition and is most certainly not a sufficient condition for acceptance of the information.
Very appropriate Far Side cartoon. I miss Gary Larson’s work. 🙂
There MUST be transparency in the entire process, otherwise it is simply a futile exercise.
Since so many climate science studies involve computer models, how does one verify the results without also reviewing the source code and the input?
Quite simply, one cannot.
Scientific journals are just another form of printing, along with publisheres of newspapers, books, magazines, or comic books. To get published you convince and editor and his reviewers. Getting something published doesn’t make it true or even good. Scientific journals are businesses, and not that altruistic. Peer review is not the holy blessing that makes the published work science. The true peer review is the discussion which happens after publication, and this is scattered.
“I am curious as to why you were for AGW legislation until the Real Climate response to the UAE data dump. Had you reviewed any of the skeptical arguements prior to this and were unconvinced or were you just taking Gavins position as valid.”
Generally speaking I trust the scientific community to operate honestly and to engage in best practices. Everytime something is announced is science, I don’t go to the source to see if I’m being deceived. In general scientists are seen a very credible. It wasn’t that I knew much about the details one way or the other as I just trusted the process – that the process was following the scientific method. Reading the answer given by RC instantly raised a red flag with how they handled the dendro because it looked like their hypothesis was invalidated (the hypothesis being that dendro – at least with the methods they used – is an accurate temperature proxy) by real world results, but instead of going back to the drawing board they kept the flawed results in their while dumping the inconvenient data that disproved their hypothesis. Based on real world data that tells me that the dendro methods used under-report warm weather, which undermines the validity of their temperature reconstructions. I wouldn’t expect an undergrad science student to do this (unless they were going to either drop out or change majors), let alone a very senior scientist who is working on setting policy for the world.
I’m not even a scientist, but I’ve had to do number-crunching as a financial analyst. Just to be a FA you have to be transparent and show how you arrived at your numbers so that way you can pass an accounting audit. I would just expect scientists to be at least as rigorous as accountants in matters of number-crunching transparency as I would expect a “scientific audit” to be wider in scope than an accounting audit – scientists all over the world trying to reproduce your results as opposed to a single audit firm double-checking your work without trying to reproduce it. Instead what I see is shades of Madoff along with the clubby atmosphere that let him get away with it.
Even with science that involves politics, I assume scientists to always exercise best practices. Like I look at the Manhattan Project scientists where many of those involved had multiple levels of political disagreement, but the scientists didn’t use their political disagreements as an excuse to fudge their science. With AGW I just thought those scientists (like any other scientist) would have science be at the forefront of what they are doing instead of trying to bend science to politics.
I think the failings of the peer reviewed process has been overstated. After all the shenanigans we have read about, skeptical papers have in fact been published. You can read emails where the likes of Santer talk about writing a paper to refute Douglass’s paper.
The real and much larger problem comes from the politicization of the whole science, from the IPCC through a fawning media and gullible learned societies. Ask yourselves, what difference would it have made if the peer review process worked perfectly? No skeptical papers would still have been included in IPCC reports, and they certainly would not have been mentioned in the media.
What to do then? The first change to make, is to make it a firm rule, that all data and methods must be archived with every peer reviewed paper. If we had had that simple mandate to begin with, we wouldn’t have needed FOI requests, we wouldn’t be struggling to reverse engineer and second guess the true meaning of various temperature trends and proxy studies. All would have been clear from the beginning, and scientists would have been able to challenge outrageous statements years ago, before any of this warmist nonsense had reached critical mass.
I agree with the article. However, one big hurdle for peer-reviewed paper access is their price. It’s way too expensive to get a copy if you don’t have a library with the journals nearby. When you think that the reviewers are not paid to review papers, why such high price for copies. Should we start a P2P torrent system to have access to peer-reviewed literature? When you are a small independent research laboratory, you can’t afford to pay for journals if your local university does not have copies.
Isn’t it the editor’s responsibility to enforce the peer process? And didn’t they violate their own written peer process by playing hide the data?
Which editors should resign first?
Open science is the only science. The rest is just a game.
For those who are liberals turned skeptics, remember that George Orwell was a socialist who was one of the most scathing critics of socialism as it was practiced.
Remember too that there are as many varieties of conservatism as there are liberalism. Being for the environment doesn’t mean that you can’t be conservative, or even libertarian.
My concern is that this farce is taking resources away from the real environmental issues. If the idiots running the government can’t even repair damage to the Chesapeake Bay, how in the world could they take on the whole climate?
“Is peer review in need of change”? – Of course.
Climategate have revealed what Wegman pointed out a long time ago “..there was too much reliance on peer review, which was not necessarily independent.”
With climategate we can with confidence say, “peer review” as used by “the hockey-stick team”, was and is, definitely NOT independent.
This “peer review” process has been just a small clique of scientists back slapping each other, covering up their mutual errors and publishing wrong and fraudulent papers.
Moreover this “peer review” has been used as an excuse and mantra to “refute” and avoid arguments against their findings.
Has that arguments been “peer reviewed”?
No?
Well then 1. We neednt answer it and 2. It must be wrong.
False – An argument fails if it is wrong in fact or logic. It has nothing to do with “peer review” and any compelling argument needs to be answered on its merits.
The team has neatly avoided this by quoting (their own) papers or referring the argument to the IPCC.
“In all other public-policy-decision arenas, there are Independent organizations that have the responsibility to Review, Verify, and Validate the information prior to making decisions that effect the health and safety of the public.”
– – – – –
Bingo!
And the difference is, in all other public-policy-decision arenas, the reviewers would never stand for the first-level testers – the scientists, the researchers, the data-gatherers – claiming some proprietary right to keep the underlying data and processes secret while revealing only their conclusions.
Historically, many old societies did look to one individual source of information concerning the natural world and their interaction with it, and did have to be content with simply accepting the proclamations from that source as having been well-founded. Those sources, called in some societies “witch doctors”, knew that the best way to secure their own power was to make their decisional processes opaque.
Jones says he’ll release most of his data “next year.” Why not “today”?
“I think AGW has been distracting us from real provable environmental problems for too long.”
I agree. One of the things that bothers me is the framing of this where those who are skeptical are labeled as political right wingers potentially in the pay of big oil. Unfortunately I see many people biting, which many of those who most go along with the framing can’t see how they are putting the environment at risk by wanting to devote so much time and resources into something that might not do much at all for the environment. I can think of a number of reasons for instance that I’d want to save forest habitats, which none of those reasons involves carbon credits. It’s rather sad thinking of all the billions/trillions of dollars that could be wasted on the environment (CO2), when true environmental problems are ignored both financially as well as not given much bandwidth in high profile discussions.
rtereive my post please
[Rescued from spam bucket & posted. ~dbs, mod.]
Double-blind is great in principle. But in practice it is an invitation for cheating. (Although there seems to have been a huge amount of cheating under the current system.)
“Climategate begs the question: ‘is peer review in need of change’?”
As cjcjc has pointed out, “begging the question” has a specific meaning in logic, a commodity sadly lacking in AGW debate, for the most part. Let’s not pollute the language of reason by loose usage of terms specific to that field.
“An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other.”
The Scheme Team would get around that in 24 hours with a few phone calls or emails.
“Unsettling as this may be for scientists, the combination of ‘post-normal science’ and an internet-driven democratisation of knowledge demands a new professional and public ethos in science.” [From Hulme and Ravetz’s piece for BBC News]
After close reading of the BBC piece, I sense that Hulme & Ravetz still haven’t understood. The fact that they seemingly accept ‘post-normal science’ as something that can be fused with an Internet mobocracy into some workable, new scientific Utopia by simply sprinkling on a little “ethos” is ludicrous. Their article does contain some valid points, but I recommend reading it with great skepticism.
Post-normal science is evil and should be rejected in toto.
In 2005, I was designated to be my team’s GHGV “expert” by my then employer British Standards in an effort to build a strong client list of Canadian “emitters” to sell our services to. Ironic though that my exploration into the field of climate research (in order to be of value to my employer and clients) brought me an understanding completely removed from where I was before I started asking questions.
As a trained process auditor, and someone who believes in continuous improvement through methodologies such as the Shewhart cyle, internal auditing, documentation of non-conformances, respective corrective actions and above all, transparency, i’m pleased to see that an ISO technical committee has been established to develop an international standard for statistical data and meta data exchange (ISO/NP TS 17369).
An external, unbiased 3rd party audit of the quality principles relating to any data and the respective code used in models would be a good start in at the very least validating the quality of the processes used to create hockey sticks.
Cheers Anthony, and keep up the great work!