The following is from the blog Master Resource. Run by Robert Bradley. There are many interesting and well written essays there, and WUWT readers might benefit from a visit. Be sure to support them by checking out their books and leaving comments. – Anthony
Climategate: Is Peer-Review in Need of Change?
by Chip Knappenberger
December 1, 2009
In science, as in most disciplines, the process is as important as the product. The recent email/data release (aka Climategate) has exposed the process of scientific peer-review as failing. If the process is failing, it is reasonable to wonder what this implies about the product.
Several scientists have come forward to express their view on what light Climategate has shed on these issues. Judith Curry has some insightful views here and here, along with associated comments and replies. Roger Pielke Jr. has an opinion, as no doubt do many others.
Certainly a perfect process does not guarantee perfect results, and a flawed process does not guarantee flawed results, but the chances of a good result are much greater with the former than the latter. That’s why the process was developed in the first place.
Briefly, the peer-review process is this; before results are published in the scientific literature and documented for posterity, they are reviewed by one or more scientists who have some working knowledge of the topic but who are not directly associated with the work under consideration. The reviewers are typically anonymous and basically read the paper to determine if it generally seems like a reasonable addition to the scientific knowledge base, and that the results seem reproducible given the described data and methodology.
Generally, reviewers do not “audit” the results—that is, spend a lot of effort untangling the details of the data and or methodologies to see if they are appropriate, or to try to reproduce the results for themselves. How much time and effort is put into a peer review varies greatly from case to case and reviewer to reviewer. On most occasions, the reviewers try to include constructive criticism that will help the authors improve their work—that is, the reviewers serve as another set of eyes and minds to look over and consider the research, eyes that are more removed from the research than the co-authors and can perhaps offer different insights and suggestions.
Science most often moves forwards in small increments (with a few notable exceptions) and the peer-review process is designed to keep it moving efficiently, with as little back-sliding or veering off course as possible.
It is not a perfect system, nor, do I think, was it ever intended to be.
The guys over at RealClimate like to call peer-review a “necessary but not sufficient condition.”
Certainly is it not sufficient. But increasingly, there are indications that its necessity is slipping—and the contents of the released Climategate emails are hastening that slide.
Personally, I am not applauding this decline. I think that the scientific literature (as populated through peer-review) provides an unparalleled documentation of the advance of science and that it should not be abandoned lightly. Thus, I am distressed by the general picture of a broken system that is portrayed in the Climategate emails.
Certainly there are improvements that could make the current peer-review system better, but many of these would be difficult to impose on a purely voluntary system.
Full audits of the research would make for better published results, but such a requirement is too burdensome on the reviewers, who generally are involved in their own research (among other activities) and would frown upon having to spend a lot of time to delve too deeply into the nitty-gritty details of someone else’s research topic.
An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other. A few journals incorporate this double-blind review process, but the large majority does not. I am not sure why not. Such a process would go at least part of the way to avoiding pre-existing biases against some authors by some reviewers.
Another way around this would be to have a fully open review process, in which the reviewers and author responses were freely available and open for all to see, and perhaps contribute. A few journals in fact have instituted this type of system, but not the majority.
Nature magazine a few years ago hosted a web debate on the state of scientific peer-review and possible ways of improving it. It is worth looking at to see the wide range of views and reviews assembled there.
As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable.
Instead of improving the process, it has nearly destroyed it.
If the practitioners of peer-review begin to act like members of an exclusive club controlling who and what gets published, the risk is run that the true course of science gets sidetracked. Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.
Certainly as a scientist, you want to see your particular branch of science move forward as quickly as possible, but pushing it forward, rather than letting it move on its own accord, can oftentimes prove embarrassing.
As it was meant to be, peer-review is a necessary, but not sufficient condition. As it has become, however, the necessity has been eroded. And blogs have arisen to fill this need.
In my opinion, blogs should serve as discussion places where ideas get worked out. The final results of which, should then be submitted to the peer-reviewed literature. To me, blogs are a 21st-century post-seminar beer outing, lunch discussion, or maybe even scientific conference. But they should not be an alternative to the scientific literature—a permanent documentation of the development of scientific ideas.
But, the rise of blogs as repositories of scientific knowledge will continue if the scientific literature becomes guarded and exclusive. I can only anticipate this as throwing the state of science and the quest for scientific understanding into disarray as we struggle to figure out how to incorporate blog content into the tested scientific knowledgebase. This seems a messy endeavor.
Instead, I think that the current peer-review system either needs to be re-established or redefined.
The single-blind review system seems to be an outdated one. With today’s technology, a totally open process seems preferable and superior—as long as it can be constrained within reason. At the very least, double-blind reviews should be the default. Maybe even some type of an audit system could be considered by some journals or some organizations.
Perhaps some good will yet come out of this whole Climategate mess—a fairer system for the consideration of scientific contribution, one that could less easily be manipulated by a small group of influential, but perhaps misguided, individuals.
We can only hope.
===
Please check out Master Resource
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
I propose “Jonesing the journals” as a new phrase, kind of like “Mann-handling the curve.” 🙂
Excess fussiness on my part, but I had to say that ‘begging the question’ is not the same as ‘inviting the question’ or ‘raising the question’ or its equivalents. Begging the questions is a rather poorly named logical fallacy.
While I have no thoughts on how to improve peer review, I do have this reference on why invoking it as proof someone is right should elicit sighs of frustration and/or laughter.
http://www.iscid.org/papers/Tipler_PeerReview_070103.pdf
Perhaps there should be more than one process, with different journals subscribing to different methods. I.e., let the marketplace of ideas sort out who’s more effective in championing good science.
All data, methods, calculations, etc. must be openly published and independently archived.
Robinson (08:54:39) :
I emailed to my MP, George Osborne on the subject of global warming, and received a letter. What has surprised and appalled me is that it is word for word , paragraph for paragraph IDENTICAL to your letter.
What an insult to the intelligence of voters, and what a total disregard for opinions of the electorate. Of course we really knew it, but it isn’t common to have it thrust in your face quite so brazenly.
The level of arrogance exhibited by these people means that there is no chance that they will take any heed of any information that will change their attitude to global warming – they are in it for the power and the public acclaim (and the money, given the performance of UK MPs on the subject of their expenses over recent years).
So James Paice wouldn’t have known about Climategate, and I suspect it is unlikley that he will even have seen your letter. At the first mention of a sceptic viewpoint your letter will have been categorised as being from a denialist crank, and the standard boilerplate response will have been generated. One might expect the signature to be someone elses too.
I will now write again and see what response is forthcoming – perhaps you will do the same with Paice and we can again compare notes.
I have also written to another MP, Douglas Carswell, who appears to have a less rigid view on global warming, but have not as yet received a reply.
After it is all said and done the net effect of the CRU whistleblowing will be as follows:
1) The current versions of the major temp data sets will have a version publicly on the web. It will be almost unusuable in its public form but it will be there to insure transparency.
2) All new studies will be based on the public datasets that the study authors “improve” in ways that are incoherent and impossible to reproduce. The studies will still be “transparent” because it cannot be helped that skeptical morons cannot figure out the amazing techniques the gods of climate employ.
3) A journal or two may change their peer review process and there may be a few actual skeptic papers published that would not have been otherwise. The “concensus” group will ignore these published studies like the plague and they will not be included in any IPCC or other summaries.
4) Phil Jones will be admonished and kicked upstairs and another “lackey” with clean credentials will take over the CRU dataset to clean it up and make it transparent. Lost data will not be recovered and the adjustments made to data will not be changed. The adjustment process will be briefly explained but no skeptical moron will be capable of duplicating the adjustments made by the climate gods.
5) People will forget about climate gate after about 6 months and things will go back to the pre-existing condition.
6) Concensus scientists will learn to write very nice politicaly correct FOIA sensitive emails while at work.
7) Concensus scientists will learn how to communicate via personal Blackberrys in order to handle all other “correspondences” off the FOIA grid.
8) In the year 2020 a hacker will discover a system cache of Blackberry text messages between AGW scientists that documents them laughing about keeping skeptics unpublished, boosting warming trends in their datasets and how they circumvented their FOIA restrictions.
9) The Court of Climate Denialism will sentence the Blackberry text hacker to death by injection in a special IPCC televised “Climate Justice” episode.
Shiny
Ed
John Egan,
You ask a legitimate question. I’m a conservative. Not a scientist but I love reading about science. Science must be kept honest and open or it will not advance. Politics should never be married to science. Nor, come to think of it, should science ever rely on mass media to tell the truths of science – journalists who do know what they are talking about (darn few) are still subject to the restrictions of the message their management supports. Frankly, I think the openness of the internet is one of the best things that has ever happened to science, politics, and anything else that needs the light of day. We have too few people in this world who understand either the scientific method or the limits of science to provide certainty.
Perhaps as part of the review process the data and methodology (software) should be distributed to reviewers. Reviewing the data and code would be a part of the process. It’s doubtful anyone would attempt to completely reprocess the data to arrive at the same analysis, but it would put the brakes on deliberate fudging. If the paper is accepted, the code/data is also made available to subscribers. This would encourage some housecleaning on the data processing prior to publication – if you know you have to package everything up for the peer review, it would discourage leaving it in a messy state.
Authors should not suggest reviewers. Editors should be familiar enough with the scientists to know who would be a good reviewer. If not, perhaps they should not edit that journal. Reviewers should be anonymous. I don’t think you could do a double-blind review as the reviewer could probably figure out who wrote the paper just from the style, citations, and emphasis in the research.
If one was publishing results of an experiment, isn’t the author generally expected to provide enough information for anyone interested to replicate the experiment, and presumably the results? Isn’t that really the final say in peer review? If independent experimenters can’t replicate results, the author needs to explain why not.
Why do the journals not hold climate science to a similar standard? (yeah, we all know why now, but 2 weeks ago I had some doubt). It’s generally not possible to prove a statement true, the best you can do is not prove it false – so the strongest support for the AGW hypothesis would have been to give the raw data to anyone interested, explain any adjustments, and why they are valid, and have people potentially hostile to the theory have at it. If the AGW theory were as ‘robust’ as we’ve been told all these years, no one would be able to refute it, and releasing data would only make the AGW arguments stronger. That would be real peer review; impossible to set up a situation where the only review is a softball by someone already sympathetic to the cause.
I can only think of one reason why [it seems like most of] the AGW side are so determined not to release their data. The Climategate emails confirmed my suspicions. It also demonstrates that the peer-review system is worse then useless, along with several scientific journals.
I think the article misses the point. It’s not peer review that is the problem- one thing that is the problem is the deliberate, repeated presentation of peer review as a system which ensures good science, and that peer reviewed papers are in some sense trustworthy or even “true”. Journals are supposed to be a forum in which scientists can present their work- whether it be ultimately correct or incorrect- to other scientists, to add to the general pool of knowledge in the science. They are not repositories of Truth. You cannot create, nor should attempt to create, a process which only allows “true” work into journals. Publication is merely the start of a process of new scientific knowledge being added to the corpus of scientific knowledge in general.
Where some process of auditing needs to occur is when science breaks out of the bounds of the scientific community to affect, in particular, politics. When collective decisions are made on the basis of it, when lives or zillions of groats are being wagered on it, then it needs to be scrutinised as finely, and with as much hostility, as possible. Effectively, when science morphs into politics, it ceases to be simply a matter for scientists. This is the central problem. Scientists jealously guard their independence (eerily, despite having political views which promote interference by the state in every other aspect of human existence) and science, itself, should indeed remain independent. But once it has become enmeshed in politics, once it becomes something which profoundly affects lives and livelihoods, it cannot any longer cling to that independence. Because it isn’t science any more, it is politics.
“Minister, you must implement this policy because scientists say so” is not good enough. This is the point when the science and the scientists must be subjected to the most rigorous examination for error and bias. The real scandal of the IPCC is that despite claiming “peer review”, in fact none of the science in their reports is checked in any way. If a paper has been published and a chapter author decides to include it, it is presumed by the system to be true. An IPCC, or equivalent, which were really doing its job would be auditing every paper, and the auditors would be hostile and looking for flaws. The journal system of peer review is no business of anybody’s but scientists and journal editors. They must be free to publish what they wish. It is at the interface with politics and society that the rest of us must get our oar in and demand the very highest of standards and most rigorous of checks and balances- because it is only at this point that the stakes become high.
The best peer review ever: WUWT…and here is the only place on earth that a scientific paper will have more than 3 millions hits!. That will surpass the pop rock stars.
A valid warning from an “old soldier” and a competent politician:
Dwight D. Eisenhower
Farewell Address
delivered 17 January 1961
Excerpt: “Akin to, and largely responsible for the sweeping changes in our industrial-military posture, has been the technological revolution during recent decades. In this revolution, research has become central; it also becomes more formalized, complex, and costly. A steadily increasing share is conducted for, by, or at the direction of, the Federal government.
“Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers. The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present — and is gravely to be regarded.
“Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.”
What Ike feared could happen did happen. Public policy has become the captive of a scientific-technological elite. The AGW proponents.
I would rather see the peer review process be done away with altogether, with all proposed papers published on the internet, for all interested people to be able to review and comment on. Then these comments could be culled to be rid of all comments (responding documents) which contained personal/ad hominem attacks. There is no place for such in a “scientific setting”.
There are a good many of us who are not involved in the “college scene” who nevertheless have a good understanding of various basics of science, and a good many years of experience, having seen many concepts which seemed like a “good idea at the time” which contained serious basic errors.
During my own professional career, I was always pleased to have “devil’s advocates” on my staff or as reviewers. A good many times these “devil’s advocates” did advance serious questions about my proposed approach to “solving problems” as I call what I did, which if not accounted for, would have resulted in at least minor disasters and large wastes of public funds.
People who work for private companies or on their “own dime” are one thing, and it is mostly “the market” which will determine the success of their intellectual endeavors. But when the taxpayer is the one footing the bill, all work done should be subject to review of those taxpayers, at least those who are interested in review.
AS lot of work, certainly. But a lot of public moneys are riding on any ultimate public policies based on these research projects and endeavors.
It seems to me that the real culprits are those non-team members who did not demand the release of the data and methods. It doesn’t seem that peer review failed, there was no peer review without the data. I think that the internet mobocracy is the best place to vet these issues. With respect to scientific arguements of sufficient import to be widely interesting, it is much more likely that the mob will get it right eventually than a couple of randomly assigned, otherwise busy, reviewers.
The free exchange of scientific data and methods over the internet will drastically improve the speed and accuracy of the scientific process
In simpler words, I much enjoy Anthony’s methodology. Anthony has been doing excellent work, his publishing of his endeavors is most interesting to those of us who have a real interest in the science of it, enjoyable to read, and comments do flow in. I am sure that Anthony does take into consideration those comments which are based on actual science, even if there are those of us who are not “professional climatologists” making comments.
cjcjc (08:36:10) :
Good article.
NB Climategate raises that question, it doesn’t beg it.
/What he said.
All good scientists should know what “begging the question” means. It is a fallacy similar to circular reasoning.
Example:
AGW is universally accepted because there is a consensus.
see why it’s a pet peeve of mine? 🙂
“Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.”
anna v wrote an interesting post here
http://wattsupwiththat.com/2009/11/27/zorita-calls-for-barring-phil-jones-michael-mann-and-stefan-rahmstorf-from-further-ipcc-participation/#comments
(22:52:36)
In her analyisis the process of peer review is subject to some very human herd behavior, but is still self-correcting in general. She gives two reasons why that process was short circuited in climate science.
“Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?”
John,
I fear you totally miss the heart of the issue. I’m not sure what you are driving at since a)most federal employees at NOAA or NASA are not appointed by any President; thier tenure usually far out lasts any sitting President. and b)much of the work done is accomplished at Universities.
Also, I’ve never heard (with the exception of Far Left blogs) of any atmospheric science professional being asked by President Bush to fudge his work. And from we can tell, this politicization of climate science appears to be mainly a problem for the Left.
To my mind, the principle problem with peer review lies in the rhetorical use the alarmist crowd has made of it in their propagandistic efforts. The mantra like repetition of the term was clearly and intentionally meant to reinforce the implicit fallacy that peer review existed to guarantee the correctness of a paper’s conclusions, something it was never intended to provide and which the present controversy has shown to have been moved much farther from its original intent within the realm of climate science. Unfortunately, they have been all too successful in imbedding this fallacy as conventional wisdom in the minds of those outside the scientific community unfamiliar with peer review’s actual purpose and ,from many of the comments I’ve seen here and elsewhere, many in the science field who should know better.
Given that a majority of the “science” produced in the climate field involves complex and arcane computer programs, which few potential reviewers have either the time or expertise to evaluate adequately, it is not clear that providing complete and open access to data and methods will significantly improve the effectiveness of peer review in this area.
Ultimately, peer review is a peripheral issue to the main problem, which is the almost complete domination of the funding of science in all fields by government provided money controlled by politicians more interested in their own agendas than in the advancement of science.
The debate that we really have to create is how we can address this very intractable quandary. Science in all areas is becoming increasingly expensive and it’s hard to envision a way to move toward other sources of funding that would be adequate to the purpose and it’s equally difficult to imagine a system which can insure that, if governments are to remain the the primary funding mechanism, safeguards can be created which will be able to remove political considerations from the allocation of grants. I don’t have any easy answers for any of this, but I believe it is something we need to be thinking and talking about on an urgent basis, as we are already far down the road to the “post science” world where all of science is driven by someone’s agenda and if that world is allowed to materialize fully, humanity will be damaged in ways that make Gore and Hansen’s most hysterical prognostications seem like a hangnail.
AP claims Phil Jones is stepping down temporarily.
Ian B (10:54:05) :
“I think the article misses the point. It’s not peer review that is the problem- one thing that is the problem is the deliberate, repeated presentation of peer review as a system which ensures good science, and that peer reviewed papers are in some sense trustworthy or even ‘true’.”
While I agree with that, you haven’t pursued it far enough. Remember that the pirate crew at CRU hijacked and perverted the peer review process. As such it was “peer review” in name only. Perhaps “co-conspirator review” might be a more apt title, with the crooks vouching for each others’ honesty.
Peer review at it’s best may need some tweaking, but when it works it’s mostly relatively effective. Unfortunately, there is never any guarantee against corruption. Only constant vigilance, of the kind we’ve seen by so many scientists and others honest citizens fearful of the consequences, can minimize the chances of this happening. And the more wealth and power that are at stake, the more vigilant one must be.
I’m liberal in many areas. So I think people will be surprised as to how many liberals will change their minds after discovering climategate for themselves. As a programmer, the code and log file is what convinced me that the model was bogus. I’ve now seen it with my own eyes. Others will also see this with their own eyes. In the past, we had nothing but people’s words (“trust us”, “it’s settled science” [red flag]). I always believed (and still do) that emissions should be reduced. Not because I still believe it warms the planet, but because I want cleaner air, cleaner oceans, etc. CO2 isn’t the only waste produced.
But the link between CO2 and warming is a hoax. This means we have time to transition into cleaner energy sources at a pace that will not have damaging effects to the economy. This is a much more reasonable approach. But credibility has been lost and I fear that any message toward pragmatic solutions will be left on the roadside between two extremes.
You won´t believe this:
http://www.foreignpolicy.com/articles/2009/11/30/the_fp_top_100_global_thinkers?page=0,4
Rajenda Pachauri nominee as one of the ten most brilliant thinkers of the world.
I am in agreement with
vboring (08:36:32) :
Improving the review process would be good, but I think the key to the
issue here is the fact that legislation is being based on hidden data and
methods.
So, what we need to do is establish standards of openness that are
required for any science that is being used as a foundation for legislation.
All data and methods being used to justify public policy must be open
source.
however, following my scientific reviewing/verification position that primarily data impacting public welfare, or receiving public funding, must be openly available (possibly after a brief time lag of say three months for some priority reasons – but no longer) I have a couple issues:
1) Peer reviewing publications will/have been themselves reviewed threw field experience regarding their own credibility in the field, however:
2) money/influence will/have always been a competing determinant in a peer reviewers conclusions vs. his/hers objectivity.
These concerns will always be present and open data will address much of that, but other matters will arise. This time thanks much to the Blogs “peer reviewing” abuse is being exposed and hopefully greatly corrected in some fashion (beyond open data).
…and the 68 runner up is….James Hansen!
I think related to this is the attempt to push “Post Normal Science” as an AGW politico-scientific methodology. As has been pointed out elsewhere, Post Normal Science sounds an awful lot like Feynman’s Cargo Cult Science along with Pathological Science. If someone tries to change the rules of science for one particular type of science, that should raise red flags (I intentionally phrased this sentence this way as changes the rules in science in general is what gave us the scientific method in the first place as that was a broad principle rather than something meant to be an exception).