Climategate begs the question: "is peer review in need of change"?

The following is from the blog Master Resource. Run by Robert Bradley. There are many interesting and well written essays there, and WUWT readers might benefit from a visit. Be sure to support them by checking out their books and leaving comments. – Anthony

Climategate: Is Peer-Review in Need of Change?

by Chip Knappenberger

December 1, 2009

In science, as in most disciplines, the process is as important as the product. The recent email/data release (aka Climategate) has exposed the process of scientific peer-review as failing. If the process is failing, it is reasonable to wonder what this implies about the product.

Several scientists have come forward to express their view on what light Climategate has shed on these issues. Judith Curry has some insightful views here and here, along with associated comments and replies. Roger Pielke Jr. has an opinion, as no doubt do many others.

Certainly a perfect process does not guarantee perfect results, and a flawed process does not guarantee flawed results, but the chances of a good result are much greater with the former than the latter. That’s why the process was developed in the first place.

Briefly, the peer-review process is this; before results are published in the scientific literature and documented for posterity, they are reviewed by one or more scientists who have some working knowledge of the topic but who are not directly associated with the work under consideration. The reviewers are typically anonymous and basically read the paper to determine if it generally seems like a reasonable addition to the scientific knowledge base, and that the results seem reproducible given the described data and methodology.

Generally, reviewers do not “audit” the results—that is, spend a lot of effort untangling the details of the data and or methodologies to see if they are appropriate, or to try to reproduce the results for themselves. How much time and effort is put into a peer review varies greatly from case to case and reviewer to reviewer. On most occasions, the reviewers try to include constructive criticism that will help the authors improve their work—that is, the reviewers serve as another set of eyes and minds to look over and consider the research, eyes that are more removed from the research than the co-authors and can perhaps offer different insights and suggestions.

Science most often moves forwards in small increments (with a few notable exceptions) and the peer-review process is designed to keep it moving efficiently, with as little back-sliding or veering off course as possible.

It is not a perfect system, nor, do I think, was it ever intended to be.

The guys over at RealClimate like to call peer-review a “necessary but not sufficient condition.”

Certainly is it not sufficient. But increasingly, there are indications that its necessity is slipping—and the contents of the released Climategate emails are hastening that slide.

Personally, I am not applauding this decline. I think that the scientific literature (as populated through peer-review) provides an unparalleled documentation of the advance of science and that it should not be abandoned lightly. Thus, I am distressed by the general picture of a broken system that is portrayed in the Climategate emails.

Certainly there are improvements that could make the current peer-review system better, but many of these would be difficult to impose on a purely voluntary system.

Full audits of the research would make for better published results, but such a requirement is too burdensome on the reviewers, who generally are involved in their own research (among other activities) and would frown upon having to spend a lot of time to delve too deeply into the nitty-gritty details of someone else’s research topic.

An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other. A few journals incorporate this double-blind review process, but the large majority does not. I am not sure why not. Such a process would go at least part of the way to avoiding pre-existing biases against some authors by some reviewers.

Another way around this would be to have a fully open review process, in which the reviewers and author responses were freely available and open for all to see, and perhaps contribute. A few journals in fact have instituted this type of system, but not the majority.

Nature magazine a few years ago hosted a web debate on the state of scientific peer-review and possible ways of improving it. It is worth looking at to see the wide range of views and reviews assembled there.

As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable.

Instead of improving the process, it has nearly destroyed it.

If the practitioners of peer-review begin to act like members of an exclusive club controlling who and what gets published, the risk is run that the true course of science gets sidetracked. Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.

Certainly as a scientist, you want to see your particular branch of science move forward as quickly as possible, but pushing it forward, rather than letting it move on its own accord, can oftentimes prove embarrassing.

As it was meant to be, peer-review is a necessary, but not sufficient condition. As it has become, however, the necessity has been eroded. And blogs have arisen to fill this need.

In my opinion, blogs should serve as discussion places where ideas get worked out. The final results of which, should then be submitted to the peer-reviewed literature. To me, blogs are a 21st-century post-seminar beer outing, lunch discussion, or maybe even scientific conference. But they should not be an alternative to the scientific literature—a permanent documentation of the development of scientific ideas.

But, the rise of blogs as repositories of scientific knowledge will continue if the scientific literature becomes guarded and exclusive. I can only anticipate this as throwing the state of science and the quest for scientific understanding into disarray as we struggle to figure out how to incorporate blog content into the tested scientific knowledgebase. This seems a messy endeavor.

Instead, I think that the current peer-review system either needs to be re-established or redefined.

The single-blind review system seems to be an outdated one. With today’s technology, a totally open process seems preferable and superior—as long as it can be constrained within reason. At the very least, double-blind reviews should be the default. Maybe even some type of an audit system could be considered by some journals or some organizations.

Perhaps some good will yet come out of this whole Climategate mess—a fairer system for the consideration of scientific contribution, one that could less easily be manipulated by a small group of influential, but perhaps misguided, individuals.

We can only hope.

===

Please check out Master Resource

Advertisements

100 thoughts on “Climategate begs the question: "is peer review in need of change"?

  1. Improving the review process would be good, but I think the key to the issue here is the fact that legislation is being based on hidden data and methods.
    So, what we need to do is establish standards of openness that are required for any science that is being used as a foundation for legislation.
    All data and methods being used to justify public policy must be open source.

  2. From my experience, peer review is vastly over rated, but it is better than nothing.
    It’s up to the editors to insure that politics does not hijack the process. Very difficult to keep politics out when research funds and power are the underlying issues.

  3. I have sent the following to my MP (UK):
    Following the release of emails from UEA CRU, I would hope that you will support a thorough, independent enquiry into both the goings on at that institution, and, more importantly, a thorough re-evaluation of the science, as far as that is possible, given that certain original data sets have been destroyed. As it appears that the peer-review process in climate science has been compromised, such an enquiry should request a re-evaluation, by independent people, of sceptical papers that were rejected by this discredited process.
    If the AGW theory that CO2 will cause dangerous climate change is proven and accepted by all, then let us get behind policies to reduce CO2. If on the other hand it is disproved, or unproven, then let us get on with saving the planet by focusing on reducing population growth (= eliminate poverty), habitat loss (especially rain forests), pollution (especially marine pollution), over fishing and the effects of by-catch and other issues of genuine importance.
    The political class seems to have ignored this scandal, and the mainstream press, where they acknowledge it, have it on page 10.
    Please don’t reply to me saying there is a “consensus”, or that the “science is settled” or that the IPCC has 2500 author’s who all agree – none of this is true.

  4. CRU is just one example of the increasing politicization of science. And, not surprisingly, it runs both ways. Under the Bush administration, numerous scientists in the Forest Service and BLM chose to leave rather than sign off on scientifically unsound decisions that were pushed by resource users – users with connections in high places.
    Most climate skeptics are conservative to very conservative. I happen to be one of the handful of left-sceptics on the planet. Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?

  5. I think all science grads should have to study Karl Popper. The logic of scientific study and the open society and it’s enemies should be compulsory.

  6. Review needs to be independent. From the e-mails, peer review in the CRU tape letters was manipulated and slanted. Corrupt is another word.

  7. Peer review definitely has limitations. With Climategate I’m reminded of Galileo and the peer review he was subjected to and I see AGW having the same type of dogma as well as excommunications. Galileo used the scientific method while the peer reviewers engaged in dogmatic groupthink. It really troubles me – particularly when I read scientific magazines – how peer review (rather than adherence to the scientific method) is used to justify the standing of AGW. I felt rather sad when I read the latest New Scientist on this topic with how they blew off “hide the decline” on the basis of peer review. Science needs to be open sourced and the scientific method will win out. Open sourced science will discourage tribalism encourage scientific excellence of peer reviewers.

  8. The Crime of the Century
    November 30, 2009 by Doctor Zero
    Few recent events have illustrated the ineptitude, and political agenda, of the mainstream media more dramatically than “Climagate.” The revelation of email correspondence from the Climate Research Unit of the University of East Anglia, documenting various attempts to suppress data and manipulate scientific “consensus” with thuggish tactics, confirms what critics of the global-warming movement have always maintained: it has a lot more to do with money and politics than science. In fact, the global-warming movement is essentially the opposite of science – the manipulation and destruction of empirical data to support a theory whose accuracy was decided in advance.
    Those tempted to compare global warming to a religion should consider that religious faith does not require the willful suppression of knowledge. Calling the belief in global warming a “religion” is an insult to religion. It’s a mistake I have made in the past, and I owe the faithful of all religions an apology for doing so.
    Global warming is a scam, pure and simple. By any objective measure, it’s the crime of the century, with a dollar value that dwarfs the sins of Bernie Madoff or Enron. People like Al Gore have become millionaires by selling books and “carbon credits” to their marks… many of whom knew perfectly well they were being taken for a ride, but felt political pressure to play along, or saw opportunities created by the exercise of raw government power. The economic damage from legislation passed in response to this hoax will run into trillions of dollars, if Barack Obama’s disastrous cap-and-trade legislation passes the Senate.
    An objective media would respond to this blockbuster news story with front-page headlines and “special report” television treatment. By now, the authors of the incriminating Climate Research Unit emails would be infamous around the world. Top operators of the global warming racket, such as Al Gore, would be hiding in their mansions, afraid to face the mob of angry reporters gathered outside. Liberals love to accuse big corporations of manufacturing crises and taking advantage of consumers with false product information and deceptive advertising. Here is the paramount example of those offenses, on a scale that would widen the eyes of the greatest titans of industry. If a private corporation had conducted a scam as vast, and as destructive to the prosperity of nations – and the aspirations of the working poor…
    … but no private corporation could do anything like this, could they? The global warming scam is the kind of crime that only Big Government can mastermind.
    Continue reading at: http://hotair.com/greenroom/archives/2009/11/30/the-crime-of-the-century/?print=1

  9. I don’t think the problem is with peer review as such, but with politicized science. Climate “science” as currently practiced blurs the distinction between researcher and political advocate. Mann, Jones, Hansen, et al. do not merely promote the alleged correctness of their research, but agitate for political change as a result of it. Fundamentally, the IPCC is a political, not a scientific, organization.
    If a biologist publishes a paper asserting, say, that the Amazonian spotted frog is the same species as the Patagonian spotted frog, no politician or activist is going to care. And it is highly unlikely that peer review would be corrupted in this case.
    The solution, then, is to separate the politics from the science. To this end, all genuine scientists should shun researchers who corrupt the scientific method for political ends and who attempt to leverage their prestige as “scientists” to agitate for social change.

  10. I finally got a response from my MP, James Paice:

    Dear Mr (me),
    Thank you for your two emails about the science of climate change, which I read with interest. However I am afraid that I do not agree with your assertion that human activity and high carbon emissions have not interfered with the stability of our climate.
    There is a wealth of scientific evidence that clearly indicates that climate change is happening, and that states we are responsible. I do accept that the Earth’s climate naturally changes over long time periods. But all previous warming and cooling periods can be explained by natural causes, whereas the work of the Intergovernmental Panel on Climate Change shows that the extremely rapid warming of our current period can only be explained when mankind’s influence is included.
    It is a sad fact that mankind currently dump 50 billion additional tonnes of heat trapping gases into our atmosphere annually, while at the same time massively reducing our planet’s natural capacity to absorb these additional emissions (by, for example destroying the tropical rainforests). With this happening every year I find it difficult to believe that this process does not influence the climate system beyond its natural fluctuations.
    I am aware that there are differences in opinion around the issues of climate science, and it is right that there is a healthy debate. However, as a politician rather than a scientist, I do not think it would be sensible to ignore the warnings of the majority of the scientific community [my emphasis].
    What we must do is assess the risks of both action and inaction, and develop policy accordingly. The overwhelming balance of evidence makes clear that our economy, our national security and our way of life are under considerable threat if we do not move to reduce the risk of ever-increasing green house gas emissions. And while action will cost money – insuring against risk always does – the economic and social costs of inaction will be far higher than the costs of action.
    Moreover, much of the action to decarbonise our economy will be economically positive. A whole new generation of well paid, long lasting jobs will be created as we develop new British industries and technologies. Our homes and businesses will be protected from ever fluctuating fossil fuel prices and will save money due to greater levels of energy efficiency. Our national security will also benefit as we will generate more of our power at home in the UK, rather than be exposed to ever increasing fossil fuel imports as our own reserves become depleted.
    Should you like to read more about Conservatives views on this issue, please consult the policy document; “The Low Carbon Economy, Security Stability and Green Growth”, which you will find under the policy section of http://www.conservatives.com
    Once again, thank you very much for writing to me on this important issue.
    Yours sincerely,
    James Paice MP

    In other words, we have to trust the Scientists. I’m not sure he even knows about the CRU leak or if he does, he didn’t mention it even though that was the subject of my second email. I suppose it’s time to wheel out that Eisenhower quote about public policy being held captive by a scientific and technological elite.

  11. List the peers. Let their reputations be at stake for poor or corrupt work. Footnote disagreements between the author and the peers. In other words, TRANSPARENCY and ACCOUNTABILITY.

  12. “Most climate skeptics are conservative to very conservative. I happen to be one of the handful of left-sceptics on the planet. Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?”
    I was for AGW legislation (in other words these were “my folks”) until I read RC’s explanation of “hide the decline,” which they turned me into a skeptic by their explanation that showed that the scientific method wasn’t practiced. What I want is good science and I hope then that good policy will come from good science. I want everyone held to high standards. Much of politics is about personal opinion based upon one’s individual values, but of the politics that is based on science, the science used for policy must be to high standards and beyond reproach. I’ve never been one to identify with political parties, but Climategate has re-enforced my belief that tribalism is harmful and will try even harder to be sure that I’m not engaging in tribalism where I put an agenda above objective facts.

  13. I disagree. Blogs are very useful to discuss things and sometimes shed light on issues, but then there should be a way to move that to the scientific magazines. In this respect, blogs are as useful as any place where people can meet and talk. Fine for making up some drafts, but not anything that intends to contribute to the scientific knowledge just because of being put in a blog. Blogs are places for discussion, not for presentation of results.
    The peer-review process has to improve. The author mentions double-blind reviews, which I agree would be a good thing. Also transparency of the process itself, which again I agree that it is needed. But most importantly, all of this scandal has to do with replicability of studies. Given that you cannot guarantee that the reviewer has done all the verifications needed, as it would be extremely time-consuming, you have to make sure that third parties can, if they want to. You have to force authors to make all data and methodology publicly accesible. As simple as that. Those who want to keep their data or methodology private, shall not be allowed to make their conclusions public, or pretend that they are scientific.
    Science is not science without TRANSPARENCY.

  14. Mike Hulme and Jerome Ravetz have written a rather excellent piece for BBC News:
    http://news.bbc.co.uk/2/hi/science/nature/8388485.stm
    Their vision for an “extended” peer review process that reaches out to the public is noble. It would certainly facilitate the trapping of errors overlooked by insiders blinded by “Groupthink.” Left unaddressed however, is the “gatekeeper” problem: how does someone outside the establishment overcome the bias of “peers” in order to publish paradigm-shattering ideas that are genuinely sound?

  15. Ultimately, it’s up to the editor. If a reviewer makes unreasonable criticisms, then the editor has to make a judgment.
    The double-blind idea is interesting, though I must note that sometimes one could still guess who wrote the paper – just as one can sometimes guess who wrote the anonymous review, based on the content.
    I don’t share your enthusiasm that blogs will replace much of anything; rather, they will add something new. Blogs can be a place where ideas are refined, informal discussion takes place, and the laymen can learn or ask questions. Sometimes, a paper can have its roots in a blog post. That’s all useful and good. But face it, most blog comments are from non-specialists, who don’t “have some working knowledge of the topic”; their helpfulness in improving the work is thus very limited. A formal peer review process is still necessary, though not sufficient.
    Open peer review is very interesting, but it would again come to the authors and ultimately the editor to decide which comments and criticisms are most helpful and worth heeding. Participation could also be sporadic; reviewing a paper takes time out of the day, and so you aren’t going to run around voluntarily reviewing everything that comes along.
    As for what the emails show, keep in mind that publishing a single paper and writing the IPCC assessments are different processes, but ultimately, the papers in question did end up being mentioned in the IPCC report. As for anything else, if you think a paper makes basic errors which are apparent on a quick reading, it’s your job as a reviewer to point them out.

  16. @John Egan
    I’m also mostly liberal in my politics, especially relative to the environment.
    I don’t think we’re so rare. I think there is a decent sized group:
    Climate Skeptics for the Environment!
    I think AGW has been distracting us from real provable environmental problems for too long.

  17. Whilst we’d all desire a fairer system for peer review, I wonder that there is a broader issue.
    Imagine for a second that there are no sceptics, and we all believe the experts on a particular topic. How does that become corrected?
    Asking for fairer peer review is not unlike asking for a fair Guru; the Guru can always turn round and say to his devotees, “I am the Guru, I know more than you, and if you disagree, it is because of your own ignorance.”
    Those who deeply disagree have no option but to leave. Perhaps they start their own alternative, and wait years for recognition. But say everyone is captivated by the authority of the Guru, and as far as anybody knows, everything he or she says makes sense. What then?
    In trying to imagine this scenario, I come to think that the real problem with AGW theory, isn’t that peer review should have exposed flaws sooner—experts will argue whether even the flaws that have been discovered are of any significance in the end anyway—the problem is more subtle and wider. I suggest the problem is that we imagine it highly unlikely that the best scientific methods and data could, even when everyone agrees they are of high quality, turn out to be wrong.
    In other words, even if 100% of all scientists (really all of them) came up with a watertight theory of global warming, and nobody had reason to be sceptical of it, we should still, for no other reason than simple, “survival”, have a plan B in case the world cooled instead of warmed. And we should not act as if we knew warming was going to happen.
    We far overestimate our collective abilities (and I’m trying to say this respectfully to all scientists) whilst underestimating the complexity of the future world.
    I realise this makes any debate much more complex, if we shift from arguing about what we think will happen, to what we don’t know might happen. But disasters are typically the nature of the unthinkable.
    “We have to act” should never have been a collective response to the knowledge of the day. I’d prefer, “we need to be ready to adapt to a variety of scenarios”.
    We have many technological marvels, but nonetheless, stuff doesn’t always work the way we expect, even when we really really expect it.

  18. Dave
    I am curious as to why you were for AGW legislation until the Real Climate response to the UAE data dump. Had you reviewed any of the skeptical arguements prior to this and were unconvinced or were you just taking Gavins position as valid. I think it would be enlightening to some of us skeptics to know as we try to have the issue debated honestly.
    thanks

  19. From the CRU e-mails, I think it’s pretty clear that the peer review process in climate science is broken, at least with respect to the journals that the climate cabal (Mann, Jones, et al) were associated with. For example, here is an e-mail from Michael Mann that illustrates the problem:
    Original Filename: 1047388489.txt
    From: “Michael E. Mann”
    To: Phil Jones ,rbradley@xxxxxxxxx.xxx, …
    .
    .
    .
    It is pretty clear that thee [sic] skeptics here have staged a bit of a coup, even in the presence of a number of reasonable folks on the editorial board (Whetton, Goodess, …). My guess is that Von Storch is actually with them (frankly, he’s an odd individual, and I’m not sure he isn’t himself somewhat of a skeptic himself), and without Von Storch on their side, they would have a very forceful personality promoting their new vision. There have been several papers by Pat Michaels, as well as the Soon & Baliunas paper, that couldn’t get published in a reputable journal. This was the danger of always criticising the skeptics for not publishing in the “peer-reviewed literature”. Obviously, they found a solution to that–take over a journal!
    So what do we do about this? I think we have to stop considering “Climate Research” as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal. We would also need to consider what we tell or request of our more reasonable colleagues who currently sit on the editorial board…
    What do others think?
    mike

  20. @Dave –
    Your reasons for questioning the AGW line are quite similar to mine. When the argument depended upon ignoring the Medieval Warm Period, dismissing the Urban Heat Island Effect, and requiring an unproven CO2 feedback loop – I started to get suspicious. When people like James Hansen and Ellen Goodman started saying that any further debate was no longer permissible and that those who did were comparable to Nazi murderers – then I was convinced.
    There is plenty of scientific evidence to suggest that global temperatures have increased and will continue to increase – albeit at a far more gradual rate than the IPCC suggests. Also, it is not prudent for humans to be fouling the only planet known to support life. Although I do not think that CO2 is the bugabear than some believe, I think it prudent to reduce emissions in a sane manner of the long term.
    And, yes, science should play a crucial role in our future – a role that is, hopefully, as free as possible from political manipulation.

  21. As a clinical medical scientist, I spent many hours as the Associate Editor of a Journal allocating reviewers to manuscripts, reading the reviews and making decisions as to whether papers were accepted or not. Occasionally, an author would make a request that his paper not be sent to a particular reviewer. This would generally be honoured but we would extra careful to ensure that the reviewers we did choose were likely to give a fair review.
    Reviewers were expected to declare any conflicts of interest, for example, the author was a former colleague or they had reviewed the paper previously.
    With the “Climategate” story, it seems to me that there has been collusion between the authors, trying to limit the editors’ choice of reviewers, the editors in giving in to their demands, perhaps out of sympathy for the authors’ stance on AGW, and the reviewers for not declaring their relationship with the authors and other conflicts of interest. The reviewers, and the editors were at fault also for not insisting on seeing enough details of the methodology to be able to be sure that they were appropriate.
    There is fundamentally little wrong with the peer review process as long as all concerned behave with integrity. The problem here seems to be that those involved regarded the ultimate message of AGW as more important than the scientific validity of the research.

  22. I’ll add one more issue with blogs: you aren’t going to see people putting too many fresh new results out there, before submitting a paper. You don’t want to get scooped.

  23. In climate science, “peer review” means the science has been approved by a peer of the British realm, most likely the Prince of Wales.

  24. >>I don’t think the problem is with peer review as such, but with politicized science. Climate “science” as currently practiced blurs the distinction between researcher and political advocate. Mann, Jones, Hansen, et al. do not merely promote the alleged correctness of their research, but agitate for political change as a result of it. Fundamentally, the IPCC is a political, not a scientific, organization.
    Winner, winner, chicken dinner.

  25. One meme from the article that is a repetition is on how onerous it would be on the peer reviewers to actually check the work.
    The onus should be on the authors to provide the material in such form as to be essentially turnkey.
    It is, indeed, quite difficult to take a journal article from the cold (that is, not precisely your own area of interest) and try to replicate the math. Not even the results – just the slogging from the raw data to the tabulations presented. The details of precisely what the process actually is tends to spread out amongst a collection of papers. “Here, I’ve used the approach of Paper3…”
    But if the journals held this standard – and bounced things for “Supplementary Material Insufficient,” the author is the one who would be doing the vast majority of the necessary extra work.
    Marching through the references you’ve used and making a neat table of precisely what data you’ve used isn’t exceedingly difficult for the author. It is excessively voluminous for inclusion in the core text of the article. But this sort of “Appendix” for the article should be compiled by the author anyway to be able to answer questions at presentations and poster sessions.

  26. “same standards applied when “their folks” are in the White House?”
    Absolutely
    The public is not that stupid, and the other choice is to let government take over our lives and make all decisions for us.

  27. Most modern desktop computers can easily burn a CD with a large amount of data, using software such as Nero, for example. Thus, raw data from even extensive projects can be supplied to anyone with a legitimate request. The FOIA disputes should be ruled out of science in the first place — any data concealment should be considered an obvious breach of ethics and the perps excluded from research work. Moreover, this is not a ‘hard-line’, conservative, or political policy — transparency is a basic condition for any legitimate academic or scientific work.

  28. The entire notion and focus on peer-reviewed papers published in journals as a basis for decisions that effect the health and safety of the public is simply wrong.
    There is no other public-policy-decision arena, not a single one, in which peer-reviewed published papers form the primary basis of such decisions. The climate change issue remains the sole, and very significant singular, exception to this statement. And it is well-established that decisions in this area will impact the health and safety of everyone in the entire planet.
    In all other public-policy-decision arenas, there are Independent organizations that have the responsibility to Review, Verify, and Validate the information prior to making decisions that effect the health and safety of the public.
    Generally, some part of the information presented for review to these Independent organizations might be based on peer-reviewed published papers, but that is not a necessary condition and is most certainly not a sufficient condition for acceptance of the information.

  29. Since so many climate science studies involve computer models, how does one verify the results without also reviewing the source code and the input?
    Quite simply, one cannot.

  30. Scientific journals are just another form of printing, along with publisheres of newspapers, books, magazines, or comic books. To get published you convince and editor and his reviewers. Getting something published doesn’t make it true or even good. Scientific journals are businesses, and not that altruistic. Peer review is not the holy blessing that makes the published work science. The true peer review is the discussion which happens after publication, and this is scattered.

  31. “I am curious as to why you were for AGW legislation until the Real Climate response to the UAE data dump. Had you reviewed any of the skeptical arguements prior to this and were unconvinced or were you just taking Gavins position as valid.”
    Generally speaking I trust the scientific community to operate honestly and to engage in best practices. Everytime something is announced is science, I don’t go to the source to see if I’m being deceived. In general scientists are seen a very credible. It wasn’t that I knew much about the details one way or the other as I just trusted the process – that the process was following the scientific method. Reading the answer given by RC instantly raised a red flag with how they handled the dendro because it looked like their hypothesis was invalidated (the hypothesis being that dendro – at least with the methods they used – is an accurate temperature proxy) by real world results, but instead of going back to the drawing board they kept the flawed results in their while dumping the inconvenient data that disproved their hypothesis. Based on real world data that tells me that the dendro methods used under-report warm weather, which undermines the validity of their temperature reconstructions. I wouldn’t expect an undergrad science student to do this (unless they were going to either drop out or change majors), let alone a very senior scientist who is working on setting policy for the world.
    I’m not even a scientist, but I’ve had to do number-crunching as a financial analyst. Just to be a FA you have to be transparent and show how you arrived at your numbers so that way you can pass an accounting audit. I would just expect scientists to be at least as rigorous as accountants in matters of number-crunching transparency as I would expect a “scientific audit” to be wider in scope than an accounting audit – scientists all over the world trying to reproduce your results as opposed to a single audit firm double-checking your work without trying to reproduce it. Instead what I see is shades of Madoff along with the clubby atmosphere that let him get away with it.
    Even with science that involves politics, I assume scientists to always exercise best practices. Like I look at the Manhattan Project scientists where many of those involved had multiple levels of political disagreement, but the scientists didn’t use their political disagreements as an excuse to fudge their science. With AGW I just thought those scientists (like any other scientist) would have science be at the forefront of what they are doing instead of trying to bend science to politics.

  32. I think the failings of the peer reviewed process has been overstated. After all the shenanigans we have read about, skeptical papers have in fact been published. You can read emails where the likes of Santer talk about writing a paper to refute Douglass’s paper.
    The real and much larger problem comes from the politicization of the whole science, from the IPCC through a fawning media and gullible learned societies. Ask yourselves, what difference would it have made if the peer review process worked perfectly? No skeptical papers would still have been included in IPCC reports, and they certainly would not have been mentioned in the media.
    What to do then? The first change to make, is to make it a firm rule, that all data and methods must be archived with every peer reviewed paper. If we had had that simple mandate to begin with, we wouldn’t have needed FOI requests, we wouldn’t be struggling to reverse engineer and second guess the true meaning of various temperature trends and proxy studies. All would have been clear from the beginning, and scientists would have been able to challenge outrageous statements years ago, before any of this warmist nonsense had reached critical mass.

  33. I agree with the article. However, one big hurdle for peer-reviewed paper access is their price. It’s way too expensive to get a copy if you don’t have a library with the journals nearby. When you think that the reviewers are not paid to review papers, why such high price for copies. Should we start a P2P torrent system to have access to peer-reviewed literature? When you are a small independent research laboratory, you can’t afford to pay for journals if your local university does not have copies.

  34. Isn’t it the editor’s responsibility to enforce the peer process? And didn’t they violate their own written peer process by playing hide the data?
    Which editors should resign first?
    Open science is the only science. The rest is just a game.

  35. For those who are liberals turned skeptics, remember that George Orwell was a socialist who was one of the most scathing critics of socialism as it was practiced.
    Remember too that there are as many varieties of conservatism as there are liberalism. Being for the environment doesn’t mean that you can’t be conservative, or even libertarian.
    My concern is that this farce is taking resources away from the real environmental issues. If the idiots running the government can’t even repair damage to the Chesapeake Bay, how in the world could they take on the whole climate?

  36. “Is peer review in need of change”? – Of course.
    Climategate have revealed what Wegman pointed out a long time ago “..there was too much reliance on peer review, which was not necessarily independent.”
    With climategate we can with confidence say, “peer review” as used by “the hockey-stick team”, was and is, definitely NOT independent.
    This “peer review” process has been just a small clique of scientists back slapping each other, covering up their mutual errors and publishing wrong and fraudulent papers.
    Moreover this “peer review” has been used as an excuse and mantra to “refute” and avoid arguments against their findings.
    Has that arguments been “peer reviewed”?
    No?
    Well then 1. We neednt answer it and 2. It must be wrong.
    False – An argument fails if it is wrong in fact or logic. It has nothing to do with “peer review” and any compelling argument needs to be answered on its merits.
    The team has neatly avoided this by quoting (their own) papers or referring the argument to the IPCC.

  37. “In all other public-policy-decision arenas, there are Independent organizations that have the responsibility to Review, Verify, and Validate the information prior to making decisions that effect the health and safety of the public.”
    – – – – –
    Bingo!
    And the difference is, in all other public-policy-decision arenas, the reviewers would never stand for the first-level testers – the scientists, the researchers, the data-gatherers – claiming some proprietary right to keep the underlying data and processes secret while revealing only their conclusions.
    Historically, many old societies did look to one individual source of information concerning the natural world and their interaction with it, and did have to be content with simply accepting the proclamations from that source as having been well-founded. Those sources, called in some societies “witch doctors”, knew that the best way to secure their own power was to make their decisional processes opaque.
    Jones says he’ll release most of his data “next year.” Why not “today”?

  38. “I think AGW has been distracting us from real provable environmental problems for too long.”
    I agree. One of the things that bothers me is the framing of this where those who are skeptical are labeled as political right wingers potentially in the pay of big oil. Unfortunately I see many people biting, which many of those who most go along with the framing can’t see how they are putting the environment at risk by wanting to devote so much time and resources into something that might not do much at all for the environment. I can think of a number of reasons for instance that I’d want to save forest habitats, which none of those reasons involves carbon credits. It’s rather sad thinking of all the billions/trillions of dollars that could be wasted on the environment (CO2), when true environmental problems are ignored both financially as well as not given much bandwidth in high profile discussions.

  39. Double-blind is great in principle. But in practice it is an invitation for cheating. (Although there seems to have been a huge amount of cheating under the current system.)

  40. “Climategate begs the question: ‘is peer review in need of change’?”
    As cjcjc has pointed out, “begging the question” has a specific meaning in logic, a commodity sadly lacking in AGW debate, for the most part. Let’s not pollute the language of reason by loose usage of terms specific to that field.
    “An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other.”
    The Scheme Team would get around that in 24 hours with a few phone calls or emails.
    “Unsettling as this may be for scientists, the combination of ‘post-normal science’ and an internet-driven democratisation of knowledge demands a new professional and public ethos in science.” [From Hulme and Ravetz’s piece for BBC News]
    After close reading of the BBC piece, I sense that Hulme & Ravetz still haven’t understood. The fact that they seemingly accept ‘post-normal science’ as something that can be fused with an Internet mobocracy into some workable, new scientific Utopia by simply sprinkling on a little “ethos” is ludicrous. Their article does contain some valid points, but I recommend reading it with great skepticism.
    Post-normal science is evil and should be rejected in toto.

  41. In 2005, I was designated to be my team’s GHGV “expert” by my then employer British Standards in an effort to build a strong client list of Canadian “emitters” to sell our services to. Ironic though that my exploration into the field of climate research (in order to be of value to my employer and clients) brought me an understanding completely removed from where I was before I started asking questions.
    As a trained process auditor, and someone who believes in continuous improvement through methodologies such as the Shewhart cyle, internal auditing, documentation of non-conformances, respective corrective actions and above all, transparency, i’m pleased to see that an ISO technical committee has been established to develop an international standard for statistical data and meta data exchange (ISO/NP TS 17369).
    An external, unbiased 3rd party audit of the quality principles relating to any data and the respective code used in models would be a good start in at the very least validating the quality of the processes used to create hockey sticks.
    Cheers Anthony, and keep up the great work!

  42. I propose “Jonesing the journals” as a new phrase, kind of like “Mann-handling the curve.” 🙂

  43. Excess fussiness on my part, but I had to say that ‘begging the question’ is not the same as ‘inviting the question’ or ‘raising the question’ or its equivalents. Begging the questions is a rather poorly named logical fallacy.

  44. While I have no thoughts on how to improve peer review, I do have this reference on why invoking it as proof someone is right should elicit sighs of frustration and/or laughter.
    http://www.iscid.org/papers/Tipler_PeerReview_070103.pdf
    Perhaps there should be more than one process, with different journals subscribing to different methods. I.e., let the marketplace of ideas sort out who’s more effective in championing good science.

  45. Robinson (08:54:39) :
    I emailed to my MP, George Osborne on the subject of global warming, and received a letter. What has surprised and appalled me is that it is word for word , paragraph for paragraph IDENTICAL to your letter.
    What an insult to the intelligence of voters, and what a total disregard for opinions of the electorate. Of course we really knew it, but it isn’t common to have it thrust in your face quite so brazenly.
    The level of arrogance exhibited by these people means that there is no chance that they will take any heed of any information that will change their attitude to global warming – they are in it for the power and the public acclaim (and the money, given the performance of UK MPs on the subject of their expenses over recent years).
    So James Paice wouldn’t have known about Climategate, and I suspect it is unlikley that he will even have seen your letter. At the first mention of a sceptic viewpoint your letter will have been categorised as being from a denialist crank, and the standard boilerplate response will have been generated. One might expect the signature to be someone elses too.
    I will now write again and see what response is forthcoming – perhaps you will do the same with Paice and we can again compare notes.
    I have also written to another MP, Douglas Carswell, who appears to have a less rigid view on global warming, but have not as yet received a reply.

  46. After it is all said and done the net effect of the CRU whistleblowing will be as follows:
    1) The current versions of the major temp data sets will have a version publicly on the web. It will be almost unusuable in its public form but it will be there to insure transparency.
    2) All new studies will be based on the public datasets that the study authors “improve” in ways that are incoherent and impossible to reproduce. The studies will still be “transparent” because it cannot be helped that skeptical morons cannot figure out the amazing techniques the gods of climate employ.
    3) A journal or two may change their peer review process and there may be a few actual skeptic papers published that would not have been otherwise. The “concensus” group will ignore these published studies like the plague and they will not be included in any IPCC or other summaries.
    4) Phil Jones will be admonished and kicked upstairs and another “lackey” with clean credentials will take over the CRU dataset to clean it up and make it transparent. Lost data will not be recovered and the adjustments made to data will not be changed. The adjustment process will be briefly explained but no skeptical moron will be capable of duplicating the adjustments made by the climate gods.
    5) People will forget about climate gate after about 6 months and things will go back to the pre-existing condition.
    6) Concensus scientists will learn to write very nice politicaly correct FOIA sensitive emails while at work.
    7) Concensus scientists will learn how to communicate via personal Blackberrys in order to handle all other “correspondences” off the FOIA grid.
    8) In the year 2020 a hacker will discover a system cache of Blackberry text messages between AGW scientists that documents them laughing about keeping skeptics unpublished, boosting warming trends in their datasets and how they circumvented their FOIA restrictions.
    9) The Court of Climate Denialism will sentence the Blackberry text hacker to death by injection in a special IPCC televised “Climate Justice” episode.
    Shiny
    Ed

  47. John Egan,
    You ask a legitimate question. I’m a conservative. Not a scientist but I love reading about science. Science must be kept honest and open or it will not advance. Politics should never be married to science. Nor, come to think of it, should science ever rely on mass media to tell the truths of science – journalists who do know what they are talking about (darn few) are still subject to the restrictions of the message their management supports. Frankly, I think the openness of the internet is one of the best things that has ever happened to science, politics, and anything else that needs the light of day. We have too few people in this world who understand either the scientific method or the limits of science to provide certainty.

  48. Perhaps as part of the review process the data and methodology (software) should be distributed to reviewers. Reviewing the data and code would be a part of the process. It’s doubtful anyone would attempt to completely reprocess the data to arrive at the same analysis, but it would put the brakes on deliberate fudging. If the paper is accepted, the code/data is also made available to subscribers. This would encourage some housecleaning on the data processing prior to publication – if you know you have to package everything up for the peer review, it would discourage leaving it in a messy state.
    Authors should not suggest reviewers. Editors should be familiar enough with the scientists to know who would be a good reviewer. If not, perhaps they should not edit that journal. Reviewers should be anonymous. I don’t think you could do a double-blind review as the reviewer could probably figure out who wrote the paper just from the style, citations, and emphasis in the research.

  49. If one was publishing results of an experiment, isn’t the author generally expected to provide enough information for anyone interested to replicate the experiment, and presumably the results? Isn’t that really the final say in peer review? If independent experimenters can’t replicate results, the author needs to explain why not.
    Why do the journals not hold climate science to a similar standard? (yeah, we all know why now, but 2 weeks ago I had some doubt). It’s generally not possible to prove a statement true, the best you can do is not prove it false – so the strongest support for the AGW hypothesis would have been to give the raw data to anyone interested, explain any adjustments, and why they are valid, and have people potentially hostile to the theory have at it. If the AGW theory were as ‘robust’ as we’ve been told all these years, no one would be able to refute it, and releasing data would only make the AGW arguments stronger. That would be real peer review; impossible to set up a situation where the only review is a softball by someone already sympathetic to the cause.
    I can only think of one reason why [it seems like most of] the AGW side are so determined not to release their data. The Climategate emails confirmed my suspicions. It also demonstrates that the peer-review system is worse then useless, along with several scientific journals.

  50. I think the article misses the point. It’s not peer review that is the problem- one thing that is the problem is the deliberate, repeated presentation of peer review as a system which ensures good science, and that peer reviewed papers are in some sense trustworthy or even “true”. Journals are supposed to be a forum in which scientists can present their work- whether it be ultimately correct or incorrect- to other scientists, to add to the general pool of knowledge in the science. They are not repositories of Truth. You cannot create, nor should attempt to create, a process which only allows “true” work into journals. Publication is merely the start of a process of new scientific knowledge being added to the corpus of scientific knowledge in general.
    Where some process of auditing needs to occur is when science breaks out of the bounds of the scientific community to affect, in particular, politics. When collective decisions are made on the basis of it, when lives or zillions of groats are being wagered on it, then it needs to be scrutinised as finely, and with as much hostility, as possible. Effectively, when science morphs into politics, it ceases to be simply a matter for scientists. This is the central problem. Scientists jealously guard their independence (eerily, despite having political views which promote interference by the state in every other aspect of human existence) and science, itself, should indeed remain independent. But once it has become enmeshed in politics, once it becomes something which profoundly affects lives and livelihoods, it cannot any longer cling to that independence. Because it isn’t science any more, it is politics.
    “Minister, you must implement this policy because scientists say so” is not good enough. This is the point when the science and the scientists must be subjected to the most rigorous examination for error and bias. The real scandal of the IPCC is that despite claiming “peer review”, in fact none of the science in their reports is checked in any way. If a paper has been published and a chapter author decides to include it, it is presumed by the system to be true. An IPCC, or equivalent, which were really doing its job would be auditing every paper, and the auditors would be hostile and looking for flaws. The journal system of peer review is no business of anybody’s but scientists and journal editors. They must be free to publish what they wish. It is at the interface with politics and society that the rest of us must get our oar in and demand the very highest of standards and most rigorous of checks and balances- because it is only at this point that the stakes become high.

  51. The best peer review ever: WUWT…and here is the only place on earth that a scientific paper will have more than 3 millions hits!. That will surpass the pop rock stars.

  52. A valid warning from an “old soldier” and a competent politician:
    Dwight D. Eisenhower
    Farewell Address
    delivered 17 January 1961
    Excerpt: “Akin to, and largely responsible for the sweeping changes in our industrial-military posture, has been the technological revolution during recent decades. In this revolution, research has become central; it also becomes more formalized, complex, and costly. A steadily increasing share is conducted for, by, or at the direction of, the Federal government.
    “Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers. The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present — and is gravely to be regarded.
    “Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientific-technological elite.”
    What Ike feared could happen did happen. Public policy has become the captive of a scientific-technological elite. The AGW proponents.
    I would rather see the peer review process be done away with altogether, with all proposed papers published on the internet, for all interested people to be able to review and comment on. Then these comments could be culled to be rid of all comments (responding documents) which contained personal/ad hominem attacks. There is no place for such in a “scientific setting”.
    There are a good many of us who are not involved in the “college scene” who nevertheless have a good understanding of various basics of science, and a good many years of experience, having seen many concepts which seemed like a “good idea at the time” which contained serious basic errors.
    During my own professional career, I was always pleased to have “devil’s advocates” on my staff or as reviewers. A good many times these “devil’s advocates” did advance serious questions about my proposed approach to “solving problems” as I call what I did, which if not accounted for, would have resulted in at least minor disasters and large wastes of public funds.
    People who work for private companies or on their “own dime” are one thing, and it is mostly “the market” which will determine the success of their intellectual endeavors. But when the taxpayer is the one footing the bill, all work done should be subject to review of those taxpayers, at least those who are interested in review.
    AS lot of work, certainly. But a lot of public moneys are riding on any ultimate public policies based on these research projects and endeavors.

  53. It seems to me that the real culprits are those non-team members who did not demand the release of the data and methods. It doesn’t seem that peer review failed, there was no peer review without the data. I think that the internet mobocracy is the best place to vet these issues. With respect to scientific arguements of sufficient import to be widely interesting, it is much more likely that the mob will get it right eventually than a couple of randomly assigned, otherwise busy, reviewers.
    The free exchange of scientific data and methods over the internet will drastically improve the speed and accuracy of the scientific process

  54. In simpler words, I much enjoy Anthony’s methodology. Anthony has been doing excellent work, his publishing of his endeavors is most interesting to those of us who have a real interest in the science of it, enjoyable to read, and comments do flow in. I am sure that Anthony does take into consideration those comments which are based on actual science, even if there are those of us who are not “professional climatologists” making comments.

  55. cjcjc (08:36:10) :
    Good article.
    NB Climategate raises that question, it doesn’t beg it.
    /What he said.
    All good scientists should know what “begging the question” means. It is a fallacy similar to circular reasoning.
    Example:
    AGW is universally accepted because there is a consensus.
    see why it’s a pet peeve of mine? 🙂

  56. “Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.”
    anna v wrote an interesting post here
    http://wattsupwiththat.com/2009/11/27/zorita-calls-for-barring-phil-jones-michael-mann-and-stefan-rahmstorf-from-further-ipcc-participation/#comments
    (22:52:36)
    In her analyisis the process of peer review is subject to some very human herd behavior, but is still self-correcting in general. She gives two reasons why that process was short circuited in climate science.

  57. “Here’s the question – are those who are clamoring for more rigorous scientific review processes willing to see the same standards applied when “their folks” are in the White House?”
    John,
    I fear you totally miss the heart of the issue. I’m not sure what you are driving at since a)most federal employees at NOAA or NASA are not appointed by any President; thier tenure usually far out lasts any sitting President. and b)much of the work done is accomplished at Universities.
    Also, I’ve never heard (with the exception of Far Left blogs) of any atmospheric science professional being asked by President Bush to fudge his work. And from we can tell, this politicization of climate science appears to be mainly a problem for the Left.

  58. To my mind, the principle problem with peer review lies in the rhetorical use the alarmist crowd has made of it in their propagandistic efforts. The mantra like repetition of the term was clearly and intentionally meant to reinforce the implicit fallacy that peer review existed to guarantee the correctness of a paper’s conclusions, something it was never intended to provide and which the present controversy has shown to have been moved much farther from its original intent within the realm of climate science. Unfortunately, they have been all too successful in imbedding this fallacy as conventional wisdom in the minds of those outside the scientific community unfamiliar with peer review’s actual purpose and ,from many of the comments I’ve seen here and elsewhere, many in the science field who should know better.
    Given that a majority of the “science” produced in the climate field involves complex and arcane computer programs, which few potential reviewers have either the time or expertise to evaluate adequately, it is not clear that providing complete and open access to data and methods will significantly improve the effectiveness of peer review in this area.
    Ultimately, peer review is a peripheral issue to the main problem, which is the almost complete domination of the funding of science in all fields by government provided money controlled by politicians more interested in their own agendas than in the advancement of science.
    The debate that we really have to create is how we can address this very intractable quandary. Science in all areas is becoming increasingly expensive and it’s hard to envision a way to move toward other sources of funding that would be adequate to the purpose and it’s equally difficult to imagine a system which can insure that, if governments are to remain the the primary funding mechanism, safeguards can be created which will be able to remove political considerations from the allocation of grants. I don’t have any easy answers for any of this, but I believe it is something we need to be thinking and talking about on an urgent basis, as we are already far down the road to the “post science” world where all of science is driven by someone’s agenda and if that world is allowed to materialize fully, humanity will be damaged in ways that make Gore and Hansen’s most hysterical prognostications seem like a hangnail.

  59. Ian B (10:54:05) :
    “I think the article misses the point. It’s not peer review that is the problem- one thing that is the problem is the deliberate, repeated presentation of peer review as a system which ensures good science, and that peer reviewed papers are in some sense trustworthy or even ‘true’.”
    While I agree with that, you haven’t pursued it far enough. Remember that the pirate crew at CRU hijacked and perverted the peer review process. As such it was “peer review” in name only. Perhaps “co-conspirator review” might be a more apt title, with the crooks vouching for each others’ honesty.
    Peer review at it’s best may need some tweaking, but when it works it’s mostly relatively effective. Unfortunately, there is never any guarantee against corruption. Only constant vigilance, of the kind we’ve seen by so many scientists and others honest citizens fearful of the consequences, can minimize the chances of this happening. And the more wealth and power that are at stake, the more vigilant one must be.

  60. I’m liberal in many areas. So I think people will be surprised as to how many liberals will change their minds after discovering climategate for themselves. As a programmer, the code and log file is what convinced me that the model was bogus. I’ve now seen it with my own eyes. Others will also see this with their own eyes. In the past, we had nothing but people’s words (“trust us”, “it’s settled science” [red flag]). I always believed (and still do) that emissions should be reduced. Not because I still believe it warms the planet, but because I want cleaner air, cleaner oceans, etc. CO2 isn’t the only waste produced.
    But the link between CO2 and warming is a hoax. This means we have time to transition into cleaner energy sources at a pace that will not have damaging effects to the economy. This is a much more reasonable approach. But credibility has been lost and I fear that any message toward pragmatic solutions will be left on the roadside between two extremes.

  61. I am in agreement with
    vboring (08:36:32) :
    Improving the review process would be good, but I think the key to the
    issue here is the fact that legislation is being based on hidden data and
    methods.
    So, what we need to do is establish standards of openness that are
    required for any science that is being used as a foundation for legislation.
    All data and methods being used to justify public policy must be open
    source.
    however, following my scientific reviewing/verification position that primarily data impacting public welfare, or receiving public funding, must be openly available (possibly after a brief time lag of say three months for some priority reasons – but no longer) I have a couple issues:
    1) Peer reviewing publications will/have been themselves reviewed threw field experience regarding their own credibility in the field, however:
    2) money/influence will/have always been a competing determinant in a peer reviewers conclusions vs. his/hers objectivity.
    These concerns will always be present and open data will address much of that, but other matters will arise. This time thanks much to the Blogs “peer reviewing” abuse is being exposed and hopefully greatly corrected in some fashion (beyond open data).

  62. I think related to this is the attempt to push “Post Normal Science” as an AGW politico-scientific methodology. As has been pointed out elsewhere, Post Normal Science sounds an awful lot like Feynman’s Cargo Cult Science along with Pathological Science. If someone tries to change the rules of science for one particular type of science, that should raise red flags (I intentionally phrased this sentence this way as changes the rules in science in general is what gave us the scientific method in the first place as that was a broad principle rather than something meant to be an exception).

  63. Dave Wendt (11:32:46)
    Very well put.
    No perfect check may be found for accenting good science. Somewhat similar to regulating the Market – all the regulations will not eliminate all the abuse or excessive exploitation.
    Re science perhaps limiting Gov. size would reduce many of the review problems.
    Increasing amounts of science/research/tech being done make for ongoing and evolving challenges to quality research. Good luck.

  64. The Fraud Is Everywhere: SUNY Albany and Queens University Belfast Join Climategate (PJM Exclusive)
    http://pajamasmedia.com/blog/the-fraud-is-everywhere-suny-albany-and-queens-university-belfast-join-climategate-pjm-exclusive/
    Posted By Douglas J. Keenan On December 1, 2009
    Some of the emails leaked in Climategate discuss my work. Following is a comment on that, and on something more important.
    In 2007, I published a peer-reviewed paper [1] alleging that some important research relied upon by the IPCC (for the treatment of urbanization effects) was fraudulent. The emails show that Tom Wigley — one of the most oft-cited climatologists and an extreme warming advocate — thought my paper was valid [2]. They also show that Phil Jones, the head of the Climatic Research Unit, tried to convince the journal editor not to publish my paper.
    After my paper was published, the State University of New York — where the research discussed in my paper was conducted — carried out an investigation. During the investigation, I was not interviewed — contrary to the university’s policies, federal regulations, and natural justice. I was allowed to comment on the report of the investigation, before the report’s release.
    But I was not allowed to see the report. Truly Kafkaesque.
    The report apparently concluded that there was no fraud. The leaked files contain the defense used against my allegation, a defense obviously and strongly contradicted by the documentary record. It is no surprise then that the university still refuses to release the report. (More details on all of this — including source documents — are on my site [3].)
    My paper demonstrates that by 2001, Jones knew there were severe problems with the urbanization research. Yet Jones continued to rely on that research in his work, including in his work for the latest report of the IPCC.
    Misconduct at Queens University of Belfast
    Arguably, the biggest concern with global warming is that warming itself will cause further warming. For example, the polar ice caps reflect sunlight back into space, thereby cooling Earth. A global warming theory suggests that if the caps shrink due to warming, then they will reflect less sunlight and so Earth will warm even further. It is possible that Earth warms so much that it reaches what is called a “tipping point,” where the global climate system is seriously and permanently disrupted — like when a glass of water has been tipped over and the water cannot realistically be put back into the glass.
    No one knows for sure how much Earth would have to warm before it reaches the tipping point — though about a thousand years ago, there was a time known as the Medieval Warm Period when much of Earth appears to have been unusually warm. It is not currently known just how warm the Medieval Warm Period was, but clearly the warmth then was below the tipping point because Earth’s climate continued without problem.
    Suppose that during the Medieval Warm Period, Earth was 1°C warmer than today. That would imply that the tipping point is more than 1°C higher than today’s temperature. For Earth’s temperature to increase 1°C might take roughly a century (at the rate of increase believed to be currently underway). So we would not have to be concerned about an imminent disruption of the climate system.
    Finding out how warm the Medieval Warm Period is thus of enormous importance for the study of global warming.
    It turns out that global (or at least hemispheric) temperatures are reflected by the climate in western Ireland (for a short explanation of that, see my site [4]). Trees grow in western Ireland, of course, and each year those trees grow a ring. Thick rings indicate climate conditions that were good for the trees; thin rings indicate the opposite. If many trees in western Ireland had thick rings in some particular years, then climatic conditions in those years were presumably good. Tree rings have been used in this way to learn about the climate centuries ago.
    Queen’s University Belfast has data on tree rings that goes back millennia — and in particular, to the Medieval Warm Period. QUB researchers have not analyzed the data, because they lack the expertise to do so.
    They also refuse to release the data. The story is scandalous.
    I have been trying to obtain the data [5] via the UK Freedom of Information Act since April 2007.
    —————————————-
    (Nature.com had a brief piece about my FOIA request [6]. The piece has statements from QUB that are dishonest: see my comment posted there. Statements from QUB therefore should be checked. My site has source documents for its claims.)

  65. For a guide on how to proceed when the peer review process becomes unreliable, look at the testing of medical therapeutics in the US. We have developed a set of rules and regulation (Good Laboratory Practices or GLP) to ensure a high standard of quality in studies involving actions that could endanger the lives of people. Peer reviewing is not a requirement. Rather, what is required is a quality assurance unit that is properly staffed, empowered, and trained. Universities need quality assurance units to protect faculty from themselves. Science will recover when institutions ensure quality.

  66. Yeah, I think this was raised before, but Climategate raises the question “Is peer-review necessary?” The climategate cabal’s insistence of publication in prestigious peer-reviewed journals begs the question since its necessity is assumed in their statements.

  67. It is a bit of a diversion to argue about how to improve peer review.
    As people have said, the problem was the taking of “peer review” as some kind of guarantee of correctness—and this article seems to be saying: well if only we could improve it when it fails, then we could take it as even more correct.
    The simple truth is that boffins can make blunders.
    It should always be acceptable for the people and leaders to say, “our strategy is to do nothing and hope that the scientists got it wrong.”
    Whomever was driving this AGW agenda, seems to have known that this is how people ordinarily see things, and decided to counter it by making scientists overstate their claims and their levels of confidence.
    “All the world’s top scientists are unanimous”…. yeah well so what? But then they accuse you of being “anti-science” and a “flat earther”.
    It is not anti-science to admit that there are limits to what can be known with high confidence. To respect those limits is to respect science, just as a film-maker respects what his lens can shoot, and an athlete respects the limits of their physical endurance.
    It is about respecting the craft, not about turning scientists into oracles for our favorite agenda.
    At the end of the day, you can listen to science, but that still leaves you with the problem of making a decision about risk. Peer review doesn’t magically disappear the risk. There is always the chance that the scientists are flat wrong, and that whatever you try to do under their advisement turns out to have unintended consequences. It is almost an existential problem—people don’t like feeling that they are flying blind through an uncertain universe. But just wait a bit, wait a bit… look here comes another surprise…

  68. This is a sober and realistic appraisal of the review system as it currently works. But I think double-blind reviewing would be an illusion. It’s usually easy to work out from the text of a paper who wrote it. Many papers emphasise that the work is a continuation of other published work (by the same authors) that is necessarily referred to in the paper.
    But I’m surprised this audience doesn’t have more respect for market solutions. There’s no prescribed peer review system at present. There are conventions, but journals operate their own processes, and are free to vary. They thrive or fail on the success of their policies in producing good papers.
    You don’t need to have a paper peer reviewed to get it out there and read, as we well know. It’s just that people do trust more papers that appear in reviewed journals. That’s their choice.

  69. Using medical terminology, when a patient presents in the ER with infection, the first thing to do is “debread” the wound (remove all the dead and infected tissue). Unfortunately, AGW is not a patient, it is a rotting corpse. There is nothing to treat.
    Alternatively, we could look at the problem as a cancer, which has metastasized. The pathology is everywhere in the body: in addition to the body scientific, it’s all throughout the body politic and the body economic. This is a very sick world, and it is their fault. The epithet “CLIMATE CRIMINAL,” which clowns like Monbiot throw at us, really aptly applies to them.

  70. Climategate begs the question: “is peer review in need of change”?
    Short answer – it depends on who you ask.
    Here’s my list:
    Patrick J Michaels, attempts made to get rid of his PhD thesis

    Stephen J Crothers, PhD denied
    http://www.worldsci.org/php/index.php?tab0=Scientists&tab1=Display&id=1119
    Vincent Courtillot, a French geo-magneticist, director of the Institut de Physique du Globe in Paris studying temp and solar activity (posted several times here) – information denied
    http://network.nationalpost.com/np/blogs/fullcomment/archive/2009/11/26/skewed-science.aspx
    Dr. Keenan, information denied
    posted above by Ed Scott
    Great posts here. I like Ike!

  71. Don’t know if anyone reads down this far, but the institutions of science that we’re all familiar with, such as peer review and the less-than-complete descriptions of data and methods, all evolved in an environment where they maximized the progress of science.
    In a pure-science world populated with independently funded curiosity- and reputation-motivated researchers and journal editors, each scientist has pretty much the correct incentives under the traditional system:
    Your goal in reading the literature is to figure out which results to build on in your own quest for discoveries and recognition.
    Publishing scientists want to be just as careful as necessary to get people to cite them and build on their work, but not so careful that they slow themselves down and get scooped.
    Journal editors use peer review so that scientists will want to read their publications in their search for useful clues for their own research.
    As long as everyone is primarily motivated to make discoveries and get cited, the incentives line up very well. Read Michael Polyani’s “Republic of Science” essay for a good description of this world and all the optimal sloppy shortcuts it entails.
    The problem is when the incentives and consequences change, so that scientists want to influence policy and attract funding by taking certain positions. Then 1) biases are likely to creep in and 2) the traditional level of reproducibility of results will be below that necessary for policy makers to make costly decisions. What’s good enough for curiosity-driven debates is nowhere near good enough for redirecting industrial civilization.

  72. Follow the money!
    Look, peer-review in reality is a control gate to publishing, and some people must do so or perish, meaning not gain tenure and lose funding. The people who control the gate ultimately control the money flow.
    It’s all about the funding. If you’re in with the CAGW cabal, you get funded and published. If you’re out, you’re out in every way imaginable, including losing your job.
    That’s what’s wrong with the system. That’s how it is politicized. Follow the money.

  73. [snip] The BBC now steps into the fold big time after keeping mum for so long, and what ist their message? They advocate more of this “post-normal science” snake-oil, so that we won’t have all this trouble about data and complicated old-fashioned science in the future but can instead rely on consensus of “social actors”!!! And they make it sound all roses and kittens, to boot.
    http://news.bbc.co.uk/2/hi/science/nature/8388485.stm
    Oh dear, oh dear. This CRU data leak may badly, badly backfire on us!
    For more, and mighty revealing, qoutes of the BBC authors’ (Hulme and Ravetz) , pls see here:
    http://buythetruth.wordpress.com/2009/10/31/climate-change-and-the-death-of-science/

  74. “is peer review in need of change”?
    I laughed when I read this.
    Changing the system won’t eliminate corruption. Corrupt players will angle any system!
    The most that can be achieved: A change in whose turn it is to be dominant, perhaps…
    the ebb & the flow – another phase …
    My preference: Liberalize the d*mn system! so that some fresh ideas can stir the stagnation. Tight-*sses can have their turn running the system again some years out after creative forces have a turn.

  75. Peer review has never been pristine. It has been used by one cabal to limit the publication of other cabals since the beginning of time (i.e. 1600). Sir Isaac Newton, one of the greatest scientists Mankind has ever produced, was not above using position to stifle a competitor. I have been subject to this type of attack in my field of engineering and I must admit that I took glee in returning the favor when the oppotunity arose. How many of us in many fields of study know of researchers, scientists, or engineers who will never sit at the same table with each other, ever! Despite the conflict, it is part of how science moves forward because the arguments get aired. BUT, it is effective only as long as the journal editors stay above the fray so everyone gets heard eventually. Have the editors of the climate journals maintained an honest debate? It does not appear so.

  76. Whether we want a mess around us is one thing. Needlessly killing animals until they become extinct is another of those one things. “Preserving as existing” is quite another. Causing the water in rivers and lakes to be so pure that it provides little or no nourishment to various “critters” is stupid. Whenever humans have attempted to do what is obviously stupid, it never worked.
    The planet turned quite nicely all by itself until humans came along, and will continue to turn quite nicely if humans get themselves gone through human stupidity.

  77. Having seen both sides of the review process, I offer the following thoughts:
    1. For the most part, serving as a reviewer is a labor of love – the principal returns being the satisfaction of doing one’s civic duty and getting an advanced peek at something new and interesting.
    2. Sometimes, however, it is also a means of promoting self-interest. This certainly seems to have been the case with Jones, Mann, etc., whose behavior is really a stain on the profession.
    3. Double blinding the process, as suggested by other respondents, sounds fine in principal, but in practice probably won’t prevent the kind of abuses revealed by the CRU emails. After all, if you’re object is to suppress results inimical to your own ideas, the author’s identity is immaterial.
    4. Archiving raw data, computer code, etc. – enough to allow the interested reader to replicate the results – is a reasonable requirement. But, under the present system, I doubt that most reviewers, having neither the time nor the incentive, would avail themselves of the opportunities access to such information would provide.
    5. Publishing reviewers’ names along with the articles might provide some such incentive; likewise, publishing the reviews themselves. In effect, one’s reviews would become a form of publication to be added to one’s CV, clucked over in the course of annual fitness evaluations, etc.
    6. The fundamental point about CRUgate, or whatever you call it, is that it’s an exceptional case. Both paleoclimate reconstructions and climate modeling are in the hands of small, tightly knit communities. Moreover, the essence of what they do cannot be replicated by individuals. To check the validity of CRUs gridding procedures, you need an outfit of roughly equivalent size and funding; ditto, for the large scale models. This allows for what amounts to monopolistic practices, the pursuit of which is favored by the enormous sums of money involved, both in the form of direct support to the investigators and overhead returns to their respective universities.
    7. Regarding the latter (overhead), I have seen little, if any discussion. Yet it is really central. If, as a Professor at the University of the Antipodes, I receive $100K to study the turtle that holds up the earth , my university tacks on an additional $50K as overhead. Overhead is the crack cocaine of the academy. The original rationale was to enable universities to provide laboratory facilities, etc., to support the research being funded. The present-day reality is that these moneys effectively go into the general fund. Many (most?) research universities are utterly dependent on them. If meaningful investigations of CRUgate go forward, one of the things that will emerge is the fact that the universities involved turned a blind eye to abuses that were apparent to anyone willing to keep his eyes open.
    8. Equally corrupting is the ideological mindset that makes AGW, and more generally, sustainability, so attractive to both faculty and administrators. Many (most?) universities are up to their eyeballs in this sort of thing; likewise, the funding institutions. So you get positive feedback: money and ideology foster the promulgation of “correct” results, which, in turn, generate more money and reinforce the ideology. The parallel with Lysenkoism is instructive.
    9. There are already rules in place that forbid practices of the sort that seem to have been revealed by the CRUgate emails. The single most important thing that can be done to restore the integrity of the science is to penalize those who have broken the rules: ineligibility for future funding, say for the next 10 years, would be a good first step, by providing an example both to individual investigators and to their institutions of what can happen when scientists and their universities become just another interest group. Michael Ruse, the philosopher (and no Creationist), has discussed this sort of thing in the context of “evolutionism,” which he characterizes as a secular religion. Essentially, Ruse argues that the people doing the science are often them same individuals promulgating this, that or the other social policy and that, in the classroom, the two often get conflated. Ruse’s point, I believe, applies with equal, if not greater, force to environmental studies generally.
    10. The other reform I would recommend is prohibiting participation of working scientists in groups such as the IPCC that, claims to the contrary notwithstanding, are effectively in the business of proposing policy. Here again, the current state of affairs, whereby individuals wear multiple hats, generates positive feedback that encourages self-serving and unethical behavior.
    With apologies for having gone on at such length.

  78. IMHO, the end state will be “Public Review”.
    We have a competition now between “blind review” and “blogs” and the blind peer review is losing. Given the petty political infighting that people seem unable to avoid, any “blind” process will end up in that state. Even double blind is no cure. Folks have a “voice” in their writings and in small fields, the positions folks have taken is pretty clear.
    So the web based review will get ever more of the “suppressed” or cutting edge raptid breaking stuff. The “end around” the political road blocks will mean that more of the interesting stuff will have bypassed the slow and politicized track.
    In the end, someone will make a “peer reviewed web publishing service” where the “reviewing” will be done real time and in the open on discussion pages. Once thrashed out (when a tread has gone stale) it will be ranked and added to the archive. Some will get highest honors, some honorable mention, and some listed under “nevermind…”. WIth all the comments, commenters, and their biases and affiliations, visible to all.
    And in that context science will advance faster and further than ever before.
    And nothing can stop it for happening. The only question is “when”.
    Frankly, the whole “secret handshake blind reviewers on paper” thing is just so archaic. It smells of old boy networks and folks wanting a political angle to work…

  79. Thanks to those who linked to UNEP.
    I was reading some of UNEP’s online material and there is mention of a strategy of uniting the world by reaching down past the government level and into the institutions in society, like academia and businesses and interest groups. In other words, I can see how this is an effort to water down the importance of nation states and their boundaries.
    There was a time when the city state was the largest social power structure, and nowadays, whilst I still pay taxes to the city or county, I also pay takes to the nation’s government. For a global system, you can add a new level of taxes on top of that, and for different things.
    We will all continue to be members of a nation, but it won’t matter much, just like I am a member of a city or county but that doesn’t matter much.
    The new global system will reach down into academia (and hence, “the science”) and business (and hence, “big energy”), and draw power from allegiances built there. In turn the global government will use controls to alter the flow of resources globally, and “manage” the world “ecosystem”.
    If we need more “rainforest” over here, we’ll pay the locals to plant trees rather than build factories—whole areas will have their development directed and coerced into whatever direction is deemed appropriate from above.
    I can appreciate the vision of watering down the nation state. After all, why should I get to have a better life just by the accident of where I happened to be born? I could easily have been born a starving kid in Africa. So I don’t want to be too pessimistic about this idea of watering down the nation state in favour of a global system; I’m sure it will work pretty well after a thousand years of trial and error.

  80. Science used to be ratified by a system of “peer usage” – that is to say other scientists actually had to find a real use of the original science in a different application – or ideally in a number of different applications. Hence Newton’s theories on gravity became accepted science because the theories could be used to make calculations in a variety of other areas where the calculations worked and worked well.
    “peer review” is a relatively new process that has only gained traction in the last 40 years or so. It seems to me to be driven by economics rather than good science because obviously funding research on the basis that the science can be proven to be worthwhile (via peer usuage) is so much more difficult than doing science and have a number of your mates give it the thumbs up and then all get funding. Peer review lends itself to exactly the kind of financiallly motivated corruption we have seen in the AGW camp. It also leads itself to what psychologists and socialogists call “an escalating commitment to a particular course of action”, since it tends to result in informal teams of pals getting together with a particular belief system all inadvertently encouraging each other to validate each other’s erroneous conclusions in order to remain part of the in-crowd. They actually don’t realise that they are lying to each other and they have built a whole set of conclusions based on lying to each other and now they can escape from their own tower of lies.

  81. A software audit could be done by “non-experts” quite easily. The authors publish what the code is supposed to do and the software auditor can review and test the software to ensure it does exactly that. The auditor has no need to know why it is being done or if it is even the right way to do it, they would simply verify that the code does what the author claims. That way the peer reviewer who can read the authors explanation of what the software is supposed to do is freed from checking if it is in fact doing it. The qualifed peer reveiwer can judge if the why is reasonable and justified.
    This is simple software quality assurance that is done everyday at thousands of software development shops, firms and independent developers.
    A software audit would include actually running the program against the raw input data the author provided and comparing that to the published data points and a review of the code itself to ensure that “special case” subroutines do not exist (i.e. ignore the post 1960 tree data) …
    Call it the pre-peer review audit, any work submitted must pass at 100% before being actually submitted for peer review.

  82. There are no doubt some improvements that can be made in the peer review process but the true nature of the beast that is besetting us is all too human. Professional societies, organizations (to include those of the hard sciences) and their “journals” can rarely take political positions or make endorsements and retain their credibility with the public (not to mention their own members). When the leadership –or outspoken members– of these organizations have obviously begun to take political stances they have currupted themselves and their organization. Consider this: the military in most advanced nations are allowed to vote freely for whomever they choose, but they are forbidden to wear their uniform and endorse –or actively canvass/campaign for– any political party or candidate over another (while in uniform). Professionals –especially hardcore scientists– and their professional organization(s) actually face similar restrictions when dealing with the public and political issues whether they want them or not. As soon as a scientist (or a professional organizations “leadership”) takes a political stand (or begins to exclude their fellows for their diverging views) the entire membership is painted with the same brush in the eyes of the public. There are “laws” in science, and there are “laws” of human nature. Unfortunately for all of us, some scientists and the leadership/editors of some professional organizations believe that they are special, that the laws don’t apply to them, that they are better and smarter than everyone else is. Life isn’t that simple. No one gets to have it both ways. No doubt, for some, the damage done will necessitate renaming their journel and/or deactivating the organization itself and beginning afresh. The two orgainzations I love to hate the most serve as perfect examples: The American Medical Association and The American Bar Association (I didn’t want to hit too close to home but I think the point is clear to all. There are others.)

  83. A small team (<12) of software developers/business analysts could be easily assembled and paid for by the vartious publications and submitting institutions to be the software audit team for all publications. Would cost about 2-3 million per year and would have caught this nonsense of Climategate years ago.
    Oh and all sumbissions and their test results would be made public. That way the world could see if a particular scientist had a track record of submitting sloppy software (i.e. code not match the explanation or hidden special case subroutines) … I'll call it sloppy software vs fraud, but in the world of software they amount to the same thing. Clean in = Garbage out.

  84. sorry for the poor spell check in my original comment 🙁
    As an aside, this software audit team could be hired directly by the various submitting scientists to do a private review of the software as well. They would be charged a fixed fee and the results would remain private. Many academic institutions would not have the QA staff available to do a good audit of their own and would benefit from this sort of division of labor. As an incentive to maintain their adversarial relationship with the scientists any submitted software that passes the data tests would be given a refund of a portion of their fee. That way the audit team would have an incentive to find errors as opposed to having an incentive to ignore them.

Comments are closed.