Margaret writes in tips and notes:
More about the failure of peer review— or more precisely its inconsistency in producing reliable assessments of the value of the submitted article
http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844
Abstract
A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.
The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.
With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.
You shouldn’t refer to something from 1982…
Yep, this paper is a classic. Things are far, far, far worse now.
Peer review is not confined to academic papers, it is a standard method of quality control used in software development. The results are very similiar to those found with acedemic papers, it depends how it is done. If the organisation requires that someone else ‘reviews’ the work, but does not specify a methodolgy it usually results in ‘pal review’ on a mutual assistance basis. You are not going to make life difficult for your co-workers in case they respond in kind. In this case it is bearly better than useless as a method of finding errors. In an organisation where you have no control over who the reviewer is and there is a defined methodology for the review process the results are far better. In all cases problems will still exist with what has been developed. Fortunately software development normally has adequate testing processes, something our friends in climate science are unable to achieve to any real extent.
@Margaret Hardman
May 29, 2013 at 1:14 am
“The paper of which this is the abstract was published in behavioural And Brain Sciences in May 1982.”
Yes, and haven’t things improved since that time?
“will keep them out somehow — even if we have to redefine what the peer-review literature is!”
– Phil Jones email to Michael Mann.
It’s worse with law review articles, which are student edited and only vetted to make sure that a cited source says what the author says it says. There is NO evaluation of the accuracy of the material presented. I’m sure it helps if the author is highly placed–a chaired position at a prestigious law school, for example. They seem to be the worst for putting stuff out without evaluation of the accuracy of the facts, knowing of course that no one else will vet it. I tried to lead a charge at my law school, with the journal I worked on, to improve the vetting for factual accuracy. No one seemed to care.
pesadia says: May 29, 2013 at 1:28 am … What is the opposite of, “you have made my day?”
Gimme back my minutes. So much to read, so little time.
In re sovereign currency, see Bitcoin, as I struggle to implement the various clients/protocols on my new-to-me linux.
Believe nothing that one reads or hears without verifying it oneself unless it fits ones preexisting worldview.
About nonsense accepted, I pray all here are familiar with the Sokal Affair, for me, the opening shots in the Science Wars. Physicist Alan Sokal wrote the essay ‘Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity’ that was accepted by journal Social Texts before he revealed his hoax. There is a large literature developed from the affair.
It is interesting that this article is from 1982, when volunteer reviewers had much more time on their hands to carefully review submitted manuscripts. Back in the early 1990’s, I had first-hand exposure to the pal review involved in pushing a manuscript onto the tenure-invoking, grant-raising pages of Science.
My suspicion is that the situation today is far worse than I thought. For example:
“I mean peer review is an utterly corrupt, ignorant stupid, mad system that we’ve created it’s just that we haven’t come up with anything better.
But let’s understand what peer review is:
Peer review is not about checking the validity of data
Peer review is not about reproducibility of data
Peer review is a check on acceptability, acceptability in the scientific community.
And that’s why I think you will still need editors, because our job is to be awkward. To say, even though you’ve got three or four peer reviewers who don’t like this, and don’t want it published. To hell with them, we’re still going to publish it. Because it still says something interesting. It looks interesting, even though it’s not accepted by this tiny group of people who we call peer reviewers.”
Richard Horton, editor in chief of Lancet, 02/04/13
Yes, that is 2013, as in this year, folks.
Yes, that is the editor in chief of Lancet, a renowned medical journal, folks.
Academic peer review is absolutely hit and miss. You get the whole spectrum from scrupulously fair, detailed, and helpful, over superficial and negligent, to dismissive one-liners and competitively motivated hatchet jobs. Just as one might expect from other experience with humanity at large.
So it’s from 1982. The Principia was published in 1687, that doesn’t mean its no longer valid. Apparently some think that anything done before the invention of the I-phone doesn’t count.
Alas, I once got rejected because of a reviewer’s statement: “He is not a member of the xxx community”. The editor in this case was guilty.
“eight of the nine were rejected. ”
Well, within error bars, that’s 80% rejected. It looks like this “80%” is arrived at by just random rejection.
For those distracted by the date of publishing there is a follow-up that is equally damning and so spot on regards the contemporary state of things it would be considered confirmation bias by Slick Lew:
http://cogprints.org/5179/
@Dodgy Geezer –
On a trip to England in 1979 I visited Nether Wallop, and do not recall seeing a university there. Am I correct in assuming there is still no university there?
Seriously, one wonders if some of the institutions spewing the AGW twaddle rather ought to be called the University of Sphincter, rather than by their present names.
David L. says:
May 29, 2013 at 2:20 am
I wonder how well Mann’s hockey stick paper would fair in resubmission.
______________________________________________________
In climate science, you’d be accused of plagiarism since everybody in the (climate science) world has seen it.
However, it would be very interesting if the underlying PCA analysis was applied to non-climate data (red noise perhaps!) and submitted in a field other than climate science. Preferably a field that is known for strong statistics.
Double-blind reviews isn’t quite the panacea it is sometimes made out to be. In fields with very many active scientists it might work. In more narrow specialties (like mine) it won’t. It would be very easy to guess who is author, and it is relatively easy to identify the referees. I can’t remember ever writing a paper where I couldn’t identify at least one of the referees, and sometimes all of them.
@dp
Your reference was published in 1982 as well. Anyone got evidence of peer review in the current century beyond some personal communications?
One would hope, perhaps desperately, that current web-based search facilities would make the results of such a test today a little more flattering to the peer review process.
I do peer review at a rate of three or four papers a year, plus short reviews (gradings really) of conference abstracts.
Much of peer review has nothing to do with the validity of the results, but rather with the organization of the paper, quality of writing, clarity of exposition, conformity to standards, whether the graphs are properly annotated, and so on. Reviewers do not have the resources to check up on the correctness of the results. If the author claims he or she did this and got that result, we pretty much have to take their word for it.
Thus an incorrect or fraudulent paper could easily pass peer review. I’m sure I could pull it off myself.
Here’s another paper that’s worth reading for those interested in the one above:
Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System
http://people.stern.nyu.edu/wstarbuc/Writing/Prejud.htm
Long story short: scientists find the conclusion of a paper more influential than its method
I wouldn’t be surprised if the same tests were conducted today, the results would be the same.
@Kuze
Your link goes back to 1977. My question earlier was to find something more recent that lets us know the current situation rather than using assertion or anecdote.
margaret,
Perhaps a better question is for you to find evidence that things have changed.
After all, like legal precedence, unless someone publshes something to the contrary, the research stands.
(If I were the preaching type, I might say “That’s how Science works!)
@chris y
Yes, editors are largely to blame.
“And that’s why I think you will still need editors, because our job is to be awkward. To say, even though you’ve got three or four peer reviewers who don’t like this, and don’t want it published. To hell with them, we’re still going to publish it. Because it still says something interesting. It looks interesting, even though it’s not accepted by this tiny group of people who we call peer reviewers.”
Richard Horton, editor in chief of Lancet, 02/04/13
I guess ‘ Because it still says something interesting’ might explain why Horton published rubbish like that on Pusztai’s potatoes, on which a Royal Society review concluded (excuse the Wikipedia):
‘that Pusztai’s experiments were poorly designed, contained uncertainties in the composition of diets, did not have a large enough number of rats, used incorrect statistical methods and lacked consistency within experiments. Pusztai responded by saying they had only reviewed internal Rowett reports, which did not include the design of the experiments or methodology used.’
The Lancet also withdrew the 1998 paper by Andrew Wakefield on MMR and autism only after this:
‘On 28 January 2010, a five-member statutory tribunal of the GMC found three dozen charges proved, including four counts of dishonesty and 12 counts involving the abuse of developmentally challenged children.’
And then there is the estimate of deaths from the Iraq conflict by opinion poll published in
The Lancet that found 601,027 deaths (range of 426,369 to 793,663) — far in excess of the estimates of the Iraq Body Count project, the UN or the Iraqi Health Ministry.
There is a serious corruption of the peer review process if editors start publishing things they find interesting – especially since those interests are likely to influence the choice of reviewers as well as decisions based upon their reviews.
As Carl Sagan once put it, ‘Where we have strong emotions, we’re liable to fool ourselves.’