Margaret writes in tips and notes:
More about the failure of peer review— or more precisely its inconsistency in producing reliable assessments of the value of the submitted article
http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844
Abstract
A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.
The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.
With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
No way!
Well it is psychology we’re talking about.
The only thing reviewed was the name of the submitting institution. No chance for backscratching -> reject.
So with the “correct” names, institutions and individuals, papers will pass peer review and be published.
Without those “correct” names, using fake ones that aren’t recognized, the same papers are now tragically flawed, rejected for publication.
What does this tell us about the peer review system?
It tells us editors are terrible at detecting plagiarism! They didn’t recognize papers their own journals had published. Don’t these editors read their own journals?
The paper of which this is the abstract was published in behavioural And Brain Sciences in May 1982.
1982 paper!?
Table of Contents – 1982 – Volume 5, Issue 02 (Special Symposium Issue)
I suspect the findings of this paper might be considered out of date, given that the research was carried out before I was born, and the peer review process was probably carried out using pen, paper and envelopes. Perhaps a case of falling for recycled news?
IKnow I shouldn’t be, but I am shocked by this article.
Is there any more information available.
What is the opposite of, “you have made my day?”
Did anyone at WUWT note the kerfuffle over the rogoff and reinhart paper on sovereign debt? They mad their data public, and sure enough some energetic grad student found a mistake in their XLS sheet. How embarrassing…esp. for paper that was massively covered in the press.
Interestingly, the main finding that gdp growth was under 2 percent w/ debt/gdp over 90 percent was changed to 2.5%. They used medians in this analysis…which tells me that their data is pretty noisy and has a large variance. (as i understand it, tho, others have found the same thing…about 2 percent with heavy debt loads)
A better description of papers,recycled or not, supporting the AGW hypothesis is “cronyism”. Peer review implies a degree of respectability, which it most certainly does not have, since the hypothesis is clearly wrong!
If there is value to this 30 year old article, it would rest with someone who repeated the experiment today to see if there is any noticeable difference.
If we only have this paper as a data point, you can hand wave that things are getting better or getting worse, or that psychology, perhaps one of the most subjective of published sciences, is much more susceptible to this kind of favoritism, which we know exists elsewhere, but to what degree?
Tim says:
May 29, 2013 at 1:21 am
I suspect the findings of this paper might be considered out of date, given that the research was carried out before I was born, and the peer review process was probably carried out using pen, paper and envelopes. Perhaps a case of falling for recycled news?
The ability to miss the point is strong in some. Tell me Tim, whilst plagiarism is far more easily spotted these days, how much do the think the institutional bias this paper highlights has changed. If you think it has changed, give us your reasons.
I wonder how well Mann’s hockey stick paper would fair in resubmission.
rk says:
May 29, 2013 at 1:32 am
“Did anyone at WUWT note the kerfuffle over the rogoff and reinhart paper on sovereign debt?”
Yes of course. The Krugmanites/Modern Monetarists need every excuse they can get to continue justifying Weimarian policies. It doesn’t matter, as the end result is known.
You call this a paper?
I have just made a modest contribution to the sum total of human knowledge; publishing a paper in the comments column of the Guardian. If anyone is interested I reproduce it here – detailed data available to bona fide researchers on request (but not if you’re looking to find something wrong with it!)
Metropolitan University of Nether Wallop
Behavioural change amongst Global Warming activists
D Geezer – Faculty of Advanced Chemistry and Roadside Catering
Global Warming seems to be having a hard time these days. The heady days of new papers proving without doubt that Global Warming is an immediate danger seem to have become rather thin on the ground, and in their place a new topic of discussion has arisen. This can best be described as a ‘Call to the Faithful’ – an exhortation not to lose heart as the science collapses around their ears. Running questionable surveys attempting to show that the hypothesis is still valid is one example of this type of behaviour. But can this opinion be backed up with evidence?
I have just conducted a short Cook-type survey to examine this phenomenon. The methodology was as follows:
1 – find an independent presenter of Global Warming news stories. I picked ‘Climate Debate Daily’ – a site which provides one pro and one anti item each day.
2 – examine and caregorise the ‘pro-global warming’ stories into three headings, using the abstracts provided. I picked the following categories:
a) – a story providing New Data on global warming (typically reports of technical papers)
b) – a story emphasising Solidity of Belief in global warming (typically reports of pro-global warming activity)
c) – Other Stories (often comment on political or industrial activity)
Note that the New Data do not have to support Global Warming theory – they just have to be stories providing new information. In practice, most of the stories under this heading actually indicated that the threat was smaller than had been assumed.
The results for the most recent 30 are as follows:
Following the Cook methodology, I dropped the ‘Other’ figure. The percentages then become (rounded to my error bars):
Thus it is shown that the Global Warming industry spends three times as much effort on preventing people leaving the faith as it does on showing that the faith is correct. Which supports the thesis at the beginning of this piece…
Let me guess, the one paper that made it through a second peer review was originally written by Psychologist, Diederik Stapel, of Tilburg University.
Old news. Paradigm paralysis – if you’re within the reigning paradigm, you’re safe. If you dissent, you’re in for some trouble.
http://en.wikipedia.org/wiki/Paradigm#Paradigm_paralysis
The only astonishing thing about this is that a 30 year old article is still paywalled (and it might not have been in the first place).
When the peer-pal reviewers see the required funding stream buzz words….
the gatekeepers cross the subliminal honesty barrier….
they then let the mono tribe droids pass….hence….
“climate change” invokes kum-baya vibes and peer-pal approval.
I can trump your 30 year old article on recycled papers with a 1 year old article on a paper consisting entirely of computer generated nonsense being accepted
http://thatsmathematics.com/blog/archives/102
It is worth reading for the extremely amusing referees report.
However this isn’t as surprising as you might think the paper seems to have gotten through only in the extreme garbage end of the academic publishing business. These people are really little more than scam artists looking to make money. The idea is to make up a plausible sounding journal name, set up a website, and send spam to what looks like every academic on the planet touting for submissions. Anyone silly enough to submit an article to a journal they found out about purely by spam is probably going to be silly enough to also pay the $500 “processing charge” which is requested once the article gets (inevitably) accepted.
You are aware that this paper is over 30 years old, right?
It’d take some sort of expert to understand this kind of mind set.
I here theres a bloke in Oz good at getting to the bottom of things.
Shouldn’t all papers submitted remain anonymous until they have passed peer review?
There seems to be a large amount of mutual ‘back scratching’ going on, as is reputed to be the case in any form of competitive ‘sport’.
StephenP: The important thing here is that these journals did not employ blind, let alone double blind review. One of the things that first concerned me about the quality of climate science was learning that several of the journals publishing papers did not blind the identity of authors to the reviewers. Some even (as do many journals) asked for authors to suggest reviewers. Easy for the editors, but very undermining of the QA value of peer review, and much easier for paradigms to be defended by circling the intellectual wagons. The problems have been exacerbated since the publication of this paper by two aspects of ‘globalisation’: cheap air travel since the introduction of the jumbo jet and e-mail, which have both meant the couple of dozen experts in specialized areas of knowledge are likely to be personally known to each other.
“Publish or perish” has a whole new meaning when the publishers play a role as “gatekeepers” to the accepted norms, and the accepted hierarchy, and the status quo.
The fact this paper is dated doesn’t mean the problem doesn’t still exist.
Rejection slips are terribly crushing to young, hopeful writers, whether they work in the arts or the sciences. Those who stick to their vision of truth need to learn to be tough, and how to live on a low income while working other jobs, while those who dance to the tunes of the gatekeepers compromise truth, become dupes, and are often looked upon in the manner people look upon Bill McKibben.