Hot of the heels of the busted “Peer Review Ring” we have this from Nature News:
In April, the US National Academy of Sciences elected 105 new members to its ranks. Academy membership is one the most prestigious honours for a scientist, and it comes with a tangible perk: members can submit up to four papers per year to the body’s high-profile journal, the venerable Proceedings of the National Academy of Sciences (PNAS), through the ‘contributed’ publication track. This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments.
For many academy members, this privileged path is central to the appeal of PNAS. But to some scientists, it gives the journal the appearance of an old boys’ club. “Sound anachronistic? It is,” wrote biochemist Steve Caplan of the University of Nebraska, Omaha, in a 2011 blogpost that suggested the contributed track could be used as a “dumping ground” for some papers. Editors at the journal have strived to dispel that perception.
…
Having control over the review process brings advantages. Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material. “Chemists have no idea about glycobiology,” says Chi-Huey Wong of the Scripps Research Institute in La Jolla, California, who studies the chemistry and biology of sugars.
More here: http://www.nature.com/news/scientific-publishing-the-inside-track-1.15424
h/t to Tom Nelson
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Do you know that every time I see the name of this site, the SNL “What Up with That” theme runs through my head?
http://www.hulu.com/watch/102975/saturday-night-live-what-up-with-that
A circle for jerks
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
Sounds like they have identified a problem with the peer review process and tried to fix it by introducing another problem.
Sounds like an organization whose time has past. Cherry picking reviewers is deceitful.
“Chemists have no idea about glycobiology,”
This speaks to another sublet but powerful bias. When I was entering graduate school we were encouraged to seek a topic or organism that no one else had studied. That way we could become the world’s leading and only expert. I suspect many so-called “peer reviewed papers” are simply rubber stamped if they salt the paper with buzz words that support the editor’s bias such as the Sokal hoax demonstrated http://en.wikipedia.org/wiki/Sokal_affair and in “Contrasting Good and Bad Science: Disease, Climate Change and the Case of the Golden Toad” http://landscapesandcycles.net/contrasting-good-and-bad-science–disease–climate.html
choose your own reviewers is the best way to ensure that referees actually understand the material.
No, the best way is to write the material so it’s legible….
Well, When you submit a paper to tetrahedron letters, you can “suggest” names of reviewers. It has been like that for some time and I have never liked that option. Nobody is going to suggest the name of a rival.
http://www.elsevier.com/journals/tetrahedron-letters/0040-4039/guide-for-authors
“This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments.”
… unless your name is Richard Lindzen.
“being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
___
So let those submitting papers choose which discipline they want to review their paper. There’s no need for them to choose which individuals they want to do the review. That invites corruption by setting up a situation of “I helped you, now you help me,” or “if I scratch your back, will you scratch mine?”
In addition, when you are elected a member of the Academy you receive a gold plated rubber stamp.
But wait..
Didn’t some online journal implode due to just this sin?
Pal review, only acceptable by the consensus.
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
I can understand that, for instance if paper on Critical Theory were reviewed by someone outside the discipline they would think it was nonsense. Same thing with a paper that on climate modeling which explains why the models are right and all the physical measurements are wrong or misleading, or why a 17 year pause in warming means its worse than we thought.
This is not new. No reason to be shocked. It was common practice in a number of international journals I was familair with at least two decades ago. For the last paper I ever published [in 2001] I was required to submit the names and addresses of three possible reviewers. The editor did not have to accept them but happily picked two of those names.
An easier solution would be to have each member list their areas of expertise into a database, and when a paper is submitted the author should list what specialties are being addressed in the paper. Then let a computer make the match ups.
‘Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material. ‘
Its also a very good way to ensure that reviewers can operate a ‘your starch my back and I will starch yours ‘ policy
Of course making public who reviews which papers would easily undermine such concerns , has ever one could see what is going on. So when will they be doing is after or before a common farmyard animal overcomes its aerodynamic problems in its reach for the sky ?
Try doing this trick has a student , only your mates can review your work , and you will told what you can do with that idea in very short terms academic terms.
I’m sure Mann and his group never did this type of thing…./
http://www.pnas.org/content/106/52/E142.full.pdf+html?sid=d6ecfae1-e6d4-4ca6-a551-e6eb86594c41
So a review by a reviewer picked by the author is about as worthwhile as that on the dust jacket of a novel. In other words, not worth the paper it’s printed on.
Peer review IS ONLY A FILTER, not the final word on a paper. It is often taken too seriously, as it is only supposed to see if the basic paper is poorly or well written (style), if it has detected errors or contradictions from known science (content), or if it is not enough material to add to the state of science (value worth wasting space on). Peer reviewing a paper reflects on the reviewer as well as the author if the paper is later found to be flawed. I don’t have a problem with this type of activity as a filter, but remember it can come back to bite you if the process is too badly abused. Assuming that if a paper is peer reviewed, it is likely correct, is a major error. Only considerable time with followup work can eventually support or reject a paper.
Another reason Feynman shunned them as useless club gatekeepers and fraternity enforcers
This is very common in health sciences. Including for fairly prominent journals.
It is often a big hassle for me, since I carry out my types of analyses across different topics, and I do not know who the decent reviewers might be. All I want is a fair shake for my decent studies.
Here is how it is supposed to work, ideally: a journal works to identify a good group of experts, and invites them to be reviewers. Many may not know how this works, but this is done for “free.”
I once did a review and shortly after received a $100 check from that journal. To this day, I have no idea why. If only we reviewers got paid. There are two reasons to review. First, nearly all academics get rated on their “research” (which means funding that includes institutional overhead – so it doesn’t matter how important or good as long as the institution is getting money and you are covering your salary), teaching, and “service.” Reviewing goes under the “service” category, as would speaking to the media, giving a guest lecture, being on some board or committee, and fulfilling various things on campus such as serving a two-year stint on some faculty committee.
So, by being a reviewer, you punch your dance card to some degree for “service.” The more prestigious the journal, the better. Serving no-pay as an editor is even more impressive.
The second reason to review is to keep abreast in yet another way of the latest research in your field. You know what is going on before it hits the journals. The other major way to keep abreast is to attend conferences.
Those who volunteer indicate their areas of focus and expertise. The journal thus builds up a stable of reviewers. When a possibly-in-range article comes in, the topics are reviewed and appropriate reviewers are sent an invitation to review. Ideally, reviewers should be somewhat eager to review, and should get this done on time.
A journal can also have ad hoc reviewers – not on their reviewer panel. This is prbbly happening a lot more lately, since it is getting more typical to ask the author to suggest three or five potential reviewers when the manuscript is submitted. This has become “optional” since the electronic submission system will not let you proceed when there are blanks in those spaces.
In my opinion, academics are getting less interested in reviewing articles, and reviewing them well. I have had editors apologize for long review time spans due to difficulty finding reviewers. I think this is part of the trend of academia moving to be very dollar-focused, with increased competition and pressure for funding dollars.
Federal funding has been flat for 20 years, but the number of PhDs has climbed the entire time. So, more academics are chasing the same grant money. This rality may have a role in the morphing of the review process.
When I nominate my three, four, or five reviewers, my guess is some or all are asked to review my article. So, I may be selecting my reviewers.
Reviewing the current review-by-buddy system scandal emerging now, I believe this is a big problem in science.
There have been other landscape-changing scandals in the recent decade that have notably changed editorial policy and customs. Reporting standards for the various types of studies now are quite well-known, and de riguer. Disclosure of conflicts-of-interest has taken a major step up after a JAMA scandal with a publication by a guy names Robinson, who carried out some obviously lousy analyses, and failed to disclose his support from the drug manufacturer whose pill his study unfairly favored.
Hopefully, academia will move away from author-nominated and selected reviewers.
There is nothing new about choosing the “right” reviewer. Some years ago a friend, with whom I had collaborated for several years, submitted a paper to a journal. The editor sent the draft to two peers, one whose views might be suspected as agreeing with the conclusions and one who was likely to disagree. The first said “publish it is an important article” (subject to some minor amendments). The second, a leading academic, said that it was nonsense and should never see the light of day. So the editor sent the draft to a third “independent” peer reviewer – myself. She knew, because of my previous work, that I would recommend publication. If she had not wanted it published she could have sent the draft to a number of others who were known opponents of the author.
Suggesting several possible reviewers should serve to limit the length of the list of co-authors on a paper, which are sometimes huge. Has a lead author ever made the error of putting a co-author on his list of possible reviewers?
Some years ago I did an article on a very touchy subject in my field and submitted it to the flagship journal. After receiving a rejection letter based on some very inept reviews , I asked the journal editor if I could publish the reviews.
That suggestion was strongly opposed. The editor said if the reviews were open to scrutiny, they wouldn’t be able to get reviewers.
I can understand why reviewers’ names shouldn’t be revealed, but I still think the review process should be more transparent. Making the reviewers’ comments publicly available would introduce more discipline into the process and also help other researchers to be more aware of problems to avoid in their own papers.
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
===
“If you cannot explain it to a 4 year old you don’t really understand it yourself.” Don’t know who said it but I think it applies.
There is a big difference between a journal asking an author for suggestions on reviewers and actually ,b>choosing them, as is practised at PNAS. It is not the first time this has been noted and in my field is a black mark against this journal.
The issue of a reviewer not being completely familiar with the subject matter is easily addressed by giving reviewers adequate instructions. I have been asked to review papers outside my field and been asked solely to comment on the area where I was familiar (in this case a technical methodology not commonly used in the actual field). This is what editors do and why they are the ones who are paid (as opposed to reviewers, authors etc.) as they are the ones who have to do the real work of identifying and instructing reviewers and then deciding when a reviewer’s comments are sufficiently supported.