Hot of the heels of the busted “Peer Review Ring” we have this from Nature News:
In April, the US National Academy of Sciences elected 105 new members to its ranks. Academy membership is one the most prestigious honours for a scientist, and it comes with a tangible perk: members can submit up to four papers per year to the body’s high-profile journal, the venerable Proceedings of the National Academy of Sciences (PNAS), through the ‘contributed’ publication track. This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments.
For many academy members, this privileged path is central to the appeal of PNAS. But to some scientists, it gives the journal the appearance of an old boys’ club. “Sound anachronistic? It is,” wrote biochemist Steve Caplan of the University of Nebraska, Omaha, in a 2011 blogpost that suggested the contributed track could be used as a “dumping ground” for some papers. Editors at the journal have strived to dispel that perception.
…
Having control over the review process brings advantages. Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material. “Chemists have no idea about glycobiology,” says Chi-Huey Wong of the Scripps Research Institute in La Jolla, California, who studies the chemistry and biology of sugars.
More here: http://www.nature.com/news/scientific-publishing-the-inside-track-1.15424
h/t to Tom Nelson
Do you know that every time I see the name of this site, the SNL “What Up with That” theme runs through my head?
http://www.hulu.com/watch/102975/saturday-night-live-what-up-with-that
A circle for jerks
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
Sounds like they have identified a problem with the peer review process and tried to fix it by introducing another problem.
Sounds like an organization whose time has past. Cherry picking reviewers is deceitful.
“Chemists have no idea about glycobiology,”
This speaks to another sublet but powerful bias. When I was entering graduate school we were encouraged to seek a topic or organism that no one else had studied. That way we could become the world’s leading and only expert. I suspect many so-called “peer reviewed papers” are simply rubber stamped if they salt the paper with buzz words that support the editor’s bias such as the Sokal hoax demonstrated http://en.wikipedia.org/wiki/Sokal_affair and in “Contrasting Good and Bad Science: Disease, Climate Change and the Case of the Golden Toad” http://landscapesandcycles.net/contrasting-good-and-bad-science–disease–climate.html
choose your own reviewers is the best way to ensure that referees actually understand the material.
No, the best way is to write the material so it’s legible….
Well, When you submit a paper to tetrahedron letters, you can “suggest” names of reviewers. It has been like that for some time and I have never liked that option. Nobody is going to suggest the name of a rival.
http://www.elsevier.com/journals/tetrahedron-letters/0040-4039/guide-for-authors
“This unusual process allows authors to choose who will review their paper and how to respond to those reviewers’ comments.”
… unless your name is Richard Lindzen.
“being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
___
So let those submitting papers choose which discipline they want to review their paper. There’s no need for them to choose which individuals they want to do the review. That invites corruption by setting up a situation of “I helped you, now you help me,” or “if I scratch your back, will you scratch mine?”
In addition, when you are elected a member of the Academy you receive a gold plated rubber stamp.
But wait..
Didn’t some online journal implode due to just this sin?
Pal review, only acceptable by the consensus.
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
I can understand that, for instance if paper on Critical Theory were reviewed by someone outside the discipline they would think it was nonsense. Same thing with a paper that on climate modeling which explains why the models are right and all the physical measurements are wrong or misleading, or why a 17 year pause in warming means its worse than we thought.
This is not new. No reason to be shocked. It was common practice in a number of international journals I was familair with at least two decades ago. For the last paper I ever published [in 2001] I was required to submit the names and addresses of three possible reviewers. The editor did not have to accept them but happily picked two of those names.
An easier solution would be to have each member list their areas of expertise into a database, and when a paper is submitted the author should list what specialties are being addressed in the paper. Then let a computer make the match ups.
‘Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material. ‘
Its also a very good way to ensure that reviewers can operate a ‘your starch my back and I will starch yours ‘ policy
Of course making public who reviews which papers would easily undermine such concerns , has ever one could see what is going on. So when will they be doing is after or before a common farmyard animal overcomes its aerodynamic problems in its reach for the sky ?
Try doing this trick has a student , only your mates can review your work , and you will told what you can do with that idea in very short terms academic terms.
I’m sure Mann and his group never did this type of thing…./
http://www.pnas.org/content/106/52/E142.full.pdf+html?sid=d6ecfae1-e6d4-4ca6-a551-e6eb86594c41
So a review by a reviewer picked by the author is about as worthwhile as that on the dust jacket of a novel. In other words, not worth the paper it’s printed on.
Peer review IS ONLY A FILTER, not the final word on a paper. It is often taken too seriously, as it is only supposed to see if the basic paper is poorly or well written (style), if it has detected errors or contradictions from known science (content), or if it is not enough material to add to the state of science (value worth wasting space on). Peer reviewing a paper reflects on the reviewer as well as the author if the paper is later found to be flawed. I don’t have a problem with this type of activity as a filter, but remember it can come back to bite you if the process is too badly abused. Assuming that if a paper is peer reviewed, it is likely correct, is a major error. Only considerable time with followup work can eventually support or reject a paper.
Another reason Feynman shunned them as useless club gatekeepers and fraternity enforcers
This is very common in health sciences. Including for fairly prominent journals.
It is often a big hassle for me, since I carry out my types of analyses across different topics, and I do not know who the decent reviewers might be. All I want is a fair shake for my decent studies.
Here is how it is supposed to work, ideally: a journal works to identify a good group of experts, and invites them to be reviewers. Many may not know how this works, but this is done for “free.”
I once did a review and shortly after received a $100 check from that journal. To this day, I have no idea why. If only we reviewers got paid. There are two reasons to review. First, nearly all academics get rated on their “research” (which means funding that includes institutional overhead – so it doesn’t matter how important or good as long as the institution is getting money and you are covering your salary), teaching, and “service.” Reviewing goes under the “service” category, as would speaking to the media, giving a guest lecture, being on some board or committee, and fulfilling various things on campus such as serving a two-year stint on some faculty committee.
So, by being a reviewer, you punch your dance card to some degree for “service.” The more prestigious the journal, the better. Serving no-pay as an editor is even more impressive.
The second reason to review is to keep abreast in yet another way of the latest research in your field. You know what is going on before it hits the journals. The other major way to keep abreast is to attend conferences.
Those who volunteer indicate their areas of focus and expertise. The journal thus builds up a stable of reviewers. When a possibly-in-range article comes in, the topics are reviewed and appropriate reviewers are sent an invitation to review. Ideally, reviewers should be somewhat eager to review, and should get this done on time.
A journal can also have ad hoc reviewers – not on their reviewer panel. This is prbbly happening a lot more lately, since it is getting more typical to ask the author to suggest three or five potential reviewers when the manuscript is submitted. This has become “optional” since the electronic submission system will not let you proceed when there are blanks in those spaces.
In my opinion, academics are getting less interested in reviewing articles, and reviewing them well. I have had editors apologize for long review time spans due to difficulty finding reviewers. I think this is part of the trend of academia moving to be very dollar-focused, with increased competition and pressure for funding dollars.
Federal funding has been flat for 20 years, but the number of PhDs has climbed the entire time. So, more academics are chasing the same grant money. This rality may have a role in the morphing of the review process.
When I nominate my three, four, or five reviewers, my guess is some or all are asked to review my article. So, I may be selecting my reviewers.
Reviewing the current review-by-buddy system scandal emerging now, I believe this is a big problem in science.
There have been other landscape-changing scandals in the recent decade that have notably changed editorial policy and customs. Reporting standards for the various types of studies now are quite well-known, and de riguer. Disclosure of conflicts-of-interest has taken a major step up after a JAMA scandal with a publication by a guy names Robinson, who carried out some obviously lousy analyses, and failed to disclose his support from the drug manufacturer whose pill his study unfairly favored.
Hopefully, academia will move away from author-nominated and selected reviewers.
There is nothing new about choosing the “right” reviewer. Some years ago a friend, with whom I had collaborated for several years, submitted a paper to a journal. The editor sent the draft to two peers, one whose views might be suspected as agreeing with the conclusions and one who was likely to disagree. The first said “publish it is an important article” (subject to some minor amendments). The second, a leading academic, said that it was nonsense and should never see the light of day. So the editor sent the draft to a third “independent” peer reviewer – myself. She knew, because of my previous work, that I would recommend publication. If she had not wanted it published she could have sent the draft to a number of others who were known opponents of the author.
Suggesting several possible reviewers should serve to limit the length of the list of co-authors on a paper, which are sometimes huge. Has a lead author ever made the error of putting a co-author on his list of possible reviewers?
Some years ago I did an article on a very touchy subject in my field and submitted it to the flagship journal. After receiving a rejection letter based on some very inept reviews , I asked the journal editor if I could publish the reviews.
That suggestion was strongly opposed. The editor said if the reviews were open to scrutiny, they wouldn’t be able to get reviewers.
I can understand why reviewers’ names shouldn’t be revealed, but I still think the review process should be more transparent. Making the reviewers’ comments publicly available would introduce more discipline into the process and also help other researchers to be more aware of problems to avoid in their own papers.
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
===
“If you cannot explain it to a 4 year old you don’t really understand it yourself.” Don’t know who said it but I think it applies.
There is a big difference between a journal asking an author for suggestions on reviewers and actually ,b>choosing them, as is practised at PNAS. It is not the first time this has been noted and in my field is a black mark against this journal.
The issue of a reviewer not being completely familiar with the subject matter is easily addressed by giving reviewers adequate instructions. I have been asked to review papers outside my field and been asked solely to comment on the area where I was familiar (in this case a technical methodology not commonly used in the actual field). This is what editors do and why they are the ones who are paid (as opposed to reviewers, authors etc.) as they are the ones who have to do the real work of identifying and instructing reviewers and then deciding when a reviewer’s comments are sufficiently supported.
jim Steele says:
July 11, 2014 at 9:28 am
In the same general domain as the Sokal Affair, it is worthwhile to look up the dada engine and the postmodernism generator (which is a specific application of the dada engine). The dada engine was first released the same year Sokal embarrassed his target journal. At the time a story circulated that an essay generated by the dada engine was employed in the same and with equal success to Sokal’s effort. That has vanished over the years so thoroughly that I can no longer find reference to it, though Sokal’s stunt remains in view. IIRC, there was a comparison between the two, where the author argued that while Sokal’s little joke was indicative, it was still a human production “attempting” to generate nonsense. The essay from the dada engine was very definitely nonsense, randomly generated by a computer, yet putative PoMo critics claimed it made important points before the source was revealed.
It’s not quite like it’s presented in the original post, check out:
http://www.pnas.org/site/authors/guidelines.xhtml
For example:
“Contributed Submission.
An Academy member may submit up to four of his or her own manuscripts for publication per year. To contribute an article, the member must affirm that he or she had a direct role in the design and execution of all or a significant fraction of the work and the subject matter must be within the member’s area of expertise.”
“When submitting using the contributed process, members must secure the comments of at least two qualified reviewers, each from a different institution and not from the authors’ institutions. Reviewers should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. Members’ submissions must be accompanied by the names and contact information, including e-mails, of knowledgeable experts who reviewed the paper, along with all of the reviews received and the authors’ response for each round of review, and a brief statement endorsing publication in PNAS. Reviews must be on the PNAS review form. Members must select reviewers who have not collaborated with the authors in the past 48 months.”
“Starting on a voluntary basis, the names and institutional affiliations of the reviewers will be listed as a footnote. The Academy member must be one of the corresponding authors on the paper. These papers are published as “Contributed by” the responsible editor. “
Science has become just another one of America’s totally corrupt institutions. And Federal Government funding for research has assisted in the corruption the same way it has corrupted everything else.
Hot of the heels of the busted “Peer Review Ring” we have this from Nature News:
In April, the US National Academy of Sciences elected 105 new members to its ranks.
It is disengenuous at best to suggest that the so-called “Peer Review Ring” reported in Nature News is comparable to the peer review process as practiced at PNAS. The number of papers withdrawn from publication by PNAS is vanishingly small compared to a large number of journals that are popping up on the internet for publishing with little or no peer review at all. The US National Academy of Sciences can, in no way, be compared to these other fly by night organizations.
It is your article, Anthony, that should be withdrawn, in my opinion.
“Those who work across disciplinary boundaries say that being able to choose your own reviewers is the best way to ensure that referees actually understand the material.”
I suppose if I were a PR type and was trying to spin and excuse bad behavior, I’d say something like this, too. Science is just too damn easy to corrupt. Reality rewards scientists who’ve given away their principles with group think, political correctness, peer-pressure and the like. These counter-productive forces have held sway for a while. How many weak authors, poor writers and poor scientific protocols have had long and happy careers because of their accommodation by other weak authors, who were themselves given accommodation? Just think of all the climate science papers that confuse and obfuscate rather than elucidate; that read like tacit, unstated or tendentious hypotheses immersed in just enough verbal diarrhea to make the ratio of obligatory weasel words a bit less obvious. And let’s not forget that that public confusion about controversial or complex science also serves the interests of Fabians, government bureaucrats, colleges administrators, media sycophants, and other sundry apparatchiks. Confusion and ambiguity permit the translation of aforementioned weak-ass BS into political crises which in turn demand immediate public policy prescriptions. In this way, the apparatchiks maintain their figurative lip lock on the public funding teat. Luckily, in the long run, science will not tolerate such malfeasance. When no one can stand on your shoulders, you’ve failed. Science will eventually correct all the bad actors.
I don’t know about you, but I take comfort in that. /dude
The NAS membership i small and definitely not made up of idiots. I think any of its members who abused the peer review process and published manifestly crap papers in PNAS would pretty soon feel the pressure of their peers. This Contributed Submission thing is a minor wrinkle in the giant problem of Peer Review
Imagine this situation. I am on the IPCC. I submit a paper to PNAS. I choose reviewers who are on the IPCC or who I know are convinced of CAGW. What a nice situation. No wonder they scream endlessly about the mountains of evidence. What they mean is the mountains of garbage produced by pal review. I scratch your back, and you scratch mine thanks. Ahhhh, that feels soooooo good. Now, where is the next funding coming from? Ahhh there’s the warehouse.
It’s not just climate science that is corrupted. The usual culprit is money. Science is for sale these days and has been for a while.
Dr. Marcia Angell, the editor of New England Journal of Medicine for 20 years, wrote the following:
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.”.
I like the “peer review” process Einstein went through on his, “On the Electrodynamics of Moving Bodies” 1905 (OK, most of you know it as, the “Special Theory of Relativity”, given that name some years later.) That “peer review” was that the journal editor thought it was interesting.
The paper is here: https://www.fourmilab.ch/etexts/einstein/specrel/www/
Please note also the “references” Einstein including in the original —- NONE! I wonder how many Einsteins there are who are excluded from the scientific community because they have completely original thinking, they have no “credentials” to speak, and the have no “tribe” to work with. (Aside from Willis E. that is…)?
Leonard Weinstein says:
July 11, 2014 at 10:48 am
Peer review IS ONLY A FILTER, not the final word on a paper. It is often taken too seriously, as it is only supposed to see if the basic paper is poorly or well written (style), if it has detected errors or contradictions from known science (content), or if it is not enough material to add to the state of science (value worth wasting space on). Peer reviewing a paper reflects on the reviewer as well as the author if the paper is later found to be flawed. I don’t have a problem with this type of activity as a filter, but remember it can come back to bite you if the process is too badly abused. Assuming that if a paper is peer reviewed, it is likely correct, is a major error. Only considerable time with followup work can eventually support or reject a paper.
=====
Exactly……how did peer review get so bastardized that it means more than spell chex…….
I work in a field that has very few qualified reviews….we will throw something out there just to see what the reaction is….we might learn something
I made a significant discovery in a field based on ideas the opposite of the entire published literature (because all the experimental results contradicted all the theoretical conventional wisdom. Could not get published anywhere as a heretical outsider. Finally went and spent a day with the oldest personal disciple of the author of the definitive ‘Bible’. After about six hours, he slapped his head, said I wish I had thought of that, and let me present at his annual technical conferences. It was not until after lab experimental results from another contracted ‘insider’ proved the discovery correct that the (small, perhaps 50 experts worldwide) establishment started to bend and I was able to get a single ONR grant. All the essential early stuff, including now issued fundamental patents, done on my own nickel. Could not even get California VCs interested despite over a $billion in potential. God forbid if that little energy storage establishment had been as large as the climate consensus.
Bucking a consensus is tough. Look at Alfred Wegener’s experience with his correct theory of continental drift. And these days, there is ‘no money in it’ for the young up and coming academic researchers. the Marcott affair is but one example of what dreck results. Pal review publication ain’t the half of it. Grant funding, post doc positions, tenure, …
But in the end, facts prevail.
lucaturin says:
July 11, 2014 at 12:19 pm
The NAS membership i small and definitely not made up of idiots. I think any of its members who abused the peer review process and published manifestly crap papers in PNAS would pretty soon feel the pressure of their peers. This Contributed Submission thing is a minor wrinkle in the giant problem of Peer Review.
Your faith in the NAS membership is touching. NAS also has applications for grants reviewed “anonymously” and the wrinkle is that the applicant may or may not get to see why an application was rejected. The NAS frequently refuses to even consider basic data gathering applications which are apparently too boring even though without that basic data, science is no different than a just-so story. The problem with high powered experts is that they begin to become quite confident of their own knowledge. That’s not a sin per se, we all do it. It is confirmation bias however, and it is something that any scientist, regardless of expertise should be wary of. Rejecting useful work simply because the proposal appears to contradict “your personal knowledge” and is therefore a waste of time, is the opposite of seeking new knowledge.
TRM says:
July 11, 2014 at 12:41 pm
———————————–
Thanks for that. A pretty crushing statement.
The name of the authors of a paper should be kept anonymous from the reviewers. I agree reviewers should know something about the topic. I think the reviewers should also be selected anonymously and by ballot. Only after the process has been completed should identities be revealed. There should be at least 1 professional statistician amongst the reviewers if the paper has statistics in it – probably most papers would need a statistician.
If reviewing papers is an obligation, best done for friends and allies…how thankless is the drudgery of replication?!
Thankfully, we have Chiefio looking at some GIStemp date stamps (and along the way “new ways to run very old code”) with a Sun SPARC bigendian emulator. (:
http://chiefio.wordpress.com/2014/07/10/down-the-rabbit-hole-with-qemu-gistemp-and-more/
Latitude said: “No, the best way is to write the material so it’s legible….”
or even comprehensible…
Cross subject research… I remember when I was at University, one of my acquaintances was a geologist. His PhD thesis was basically writing a program (in fortran, of course) to simulate a rock.
Really … using a bunch of very slowed down fluid mechanics equations to simulate the behavior of the said rock over millions of years.
I probably could have written and tested his program in a couple of weeks (using something other than fortran though).
I believe he received his PhD.
Al Capone always insisted on picking his own judge and jury.
“Editors at the journal have strived to dispel that perception.”
Editors should know that the past participle is “striven”.
Science.
ROFLMAO
John Barnes long established is no guard at all form being rubbish , plenty that was long held consensus has turned out to be ‘dead wrong ‘ Meanwhile ‘old boys clubs ‘ come with their own problem . If their ‘the best ‘ then they should have no issue with meeting ‘the best standards possible and a closed shop of friends is a long way from that . But let me ask you would think it OK that student should pick who can mark their work from a list of their ‘preferred people ‘?
It’s not quite like it’s presented in the original post, check out:
http://www.pnas.org/site/authors/guidelines.xhtml
For example:
“Contributed Submission.
An Academy member may submit up to four of his or her own manuscripts for publication per year. To contribute an article, the member must affirm that he or she had a direct role in the design and execution of all or a significant fraction of the work and the subject matter must be within the member’s area of expertise.”
“When submitting using the contributed process, members must secure the comments of at least two qualified reviewers, each from a different institution and not from the authors’ institutions. Reviewers should be asked to evaluate revised manuscripts to ensure that their concerns have been adequately addressed. Members’ submissions must be accompanied by the names and contact information, including e-mails, of knowledgeable experts who reviewed the paper, along with all of the reviews received and the authors’ response for each round of review, and a brief statement endorsing publication in PNAS. Reviews must be on the PNAS review form. Members must select reviewers who have not collaborated with the authors in the past 48 months.”
“Starting on a voluntary basis, the names and institutional affiliations of the reviewers will be listed as a footnote. The Academy member must be one of the corresponding authors on the paper. These papers are published as “Contributed by” the responsible editor. “
Members of the National Academy of Sciences jealously guard their privileges and responsibilities. While there may be an occasional paper published essentially as support to a friend, even in those cases, if the science can’t stand up to proper referee review, the paper will not be published. I worked for number of years for a well-respected member who jealously guarded his reputation and I can tell you he did not let anythng slip under the door. He had us poor post-docs slaving at his beck and call to review articles. He judged our reviewing capabilities based on his own. This is typical for the most highly regarded members of the National Academy.
John
I recently submitted a Letter to the Editor of PNAS as a lay member of the public. Let’s just say that the conclusions reached by the authors of the paper in question were patently absurd and could almost be dismissed by inspection. I found the submission process to be Byzantine and extremely discouraging to one who is not already a member of the club.
pokerguy says:
July 11, 2014 at 10:03 am
Mark W. writes: “It really is sad when bigots come out to play.”
Let’s be clear. I’m a bigot because I think the belief in as Wikipedia defines them, “humanoid forms with feathered wings on their backs and halos around their heads,” is half way to nuts?
——————————-
So your source for theological concepts is Wikipedia?
I don’t know a single Christian who believes in angels as per that definition. Even children know better.
And I also don’t know any single Christian who believes this is a scientific topic.
I don’t know if misrepresenting what other people believe to be able to sneer them and talking about something one knows nothing about is “bigotry”,
But it surely looks like what warmists do all the time.
Posted on wrong thread, sorry.
Letter from 2012, still apparently relevant:
http://wattsupwiththat.com/2012/08/08/an-incovenient-result-july-2012-not-a-record-breaker-according-to-the-new-noaancdc-national-climate-reference-network/#comment-1054285
http://www.drroyspencer.com/wp-content/uploads/UAH_LT_1979_thru_July_2012.png
To:
Heads of Departments,
Proceedings, National Academy of Sciences (PNAS)
Dear PNAS Heads:
UAH Global Temperature Update for July 2012: +0.28C,
COOLER than June, 2012: +0.37 deg.
If one wants to argue about GLOBAL warming, should one not look first at GLOBAL temperatures?
Respectfully, Allan
I would agree with several posters that in the biosciences at least it is not so unusual to be invited to suggest your own reviewers. I proposed reviewers in this way for a paper I submitted recently in the osteoporosis field – it got rejected.
Well it would seem to me, that the NAS, being a National Institution, would simply have membership requirements that are documented just as are the US tax laws.
Then any person who can document compliance with those National standards, would automatically be eligible for membership, and requiring only submission of credentials, and a one time fee for review of those credentials.
And all matters submitted as advice to government, either POTUS or the Congress, would require also a minority (presumably dissenting ) report, as well as the “consensus” report.
Isaac Newton hated Robert Hooke. His most famous saying, “”If I have seen further it is by standing on the shoulders of giants” was a jibe at Hooke, who was a hunchback.
History has almost forgotten Hooke, Britain’s answer to Leonardo, not least because Newton outlived him and used his immense influence to bury Hooke’s legacy to science.
The major flaw in peer review is that scientists use it to bury work that competes with their own. Cliques of scientists bury the work of cliques of other scientists. Science itself is diminished by these practices.
Einstein regularly withdrew papers when reviewer insisted on changes because he believed he knew better than the reviewers. Yet he was not immune to detailed corrections, in particular corrections to errors in his mathematics, some of which went undetected for years.