Another peer reviewed science failure

From the Ed Begley Jr. department:

“If these scientists have done something wrong, it will be found out and their peers will determine it,” insisted Ed. “Don’t get your information from me, folks, or any newscaster. Get it from people with Ph.D. after their names. ‘Peer-reviewed studies is the key words. And if it comes out in peer-reviewed studies . . . “

Professor of Economics George J. Borjas writes on his blog:

I have a few pet peeves. One of them is how “peer review” is perceived by far too many people as the gold standard certification of scientific authority. Any academic who’s been through the peer review process many times (as I have) knows that the process is full of potholes and is sometimes subverted by unethical behavior on the part of editors and reviewers.

The reason I bring this up is because of a brewing scandal in my own discipline, economics. There has been online discussion about this for over a month…

…

The facts seem easy to summarize.

1. Two young economists, Petra Persson and Maya Rossin-Slater, wrote a paper entitled “Family Ruptures, Stress, and the Mental Health of the Next Generation.” They submitted the paper for publication to the American Economic Review (AER), the premier publication of the American Economic Association.
2. The paper was handled by AER co-editor Hilary Hoynes, an economist at UC Berkeley. All of the available information indicates that the paper went through normal reviewing procedures. Hoynes sent out the paper to four referees that she specifically selected to give her advice on whether the paper was sufficiently important and original to be published in the AER.
3. After the referees wrote their reviews and the authors addressed the various issues raised by the reviewers, Hoynes accepted the paper for publication.

…

And here’s where things get really interesting. It turns out that Hilary Hoynes, the co-editor at the AER, happens to be a current coauthor of one of the junior economists who wrote the Family Ruptures paper. (Page 12 of her CV dated October 13, 2015, indicates Hoynes was working with one of the coauthors while the review process was ongoing). This is a big no-no. The editor, by selectively picking which referees will review the paper, has a lot of influence over how the “peer review process” turns out. A good editor has a feel for how particular economists will react to particular kinds of work, so that by choosing the right reviewers the editor can “nudge” the final assessment in a particular direction. The conflict of interest is so large and so obvious that the AER has written guidelines about this:

There are several rules that affect assignment of manuscripts. Coeditors are generally not assigned manuscripts authored by an individual at his or her institution, by an individual with whom the Coeditor has been a recent coauthor, by an individual who has a close professional or personal relationship with the Coeditor, or by an individual who has served as a graduate student advisor or advisee of the Coeditor.

To make matters worse, after the barrage of posts at EJMR pointed out that there existed at least one paper in the medical literature that resembled the now-forthcoming AER paper, Hoynes (and perhaps otherAER editors) attempted to resolve the problem by allowing the authors to add footnotes and a new section to the Family Ruptures paper. These post-acceptance revisions were apparently added sequentially in different rounds. Despite the additions and despite the new information, the paper was never again sent  to the four referees to determine if the nature of the contribution had changed in light of the new information. Instead, the to-be-published version of Family Ruptures contains added-on passages with “Consumer Warnings”-like notes that stick out like a sore thumb.

(As an aside, EJMR has been referred to as a cesspool by some commentators. Retraction Watch published an article about the brouhaha last month, and quotes Hoynes dismissing EJMR because it is “unmoderated” and “not a legitimate source of information.” Unfortunately, she does not address how this unmoderated forum of illegitimate nonsense led to revisions in an already-accepted paper at theAER.

Read the entire post here: https://gborjas.org/2016/06/30/a-rant-on-peer-review/

Why is the blatant failure of peer review of an economics paper in that fields most prestigious journal relevant to us here where we discuss climate? Well for one, we saw failures of peer review such as gate-keeping and favoritism on display in the Climategate episode. Remember this one from Phil Jones at the Climate Research Unit of

East Anglia University?

“I can’t see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow — even if we have to redefine what the peer-review literature is!”

This failure in economic literature is relevant to climate science because not only are the dynamics we’ve seen in economic science demonstrated similar to climate science, but also both fields have large influences on public policy — in part due to politicians confidence in peer reviewed literature. (h/t to Larry Kummer)