From the Ed Begley Jr. department:
“If these scientists have done something wrong, it will be found out and their peers will determine it,” insisted Ed. “Don’t get your information from me, folks, or any newscaster. Get it from people with Ph.D. after their names. ‘Peer-reviewed studies is the key words. And if it comes out in peer-reviewed studies . . . “
Professor of Economics George J. Borjas writes on his blog:
I have a few pet peeves. One of them is how “peer review” is perceived by far too many people as the gold standard certification of scientific authority. Any academic who’s been through the peer review process many times (as I have) knows that the process is full of potholes and is sometimes subverted by unethical behavior on the part of editors and reviewers.
The reason I bring this up is because of a brewing scandal in my own discipline, economics. There has been online discussion about this for over a month…
…
The facts seem easy to summarize.
- Two young economists, Petra Persson and Maya Rossin-Slater, wrote a paper entitled “Family Ruptures, Stress, and the Mental Health of the Next Generation.” They submitted the paper for publication to the American Economic Review (AER), the premier publication of the American Economic Association.
- The paper was handled by AER co-editor Hilary Hoynes, an economist at UC Berkeley. All of the available information indicates that the paper went through normal reviewing procedures. Hoynes sent out the paper to four referees that she specifically selected to give her advice on whether the paper was sufficiently important and original to be published in the AER.
- After the referees wrote their reviews and the authors addressed the various issues raised by the reviewers, Hoynes accepted the paper for publication.
…
And here’s where things get really interesting. It turns out that Hilary Hoynes, the co-editor at the AER, happens to be a current coauthor of one of the junior economists who wrote the Family Ruptures paper. (Page 12 of her CV dated October 13, 2015, indicates Hoynes was working with one of the coauthors while the review process was ongoing). This is a big no-no. The editor, by selectively picking which referees will review the paper, has a lot of influence over how the “peer review process” turns out. A good editor has a feel for how particular economists will react to particular kinds of work, so that by choosing the right reviewers the editor can “nudge” the final assessment in a particular direction. The conflict of interest is so large and so obvious that the AER has written guidelines about this:
There are several rules that affect assignment of manuscripts. Coeditors are generally not assigned manuscripts authored by an individual at his or her institution, by an individual with whom the Coeditor has been a recent coauthor, by an individual who has a close professional or personal relationship with the Coeditor, or by an individual who has served as a graduate student advisor or advisee of the Coeditor.
To make matters worse, after the barrage of posts at EJMR pointed out that there existed at least one paper in the medical literature that resembled the now-forthcoming AER paper, Hoynes (and perhaps otherAER editors) attempted to resolve the problem by allowing the authors to add footnotes and a new section to the Family Ruptures paper. These post-acceptance revisions were apparently added sequentially in different rounds. Despite the additions and despite the new information, the paper was never again sent to the four referees to determine if the nature of the contribution had changed in light of the new information. Instead, the to-be-published version of Family Ruptures contains added-on passages with “Consumer Warnings”-like notes that stick out like a sore thumb.
(As an aside, EJMR has been referred to as a cesspool by some commentators. Retraction Watch published an article about the brouhaha last month, and quotes Hoynes dismissing EJMR because it is “unmoderated” and “not a legitimate source of information.” Unfortunately, she does not address how this unmoderated forum of illegitimate nonsense led to revisions in an already-accepted paper at theAER.
Read the entire post here: https://gborjas.org/2016/06/30/a-rant-on-peer-review/
Why is the blatant failure of peer review of an economics paper in that fields most prestigious journal relevant to us here where we discuss climate? Well for one, we saw failures of peer review such as gate-keeping and favoritism on display in the Climategate episode. Remember this one from Phil Jones at the Climate Research Unit of
East Anglia University?
“I can’t see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow — even if we have to redefine what the peer-review literature is!”
This failure in economic literature is relevant to climate science because not only are the dynamics we’ve seen in economic science demonstrated similar to climate science, but also both fields have large influences on public policy — in part due to politicians confidence in peer reviewed literature. (h/t to Larry Kummer)
I’ve met Ed Begley Jr. He prays to the living deity of Climate Alarmism in a need to feel good about himself as self appointed savior of planet Earth…..
He’s scientifically illiterate and only listens to opinions that re-enforce his own.
You can quote Phil Jones about his desire to keep papers out of the IPCC and change the definition of peer review, but the proof is in the outcome.
What were those two papers? Were they included in the IPCC? Does Phil Jones have standing to change peer review and did he successfully change the definition of “peer review”?
We all express frustrations in e-mail — but was there any practical effect of these e-mails?
http://www.cato.org/publications/commentary/climate-scientists-subverted-peer-review
https://wattsupwiththat.com/2011/11/27/the-tribalistic-corruption-of-peer-review-the-chris-de-freitas-incident/
Can you delete any emails you may have had with Keith regarding the latest Intergovernmental Panel on Climate Change report? Keith will do likewise.
Phil Jones email to Michael Mann May 29 2008.
I’ve just completed Mike’s Nature trick of adding in the real temperatures to each series for the last 20 years (i.e. from 1981 onwards) and from 1961 for Keoth’s to hide the decline.
Phil Jones email November 16, 1999
Just expressing more frustration I reckon
Of course peer review is not perfect, as no system is. I think one change that would improve quality is for the authors to be anonymous to the reviewers. That would make it hard to nod through papers based on reputation or nepotism.
Gold standard? Well, perhaps a tarting point for deciding whether to take something seriously.
One thing that peer review generally does is ensure that the paper is put into proper context. The reviewer is able to check that the introduction provides a reasonable summary of the state of knowledge. Many non-peer reviewed articles are simply far too selective in the literature they cite, selecting only those that back up their conclusions. Obvious gaping holes will usually be picked up by the reviewers.
“I think one change that would improve quality is for the authors to be anonymous to the reviewers.”
—
That is difficult to ensure in practice. Editors select reviewers who are familiar with the topic at hand, and they would very often be able to figure out who wrote the paper from the specific angle, methodology, writing style etc. Papers are often incremental, and where that is the case, a new paper will refer to the previous one (which is published and no longer anonymous) all over the place. Even without such a specific reason, authors tend to cite their own works more heavily than those of the competition, which will give the game away.
It also works the other way around. I can guess who my reviewers were in at least, say, 3 out of 4 cases.
Ultimately, the peer review system is built on honor and integrity. Where these are lacking, the process becomes dysfunctional, and no formal rules and stipulations can save it.
Rather than the author being anonymous, would it be better [*] for the reviewers to be known and for their reviews to be published with the paper? The argument that it would take up too much space in the journal would no longer apply in our on-line world.
[*] NB. “better”, not “perfect”. No matter what method is used, there would be scope for a determined group to corrupt it.
A better solution is to have a team of engineers, statisticians, etc., preferably who actively hate scientists, review all government science studies. By law give them access to everything, data, code, etc. A line by line hostile review of any publicly funded science study.
Estimates of reproducible studies are as low as 11% and certainly less than half. There are so many problems from sample size to misuse of statistics that only a minority of studies have any value. There is no reason the let all those bad studies to escape, and be published, to be used as the basis for more bad studies. Drive the stake in early.
We are paying for good science. We should only get good science.
Peer review is ridiculous, not only does it allow nonsense to get through, it also blocks science which proves it wrong. Take Clovis first, peer reviewed and wrong. 20 years pass with peer review debunking all evidence which showed Clovis first was wrong. Ill also point out peer review,and consensus was ready to vote Einstein’s theory as wrong, only irrefutable proof stopped them, which brings me to point two. “Studies” are not science. The vast majority of science which is peer reviewed and published as fact are nothing more than studies.clinical studies are at best indicators of where to direct real scientific activities. If science was doing science, peer review would be simply recreating and confirming results of real experiments. Studies are nothing factual, which is why it is so easy to do them and pretend,they are,fact
Publish or perish, or so I’ve heard.
Take that for what it’s worth, coming from someone that struggles to complete a sentence.
EJMR? An explanation or link would be useful.
Earth Journal for Making Renimbi.
Each Journey Minimises Rebuttal.
English Jock-footballers Massively Redundant?
I have no idea, and wondered, likewise.
Auto
Google is your friend !! LOL
Economics Job Market Rumors
http://www.econjobrumors.com/topic/best-of-ejmr-the-archive-thread
TB,
Fundament. I like that. Much more acceptable in polite society than my usual terminology.
Hmmm. A term of endearment, in the genre of cryptic namecalling.
Peer Review??? The most useless crap going around. A bunch of fellow idiots thinking they know something.. A peer is nothing more than an idiot,following another idiot.
Peer review is a “publication” issue.
All it is doing is getting someone else, who may or may not actually check the work thoroughly, to say if they think your work should be entered into the scientific literature.
Peer review DOES NOT in any way imply that your paper is actually correct.
Yes, the purpose of publication is put the material out there for a wider range of experts to review. Peer review should be an ongoing process that continues long after publication since most pre publication review is incestuously performed by like minded people whose group think can get in the way of objectivity. IPCC driven climate science only tolerates post publication review when it doesn’t conflict with the narrative which seems to amplify this effect. Can you see the positive feedback acting on the science itself? This is also called positive reinforcement and unfortunately, some very bad behavior is being positively reinforced.
I’m guessing that if you look at past history of peer reviewed “climate science” papers…
…. then at least 97% would be incorrect in some way. 🙂
The phrase “appearance of impropriety” is unfamiliar to way too many in the peer review process.
There is nothing wrong with peer review per se. I do peer review at the rate of four or five submissions per year. Peer review is, on the whole, an extremely valuable service that greatly improves the quality of journal papers,
But let’s be clear what peer review is. It is an experienced set of eyes giving opinions and recommendations on submitted papers. Reviewers do not normally have the opportunity or resources to inspect raw data or repeat experiments. As such, reviewers have to trust that the authors are behaving in good faith.
Someone has to decide whether a paper meets the proper standards to be published in a given journal. The peer review system is as good as any. If sometimes things go wrong, well that is because humans are involved.
As I said above.
It is a review to see if the paper should be entered into the official journal/scientific literature.
Nothing more
The people often doing climate science review are called “gatekeepers”
Well someone has to decide whether a paper is suitable for publication. If not editors and peer reviewers then who?
And the term “gatekeeper” is obnoxious, an insult to the hundreds of thousands of peer reviewers who sacrifice their time to keep modern science moving forward, and who do their job with integrity. Unless you make a similar sacrifice of time, I recommend you back off from making flippant insults.
Re: AndyG55
THEN we have those who rush to another’s defense, short circuiting ANY POSSIBLE OCCURRENCE of critique.
Ring any bells?
@Killer Marmot et al,
Chill dudes. AndyG55 was specifically referring to climate science reviewers as ‘gatekeepers’, not to reviewers in general.
Those who hold up “peer-review” as being the be all and end all of debate, especially in an area like “climate science™”, they are the un-scientific people, who understand very little about peer-review.
Much of what does get through is manifestly JUNK-science…
Much of the real science gets blocked.
Have you ever done peer review? Have you ever had a paper peer reviewed?
Reading your posts, I would guess no. I can tell you that it is not how you imagine it.
I second Killer Marmot’s statements. What you’re saying bears no relation to my experience over decades of reviewing and being reviewed – and I say that having had some fairly bad reviews from time to time.
And I should add that I’ve had some great (and severe!) reviews that made a hell of a difference in the quality of the final work. So there are people who really apply themselves to the task of reviewing.
Perhaps if someone could link to the peer review comments of an actual climate paper it would be informative.
Linking to those comments is probably considered a micro aggression as it exposes faults.
Much of the junk that gets through is not new junk, but rehashed and recycled old junk which is a by-product of peer review insisting on the kind of references it does. It doesn’t help when so much of what you can reference is so obviously wrong, yet believed by many to be gospel.
Microaggressions are fine, only macroaggressions concern me.
There should be at least some peer review comments that are publicly accessible.
The published responses to papers and the comments of the current GISS director, to the extent they reflect actual peer review, are bullsh*t.
If that is the best science can do, science should be taken away from scientists, and the grants given to engineers who at least have a clue what they are doing (or at least fund them to do peer review)..
But I would be delighted to see an actual sample of peer review that looks intelligent and proves me wrong.
AndyG55 July 3, 2016 at 9:55 pm: “Much of the real science gets blocked.”
How ironic it would be you who pen this …
wogga wogga sounds like the sound that PacMan makes
After a paper is published, a paper *should* then be subjected to a massive peer review by everyone in the field. If you want to see the quintessential example of such a review failing, research Michael Bellesile’s work. It exposes the group-think of an entire discipline (history). Even those in the same field of research accepted his claims and wondered how THEY could have been so wrong. The parallels with that episode and climate change research is frightingly similar on multiple levels.
As far as getting published, I’m sure most of you are familiar with the DDT-eggshell-thinning research that helped launch the environmental movement. The original researcher (Bitman?) could not get his corrected research, showing NO such thinning, published in the same journal as the orignal paper because it was at odds with ‘accepted science’ that was primarily based on his flawed research!
I can see some problems with that.
First, it would be nice if obvious deficiencies were corrected before publication. People reading journal articles could then focus on more more important criticisms than, say, that the article was poorly organized.
Second, it costs money to publish. Articles have to be properly formatted, and this takes up the time of paid employees. And most journals still have paper versions. Indiscriminate publishing would add to these costs.
Third, when I subscribe to a journal I don’t want to have to read through a lot of dross to find the gems. I am paying the journal for its selectivity.
Fourth, it would mean researchers could fatten up their resumes by spraying poor articles all over the place without regard to quality.
No, I’m not arguing against anything you wrote. I’m saying that even after proper peer review, once the paper is published, all those in the appropriate field *should* look at the paper *skeptically*, and weigh it against their own expertise and research. The Bellesiles debacle showed that when the results confirmed personal biasis, experts ignored facts as they, themselves, had thought them to be, and were willing to believe that their facts were in error. Rather like discarding all known evidence of the MWP based on a single paper showing Man is creating climate change.
Jtom – this comment is a couple days late! But hopefully you can follow up.
No, most of us readers do not know the original DDT article or articles, and we are not familiar with the debunking or correcting article or articles.
You can spin any story you want this way. Could you simply add a bit more info, for anyone who wants to investigate further?
Since you did have a name in your first anecdote, I was able to google. Bellesile is a historian. He takes some aspect of our culture, considers one or more explanations to account for why things are the way they are, then examines a set of existing pieces of evidence to see how well the idea is supported, or which idea is more supported.
That is not science. And, as far as I can tell, his controversial book did not go through peer review – he found a publisher of a history book and it was published; for political reasons, it was later discredited in various ways, and a panel was composed to address the quality of his work, long after publication.
And, this was all quite unusual. Not the regular peer-reviewed-journal article reviewing I get asked to do regularly, and do 5-6 times per year.
Please add just a bit more info on anecdotes or cautionary tales such as these. thanks!
who Ed Begley? The name rings no bells on this side of the Atlantic.
He is an actor:
But here’s a sample of his thinking”
I would love to see journals insist on at least one reviewer from outside their community; preferentially a professional in the basic fields of math and statistics, a professional engineer with experience reducing claims like the paper to practice.
“……….This failure in economic literature is relevant to climate science because…………;” because in neither economics nor climate science is it possible to perform controlled experiments to test the validity of any hypothesis.
Thus the most influential members of each field, by throwing their support to any particular thesis, to a great extent determine and define what is considered the “correct” thinking.
First, you have to find honest peers who refuse to play the you-scratch-my-back-and-I’ll-scratch-yours game. Then you have to find peers who are not fearful of being taken to court, either by Mr. Mann for implying his research might be flawed, or by liberal Attorneys General who could demand that you turn over 40 years of correspondence to prove you’re not in league with ExxonMobil or any such ilk. Those who claim there are no consequences for honest scientists who rock the boat haven’t been paying attention lately.
First and foremost, peer review is political. Not conspiratorial political, just personality political. Also, who you are matters the most, not what you have to say. Third, this article and some scanned over comments here appear to give the impression that people are waiting around dying to give reviews. The typical case is that getting a paper reviewed is tedious and time-consuming, there is almost never 4 reviewers, and a review I had for a paper appeared to consist of reading the first sentence of each paragraph then asking questions answered in the following paragraph. Peer review is just to try to stop blatant quackery, and obvious goofs. It does nothing else on a consistent basis. The issues are so complex and the data so convoluted and noisy that interpretations are really just opinions of what may be going on.
A Toronto based CBC science reporter decided to take his 5 year old son on a Western Canadian train trip. They shared a seat with an Albertan Civil Engineer. As the train entered Alberta, the boy saw a black Angus bull in a pasture. He was so excited that he exclaimed to his Dad “look, Dad the cows in Alberta are black! The Dad got all excited too and fired up his lap top and wrote an article for the CBC based on the questions “Why are the Alberta cows turning black? Is it climate change? What has Stephen Harper been doing to hide this from working families? What kind of taxes do we need to implement to make this stop?”
As he finished his article he asked the Engineer to peer review it for him. The Engineer took the lap top and tapped away for a couple of minutes and handed it back to the CBC Science guy. The revised article read “What we know so far is that there is at least one bovine in Alberta, it is a bull, and it is black on at least one side”.
Gerry
A Toronto based CBC science reporter decided to take his 5 year old son on a Western Canadian train trip. They shared a seat with an Albertan Civil Engineer. As the train entered Alberta, the boy saw a black Angus bull in a pasture. He was so excited that he exclaimed to his Dad “Look, Dad the cows in Alberta are black!” The Dad got all excited too and fired up his lap top and wrote an article for the CBC based on the questions “Why are the Alberta cows turning black? Is it climate change? What has Stephen Harper been doing to hide this from working families? What kind of taxes do we need to implement to make this stop?”
As he finished his article he asked the Engineer to “peer review” it for him. The Engineer took the lap top and tapped away for a couple of minutes and handed it back to the CBC science guy. The revised article read “What we know so far is that there is at least one bovine in Alberta, it is a bull, and it is black on at least one side”.
Gerry
Gerry, Now that is hilarious and thank for the laugh, but your statement is so true.
Bravo! Jolly good speech. Now if we can just get people to take our stuff as seriously as the stuff that did actually get peer reviewed, we can end the scourge of vaccinations being forced on us by the NWO and UN and…. er….. sorry, wrong blog… my bad.
😛
And let us not forget Poor Phil Jones, who once declared (inter alia):
Full quote, source (and more!) at Phil Jones keeps peer-review process humming … by using “intuition”
Huh? No code, no data, no due diligence? What are they reviewing, writing style?
How does peer review work if the peer reviewers are receiving larger grants for the same field of work than the person being reviewed hopes to receive.
Economics contains elements of science and math but it is neither math or science. Like an oil painting may contain the mathematics of perspective and the theory of color and light, it is neither math or science. It is not a bad thing, just that to me it is closer to liberal arts, such as the study of sociology or religion.
The problem is as with certain fields of science, many fields in liberal arts study has replaced academic integrity and ethics with activism and political expediency. Look at democratic campaign tactics for the usual suspects that depend on weak, shoddy, unethical or plain stupid academics.
I refuse to bow at the feet of the priests of climate change. God gave me common sense – and the power of observation. I see Winter, Spring, Summer, and Fall. Nothing has changed – and if we can deal with those extremes, we will have no problem dealing with a couple of degrees difference in temps. We don’t control the climate; it controls us.