The Absurdity of Peer Review

What the pandemic revealed about scientific publishing

Here are a few selected excerpts from this excellent article from elemental

Mark Humphries Jun 3 · 7 min read

I was reading my umpteenth news story about Covid-19 science, a story about the latest research into how to make indoor spaces safe from infection, about whether cleaning surfaces or changing the air was more important. And it was bothering me. Not because it was dull (which, of course, it was: there are precious few ways to make air filtration and air pumps edge-of-the-seat stuff). But because of the way it treated the science.

You see, much of the research it reported was in the form of pre-prints, papers shared by researchers on the internet before they are submitted to a scientific journal. And every mention of one of these pre-prints was immediately followed by the disclaimer that it had not yet been peer reviewed. As though to convey to the reader that the research therein, the research plastered all over the story, was somehow of less worth, less value, less meaning than the research in a published paper, a paper that had passed peer review.

Imagine reading about the discovery of the structure of DNA with that same reticence we use today: “In a recent Letter to the journal Nature, Cambridge University scientists James Watson and Francis Crick proposed a new structure for DNA (not yet peer reviewed). They claim their “double helix” model, a spiral of two strands of bases, both explains decades of experimental work, and provides a clear mechanism for copying genes. Their proposal drew heavily on data contained in Letters in the same issue of Nature from the teams of Rosalind Franklin (not yet peer reviewed) and Maurice Wilkins (not yet peer reviewed).”

Or consider this modern take on a certain scientist’s annus mirabilis:
“The past year of 1905 has been a remarkable for one Herr Einstein, who proposed no less than four theories new to modern physics in a series of papers. His first was on a much-anticipated explanation of the photoelectric effect (not yet peer reviewed), the second on how Brownian motion arises from the collision of invisible particles (not yet peer reviewed), the third on the equivalence between mass and energy (not yet peer reviewed), and the final paper updates Newton’s mechanics to be more accurate for objects moving close to the speed of light (not yet peer reviewed).”

These imagined reports are both eye-wateringly ridiculous.

Continuing…

Pandemics don’t conveniently hang around waiting for that slow, grinding peer review process to judge science. Science in the time of Covid-19 has had to be nimble, quick on its feet, has had to show its findings to the world without the layers of review-and-revise. We’ve needed rapid research into models of transmission, into immunity and reinfection, into public health messaging and effective interventions, into drugs to treat symptoms, machines to support the severely ill, and the development of radically new types of vaccine. So pre-prints, those manuscripts put on the internet for all to read before peer review, became the weapon of choice. Long common in physics, pre-prints in biology and especially medicine exploded in number during the pandemic.

And the media have dealt with this explosion by consistently pointing out when research has not yet been peer reviewed. Presumably they do this to warn the reader that the research lacks the safeguards that peer review brings. The problem with that warning is peer review guards against nothing.

Does it catch fraud or manipulations of data? No, patently not: peer reviewers are not omniscient, so they cannot divine made-up data, nor can they check all the outputs of a lab to see when they’ve simply copy-and-pasted data between papers. If they were, we wouldn’t have the website PubPeer stuffed to the gills with people flagging potentially serious misdemeanors in published papers, nor Retraction Watch’s endless reporting of papers so dodgy they’re expunged from the literature.

And finally concluding. This is an excellent piece and should be read in its entirety at the source.

This then is why I was so bothered about how Covid-19 research is reported: peer review is no guard, is no gold standard, has little role beyond gate-keeping. It is noisy, biased, fickle. So pointing out that some piece of research has not been peer reviewed is meaningless: peer review has played no role in deciding what research was meaningful in the deep history of science; and played little role in deciding what research was meaningful in the ongoing story of Covid-19. The mere fact that news stories were written about the research decided it was meaningful: because it needed to be done. Viral genomes needed sequencing; vaccines needed developing; epidemiological models needed simulating. The reporting of Covid-19 research has shown us just how badly peer review needs peer reviewing. But, hey, you’ll have to take my word for it because, sorry, this essay is (not yet peer reviewed).

Mark Humphries researches computational neuroscience at the University of Nottingham, U.K., and is the author of “The Spike: An Epic Journey Through the Brain in 2.1 Seconds” (Princeton University Press) out now.