The Problem with ‘Peer Review’

From THE DAILY SCEPTIC

by James Alexander

In May 2017 a sculpture was displayed at the Moscow Higher School of Economics, believe it or not, in honour of the great secular-rational god Peer Review. The sculpture takes the form of a die displaying on its five visible sides the possible results of review — “Accept”, “Minor Changes”, “Major Changes”, “Revise and Resubmit” and “Reject”.

Peer review. What is it? Why does it matter? Where did it come from? How old is it?

A fairly solid academic article – Noah Moxham and Aileen Fyfe, ‘The Royal Society and the Prehistory of Peer Review, 1665-1965’, published in The Historical Journal 61 (2018), pp. 863-889 – begins with an untruth stated by the House of Commons committee on Science and Technology in 2011.

In one form or another, peer review has always been regarded as crucial to the reputation and reliability of scientific research.

Always? Fact-check: False. It’s a lie, or an error. Apparently, many people think that peer review was invented in the 17th century. Not so. Let me summarise Moxham and Fyfe’s findings:

• “Peer review” was not named, they say, until the 1970s.
• In relation to the Royal Society the first editor of the Transactions in the 17th century actually sought copy from authors.
• In 1751 someone wrote a satire exposing some of the very silly papers that had been published in the Transactions.
• In 1752 the Royal Society took responsibility, financial and editorial, for the Transactions which, until then, had been informally arranged.
• In 1774 the Royal Society refused, however, to take collective responsibility for what was published: saying the responsibility remained with the author.
• In 1831 Babbage asked for more careful consideration of papers and in 1832 written reports on papers were asked for.
• In 1896 Joseph Lister created committees to deal with submitted papers.
• In 1936 someone complained that too much “routine research” was being published. This sounds like we are approaching our modernity.
• By the 1990s peer review was seen as normal.

In other words, what we call refereeing was sometimes evident in the early days, when an editor sought someone else’s opinion, but everything was very informal until the 19th century, and, in fact, there was no firm protocol until the late 20th century.

That is, almost all the glories of science were achieved when the system was informal, or before there was even such a thing as an informal system.

Along the way there is an interesting tale about one John James Waterston, a 19th-century Scot. He was first to venture a kinetic theory of gases. But his paper was dismissed as rubbish by Sir John William Lubbock: indeed, it was archived by the Royal Society, i.e., not returned to the author. Since Waterston had made no copy for himself, he had to rewrite it from scratch. Fifty years later, Lord Rayleigh, recognising the value of the archived original, had it belatedly published. Rayleigh wrote:

The history of [Waterston’s] paper suggests that highly speculative investigations, especially by an unknown author, are best brought before the world through some other channel than a scientific society, which naturally hesitates to admit into its printed records matter of uncertain value. Perhaps one may go further and say that a young author who believes himself capable of great things would usually do well to secure favourable recognition of the scientific world by work whose scope is limited, and whose value is easily judged, before embarking upon higher flights.

To translate into modern terms: send predictable box-ticking rubbish to Nature or Science or the Transactions or the Lancet. And send original science, unless you are well-known to the editors and can overcome their prejudices using wine and biscuits, er, we are not sure where, but somewhere, and good luck.

In conspiratorial circles this state of things is sometimes blamed on, believe it or not, Robert Maxwell. Robert Maxwell, who he? Well, Jeffrey Epstein’s spiritual father-in-law. Both remarkable Great Gatsbys, and, as such, still unexplained. No one knows how R.M. died. No one knows how J.E. died. They were in Intelligence: query, whose intelligence? They were in Science, query, but what was their interest in it? They had Money: query, whose money? Private Eye made much fun of Capn Bob, but was perhaps less acute, or simply less interested, in Squire Jeff. I thought I’d look into the dark side of this story, but failed, as my library here does not have a copy of John Preston’s recent biography of Maxwell, the older biography is missing, and the only library books about Maxwell were two, yes, two, separate books of jokes about him, both published in 1992, shortly after his death. Incidentally, the jokes were variable: but perhaps the best one I read this afternoon was: “What is the difference between Robert Maxwell and the Royal Navy? One rules the waves and one waives the rules.” Oh go on, there was another one: “What is long, brown and goes fft as it hits the water? Robert Maxwell’s last cigar.” Why have no similar joke books been published about Jeffrey Epstein? There, my friends, is the difference between the 1990s and the 2020s, and also the difference, perhaps, between the UK and the USA.

Anyhow, isn’t it fascinating that, in the more conspiratorial corners of our attempt to make sense of the world, ‘peer review’ is sometimes tangled up in the spokes of Jeffrey Epstein’s fatal bicycle? Not only because Gates, Chomsky et al. were Epstein’s brothers-in-arms, but for the very odd reason that we hear that Robert Maxwell, Ghislaine’s doomed father, not only made his fortune from academic publishing but also invented ‘peer review’.

Let us look into Maxwell. I remember that when I was young Maxwell was one of those unaccountable and unarguable titans, the equal of Murdoch, but that this all exploded when he suddenly died and his empire was found to be established on embezzlement. He died in 1991 (incidentally, when I was an undergraduate at Lubbock’s and Rayleigh’s old college). Born Jan Hoch in 1923, Maxwell has a story which, in its Wikipedia version, is a bit obscure. He was Czech, served in the British army, and then, after the War, used ‘contacts’ in the Allied Occupation authorities to buy his way into publishing, becoming distributor for Springer-Verlag, publisher of scientific books, and eventually in 1951 buying three-quarters of a publishing firm which he renamed Pergamon Press – oddly, named after a city in Turkey where there used be a king with a good library. Pergamon Press was apparently the source of his fortune: he was eased out in 1969, but bought it again in 1974, and eventually sold it in 1991 for £440 million to Elsevier, which still possesses all its extant titles, all those hundreds of academic journals that Capn Bob launched across the seven academic seas. In the meantime he bought the Mirror Group in 1984, and this is what made him famous, though it was not what made him rich.

Now, there are all sorts of dubious things about Maxwell. Intelligence: on which side? MI6? KGB? Contacts: who exactly? Involved in German scientific publishing? How? Found the money to buy out Butterworth-Springer. From where? But who cares: he was obviously a hustler, the sort of person who generates money simply by moving about and scraping about. The fact is, there will be dubiety in the original story, but it is very unlikely to concern ‘peer review’. There was no money in peer review, and I doubt Maxwell, though he obviously had some genius, got as far as thinking that one through. He was interested, as I say, in quality (and money) and probably left quality to his well-wined scientist guests to think about.

Nay, the fact is that ‘peer review’ is all about self-affirming bureaucracies. Old journals were amateur and aristocratic affairs. The original Transactions of the Royal Society was paid for by individual subscribers (about 750 in the 19th century), all of whom received a copy, an extra 100 copies printed for use rather than sale. But nowadays the whole thing is a grotesque machine. Elsevier, we are told, has a higher profit margin ­– of course, not absolute profit – than Google or Amazon, because governments, universities and scientists themselves pay for the matter published and also the editing of that matter. It is gratis, goes in a circle, and the unpaid labour then pays for it, via a set of institutions dominated by a logic no one can quite dismiss. All Elsevier has to do is publish something and sell it back to universities whose members have laboured, unpaid, to write for it, and also edited it, reviewed it, etc. It is a grand monopoly capitalist conspiracy surviving on the corpse of collective responsibility.

And peer review costs ‘em nothing.

I don’t have much peer reviewing to do, as I am not a success in any narrow modern academic field: my interests are, cough, too broad (that’s my way of putting it: others would say too shallow). I seem to be on someone’s list for conservatism and tradition, since I published things on these once or twice, and obviously I am 20th on a list of possible names for these subjects. Most of the stuff I read is worthy rubbish, and some of it unworthy. I think only once was I enthusiastic – about a short article in which someone kicked the shins of Jeremy Waldron.

I don’t think anyone complains too much about writing for peer review, and there is certainly not enough opposition to the entire system of peer review (there should be more). What everyone dislikes is the fact that their research is dismissed by peer reviewers for rather arbitrary reasons. Even the most liberal academic complains about this: it is one subject on which almost everyone can agree. Everyone wants peer review for others and intellectual freedom for themselves.

The Guardian has a couple of good articles on peer review and academic publishing. As early as 2003 Michael Eisen argued that everyone should be freely available to everyone. And in 2017 Stephen Buranyi pointed out not only that publishers extract much free work from scientists, but, worse, that “the publishing industry exerts too much influence over what scientists choose to study, which is ultimately bad for science itself”. I’d say that this is part of the problem: the other half is centralisation, not only of publication but also of funding. Research grants are an equally major form of monopoly control. Repeat old research? Here’s a grant. Trying something original? Silence. And worse: in the humanities, trying something controversial? Rejection and perhaps even the beginning of reputational death and cancellation.

It is the Buranyi piece from 2017 that argued that “few people in the last century have done more to shape the way science is conducted today than Maxwell”. Let us see what the argument is. He explains that there was a view after the Second World War that British science was good but British scientific publishing was bad. The directors of Butterworths were ex-intelligence: the Government combined Butterworths with renowned German publisher Springer and hired Maxwell to manage it. When Butterworth decided to get out, Maxwell found £13,000 to buy it. This was in 1951. It was Maxwell’s collaborator, Paul Rosbaud, the scientific editor, who had also previously worked in Intelligence during the War, who worked out what to do.

The scientific societies that had traditionally created journals were unwieldy institutions that tended to move slowly, hampered by internal debates between members about the boundaries of their field. Rosbaud had none of these constraints. All he needed to do was to convince a prominent academic that their particular field required a new journal to showcase it properly, and install that person at the helm of it. Pergamon would then begin selling subscriptions to university libraries, which suddenly had a lot of government money to spend.

Interestingly, Maxwell actually at this stage operated a seller’s market, more or less approaching scientists to see if they had anything they had to hand that he could publish, as Oldenburg of the Royal Society had done for the Transactions in the 17th century. O tempora, O mores! I know I wish that system was still in place.

Apologies for returning to the Ghislaine-Epstein-Gates-Mandelson-Andrew story, but doesn’t the following excerpt sound as if Maxwell Senior invented the protocol that young Jeffrey was to inherit: with all that hobnobbing with the great and good, including the scientific great and good?

By then, Maxwell had taken Rosbaud’s business model and turned it into something all his own. Scientific conferences tended to be drab, low-ceilinged affairs, but when Maxwell returned to the Geneva conference that year, he rented a house in nearby Collonge-Bellerive, a picturesque town on the lakeshore, where he entertained guests at parties with booze, cigars and sailboat trips. Scientists had never seen anything like him. “He always said we don’t compete on sales, we compete on authors,” Albert Henderson, a former deputy director at Pergamon, told me. “We would attend conferences specifically looking to recruit editors for new journals.” There are tales of parties on the roof of the Athens Hilton, of gifts of Concorde flights, of scientists being put on a chartered boat tour of the Greek islands to plan their new journal.

This, apparently, is the story. Maxwell: journals, more journals, grifting and huckstering, at first 40 journals, then 150, sharpening his suits, buying a Rolls Royce, leasing Headington Hall. He should have stuck to this: he was evidently quite good at it, and could have remained respectable, especially if he had avoided lapsing into embezzlement. (There is no evidence he added young women into any of his inducements.) Not that it was admirable. It was, of course, not admirable at all. Academics did not know how to resist. “If a serious new journal appeared, scientists would simply request that their university library subscribe to that one as well.” And everyone still wonders why Thatcher et al. had to scale the cost of the universities down from the 1980s onwards…

One thing about Maxwell, though: he published what scientists independently wanted to publish. He left editorial decisons to scientists. So, on balance, I think it is unlikely he had anything to do with ‘peer review’. No one mentions ‘peer review’ in relation to Maxwell. Even Brian Cox (no, not that Brian Cox, and, no, not that Brian Cox either), who worked with him at Pergamon for 30 years, and wrote a brief account of their collaboration in 1998, republished in 2002, mentions nothing about peer review. Instead, the achievement is:

In the 40 years between its founding and its sale to Elsevier in 1991, Pergamon published over 7,000 monographs and reference works, and launched 700 journals, 418 of which were still current titles when the company was sold and 400 of which continue to this day to be sold under the Pergamon imprint within Reed Elsevier.

It is about quantity, not quality. And peer review is – supposedly – about quality. I think Maxwell probably trusted the scientists with the science. Peer review was probably just a side-effect, a sort of mechanisation, a way of editors convincing themselves that they had a distinctive ‘field’ for their new journal. If Maxwell was at fault for anything it was for imposing quantity on everyone, after which they worked hard to make the quality match the quantity, which they could only do by narrowing down and intensifying study and increasingly constructing it in terms of what we now, in the great 21st century, call ‘outputs’ and ‘impacts’.

But the story does not matter. Peer review is problematic for reasons not to do with its origin.

Here are some quotations.

The first was uttered by Eric Weinstein, about an hour and a half into a podcast with his brother Bret Weinstein, first published on February 20th 2020:

Peer review is a cancer from outer space. It came from the biomedical community. It invaded science. The old system, I have to say this because many people who are now professional scientists have an idea that peer review has always been in our literature, and it absolutely mother-fucking has not. It used to be that an editor of a journal took responsibility for the quality of the journal, which is why we had things like Nature crop up in the first place, because they had courageous, knowledgeable, forward-thinking editors. I just want to be very clear, because there is a mind virus out there that says that peer review is the sine qua non of scientific excellence yada, yada, yada, bullshit, bullshit, bullshit… and if you don’t believe me go back and learn that this is a recent invasive problem in the sciences. … When Watson and Crick did the double helix, and this is the cleanest example we have, the paper was agreed should not be sent out for review, because anyone who was competent would understand immediately what its implications were. There are reasons that great work cannot be peer reviewed. Furthermore, you have entire fields that are existing now with electronic archives that are not peer-reviewed. Peer review is not peer review. It sounds like peer review. It is peer injunction. It is the ability for your peers to keep the world from learning about your work… because peer review is what happens, real peer review is what happens, after you’ve passed the bullshit thing called ‘peer review’.

Here is another quotation. As Weinstein says, original thought and speculation and brilliance has a vexed relation to peer review. But, as I have said, there is also a problem about money. This is because, unfortunately, all the money comes from great centralised caches nowadays, and this money is thrown about on the basis of peer review. The quotation below is very good, and shows that the Guardian, in the old days, used to put out good opinion pieces. Here is Philip Ball in 2011:

The moral seems to be that really innovative ideas don’t get funded – that the system is set up to exclude them. To wring research money from government agencies, you have to write a proposal that gets assessed by anonymous experts (‘peer reviewers’). If its ambitions are too grand or its ideas too unconventional, there is a strong chance it will be trashed. So does the money go only to ‘safe’ proposals that plod down well-trodden avenues, timidly advancing the frontiers of knowledge a few nanometres? There is some truth in the accusation that grant mechanisms favour mediocrity. After all, your proposal has to specify exactly what you are going to achieve. But how can you know the results before you have done the experiments, unless you are aiming to prove the bleeding obvious?

And if that was not pithy enough, here is Eric Voegelin, one of those great émigre scholars who went to America. This is from Robert B. Heilman and Eric Voegelin: A Friendship in Letters, 1944-1984, p. 312:

If you place money in the hands of academic mediocrities, it will hardly improve scholarship or advance science, but rather increase the social power of mediocrity.

I doubt Maxwell intended that. We might blame him for it: but it was a consequence of what he and others did, not what they actually wanted from the system.

The truth of ‘peer review’ was that it was the means by which bureaucratic controls were imposed, indeed, self-imposed, on intellectual activity. That is it.

James Alexander is a Professor in the Department of Political Science at Bilkent University in Turkey.