Guest essay by Abzats.
The most exciting period in science was, arguably, 1895-1945. It was marked by discoveries that changed the foundations of modern science: X-rays, quantum mechanics, superconductivity, relativity theory and nuclear energy. Then, compare this with the next 50 years in science. Incomparable. Nothing of that scale or impact. Yes, technology has advanced, but fundamental science – has come to a crawl. Have you ever wondered why? What changed as the 20th century grew older? Among other things, research budgets and the number of PhDs increased exponentially. This cannot be bad.
Well, it can. All depends on the rules of the game. And they have changed. The change went largely unnoticed by the general public. In this article I will try to bring everyone up to speed. I will explain to non-scientists the “business model” of modern science. People may want to know. After all, scientists are burning public money, billions a year. And, I am quite sure, those who get my message will react with “you cannot be serious!” And leaders of organized crime will be pulling their hair out in despair: “why did not we think of this first?”
Single most important element of the modern science machinery is the peer review process. It was introduced a long time ago, but it took over the scientific community at about mid 20th century. Why is it important? Every scientist must publish his or her work. If you do not publish, you will not advance your career. This works the same way as it does, say, for a businessman – if you cannot close a single deal, you are finished. Most journals have adopted peer review policies. Peer review process is also standard for research grants competitions. It is also the foundation of the tenure and promotion process at universities.
Well then, what is it exactly? To save time, let me explain peer review of papers submitted for publication in scientific journals. Once a journal receives a manuscript the journal sends it to 2-3 reviewers, who are experts in the field. Each reviewer writes a report that includes a recommendation on whether or not the manuscript should be published and advice to the author on how the manuscript can be improved. So far so good. Nothing seems wrong. This should work wonderfully. Well, in theory only. In reality it does not. In reality it is more of a disaster.
Let me explain. All the reviewers are anonymous. That is, they know your name but you do not know theirs. This is the first red flag: unless you plan to do something really bad, why do you insists being anonymous? The second red flag is that none of them gets paid. Those who believe in Santa Claus will say, well, they are just nice people volunteering their time to help advance science. Those who work for a living will smell a rat. I can give you one reason: being a reviewer gives you power over other people. Some just enjoy it, others use it to advance their own agenda. Such as approve manuscripts that praise reviewer’s own research and reject those that criticize it.
The power reviewers have is enormous. Put yourself in author’s shoes. You worked hard for six months on a manuscript. Your work is brilliant, if you publish it, not only will you advance your career, it will make you a leader in the field. Then, the manuscript goes to a reviewer who just happens to be having a bad day. He browses through the manuscript for 20-30 minutes, does not like the name of the author (never heard of him, “wrong” ethnicity, or … whatever), and rejects the paper. Can you appeal? No. You can write an angry letter, but you cannot call your attorney. Because nobody is breaking the law. because there isn’t any.
They can ruin your career and drive research, often funded by the public, to a dead end, and they are not accountable to anyone. In such a system, for most scientists the best, or should I say the only, way to advance their careers is by kissing up to those in higher positions: in person, in manuscripts, and in the whole research strategy. This has been going on for decades. As a result of this “natural selection”, the scientific community has been consumed by cronyism. Parts of it are rotten to the core.
Let me give you one example. Last year I attended a Radiation Research Society meeting. It was held in Maui, Hawaii. Why? Obviously it is a great spot for a vacation. You will not find any major research centers in the neighborhood. If you are still thinking of defending this choice, get this – the conference was held at the Grand Wailea Resort. The thing about this place is that luxury here is obscene. It is a kind of place a bum would go to after winning a lottery. And, guess what, I believe I have seen a few. Never before had I seen an invited speaker at a major conference making bodily function jokes. Here I had seen more than one, including a recipient of a lifetime achievement award spelling a word for body waste and thinking it was funny. Do not get me wrong, I am not judging here. But, if he jokes at a preschooler level, would you trust him to be a reviewer of your work? Do I need to mention who paid for the event? Or, that it took place during the worst economic crisis in decades?
A couple of other problems. Reviewers have no real motivation to work fast. Here is what you would see when checking status of your manuscript on a journal’s web site: manuscript to referee, unable to report – sent to another referee, and so on, several times, for weeks and months. Nonsense. With all the technology available, a manuscript can be published within hours. But, no, it has to sit for weeks on somebody’s desk. Somebody who just does not care enough. Or, worse, someone who is interested in delaying the process. The reviewer may be working on exactly the same problem and wants his paper published first.
Another problem that extremely frustrates me as an author are suggestions reviewers make on how I should improve my manuscript. Originally, may be, it was a good idea – your peers offering you advice that will help you improve your work. But it all has gone very wrong. These days these are not suggestions or advice – these are demands. You change your manuscript exactly as you are told, or it will be rejected. I am a well established scientist, why do I have to take advice from someone who would not even reveal his identity or credentials? And, finally, this system is perfect for stealing ideas. After you submit your manuscript you have no control of who will access it. All you can hope for is human decency, and it is not always there.
This brings us to the root of the problem. People, including scientists, are flawed. Few will miss a chance to stab competition in the back and abuse whatever little power they may have. I am not the first to criticize the peer review process. But I am not. Criticizing implies it can be fixed. It cannot. It was a bad idea all along. Then, what can be done? There is no quick and easy solution.
But I know where to start – ban peer review. And I know this can be done, this nonsense can be dealt with. This is not brain surgery, this is all about leveling the playing field, making rules for fair and open competition. These problems have been solved in all other spheres. Only scientists for whatever bizarre reasons received a special treatment and the right to live in lawlessness. Which is so wrong, I cannot find words to describe. Science is one most important sphere of human activity.
Who will find cure from cancer? Who will prevent the planet from becoming uninhabitable? Scientists. Not those from the beaches of Grand Wailea. Real ones. I hope we can still find some and reverse, before it’s too late, the depletion of brains. Let’s get started. Ban peer review!
An interesting post – thanks. By chance, I recently read a couple of articles on related lines, see here and href=http://milesmathis.com/phycor.html>here. Sadly, the science garden is not as pretty as we all want to believe.
90% lies and BS with 10% EPA and there you have it.
I’ve been reviewing papers in my field for about 30 years, probably about 5-10 per year. Most of the time I believe I either helped make the paper better or else I saved the journal from publishing a below-par paper. I have some evidence for the first case, since authors have sometimes given the anonymous peer reviewers credit for having improved their paper. The many people in my field that I know personally have much the same experience as I do.
However, the climate science field is different. Here we have rather naive generalist scientists who happened to strike it rich in terms of public awareness, savvy politicos sensing an opportunity, a corrupt UN seeking global power, powerful NGOs like WWF and Greenpeace riding the wave, administrator-heavy scientific organizations seeing ways to funnel money to their membership, poor island nations seeing a way to shame richer ones, and just in general a perfect storm of chance variables coming together. Peer review became pal review for this branch of science. Perhaps a similar branch of corrupt science is the pharmaceutical research studies, where the payoff for getting a drug through clinical trials is worth billions.
So I would argue that peer review is not necessarily broken for all of science, only for the corrupted branches.
The poster Abzats above says “Ban peer review!” Interestingly, this has been done for more than a decade now for high-energy physics and cosmology–it’s called Arxiv, where anyone can post a paper. Top physicists use it because they like to get their ideas out there instantly and see those of their colleagues as soon as possible. Yes, there are plenty of kooks doing the same, but no one reads their stuff, so it does no damage. Some of these “preprints” get argued about, changed, and eventually published in a peer-reviewed journal, but in many cases, the publication does not occur, yet the Arxiv version has a powerful ripple effect.
Whether this would work for climate science is unclear to me. Perhaps we already have a form of this, since persons like McIntyre, Nic Lewis, etc. tend to publish on the blogs more often than in the journals.
Goldie says:
June 25, 2013 at 12:55 am
A somewhat one sided perspective. I have worked for two journals and published many papers and I have to say that I have never found it to be that way.
What was the subject matter?
“Not those from the beaches of Grand Wailea. Real ones.”
Idealist! Where have you seen a “real scientist” and what you understand under that?
The guys you met ARE REAL SCIENTISTS.
You are unhappy about a scientist making “body jokes”?
Take Einstein.
http://www.cracked.com/article_18559_6-famous-geniuses-you-didnt-know-were-perverts.html
You are unhappy with peer review?
Sorry, there are only two ways proving you are right:
1. Design and build a working device (e.g. H-Bomb)
2. Build a scientific mafia supporting your weird (sorry, ingenious!) work
You remember, in Soviet Union before 1953 there was no “pal review”.
Just if you failed with the task assigned you by the Kommunist Party (aka Stalin), you will be shot in the best case, or put into GULAG in the worst case.
If a cost/benefit analysis has not been run, it must be a (taxpayer) Government project.
No business would ever attempt it without all the free money.
If the spigot is left fully open, the funds/funding will run out. Leaving the hypothetical children in the lurch.
Which won’t really be a problem, as long as we keep funding our politicians.
All they need is more of our money, to solve our/their woes.
I’m sick of it.
If the quality of Mr. Abzats scientific publications can be judged by his proof-reading skils demonstrated here today, then this might explain some of the issues he has encountered.
Oh my, the defenders of anonymous peer review are not pleased, but are reduced to sniping about grammar. That is a good sign that the author is on target.
Anonynous peer review, plus governments dedictated to using tax payer money in the hundreds of billions per year is not a good recipe.
Big science research and big medical research are rife with intefrity issues world wide.
Abzat’s pointing this out is nice way to start a conversation. Dismissing the essay over grammar while avoiding the point is rather shallow, to be diplomatic.
Increase the number of reviewers to 10
Name a time limit
Pay them the max rate if they make the time limit, less if they delay
Rate the paper on the number of positive reviews 3/10 or 7/10, even 0/10 if some wants to revive the flat earth theory
Publish on a Government website so it is accessible to all.
Name the reviewers and their rating, they’re being payed now.
JDN says:
June 25, 2013 at 1:18 am
“@Anthony: This article was a rant. It would never pass peer review because it lacks a testable hypothesis.”
Yeah well the peer review system would do everything to preserve its existence wouldn’t it.
And basically it’s all government science. According to Parkinson’s Law it must become ever more inefficient. We see that in climate science or cosmology – which is wholesale conjecture – or string theory – which is unfalsifiable because it delivers every possible theory all at once – a parametrization explosion like in climate science.
One could say that the marginal use of further science spending becomes zero or negative. Another manifestation of the Keynesian endpoint.
Anyone found Dark Matter already?
Crick and Watson. The genome project. Shockley’s transistor. A few things that followed on from that… Vaccines for polio and a few other maladies. A trip to the moon. Some interconnected thingytube that lets machines communicate with each other. Faxes! How can we forget Faxes! Solar power. Nuclear power. Golden rice.Genetically Modified Organisms.
Just sayin’
I would say that it is not only science that appears to have stalled in the mid 20th century.
Consider music. All the truly great composers pre-date the 1930s, with the 19th century being the zenith.
Also art. What art today, really compares to the great masters of yore? The same, to a lesser extent, apples to goldsmiths and silver smiths, and the finest painted porcelain. Is there really any truly great art post the art deco period?
And perhaps also literature and poetry. Of course there are some good modern day stories, but the art of story telling and accomplished writing, in particular, seems to have diminished these past 100 years.
Has the world become too cynical, are there no true romantics? I do not know the answer, but there appears to be a greater number of people of average talent and ability, and far fewer truly exceptional geniuses amongst us.
Scientific enquiry wears more than just this shackle. Some countries do not
fund research They (enter your favourite bogeyman here) do not think can
be ‘monetised.’ This would have successfully scuttled Relativity and all its
spin-offs completely.
It’s little wonder ‘Buffy the Vampire Slayer’ was such a successful TV series.
JP Says
“Current “fundamental science” is doing just fine.”
Maybe, but just maybe. On the other hand, all the politicians we elect come
almost exclusively from Social Science, Political Science, Psychology, Economics
and Law. Same peer peview process and none of it Science. So fundamental Science is not doing just fine. Look at the trillions going awol in “Govt Science” grants, meanwhile they want us imprisoned on Earth and not to do manned space exploration and space mining? No sense of adventure anymore.
“Only scientists for whatever bizarre reasons received a special treatment and the right to live in lawlessness. Which is so wrong, I cannot find words to describe.”
I found words to describe it: religion, totalitarianism and some other words that will be snipped.
u.k.(us) says:
June 25, 2013 at 12:55 am
“But I know where to start – ban peer review.”
=============
To replaced by………what?
Consensus ?
–
How about “ban” press releases ?
It might slow down some of the stupid policy decisions being made.
The operative word being “stupid”.
///////////////////////////////
How about publishing everything?
Why not simply, publish the paper and the revieiwers’ comments on the paper.
Then let the reader make up their minds as to whether anything (ie., the paper, some part of it, the reviewers’ comments) is of interest and whether the science has been advanced by the paper and/or by the reviewers’ comments. After all, to see what has failed, can often be extremely useful in advancing understanding, and sometimes it can be useful to revisit views that were dismissed say 20 or 30 years earlier.
You bring up a fundamental issue in this post: how is knowledge validated?
By extension, since the manner of both becoming aware of knowledge and of confirming it cannot be separated from the conditions required to create or discover it – as you allude to in your comments relating to the structural demands that have developed as a con-commitment part of the current system – what are the conditions that allow knowledge to develop?
“Publish or perish” has been a term summarizing the decay of scholarly vocation for decades – at least for those most aware of its implications in demanding implacable compliance with what is intrinsically a bureaucratic and careerist mentality: for others, it provides a welcome format. The consequent rewarding through promotion, of person or product, and further opportunities and money is obligatory.
To get this process wrong can only be corrupting – it is inevitable.
Your focus on the nature of the review process is a good start.
It is quite bizarre. It should be immediately evident that any component of anonymity is being applied in the completely opposite manner to which it should.
It is the person submitting the work who should be anonymous. The work must be judged IN ENTIRETY on what it is in itself. There is NO other element that should intrude. ANYTHING that does intrude, including reputation of person or association, is corrupting. ALWAYS.
Any reviewer, if they are to be anonymous, must produce a cogent analysis that can be seen and judged against the submitted work by all – not just the submitter, and not just by any “decision maker” be they editor or other. The adjudication can otherwise only be meaningless since there is very obviously no basis at all for anyone to believe in it.
Except of course, as being the Judgement of Authority. Which means the control of knowledge by those whose structural positioning is that of the functionary. The result is at best stagnation but with a ready propensity to the perversion of the pursuit and maintainance of knowledge, and the twisting of interpretations of realities. As we see.
If any reviewer, or adjudicator, does NOT provide an evaluation that can in itself be judged, they must, at the least, be known. So their capacity in this role is accountable.
It should be both: they should be known and their reasoning seen and published. If anyone is not prepared to do this, but at the same time puts themselves forward for this responsibility, an immediate conclusion can be reached. They are unfit.
The above thoughts are not intended to be some sort of definitive solution. They address only an element in a system or tradition which may be entirely redundant or may have never been other than corrupting. It may be that it can be seen that a culture based on free publication and evaluations that do not have to be in effect ratified prior to general availability is now possible. Not just possible, but essential. That is after all, the very essence of inquiry and always has been.
Eric Worrall,
” If there is no peer review, how can you prevent creationists and people with a perverse agenda, such as promoters of fake medical treatments, from flooding credible journals with junk science?”
The Editors of the Journal! What do you think they are paid to do? Do you really think “Nature” would print a creationist story just because there is no peer review? How far do you think “promoters of fake medical treatments” would get trying to get published in The Lancet or the British Medical Journal even if there were no peer review?
The point of Peer review should be to stop silly mistakes appearing in otherwise genuine research papers. What it is wrongly being used for is stopping the “wrong kind of science” getting published. To be frank it is fairly pointless submitting a sceptical paper with merit to Nature where the editors are disinclined to publish anyway. Peer review is just being used as an excuse to refuse publication because the paper does not follow the party line. It is just being used as a way of abdicating proper editorial decision making.
What is needed is more journals, or journals brave enough to print articles that challenge the orthodoxy.
IMHO the big problem is anonymity. Remove that and half the problem disappears.
At least you know who the editor is and what his agenda is. In my opinion the best kind of review is open on-line review. “credible journals” should have an open on-line forum where scientists can submit papers for review. All reviews must be published so we can see what they say and reviewers names and credentials should be known and not anonymous.
If a journal wishes they could “vet” reviewers so that only trained professionals are allowed to review but the writer will know who they are. The editor will then decide whether the reviews have merit and whether the paper should be amended. If the writer wants to, they can refuse to change the paper in which case the paper will remain perpetually “in review” and all the reader will see is the on-line un-amended version alongside with the reviews. They can then make their own decision as to whether the reviewers have a point.
If the paper is amended to the Editor’s satisfaction it is either “published” or tagged as final and becomes part of the journal proper.
Gradually readers will want to read, and scientists submit their papers to, those journals that have a clear, open, fair and effective review process. Those that print random junk science will end up ignored and no decent scientist would ever submit a paper to them. Likewise those that refuse to publish obviously meritorious papers because the editor or reviewers do not agree with the conclusions will find other more sympathetic journals willing to publish.
Ultimately it is the readers who will make the decision which journals are successful and which are not.
This has been the case in general print journalism for ages. That is why we have The Wall Street Journal and The New York Times alongside rags like The National Enquirer or, here in the UK, The Daily Telegraph and The Guardian alongside the Daily Sport.
Err.. I can do jokes about bodily fluids and 100x more disgusting things; and if I say someone’s work sucks, then it does. Because the two things are COMPLETELY unrelated. Just saying, because you asked.
This was an overall terrible ‘news’ item – why did you even post it?
richard verney says:
June 25, 2013 at 2:14 am
“And perhaps also literature and poetry. Of course there are some good modern day stories, but the art of story telling and accomplished writing, in particular, seems to have diminished these past 100 years.
Has the world become too cynical, are there no true romantics? I do not know the answer, but there appears to be a greater number of people of average talent and ability, and far fewer truly exceptional geniuses amongst us.”
============
I was just thinking the same sort of thing, only different.
Mostly the cynical part, genius is just knowledge with a foundation, it will never go anywhere without curiosity. Not sure there ever were many geniuses to start 🙂
For those who watch “Big Bang Theory”, or those who don’t:
“What’s new in the world of Physics?”
“….. Nothing!”
Principea Scientific International are publishing on the internet where anyone can ”peer review” but not cause the problems cited above. They started because of the flawed peer review process. I think it is possible to publish through them.
Right–Organized Science has taken over from Organized Religion.
If papers are posted online, prior to peer review, registered PhD readers could give them thumbs up / thumbs down, which would tend to filter out the wacko stuff.
Henry Bauer suggested that a “Science Court” (first proposed in the 1970s) could be a court of last resort, to appeal to when peer review goes wrong. It wouldn’t be perfect, but it could only improve the current system. See the thread devoted to Bauer’s recent book, Dogmatism in Science and Medicine, at http://wattsupwiththat.com/2012/11/05/dogmatism-in-science-and-medicine-how-dominant-theories-monopolize/
And that doesn’t even cover the awarding of government monies to projects.
Anthony, WUWT already, in effect, provides an open, transparent peer review forum but you could formalise that a bit.
If you had a new tab for ‘Peer Review’ a draft paper could be published as a thread. Two or three people could be invited to become reviewers/moderators. Only comments that were part of the peer review process would be kept and the original article/paper would be changed over time. At some point the paper would be ‘published’ which would mean that comments are closed (and the people involved can get on with their lives). You could even publish it behind a paywall if you wished by removing the thread and putting it on Amazon as an eBook (with all the comments) for a modest price.
Papers which have already been published elsewhere could also be peer reviewed by this process.
I think that a process like this could set a new standard for the peer review process in any discipline.
You should be aware that I asked three of my closest friends for their opinion before posting this and they all agreed so I know that I’m right!