Guest essay by Abzats.
The most exciting period in science was, arguably, 1895-1945. It was marked by discoveries that changed the foundations of modern science: X-rays, quantum mechanics, superconductivity, relativity theory and nuclear energy. Then, compare this with the next 50 years in science. Incomparable. Nothing of that scale or impact. Yes, technology has advanced, but fundamental science – has come to a crawl. Have you ever wondered why? What changed as the 20th century grew older? Among other things, research budgets and the number of PhDs increased exponentially. This cannot be bad.
Well, it can. All depends on the rules of the game. And they have changed. The change went largely unnoticed by the general public. In this article I will try to bring everyone up to speed. I will explain to non-scientists the “business model” of modern science. People may want to know. After all, scientists are burning public money, billions a year. And, I am quite sure, those who get my message will react with “you cannot be serious!” And leaders of organized crime will be pulling their hair out in despair: “why did not we think of this first?”
Single most important element of the modern science machinery is the peer review process. It was introduced a long time ago, but it took over the scientific community at about mid 20th century. Why is it important? Every scientist must publish his or her work. If you do not publish, you will not advance your career. This works the same way as it does, say, for a businessman – if you cannot close a single deal, you are finished. Most journals have adopted peer review policies. Peer review process is also standard for research grants competitions. It is also the foundation of the tenure and promotion process at universities.
Well then, what is it exactly? To save time, let me explain peer review of papers submitted for publication in scientific journals. Once a journal receives a manuscript the journal sends it to 2-3 reviewers, who are experts in the field. Each reviewer writes a report that includes a recommendation on whether or not the manuscript should be published and advice to the author on how the manuscript can be improved. So far so good. Nothing seems wrong. This should work wonderfully. Well, in theory only. In reality it does not. In reality it is more of a disaster.
Let me explain. All the reviewers are anonymous. That is, they know your name but you do not know theirs. This is the first red flag: unless you plan to do something really bad, why do you insists being anonymous? The second red flag is that none of them gets paid. Those who believe in Santa Claus will say, well, they are just nice people volunteering their time to help advance science. Those who work for a living will smell a rat. I can give you one reason: being a reviewer gives you power over other people. Some just enjoy it, others use it to advance their own agenda. Such as approve manuscripts that praise reviewer’s own research and reject those that criticize it.
The power reviewers have is enormous. Put yourself in author’s shoes. You worked hard for six months on a manuscript. Your work is brilliant, if you publish it, not only will you advance your career, it will make you a leader in the field. Then, the manuscript goes to a reviewer who just happens to be having a bad day. He browses through the manuscript for 20-30 minutes, does not like the name of the author (never heard of him, “wrong” ethnicity, or … whatever), and rejects the paper. Can you appeal? No. You can write an angry letter, but you cannot call your attorney. Because nobody is breaking the law. because there isn’t any.
They can ruin your career and drive research, often funded by the public, to a dead end, and they are not accountable to anyone. In such a system, for most scientists the best, or should I say the only, way to advance their careers is by kissing up to those in higher positions: in person, in manuscripts, and in the whole research strategy. This has been going on for decades. As a result of this “natural selection”, the scientific community has been consumed by cronyism. Parts of it are rotten to the core.
Let me give you one example. Last year I attended a Radiation Research Society meeting. It was held in Maui, Hawaii. Why? Obviously it is a great spot for a vacation. You will not find any major research centers in the neighborhood. If you are still thinking of defending this choice, get this – the conference was held at the Grand Wailea Resort. The thing about this place is that luxury here is obscene. It is a kind of place a bum would go to after winning a lottery. And, guess what, I believe I have seen a few. Never before had I seen an invited speaker at a major conference making bodily function jokes. Here I had seen more than one, including a recipient of a lifetime achievement award spelling a word for body waste and thinking it was funny. Do not get me wrong, I am not judging here. But, if he jokes at a preschooler level, would you trust him to be a reviewer of your work? Do I need to mention who paid for the event? Or, that it took place during the worst economic crisis in decades?
A couple of other problems. Reviewers have no real motivation to work fast. Here is what you would see when checking status of your manuscript on a journal’s web site: manuscript to referee, unable to report – sent to another referee, and so on, several times, for weeks and months. Nonsense. With all the technology available, a manuscript can be published within hours. But, no, it has to sit for weeks on somebody’s desk. Somebody who just does not care enough. Or, worse, someone who is interested in delaying the process. The reviewer may be working on exactly the same problem and wants his paper published first.
Another problem that extremely frustrates me as an author are suggestions reviewers make on how I should improve my manuscript. Originally, may be, it was a good idea – your peers offering you advice that will help you improve your work. But it all has gone very wrong. These days these are not suggestions or advice – these are demands. You change your manuscript exactly as you are told, or it will be rejected. I am a well established scientist, why do I have to take advice from someone who would not even reveal his identity or credentials? And, finally, this system is perfect for stealing ideas. After you submit your manuscript you have no control of who will access it. All you can hope for is human decency, and it is not always there.
This brings us to the root of the problem. People, including scientists, are flawed. Few will miss a chance to stab competition in the back and abuse whatever little power they may have. I am not the first to criticize the peer review process. But I am not. Criticizing implies it can be fixed. It cannot. It was a bad idea all along. Then, what can be done? There is no quick and easy solution.
But I know where to start – ban peer review. And I know this can be done, this nonsense can be dealt with. This is not brain surgery, this is all about leveling the playing field, making rules for fair and open competition. These problems have been solved in all other spheres. Only scientists for whatever bizarre reasons received a special treatment and the right to live in lawlessness. Which is so wrong, I cannot find words to describe. Science is one most important sphere of human activity.
Who will find cure from cancer? Who will prevent the planet from becoming uninhabitable? Scientists. Not those from the beaches of Grand Wailea. Real ones. I hope we can still find some and reverse, before it’s too late, the depletion of brains. Let’s get started. Ban peer review!
@ur momisugly DirkH says:
June 25, 2013 at 5:32 am
“Human interactions depend on trust. Trust becomes impossible in a society under the control of a secret police.”
———————————————————————————————————————–
Trust requires belief. Belief requires confidence that what is seen is true. When presented by another person, confidence in what is true depends on confidence that the person presenting it firstly knows what is true and secondly will present it honestly.
In a situation, a group, a sect, a culture, a world, that does not allow others to be in a position to evaluate the capacity for knowledge and the commitment to honestly, there can be no human trust. There can only be an imitation of it showing itself in acceptance buttressed by necessity, desire generated by powerlessness or need for identification, and hope.
Policing comes in many forms and its activity does not have to be secret in any one action, but can be in effect secret in that its claims to legitimate existence are not themselves properly seen or even conciously understood by the proponents. Where there is this effective secrecy there will always be an abuse and betrayal of trust.
Whatever opinions might be proffered about policing of anything else, science, more so than anything else, has as its first demand that truth or otherwise be seen and that secrecy is anathema. If there is to be any reference or judgement about the role of societal and cultural norms, and the requirement for honesty and openness in these, then science must – not in its activities, but in its expression of these values – exist as it should.
Science cannot and should not assume the role of dictating societal structures and expectations. A significant part of the very degradation that can be seen now comes from the attempted utilization of science for legitimizing things beyond its ambit, built easily on the hubris and ignorance of some of those practicing as scientists themselves.
But science, having a distinctive role in the application of these principles, with a discipline demanded of it that, to a degree not present in other areas, should exclude those people and propositions that fail in this, cannot exist in a more general degradation, and conversely by being what it should be, can form a major part, by way of a cultural and personal imperative, in how societies exist more generally.
And it is true that secret policing whatever form that takes, as a nominal scientific adjudication, or as a general suppression of human capacity for confidence in the ability to make judgements and therefore trust is incompatible with human potential and therefore the ability to exist.
The result has always been and always will be decay, and not least for those who support and participate in this. Any group of people, or society, subject to this has a future of increasing barbarism. It is capitulation to the primaeval.
“johnmarshall” above has it right. Check out the Prinicipia Scientific International’s peer plan with a set window during which anybody can give in put, after which input is considered and revisions made. There might even be a second round of review. They also think normal peer review is more like pal review and gives the reviewer too much power to do unethical things.
Jim Rose says:
June 25, 2013 at 6:40 am
“There are “hot” topics in physics where several groups are publicly known to be pursuing the answer to the same problem. The solution often results in publication in one of the top ranked journals. The possible conflict of interest for the reviewer is obvious. What to do? Well, when I submitted a manuscript in that circumstance, I would include a very short list of competing groups and ask that the reviewer not come from one of these. As fas as I could tell the editors (e.g. PRL) generally met my request.’
Good point.
Ditto.
Day to day we live in and use items that are old inventions, improved. Cell phone are an improvement on Bell and Marconi’ s work. Modern airplanes are improvements of planes from the 20’s. It is a much bigger step from no aircraft to one, than it is to improve the first one. Cars, trains, internal combustion engines, cameras, computers photovoltaics, wind turbines etc. are all inventions that have been improved drastically. But improving is not inventing, the days of the single bright person putting his or her name on something is gone, too afraid of failure and lawsuits. I don’t think we could build new railways or highway systems anymore, the transnational railways and Interstates were built in a decade, now approval for a single bridge may take that long. Most of the roads in my area are 150 or more years old, they have been improved but not many new ones built. People like, Benz, Diesel, Bell, Faraday, Volta, Tesla, Edison, Curry, Einstein, Atanasoff and Ford all provided a new idea all together not improvements ( except Ford). Texting may be a more mobile form of the telegraph and Morse Code, but it is just an improvement. In decades and centuries to come what indivuals will be remembered? Maybe Gates and Jobs?
richard verney says:
June 25, 2013 at 2:14 am
I would say that it is not only science that appears to have stalled in the mid 20th century.
Is there really any truly great art post the art deco period?….
Has the world become too cynical, are there no true romantics? I do not know the answer, but there appears to be a greater number of people of average talent and ability, and far fewer truly exceptional geniuses amongst us.
>>>>>>>>>>>>>>>>>>>>>
E.M. Smith covered just this question at isms, ocracies and ologies In short the entire philosophy has changed since The Enlightenment gave us the Scientific Method as a way of thinking and the US Constitution.
What we have now is a swing back to using ‘Feelings’ instead of reason which allows ‘Practical Politics’ to use propaganda to control people. SEE: Dumbing Down America by Dr. Samuel Blumenfeld for how John Dewey’s Progressive school system was used to create unthinking individuals. Robin goes into greater detail at Invisible Serfs Collar
“The whole aim of practical politics is to keep the populace alarmed (and hence clamorous to be led to safety) by menacing it with an endless series of hobgoblins, all of them imaginary.” ~ H. L. Mencken
Those in control feel threatened by genius and normally will do anything to control and kill it. So now Gifted children and adults are at high risk for being identified as ADD. and being ‘Medicated’ at ages as young as 6 years via government schools.
SIDE EFFECTS
EM Smith goes on to discuss the “Counter-Enlightenment” This is what we are currently seeing effects of in our current crop of ‘Scientists’ using Post-Modern Science instead of the scientific method.
To put it bluntly Abzats has identified a symptom of a much wider problem.
***
MattN says:
June 25, 2013 at 5:09 am
Every whacko theory will get published and no one will have any idea what’s legitimate or not.
***
Maybe you wouldn’t….
DirkH said: MattN says:
“June 25, 2013 at 5:09 am
“What exactly will banning peer review achieve? Every whacko theory will get published and no one will have any idea what’s legitimate or not. ”
Totally unlike today…
http://en.wikipedia.org/wiki/Strange_matter”
Did you read my whole post? 3rd sentence that you didn’t include says exactly: “Not that I’m saying the current system does a stellar job of that now.”
The major problem is obvious. The transparency of the peer-review system. It should be open and transparent, and all parties must know the names and associations that are making advancement. The community being what it is, the academic rivalry and jealousy is rife. And this is something which is not healthy for the overall scientific process.
I disagree with the author that peer review is always anonymous. I tell people who I am after I’ve reviewed their manuscript (the journal sometimes gives you the option of doing this). And as other commenters have said, not all scientific fields are alike. Focus on the problem “children”, as it were, and let the relatively well-behaved kids continue on uninterrupted.
The reason science went downhill is because the theory of evolution became popular. Why would anyone think they can find the laws of science when they believe everything originated from chaos and disorder? “God does not play dice with the universe”, said Einstein. “Don’t you tell God what to do”, replied Neils Bohr. The new generation of scientists regard the universe as a casino rather than an orderly functioning machine. Why waste your time finding the laws of science. Just plug your random data into a statistical equation and your science can be a 9 – 5 job.
The alternative to peer review is independent confirmation / replication of results. There is no reason why any hypotheses should be squelched from publication. The wheat is separated from the chaff by waiting to see if subsequently another independent scientist can confirm or replicate the results / predictions. This may take the form of running an experiment to confirm (or deny) a theoretical hypothesis, re-running a complex experiment to verify that the results are repeatable, or independently analyzing a given dataset to see if the conclusions drawn from such are robust. This is how the scientific method works. Peer review is worthless — its just opinion vs. opinion.
I’m not sure I buy totally into the idea that peer reviews have totally destroyed science. Peer review is important if one is on the faculty of a university, or belong to a public scientific organization. But, not all scientists work this way. Many work for private corporations and research arms of these organizations. Ground breaking research in some field can turn out to be valuable to these firms bottom lines. Additionally, not all nations use peer review in the same we we do. We should also remember that it took Einstein 15 years to get his initial theories on Relativity proven in the field. Yes, the Great War put a damper on things. But the War only accounted for 4 of those 15 years.
I think there are 2 other facets that also come into play. The first is that most of the ground breaking discoveries in physics and chemistry have already been discovered. Both fields since 1900 have become more and more specialized. Perhaps it is more difficult for a theoretical physicist to theorize, as the depth of these specialties have become so complex. Secondly, the “best and brightest” of our society no longer go into the hard sciences (at least not in the West). At least in the United States, the primary and secondary levels of education have become so poor that many intellectually gifted students fall through the gaps. One hundred and twenty years ago there existed a very large pool of under class students in New York, Boston, Chicago, and Philadelphia who came up through the public school systems and easily transitioned into our elite institutions of higher learning (see the the alumni of Dewitt Clinton High in Manhattan and you will see my point). But those days are long gone. Yes, there are brilliant scientists today. But, I believe their numbers are small in comparison to say 1920.
I am not convinced that the current peer review process is the problem. In many areas of science and particularly in the new field of climate science governments all around the world have thrown huge wads of cash towards climate research. With huge sums of money and research geared towards finding problems affecting the climate then by god problems can and will be found. It can’t really be described as corruption in a narrow sense, because researchers are merely following the direction they have been given by government.
The biggest reform you could make to the present system without completely abandoning it is to make the reviewers’ identities known to the authors. Now, having said that, I consider the peer review process to be nearly worthless in every regard and I don’t think the quality of science would go down if it were abandoned completely.
Henry Bauer’s name was mentioned briefly above, but I think that his arguments deserve a wider audience, especially amongst the commentators here who are assuming that climate ‘science’ is a unique case of rogue science, and everything else is tickety-boo.
A few people added medical science to the list of rogue sciences, not enough I’m afraid.
Bauer’s argument is that the entire social set-up of science, too many people chasing too few grants and not enough prestige to go round, has thoroughly corrupted the whole scientific process from top to bottom in every field. Peer review is just one subset of what is going wrong, the dirty tricks that researchers use to gain and keep status apply to every field in every possible way. He cites examples from medicine, geology, cosmology, physics, and I would happily add psychology to the list.
If only there were some large medium for information to be shared with others all over the world by simply uploading the information and transferring the data over cables and digitally through the air. Then peer-review could be unanonymous and be made broader by being reviewed by the entire research community in that field. It could be called the interweb or world wide net.
Thanks, Abzats. Very interesting article.
Science publishing should be open, let the whole community be the judges, in a free-market style.
Anonymous, unpaid reviewers take too long and can easily be corrupted.
Transparency is the key. Free, public publishing in the Web is a likely way out of this problem that is at its worst in climate science.
Complete and utter nonsense.
This starts from a false premise, namely that science has stagnated. In reality there are whole fields of study which simply did not exist pre 1945 – all of molecular biology for example, anything to do with lasers, all computational based science. Everything from DNA to Black Holes. Nearly all particle physics. All modern electronics. Nearly all polymer chemistry. Quantum electrodynamics. All of astronomy other than that with visible light (which is most of it). Plate tectonics.
In nearly all fields the peer review process works fine. Just because problems exist in one or two areas, climate science and some medical sciences, there is no need to throw out the baby with the bath water.
This is just another ‘golden age’ fallacy.
There is one enforcement of reality, of true science: when results must be right due to direct ties to applied science, to engineering working in the real world.
But that applies to some fields far less than others, like in climatology having a model with repeated wrong predictions is no problem to those involved.
“The second red flag is that none of them gets paid. Those who believe in Santa Claus will say, well, they are just nice people volunteering their time to help advance science. Those who work for a living will smell a rat. I can give you one reason: being a reviewer gives you power over other people.”
Doing work without pay is also an aspect of the highest-postcount members on various web forums and the dominant deletionists on wikipedia controversial articles. Many of the former, on anything from politics to computer games, tend to be on average the most ideologically motivated, groupthink-enforcing, ego-driven in the worst way, and dishonest subset of the population I’ve ever seen, utterly into bashing the writing of those they dislike for reasons which have jack to do with pursuit of objective truth…
If anything, the only surprise is that science hasn’t been harmed worse by the present peer review system’s design, which must be credited to the better nature of many scientists, though not everyone who calls himself or herself a scientist deserves the term.
“It was introduced a long time ago, but it took over the scientific community at about mid 20th century. Why is it important? Every scientist must publish his or her work. If you do not publish, you will not advance your career.”
Aside from other problems, it encourages the scientific equivalent of patent spam.
Anyway, the present system is indeed broken. Reviewers should be non-anonymous and paid, among other changes (such as those Zalotocky suggested).
I am an academic in a social science field. All our journal articles are double blind. It seems like such a simple fix. I naively thought all scientific fields used double blind reviewing.
I’m sorry but I disagree.
I do agree that peer review as it stands is shockingly weak as a standard of scientific excellence and has been corrupted beyond recognition in several disciplines.
It’s like democracy; the worst form of government imaginable, and yet better than anything else that’s been tried.
Overhaul is clearly in order; no publication without full disclosure of all data and methodology. ‘Proprietary data’ claims should result academic dismissal.
Also; no more unpaid and anonymous peer-review. Paid reviews published alongside the final work. You want to get rid of the stink; sunshine is the best disinfectant. The pay will assure that people are willing to do the work. There are always hungry newly minted PHd’s out there willing to actually do the groundwork necessary to ensure a competent review. But they aren’t too interested in doing it for free.
Finally; if you want to be a peer reviewed journal; fully disclosed board, including their involvement with academic and private institutions and no-pay wall. Period. Universities can and will pay the subscription price, regardless of on-line availability, and availability of the paper, its data, and methodology should be freely available to any who wish to check the math. Otherwise you are the great and powerful Oz, standing over there behind the curtain.
As far as the board members themselves; the first time they publicly advocate a political position or policy that relates to their field; they should be cut off from all government funding. If they are a professor at a public institution; gone. It is okay for a scientist to campaign, but not on the public dime. Go to a private institution if that is what you want to do. It isn’t a question of academic freedom, it’s about using government (tax) money to campaign. If you want to use your government paid position as a bully pulpit; you need to first get yourself elected to office.
That isn’t to say they shouldn’t be allowed to write a letter to their representative if they feel so inclined. But to testify in front of congress is an overtly political act. Congress doesn’t do science. To engage in the kind of rhetoric and outright obfuscation as some of these people have amounts to official misconduct. Government funded ‘proprietary’ research? The mere claim demands a criminal sentence of jail time.
Of course, given the recent levels of official misconduct currently being ignored or excused; my final observation is the real solution to all of this is to get the government out of ‘soft science’.
If the Army needs a new laser or the Navy wants a new propulsion system; go for it. If non-deterministic models of massive chaotic systems who’s inputs are not well understood are the topic; forget it. The government has no place in paying people engage in either tasseography or haruspicy. Such is the state of modern climate ‘science’.
We need to learn to distinguish between real science and the use of complex statistical models to forecast the behavior of massive chaotic systems whose inputs are complex to the point that even they cannot be reliably modeled. That is shamanism, not science. (Picture Mike Mann in a full zulu-mask dancing around a bunch of bristle-cone pine cores while grad students holding fluorescent light bulbs recite an al-gorian chant.) If you think about it; the reading of entrails and tea leaves to predict the future is a really good analogy for the ‘science’ of GCM’s.
Anyway; back to my original point; you have to name some better form of review, you cannot simply suggest we do away with the current method of scientific refinement without suggesting an alternative means. Otherwise you have cast us back into the dark ages with no mechanism by which we can agree to improve our scientific understanding.
Interesting point of view. Indeed, if you read through Chemical Abstracts, there are far fewer, but more interesting, and better written entries before, say, 1960, than afterwards. About 90% of what appears in present day research is useless nuance with more useless nuance as references. Not only that, the articles and abstracts are written in language that is arcane, and yet often imprecise.
I couldn’t blame it all on the peer review process, but there’s something to think about.
In Re; Austin6/25, 6:22 am;
Austin, you’ve hit the nail on the head. I got my bachelors in business, with an ’emphasis’ on finance. Yes, it is a similar problem. What the good professor most likely realized is that, once you’ve done massive regressions on whole industries, all you’ve really accomplished is forecast one of your major inputs; economic growth and development. It is circular reasoning. If you knew the answer (how much economic growth will occur in the next quarter, ceterus peribus), you could probably have a model predict fairly accurately how any particular industry would do over that period of time. Less so on a particular stock, but still; once you agregate all the stocks in an industry, and correlate all the industries; you have modeled the economy. Economic growth is simply the aggregation of how each economic productivity unit performed.
Except for one problem; your model assumes a whole bunch of relevant inputs that are simply not known; the economic implications of elections and legislation, the impact of regional weather on agricultural output and construction productivity. Demographic patterns, fed monetary policy and paradigm shifts in technology all play into it. It is simply not possible to model these things with any degree of accuracy.
So you are left with the circular reasoning of the CAPM model and the weak form efficient market hypothesis. No matter how large your data set, there is simply no place in CAPM for Fannie May’s ‘liar’ loans, Obamacare, google glasse, or hurricane Sandy.
@ur momisugly Eric Worrall
June 25, 2013 at 1:06 am (and others)
I am a scientist working in the field of cardiovascular genetics. Even here we often have to deal with biased, unfair, and superficial reviews. I had papers rejected only becasue somebody decided that they had a “low priority” without any serious methidological issue. Surely blind peer review is not the perfect way to select good science out from the bad one. Perhaps one should think to develop a kind of ranking allowing to select the most correct and professional reviewer.
However there is another issue I think we have to consider: the pressure for publication that scientists have to face every day, Publications are needed for career, publications are needed to get money to found other research and for young fellows to grorw. The research costs are continuously increasing while the funds are shrinking. So, there is a more substantial issue than the peer review. It is the entire way of how science production is organized that needs to be reassessed and redesigned
I bet that if we remove the peer review process the amnount of junk science oublished will further increase. Duble blind review process is not a solution as well since it is very easy to spot papers from competitors once you know them.
I cannot agree with the author’s statement that “these problems have been solved in all other spheres.”
I’ve been earning my living as a professional translator for more than 30 years. In translation business, some agencies (middlemen who find translators for corporate clients) are using anonymous “reviewers” to evaluate translations. The problem here is the same as in the scientific peer review process. American translation agency owners, as well as their employees that coordinate translations, usually don’t know any language but English. “Reviewers” are often competitors who cannot find work, because their translations are of lower quality that those they review. You can imagine the consequences.
Personal vendettas are also abound. I’ve been working with a certain editor for more than 20 years. He is an intelligent person but cannot write very well. For this reason, he has been taking editing work instead of translation work. (Editing pays much less than translation but gives one a power to tell the client if this or that translator is good or bad. Strange, illogical, counterproductive situation, but that’s how it is.) In the past, I never had bad reviews from this editor. As years went by, we have become closely acquainted. Recently, while talking to him on the phone, we had a sharp disagreement about a personal matter that had nothing to do with our translation work. Et voila! I suddenly stopped getting work from an important client who uses him as a translation reviewer, despite the fact that this client and this reviewer were happy with my translations for many years.
I think this situation is very similar to the evil peer review situation in science. I am sure it exists in other professional fields as well.