Cut-throat academia leads to ‘natural selection of bad science’, claims study
Ralph Westfal submits this story Via the Guardian
Scientists incentivised to publish surprising results frequently in major journals, despite risk that such findings are likely to be wrong, suggests research.
Getting stuff right is normally regarded as science’s central aim. But a new analysis has raised the existential spectre that universities, laboratory chiefs and academic journals are contributing to the “natural selection of bad science”.
To thrive in the cut-throat world of academia, scientists are incentivised to publish surprising findings frequently, the study suggests – despite the risk that such findings are “most likely to be wrong”.
Paul Smaldino, a cognitive scientist who led the work at the University of California, Merced, said: “As long as the incentives are in place that reward publishing novel, surprising results, often and in high-visibility journals above other, more nuanced aspects of science, shoddy practices that maximise one’s ability to do so will run rampant.”
The paper comes as psychologists and biomedical scientists are grappling with an apparent replication crisis, in which many high profile results have been shown to be unreliable. Observations that striking a power pose will make you feel bolder, smiling makes you feel happy or that placing a pair of “big brother” eyes on the wall will protect against theft have all failed to stand up to replication.
Sociology, economics, climate science and ecology are other areas likley to be vulnerable to the propagation of bad practice, according to Smaldino.
…
Smaldino cites an experiment by the American psychologist Daryl Bem, who purported to show that undergraduates could predict the future and published the result in a prestigious journal.
“What he found was the equivalent of flipping a bunch of pennies, nickels, and quarters, asking students to guess heads or tails each time, and then reporting that psychic abilities exist for pennies, but not nickels and quarters, because the students were right 53% of the time for the pennies, rather than the expected 50%. It’s insane,” said Smaldino. “Bem used exactly the same standards of evidence that all social psychologists were using to evaluate their findings. And if those standards allowed this ridiculous a hypothesis to make the cut, imagine what else was getting through.”
Yes, imagine. Full story at the Guardian
In a paper published earlier this year, Smaldino sums up the problem:
Scientists often learn more from studies that fail. But failed studies can mean career death. So instead, they’re incentivized to generate positive results they can publish. And the phrase “publish or perish” hangs over nearly every decision. It’s a nagging whisper, like a Jedi’s path to the dark side.
“Over time the most successful people will be those who can best exploit the system,” Paul Smaldino, a cognitive science professor at University of California Merced, says. To Smaldino, the selection pressures in science have favored less-thanideal research: “As long as things like publication quantity, and publishing flashy results in fancy journals are incentivized, and people who can do that are rewarded … they’ll be successful, and pass on their successful methods to others.”
Many scientists have had enough. They want to break this cycle of perverse incentives and rewards.
This is going on my list!
daveburton…..the link (my list) = Oops! That page can’t be found.
Um. Try this one (had to adjust the modification that WordPress put on Dave’s original)
http://www.sealevel.info/papers.html#whitherscience
I hope WP doesn’t scrounge this one.
Thank you, kokoda and OldUnixHead. I botched the link, it wasn’t WP’s fault. Sorry!
[Reply: Fixed it for you. Was missing the http:// part. -ModE ]
Thank you, how refreshing to see this publication. Real science exposing pseudo-science AND openly discussing the perverse society that academia has become.
Don’t get too excited. It’s the Guardian! Worst eco- Commie rag on the planet outside of Russia or China! Notice they totally fail to make the link to the garbage that gets published as climate science. Every one of those bogus studies with outrageous claims gets prominent play in the Guardian. No wonder their circulation and finances are on the rocks!
“””””….. Paul Smaldino, a cognitive scientist who led the work at the University of California, …..”””””
So are they saying that cognition is a science, or are they just saying this guy is “cognitive” as in “Awake ” ??
g
But but but climate scientists are dispassionate, incorruptible gods of truth!
At least that’s the impression I get from all the millennials who shriek ‘CO2!’ at me.
And, of course, all their studies/papers are capable of replication. /s
Climate scientists have “devoted their lives” to it so we should believe them.
http://www.slate.com/blogs/bad_astronomy/2016/09/20/august_2016_was_the_hottest_august_on_record.html
No need for sarcasm, in climate related sciences like paleoconstructions even methodologically flawed papers are reproduced with independent methods. Any critics are automatically considered to be crackpots or misinformers paid by Big Oil. Sea level rise, ocean alkalinity neutralization, same thing.
Like the MBH hockey stick is nicely reproduced by Marcott & al. The science is settled. Connolley spells it on Wikipedia quoting the authorities.
‘We’ believe in Hansen since we believe in authority. There is no need for real replication, and there is no room for doubt since it dilutes the message. Remember the fine balance between honest and effective!
Perhaps Prof Brian Cox should take this onboard before he spouts off next time about people not trusting what ‘the science’ tells them.
Him, and a couple dozen trolls.
All wrong !
Human nature is the problem, ( the bigger the lie, the better ) just look at the worlds politicians and you find that the idiotic humans are all, repeat all, on the band wagon !
Was it not one A. Hitler who said, “The mass of the people are far more likely to believe a really big lie than a small one!”?
And what lie can be bigger than that the source of carbon to carbon based life is inimical to that life .
Science establishes a set of objective rules that are a check on our animal instincts. Human nature corrupts academia.
“Cut-throat academia leads to ‘natural selection of bad science’, claims study”
Interesting idea. In economics, Gresham’s Law states that bad money drives out good money, presumably because, if somebody was willing to accept paper money, people woild hold onto their gold an silver.
However, in science we have President Eisenhower stating the problem better.
“In this revolution, research has become central; it also becomes more formalized, complex, and costly. A steadily increasing share is conducted for, by, or at the direction of, the Federal government.
Today, the solitary inventor, tinkering in his shop, has been overshadowed by task forces of scientists in laboratories and testing fields. In the same fashion, the free university, historically the fountainhead of free ideas and scientific discovery, has experienced a revolution in the conduct of research. Partly because of the huge costs involved, a government contract becomes virtually a substitute for intellectual curiosity. For every old blackboard there are now hundreds of new electronic computers.
The prospect of domination of the nation’s scholars by Federal employment, project allocations, and the power of money is ever present
and is gravely to be regarded.
Yet, in holding scientific research and discovery in respect, as we should, we must also be alert to the equal and opposite danger that public policy could itself become the captive of a scientifictechnological elite.
It is the task of statesmanship to mold, to balance, and to integrate these and other forces, new and old, within the principles of our democratic system — ever aiming toward the supreme goals of our free society.”
http://coursesa.matrix.msu.edu/~hst306/documents/indust.html
The Spanish have a phrase for this: el sucko baddo.
Given the spate of irreproducible results in pharmacological research, the documented ability to get nonsense papers published in “juried” journals, and total inability of climate models to predict “dique” (as they say in French), you would think the good guys and gals in science would be warming up their torches and pitchforks and demanding an end to this nonsense.
A thought: most of the denizens here have heard of CAD (computer aided design) and CAM (computer aided manufacturing). In that spirit I would like to suggest a new category for the current state-of-the-art in climate modeling: CAG (computer aided guessing)
I suggest a different term: CAPR (computer aided predetermined results).
Rather: Computer Results Are Preprogrammed.
RACookPE1978 September 21, 2016 at 5:47 am
I like it! An acronym that accurately predicts the output.
How about GAG? Guessing at Garbage!
That is a large part of the problem, presenting a negative result as a “failure”. Establishing a negative result is SUCESSFUL result.
The 53% is a failure despite being a positive result.
There are a couple of good Edison quotes:
“I can never find the things that work best until I know the things that don’t work.”
“I have not failed. I’ve just found 10,000 ways that won’t work.”
I suggest that 53% is, at the most generous, neutral as it within the normal margin of error.
“””””….. Scientists often learn more from studies that fail. …..”””””
I see a typo crept in there.
“””””….. Scientists often earn more from studies that fail. …..”””””
There; fixed that !!
g
People, scientists included, are motivated by incentives.
According to a Dec 13, 2010 New Yorker article, once there is a predominant opinion in a field of science, the published literature, the funding, the studies conducted, the individual decision to publish or not and the likelihood of getting published all become skewed in favor of it.
The name of the article is “The Truth Wears Off – Is there something wrong with the scientific method?”
http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off
archived here: http://www.webcitation.org/6SgCvSc3w
I won’t bother to go into the details because it may sway some from actually reading the article which makes a far stronger case than I could here, with numerous examples. It never mentions climate science, but it is obvious that the same problems exposed in it apply to climate science as well.
Here are the first two paragraphs:
“On September 18, 2007, a few dozen neuroscientists, psychiatrists, and drug-company executives gathered in a hotel conference room in Brussels to hear some startling news. It had to do with a class of drugs known as atypical or second-generation antipsychotics, which came on the market in the early nineties. The drugs, sold under brand names such as Abilify, Seroquel, and Zyprexa, had been tested on schizophrenics in several large clinical trials, all of which had demonstrated a dramatic decrease in the subjects’ psychiatric symptoms. As a result, second-generation antipsychotics had become one of the fastest-growing and most profitable pharmaceutical classes. By 2001, Eli Lilly’s Zyprexa was generating more revenue than Prozac. It remains the company’s top-selling drug.
But the data presented at the Brussels meeting made it clear that something strange was happening: the therapeutic power of the drugs appeared to be steadily waning. A recent study showed an effect that was less than half of that documented in the first trials, in the early nineteen-nineties. Many researchers began to argue that the expensive pharmaceuticals weren’t any better than first-generation antipsychotics, which have been in use since the fifties. “In fact, sometimes they now look even worse,” John Davis, a professor of psychiatry at the University of Illinois at Chicago, told me”
“Sociology, economics, climate science and ecology are other areas likely to be vulnerable to the propagation of bad practice”. None of the subjects is a science. The title should be changed.
Economics is – it is the dismal science.
“Any science with an adjective in front of it isn’t Science.
“History is the most ‘dismal science.'”
https://www.theguardian.com/science/head-quarters/2015/jun/25/the-first-imperative-science-that-isnt-transparent-isnt-science
The first imperative: Science that isn’t transparent isn’t science
Eisenhower warned about this. Obama and Clinton exploited it.
What is really being said here is that now science is captive to the trivialities of the world of marketing as if it were just another consumer product for the supermarket shelves, more content for the media to puff out with advertising and more huff and puff from politicians and activists pushing their own barrow. We have moved into a floating world of hyperchoice and the marketing industry is pushing the boundaries further and further to grab our attention.
Actual intellectual progress taking time to confirm, more time to move to a safe product just does ‘cut it’ anymore, it has to be a marketing bliss bomb to get attention these days, it has to jump off the tiny screen of a ‘smart’ phone into the tiny mind of the e-consumer who then struts around as if they know everything that’s trending and therefore that matters.
The reality is that it is all just a big empty barrel rolling down at hill to a ditch at the bottom.
‘Publish or perish’ also leads to ‘Pal review’ in smaller fields (perhaps in the larger ones as well), like AGW as scientists ‘scratch each others backs’ to get published.
Let me tell you how it works Mr. Smaldino.
As Saul Alinsky asserted, keep em guessing, keep em worried, rules for radicals states:
“Whenever possible, go outside the expertise of the enemy.” Look for ways to increase insecurity, anxiety and uncertainty.”
The politicians and Internationalist ‘liberal’ cultural Marxist college and the establishment thought that their hegemony was threatened and powerful, assertive nations states in the west were leaving the undeveloped world behind and nigh a man of these ‘limousine liberals’ deemed that, the west must be made to atone for the sins of their fathers. Sins? our forebears who actually did quite well for us – yer know the industrial revolution and cheap power for hospitals, schools, and homes – all that rot.
Post WWII, things were going along quite nicely….too nicely.
Until, the chumps [club] of Rome and the Malthusians like Paul Ehrlich commenced the doom and the related litany of immiseration prophesies.
The proles were too uppity deemed our masters and a new way to cow, to browbeat the people was needed, after dabbling with cooling [it is]. During the Eighties, the politicians alighted on a new ‘threat’ and man made global warming was supposed/hypothesized to be the great new
chimeraevil and by God – it was all our [you, yes YOU!]……fault!It plays well to the ingrained western psyche – the original sin and “it’s all going to well!”…..the west’s unnatural urge to apologize for being successful, hard though that is to fathom.
The politicians had their myth to beat a gullible but sadly ignorant public up with, the investment banks lapped it up, the corporate elites loved it and Goldman Sucks made a killing! what was not to like about globull warming?!!
Funds were pumped into climate change labs [and teaching kids about Keynes-ian economics and the printing press is our saviour!] and all sorts of other cow droppings.
Cr*p hypotheses demand cr*p papers……………………
After 50 years of education dumbing down, these days no one knows any better and crikey publish that paper BUT no need/ don’t publish your data – it’s so easy ain’t it – Mike?
They call it “progressive thinking”, it’s anything but.
Well, why should they give you their data when all you want to do is prove them wrong? 🙂
Xist, yes, that is one thing, but knowing to be wrong and not just being ignorant and incompetent is another thing.
Hmm, a good point Bill;-)
Though, what ever happened to publicly sharing your results and wanting to be proven correct [incorrect]. To being robust enough to be confident in your own abilities, methodology and your evident expertise, not least in recording your data sets and its consequent computer statistical analysis?
Or, you might just be a snake oil pedlar and or, a spiv selling poorly crafted goods, a charlatan if you like.
Over to you, Mike.
I believe the thrust of this article is true. Especially with the more prominent journals in various fields. My daughter is an Infectious disease Doctor currently engaged in AIDS research (specifically in seeking a cure for infants born with it). She has been published, but, often her work, which is incremental in nature, is rejected because, while accurate, ‘isn’t significant enough’ to publish in Journal ‘A’. They suggest she submit it to another, less prominent journal.
You make a valid point, though imho – we must make a distinction between pure, medical science and climatology quackers.
Indeed, all valuable medical research effected by qualified, competent and diligent medical researches be it investigation of Aids, and indeed all pathologies physicians, should, must endeavour to have their work published. Because, from small intensive studies, seeds grow into great advances, collaboration is as vital as it also, shines a light on your daughters’ dedicated expertise.
The original Guardian article refers to small sample sizes. The classic is a study that purported to find health benefits of eating chocolate.
It was a real study. It gathered real data. The problem was a small sample size and a large number of parameters. It is likely that if you examine enough parameters you will find some apparent correlations. It’s like flipping coins. Eventually you will flip five heads in a row. It doesn’t mean anything.
The trouble is that journals will publish results that are not statistically robust, and journalists will gleefully spread them to the public. link
Thinking about Dixon and Jones working with Lew paper… shudder.
I noticed this was an article in the Guardian. Meanwhile, they also publish this: https://www.theguardian.com/environment/2016/sep/20/hillary-clinton-dropped-climate-change-from-speeches-after-bernie-sanders-endorsement.
Either the left hand doesn’t know what the right hand is doing or the Guardian is so blind they don’t see the irony is decrying the publish-or-perish system while endorsing something that is still alive because of publish-or-perish is allowed.
The Guardian is not real big on noticing irony. Heck, the name “Guardian” itself is pure irony to start with – it is a mouthpiece for the barbarians at the gates of civilisation. Those cretins think that Pontius Pilate was the true messiah and that hand wringing and washing will get them to green left heaven.
Ironic that this surfaces in the Guardian. They should know. They publish enough articles about climate “Science” that is contrary to common sense, let alone reproducible, and often based on models without feeding in a single piece of real-world data..In that fantasy world, model results are considered to be “data”, and are used as the basis for further modelling.
The Guardian has the extraordinary talent of backing the wrong horse every single time.
Climate “science” gets a free ride, because saving the planet.
Publishing results based on skimpy or tortured evidence is bad enough, but the crucial flaw in the system is peer-review gate-keeping that gives bad research cover by discouraging challenges from other researchers. When small groups dominate specialized area of inquiry and when science-by-press-release overwhelms careful sifting of findings, the errors don’t get corrected quickly or appropriately.
Yes with most of the money going to institutions, peer conspiracy and networking becomes the way to game the system. So you get “pal review” with trickle down effects for the pals.
Bingo! This is a keeper.
The world is a little brighter today with the problem identified. Thanks.
Is it a double negative in the case of climate sociology?
Bad science has gone much further down the road with this administration’s reward system to meet advocacy targets. It’s comparable to courtroom attorneys paying extra to biased expert witnesses for their performance, and at the expense of the truth.
Even if, like VW, you have to fudge the data to meet some useless EPA regulation.
“The paper comes as psychologists and biomedical scientists are grappling with an apparent replication crisis”. Psychologists eh? I’m thinking John Cook and his mentor, Lew, would have more than just a little trouble having their work replicated.
Dr. Dave wrote in part:
Give me enough money and I’ll replicate their work. (Didn’t say my work would validate their findings; just replicate them. Also didn’t claim my work would be valid.)
I posted this in Tips and Notes, but I’ll repeat it here.
Here’s another story about fashionable science.
http://www.theregister.co.uk/2016/07/07/the_great_brain_scan_scandal_it_isnt_just_boffins_who_should_be_ashamed/
“Publish or perish” is simply another flawed job performance metric.
The recent scandal involving Wells Fargo Bank, the firing of 5300 employees (!) over the creation of 2 million phony accounts using stolen customer identities, stealing over $200 million from their customers, and the assessing of $185 million in fines (prison terms yet to be determined) was all the result of a flawed job performance metric, left in place over several years.
Employees were rewarded based on the number of new accounts they established.
What science lacks presently is an oversight mechanism that can identify fraud and effectively punish misbehavior.
Well, Wells fargo might feel they “bought” immunity from the current administration, and from the next administration as well. Many “small donations” by credit card via Wells Fargo to the Hillary campaign were double and triple billed (more money was sent to the campaign than was promised by the donor’s phone call), and – unless the donor complained – the extra money was kept by the campaign to justify “matching donations” from the federal government! Now, many thousands DID complain, so many that the corrections were made automatic. IF a call was made. Probably fewer than 4 in 50 noticed the multiple billings, got mad about it, and actually followed through to the Wells Fargo collection agencies and call centers in protest.
Where on earth did you get such a fabricated story?
Here:
http://observer.com/2016/09/exclusive-hillary-clinton-campaign-systematically-overcharging-poorest-donors/
In 2008 and 2012, the Obama campaign turned off address checking for credit card donations to their web site. This allowed people with foreign cards to donate to his campaign.
They also allowed donations from gift cards, which would allow individuals to donate as much as they wanted, under as many names as they wanted, without any ability on the part of the campaign to determine that they were doing it. Great way to get around donation limits.
I wonder I Hillary is using any of these lessons learned.
Thank you Jeff F, I’ll know never to go to that site for credible info
Richard – just because it is not what you wanted to hear doesn’t mean it is not credible. The Observer link provided included documentary evidence, ie a bank statement. You on the other hand have provided no countering evidence.
I’m not sure it matters or if it’s even true but from the link:
“Disclosure: Donald Trump is the father-in-law of Jared Kushner, the publisher of Observer Media.”
Mike,
A conflict of interest it may be but the evidence still stands. Unless, you think the father in law is not telling the full story?
Why the surprise? Since the turn of the century tens of thousands of Phd’s have been produced per year like sausages from a factory. A doctorate today is a debased currency. But I will tell you about two that in spite if their doctorate were not debased.
The first one had a doctorate in radio astronomy. When I asked what he was doing in our group writing code for a process control computer? He said “there are much brighter people in the field than I so to make a mark is very difficult”
The second was working with me checking out a control panel of instruments. He had a doctorate in chemical engineering. I asked the same question, what was he doing in the instrument department? He said ” I like to get my hand dirty doing practical things.
Unfortunately there are not many like them among the sausages.