From the National Association of Scholars via an article in the Wall Street Journal.
Policy makers often cite research to justify their rules, but many of those studies wouldn’t replicate
Half the results published in peer-reviewed scientific journals are probably wrong. John Ioannidis, now a professor of medicine at Stanford, made headlines with that claim in 2005. Since then, researchers have confirmed his skepticism by trying—and often failing—to reproduce many influential journal articles. Slowly, scientists are internalizing the lessons of this irreproducibility crisis. But what about government, which has been making policy for generations without confirming that the science behind it is valid?
The biggest newsmakers in the crisis have involved psychology. Consider three findings: Striking a “power pose” can improve a person’s hormone balance and increase tolerance for risk. Invoking a negative stereotype, such as by telling black test-takers that an exam measures intelligence, can measurably degrade performance. Playing a sorting game that involves quickly pairing faces (black or white) with bad and good words (“happy” or “death”) can reveal “implicit bias” and predict discrimination.
All three of these results received massive media attention, but independent researchers haven’t been able to reproduce any of them properly. It seems as if there’s no end of “scientific truths” that just aren’t so. For a 2015 article in Science, independent researchers tried to replicate 100 prominent psychology studies and succeeded with only 39% of them.
Further from the spotlight is a lot of equally flawed research that is often more consequential. In 2012 the biotechnology firm Amgen tried to reproduce 53 “landmark” studies in hematology and oncology. The company could only replicate six. Are doctors basing serious decisions about medical treatment on the rest? Consider the financial costs, too. A 2015 study estimated that American researchers spend $28 billion a year on irreproducible preclinical research.
The chief cause of irreproducibility may be that scientists, whether wittingly or not, are fishing fake statistical significance out of noisy data. If a researcher looks long enough, he can turn any fluke correlation into a seemingly positive result. But other factors compound the problem: Scientists can make arbitrary decisions about research techniques, even changing procedures partway through an experiment. They are susceptible to groupthink and aren’t as skeptical of results that fit their biases. Negative results typically go into the file drawer. Exciting new findings are a route to tenure and fame, and there’s little reward for replication studies.
American science has begun to face up to these problems. The National Institutes of Health has strengthened its reproducibility standards. Scientific journals have reduced the incentives and opportunities to publish bad research. Private philanthropies have put serious money behind groups like the Meta-Research Innovation Center at Stanford, led in part by Dr. Ioannidis, and the Center for Open Science in Charlottesville, Va.
There’s more to be done, and the National Association of Scholars has made some recommendations. Before conducting a study, scientists should “preregister” their research protocols by posting the intended methodology online, which eliminates opportunities for changing the rules in the middle of the experiment. High schools, colleges and graduate schools need to improve science education, particularly in statistics. Universities and journals should create incentives for researchers to publish negative results. Scientific associations should seek to disrupt disciplinary groupthink by putting their favored ideas up for review by experts in other sciences.
All government agencies should review the scientific justifications for their policies and regulations to ensure they meet strict reproducibility standards. The economics research that steers decisions at the Federal Reserve and the Treasury Department needs to be rechecked. The social psychology that informs education policy could be entirely irreproducible. The whole discipline of climate science is a farrago of unreliable statistics, arbitrary research techniques and politicized groupthink.
Mr. Wood is president of the National Association of Scholars. Mr. Randall is the NAS’s director of research and a co-author of its new report, “The Irreproducibility Crisis of Modern Science.”
A reproducibility crisis afflicts a wide range of scientific and social-scientific disciplines, from epidemiology to social psychology. Improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results together have produced a critical state of affairs. Many supposedly scientific results cannot be reproduced in subsequent investigations.
This study examines the different aspects of the reproducibility crisis of modern science. The report also includes a series of policy recommendations, scientific and political, for alleviating the reproducibility crisis.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Federally-funded research is guided by the distribution of research funds
1. Funds are limited to research studies that support standard models of Nature
Well, you get what you pay for. The gov’t likes to pay scientists to scare, as a result we’ve lots of scaredy cats. Emotion gets the job done where reason doesn’t.
Actual science is having a rather abused life, given gov’t johns and prostitute scientists are claiming the ‘street’ domain.
Thank you omanuel. You are correct. But if you want to see vile hatred, deliberate attacks, and chicanery of all types, just try to publish a paper straying too far from the standard models of Nature. And the standard models of Statistics.
Another thing hurting is people using statistics to deceive not enlighten. This is done out of ignorance or malice, but the results are the same. People will claim that the lower the p value the greater the significance of the results (even approaching the TRUTH). Far too many scientists have no idea of the relationships between Type 1 and Type 2 errors. They need to consult any good probability and statistics book. The best books are probably not Statistics in _____) but just probability and statistics.
To illustrate:
The Arctic Sea Ice oscillates between 3 Mkm^2 and 14 Mkm^2 every year. It has been decreasing recently through the satellite era, but we have not gone through even half of a single PDO or AMO cycle. We simply do not know what its usual cycle is – if indeed the Arctic sea ice has period changes in extent and thickness – nor the length of even one cycle, much less several cycles to make an estimate of the average length.
The Antarctic sea ice also oscillates between 3.0 Mkm^2 and 18 Mkm^2, but has been increasing steadily since 1992.
But the CAGW propagandists report “Artic sea ice has decreased 30%” using the MINIMUM sea ice values and a starting point at an all-time high period in 1979-1983!
Thus, they emphasize the MAXIMUM percent sea ice change: -1.5 Mkm^2 from 4.0 Mkm^2 is a much larger (More impressive!) decrease than a 1.5 Mkm^2 decrease from 14.0 Mkm^2 maximum, isn’t it?
But it is only a 7% decrease of sea ice maximum over the entire 38 year-long record.
But the Antarctic sea ice increase is trivialized by reporting ONLY the decadal rate of the Antarctic sea ice MAXIMUM!
In truth, the excess sea ice around Antarctica as recently as 2014 was larger than the entire area of Greenland. But the CAGW publicity-fundraising-crisis manufacturing machine does not want that fact known.
Yes, I’ve bookmarked retractionwatch.wordpress.com, but as I’ve noticed “climate science” seems to be getting a pass there.
Its a miracle that the (in many ways) least understood and arguably newest “science” gets it correct when some sciences that are centuries (or more) older are having so much trouble…
No one dares even mention CliSci in a negative way …. career destroying attacks quickly follow.
I do, every day. But yeah, my meteorology career is destroyed by the likes of Mann et al.
and we both (we all….) know that the list, with your name at the top, is long with good men and women whose careers have been intentional destroyed, in some cases by paid political operatives, for speaking out.
The basic problem is that apart from many researchers not understanding statistics (what the p score and CHI squared statistic really mean) is that the base tests should be run at the 5 sigma level instead of the 2 sigma level that a lot of sciences are using. I realize that the costs involved to run at 5 sigma are much greater but what do you want? Cheap flawed science or expensive good science. What has happened is that we went for the cheap science but that ultimately leads to 100% failure to reproduce results.
Like flying in the Space Shuttle.
A Machine with 2,500,000 moving parts built by the Lowest Bidder
Anthony,
I don’t think your career has been destroyed by Mann. You are now one of the most well known and respected meteorologist in modern history. There is not a meteorologist or “climate scientist” on the planet that doesn’t know your name or WUWT. But, I understand where you’re coming from.
Engineers, physicists and many other interested parties know your name and respect you for speaking out. Your efforts will long outlive you!
http://dels.nas.edu/resources/static-assets/exec-office-other/climate-change-full.pdf
An 80 year old troll that opines on both NSPE & AIChE forums suggests that HIS WORLD SCIENTISTS know more than ours. Does anyone ever reviewed this link. I don’t think I’ve ever seen this ever discussed on any of the websites listed on Anthony’s right hand column. You could respond directly to him @ur momisugly NYeoman@ur momisuglymsn.com
Hi Kip, I was just today thinking I hadn’t seen you around or a while.
What exactly do you mean by ‘mention CliSci in a negative way”?
“and we both (we all….) know that the list, with your name at the top, is long with good men and women whose careers have been intentional destroyed, in some cases by paid political operatives, for speaking out.”
Who are these paid political operatives? Paid by whom? Of course, “mainstream” scientists have been harrassed by the government, too.
I don’t actually know that list. Could you give a few examples, and how it happened? I’m interested. I wonder in particular how often it was a matter of disagreeing with the science, and how often it went beyond that. Thanks. I know the story of Dr. Ridd, that doesn’t seem to qualify. Judith Curry left, but was her career “intentionally destroyed”?
That’s quite an accusation. I don’t know Anthony’s history.
Kristi ==> The list is quite long — Curry, Crockford, soon, Lindzen, Watts, Ball, Pielke (Sr.) and Pielke (Jr). Pielke Jr was attacked at the behest of Podesta by Romm et al at ThinkProgress. Look for my further comment below
Kristi ==> I forgot to point you to the story of Lennart Bengtsson — see https://wattsupwiththat.com/2014/05/22/lennart-bengtsson-speaks-on-the-gwpf-and-furor-over-his-bullying-by-his-climate-science-peers/
@Kristi Silber
just an example. now do your home work and find others all by yourself. You see, you shouldn’t rely on information source you don’t actually trust, like people here. You should think by yourself, not parroting any side opinion, and that includes finding relevant information yourself. Most of the time, people won’t spread blatant falsehood, they will omit inconvenient truth, and rely on ignorant people to spread it further in good faith (like yourself: no offense, but you are ignorant, which is easily cured if you will…).
https://wattsupwiththat.com/2018/04/09/pielke-harasser-judd-legum-turns-tail-and-runs-from-debate-challenge/
paid operative: Judd Lugum
payer: Center for American Progress (CAP)
Why do you think Ridd doesn’t qualify? Would even Galileo qualify, as per your standard?
@Kristi Silber
There was also Emeritus Professor Bob Carter of James Cook University o
fNorth Queensland (my alma mater, much to my chagrin, after the way they treated him). Though I guess, as he was Emeritus Professor his career wasn’t destroyed. However, in an act that was truly petty, spiteful and vindictive, they revoked the few perks he had by way of his Emeritus status.
It gets a pass because none of the climate scientists are policing their discipline in the way other scientists police theirs. It is one of the key reasons I am sceptical.
There are people who police climate “science”. Just try to publish a paper that conflicts with the “consensus” and see how fast your career is destroyed.
Yes, when the so-called “scientists” are all nodding in approval of each other’s work, you know it isn’t *actual* science.
Retraction Watch monitors retractions.
A climate paper may be crap, but, unless it’s retracted, it won’t appear on that web site. And teh Climat gang would never allow a paper to be retracted. The Mann et al hockey stick is a case in point – it’s obviously wrong, no one uses it any more, but it won’t get retracted…
Pfffft. Science isn’t about reproducibility… It’s entirely about funding. Just ask any Climate scientist.
😀
Can we at least agree its about the reproducibility of funding?
NAILED IT!
I see your game… and you won’t get away with it. The reproducibility of climate science funding consensus means it cannot be about science… therefore it is all about the funding in climate science.
:p
No, wait… the reproducibility of climate science funding consensus means it cannot be about science or funding, so it’s all about the reproducibility of consensus. But the consensus of reproducibility of consensus means it cannot be about reproducibility or consensus, so it’s about nothing whatsoever… or something.
Now I’m confused.
Please. Not while I’m drinking beer 🙂
Government science is not about reproducibility. Science is related to a government science just as a jacket is related to a straitjacket.
the best analogy i have ever heard bravo!!!!!!!!!!!!!
government science is to science what military music is to music.
And funding is decided by politicians, or their appointees.
How many have even a decent scientific back-ground?
Say to age 18 – I am not asking Doctorates or Masters – but just knowing scientific method – the Feynman video covers that well in a few minutes!
Auto
How is climate “science” doing on that scorecard? Lol!
It’s pretty bad when all the peer reviewers and people who might attempt replication are your friends and use the same jiggered data and lousy methodology and can’t reproduce your results but I suspect that is the state of things.
I can’t see how any genuine researcher could replicate Mann’s work without a massive fiddle.
Do they make fiddles out of Yamal trees?
Maybe Yamala guitars?
They are to be played with hockey-stick shaped bows by people trained in fiddling at Penn State.
John harmsworth In Cargo Cult Science, Richard Feynman goes into detail on the great effort required to replicate mice negotiating a maze. But then how that detail is not being used in subsequent studies. http://calteches.library.caltech.edu/51/2/CargoCult.htm
Speaking of which, on a subject near and dear to me since my wife started to nag me having read about the original lancet article in the gullible sensationalist media:
https://www.acsh.org/news/2018/04/13/new-alcohol-study-mostly-hype-journal-authors-media-blame-12839
There’s too many examples of “science” that has been proven wrong….that is not retracted
From the last paragraph of the article: “All government agencies should review the scientific justifications for their policies and regulations…” Let’s start with the EPA’s Endangerment Finding.
The number of successful geoengineering projects have been very limited ( cloud seeding , building dams and levees ….etc) but still; that qualifies as a separate science as does meteorology. However climate science should not exist as a separate discipline since it takes knowledge of about 20 disciplines to even have a basic understanding. Moreover, there is nothing we can do about the climate and since the whole of climate science is now based on flawed computer simulations, universities should not allow any university resources to be used in any study that uses computer simulations and thus not allow funding for it in the 1st place.
And, those irreproducible studies have been cited in so many follow on studies which in turn have been cited many times, which in turn…
Just like with the climate models, we simply do not have enough computing power to unravel this confusion.
You are right, but the real dirty little secret is we will never have the computing power to unravel the confusion.
And if they won’t disclose their data it makes life even more problematic.
I’d be curious, as a non-scientist, to know how much of this publication of non-replicable studies, i.e., studies that draw incorrect conclusions, is due to an outlier effect. A study that produces an outcome that falls in the tail of the distribution of possible outcomes is surprising, and is therefore interesting, and is published.
I read of this phenomenon in the comments of WUWT years ago and it struck me as reasonable. If 50% of studies produce an incorrect result, couldn’t that be because it’s far easier to get an outlier result, i.e., an unexpected result, published? (And spread far and wide as “Breaking News!!”)
Rod ==> This is a very hot topic in medical research and psychology in particular. There are a lot of studies on the why’s and and in what ways the studies go astray. A little Googling will bring you enough to read for a month or so. Start with Ioannidis .
There are lots of reasons, including publication bias (publishing positive results but not negative) and the awful p hacking and correlation fishing that goes on.
I think it was one on of the cardiac journals that changed its policy a year or two ago and said it would only publish papers where what was being looked for was stated BEFORE the study took place. The number of studies with positive correlations fell from well over 60% to under 10%.
Like the “No Stairway” sign in the Wayne’s World guitar shop, they ought to put up a “No Texas Sharpshooter” sign in their submissions office.
Rod, to add to Phoenix’s comment, another idea that’s been floated (and practiced) is having researchers submit their rationale, justification and some background as well as methods, and if accepted, the journal would guarantee publication before the study began. This would eliminate the problem of negative results not getting published.
As Kip implied, it’s important to remember that the problems are worse in some fields than others. There are a ton of papers in medicine that have very small sample sizes. Control groups aren’t always possible. Medical doctors may not have a good grounding in experimental design and statistics.
And of course, the social sciences are infamously “soft.”
Proper use of statistics is way harder than it looks. Some research groups – climate modelers, for instance – include statisticians with the kind of expertise needed for complex studies. This is one reason I argue that expertise is important, and laymen seldom have the depth of knowledge and experience to practice meaningful, original science. The kind of knowledge one needs these days can be so esoteric, even those in related fields don’t have the same understanding.
That said, I don’t think the situation is quite as bad as it seems. I have read the paper about the “50% of studies are wrong.” It’s based on hypothesis and calculation, sometimes using arbitrary quantities to represent a parameter: 10% bias, for example. The paper makes many good points about problems in research, but few were news, and the title was “alarmist.” it got huge press, of course. Some questioned whether his paper were part of the false 50%.
Proper use of statistics is way harder than it looks.
SomeA few research groups – but no climate modelers (that we know of), for instance – include statisticians with the kind of expertise needed for complex studies.Rod, John Ioannidis paper “Why most published published research is false” is available at http://www.google.com/url?q=http://journals.plos.org/plosmedicine/article%3Fid%3D10.1371/journal.pmed.0020124&sa=U&ved=0ahUKEwiWx6GP3cHaAhVruVkKHWnmCUEQFggZMAA&usg=AOvVaw1wsApJxX7HgiWhiIJZ0Yhs It’s well worth reading, but it’s tough going.
Rather remarkably,It was well received and the medical folk at least seem to be trying to do better. Ioannidis wrote a ocuple of follow-up papers BTW.
One thing though. The paper primarily addresses experimental science. You run an experiment, analyze, and report the results. Most climate “research” doesn’t involve experimentation. It’s more like Geology and Astrophysics. It consists of analyzing (often dubious) existing data or of running elaborate unvalidated computer models. The problems that Ioannidis identified with experimentation — especially statistical naivete — often seem to show up in Climate Science, but not in the same way. And there appear to be a bunch of largely unacknowledged problems unique to observational science.
And Climate Scientists, unlike medical researchers, definitely do NOT like to be told that they might not know what they are doing.
I hope they got it published in the journal of irreproducible results.
Wait, are you suggesting they can not reproduce the irreproducible?
The existence of JIR suggests that scientists of previous decades knew that some published research was bad. Unable or unwilling to call it out in a serious way, they took to mockery. However, instead of causing a serious look at reproducibility, JIR made it easier to laugh off the misfeasance.
Of course we should share the link to the journal of irreproducible results: http://jir.com/
The biggest myth going around for the last 100 years is that drugs cause addiction. There is no evidence for that.
Addiction is a symptom of PTSD. Look it up.
Dr. Lonny Shavelson found that 70% of female heroin addicts were sexually abused in childhood.
…Dr. Lonny Shavelson found that 70% of female heroin addicts were sexually abused in childhood…..
Not a proper statistical argument on its own – this statement shows us in a nutshell what is wrong with using stats.
Supposing that normally, 80% of females reported being sexually abused in childhood? Which is certainly possible, in teh current climate. Then, 70% begins to look below average…
Another possibility is that those who were sexually abused in childhood were more likely to try drugs.
or even that children who were likely to try/overdo drugs came from an environment that was more likely to lead to abuse.
This is an interesting idea. To be sure, there is such a thing as chemical dependence, but it may be much easier to break without and underlying emotional trauma. It would also explain why no everyone who has taken an addictive substance becomes “hooked” or why the addictive effect varies wildly between individuals. Also why some people can become addicted to non-addictive substances and behaviors.
As part of a college class, I was required to attend an AA meeting. Nearly all the recovering alcoholics appeared to be heavy smokers and coffee drinkers – if the meeting was a reasonable example of their normal habits. I’ve often wondered if they were just trading one addiction for another. If solving their PTSD (or other psychological) issues would help end their addictions, this could help many people. I hope this approach is being studied.
There are definite differences in population sub-groups. Native Americans and the Irish have statistically greater rates of alcohol addiction, for instance. There is no commonality of socio-economic grouping, so it suggests, not proves, there is a genetic component that interacts with alcohol intake.
It is very hard to argue that tobacco does not cause addiction. I have seen it over a lifetime where a lot of my friends started off slowly smoking and soon had cravings that could not be stopped. Of course some people can stop cold turkey but for most it is a losing game. Also there are some modern drugs where after 1 dose you would sell your own mother for another fix.
Careful, you are treading very dangerous ground. After all, we can’t be pushing for drug legalisation since it is ‘not chemically addictive’ and still continue the narrative that the good’ole englishmen intentionally plied north american indians with alcohol to exploit their genetic pre-disposition, but this is getting off topic.
On the vane of good climate science and bad faith, maybe there is a study that shows certain races of people exhale less C02 than others and are therefore holier than others.
And who has reproduced and verified that claim?
We have dozens of automatic homeostatic processes that attempt to keep our bodies in a sustainable, balanced state of health. When you introduce a foreign chemical like a recreational drug, it tilts the scale. In the mind of the user, the “high” is pleasant and desirable, but to the body it’s an unsustainable state and undesirable. So the body works to restore homeostasis by adjusting the aforementioned processes to account for the drug. This is colloquially known as “gaining a tolerance”. When the drug is no longer present, the body starts to roll back the changes, but until it does you feel terrible, even to the point of becoming seriously ill. This is colloquially known as “withdrawal symptoms” or in the case of alcohol, “a hangover”. Alcohol is somewhat unique in that it metabolizes out of the body so quickly (within 24 hours in most cases), that even steady drinkers rarely build up a significant tolerance.
Drug addiction can be just as much about “running from the crash” as about “chasing the high”.
A hangover, medical term veisalgia, is a combination of symptoms. Withdrawal may be a part of it, but the symptoms felt by most are headache and nausea. The headache is caused by vasodilation and inflammation induced by the alcohol. The vasodilation causes the “sinus headache” portion of the hangover. The inflammation causes the headache at the base of the skull. Oral decongestants and anti-inflammatory medications can alleviate and even prevent nearly all hangover symptoms.
Nausea is caused by alcohol induced stimulation of the chemoreceptor trigger zone. Overstimulate the CTZ and nausea/vomiting result.
Or addiction in parents leads both to abuse of offspring and the model for their addiction.
The idea that addiction comes from PTSD has been shot down by the opium epidemic, which can strike people of all backgrounds. It’s a greater killer than road accidents. And we still don’t have a drug czar.
Drugs don’t necessarily cause addiction, but some drugs are definitely addictive.
Recognizing this is fine, except beware the cures. Possible problems in some fields include rare events, unpopular area for funding (lack of researcher control over process), inappropriate for statistical analysis, lack of a real problem, changes in study environment, difficulty in precise standardization, etc. Then I looked at the article summary.
For example–“Uncontrolled researcher freedom makes it easy for researchers to err in all the ways described above.” …”6. Researchers should pre-register their research protocols, filing them in advance with an appropriate scientific journal, professional organization, or government agency. 7. Researchers should adopt standardized descriptions of research materials and procedures.”
Science went down with such centralized control. Works fine with easy to standardize situations, others not. One might think those in medicine would know about curing symptoms. Might keep you alive though. Some suggestions Ok, but this needs real peer review.
HDHoese ==> “6. Researchers should pre-register their research protocols, filing them in advance with an appropriate scientific journal, professional organization, or government agency.” A huge part of the problem is failures in study design — and the pre-registration of study design allows peers to review the design, point out faults that will make the study irreproducible or irrelevant, and defines what end points are being looked at, statistical methods to be used to analyze results, etc. [This doesn’t apply to “blue-sky” research, of course].
Ocean Acidification research wasted years and a lot of money before the field self-regulated to standardize and establish appropriate methods — most researchers didn’t even have the basic sea-water chemistry right. Those that did could have helped those that has grants but lacked the necessary knowledge.
A lot of medical research that is garbage is the result of data torturing — data dredging — in attempts to find something — anything — “publishable” from some long-term expensive study that in reality, found nothing at all interesting. Post Hoc redesigning of a study to get a result.
Back in the early 1980s I spent a week working and learning from one of the top biometricians in the country. He was the chief statistical consultant for several medical research institution. We had lunch most days that week. He told me and others in the class he was teaching at our marine research institution that medical doctors were some of the worst scientists he had ever dealt with. He was on retainer, they could call him at any time. He held meeting each year at each institution to advise them of his services and they should come to him during the experimental design phase. Yet the doctors would still conduct research or do an “experiment” and then after they were finish come to him and say, “Now what statistics should I use?” He told me that if it happened just once it would have been one thing but some went through this scenario repeatedly. Some would even get angry when he told them he couldn’t help them.
Edwin ==> One of the major recommendation is that experimental design include what statistical methods will be used — and why — to analyze the resultant data.
Your friend is absolutely correct — the use of ad hoc statistics means the experiment is already compromised.
KH
You are entirely correct. We had a good statistics department where we sent students to help with design. When the ocean pH nonsense started I got into it as a reviewer for the NIPCC. I thought I didn’t know much about it until I read a number of papers. Having centralized reviewers that know little of the basics won’t help, I would argue, and as others note problems with post hoc application and statistics shopping used to be well known.
I worked with a couple of very good government modelers and many scientists. When you run into a bad one, they are really, really bad. Back in those days we did not have press releases and I may have been wrong thinking that we needed to put out more good science to the public. Best available ‘science’ can have lots of problems.
Edwin
We have a doctor with a hobby, he says, to read the medical literature. He is very good and admits shortcomings. We hope he will survive the deluge of paperwork.
“most researchers didn’t even have the basic sea-water chemistry right.”
I think peer review is supposed to catch problems like acidifying with hydrocholoric acid instead of CO2 /arbonic acid when you are studying carbonate chemistry. But it doesn’t seem to have done so. That seems to me to be a bit disconcerting. Hard not to conclude that the peer review process is broken.
Edwin, that doesn’t surprise me at all. Doctors aren’t taught how to do science, they are taught how to cure people. They might study some statistics for epidemiology, but not things like experimental design. I imagine in some instances they did some procedures, found something interesting, then wanted to publish it. Which is fair enough, but it’s observation rather than science.
Kip, I think the idea of publishing methods online before the study has the fatal flaw that people could steal ideas. It’s a matter of intellectual property rights. The methods and rationale should be recorded and submitted beforehand, but not to the public. Often this is done at least to some extent in a research proposal on a grant application.
…A deeper issue is that the irreproducibility crisis has remained largely invisible to the general public and policy makers. That’s a problem given how often the government relies on supposed scientific findings to inform its decisions….
On the contary! policy makers are not interested in science INFORMING their decisions. They are interested in science SUPPORTING their decisions.
Policy decisions invariably benefif someone, and you can lay odds that that someone has ‘commissioned’ the policy decision, usually through lobbying. All science is there for is to provide a justification…
+100 from another geezer.
How old do you have to be to qualify as a geezer?
“A deeper issue is that the irreproducibility crisis has remained largely invisible to the general public and policy makers…”
I don’t think it is invisible to the generally public. After being bombarded by a seemingly endless series of studies with contradictory findings on, for example, what is healthy to eat and what is bad to eat, the General Public often just shrugs their shoulders and ignores the whole sorry lot.
Our sons girlfriend is studying Bio Medicine at university and has just had to write a lab report in it she must have 40 citations, when she told me that all I could think was that it would almost create a merrygoround of new researchers being forced to copy the mistakes of those that have gone before in order to pass the course.
James Bull
James, Did she read all 40 citations? I have reviewed papers, sat on editorial boards, where the scientists quoted lots of papers but had actually read hardly any of them. They would read a paper that quoted other papers, which they didn’t read, then they would cite all of them. There have been times when people quote papers, that quote papers but never bothered to look at the original source. Therefore mistakes or misinterpretations took place and were perpetuated through time.
It’s worse than that. Often the paper being cited contradicts the conclusions being made by the author and in no way supports their research.
Another major problem is with papers that have many authors. If you join a team like this you may be 10th in the list of authors but wind up with hundreds of citations if any one of the papers becomes popular. Citation indices have lost their value from long list of authors just as peer review by pals has lost its value.
BINGO!
One of the ‘dirty little secrets’ of science is just how rubbish peer review can be in practice. It is sold as a check on BS but in reality it can act as as actually promotor of the same. Climate ‘science’ being the classic example, any old sh*t both can and does get through peer review, if it supports the consensus.
We had a scientist at our institution that deliberately, at the prompting of several in the federal government, bypassed our editorial board and review process. Their paper was to be published in a major federal journal that claimed to be peer reviewed. I found out that the paper had bypassed our process when one of the peer reviewers that I knew well called and ask how we let such a poor paper out of house, though he bet I didn’t know. Later another peer reviewer called to complain that none of his comments had been addressed by either the author or the journal editors, not even the math errors and misuse of at least two statistical models. He was upset that the federal journal editors were basically ignoring his telephone calls and letters. This took place “way back” in the 1980s. Over the years I was a peer reviewer on several papers for the same journal where the authors and editors ignored my comments and corrections. I finally quit reviewing for the journal. I wrote them a nasty letter about their failure to properly use peer reviewers. I got no response. So such problems have been going on for a good while. As we use to say in the Navy when we repeatedly did dumb things mandated or uncorrected from on high, “200 hundred years of tradition, unhampered by progress.”
“He uses statistics as a drunken man uses lamp-posts.. . for support rather than illumination.”
(Andrew Lang)
“Statistics used to garner attention should remain suspect. 50% of the people in this classroom have only one testicle … you in the back, are you listening now?”
(surveying professor … paraphrased)
‘Acceptance’ of glib theory and daft word-view interpretations of ‘data’, is where it all falls down.
The only people who don’t accept so easily are those who have both a clue, and a spine.
It makes little difference if what’s being ‘accepted: is completely false, or not. Cohesion to the BS is valued far more highly than facts, or messy stuff like repeatability, because sucking-up gets the public money.
Unaccepting is unacceptable to the accepting— it doesn’t pay $$$ to be right.
There are no watchdogs for this stuff. The press and politicians simply corrupt science to push political agendas. There should be a standard requirement that any research used to support public policy:
1) Releases their data to the public
2) Must be independently verified and reproduced
3) Must be done in a double-blind manner so the verifier doesn’t know what they are verifying
4) Penalties must be stiff for scientific fraud
I’d change the first one to “Releases their data AND METHODS to the public.”
There’s so much guesswork in statistics that they’re nearly useless. I think that they are still useful in creating a hypothesis, but shouldn’t be used as evidence to prove anything. A good starting point, but often wrong.
Well, you’re right about one thing – statistics don’t prove anything. That’s not what they are meant to do. That’s not what science does.
But you are dead wrong about them being nearly useless. They are not guesswork when used appropriately. They are a tool that can be misused and abused, but that doesn’t make the tool useless.
HOW BAD IS THE GOVERNMENT’S SCIENCE? It’s VERY BAD, but we knew that!!!!!!!
Yes Aurora: And that definitely is REPRODUCABLE and repeatedly so.
How discouraging to find Watts Up With That using an apparently thoughtful article as bait to land a WSJ subscription.JGL
>
?
“fishing fake statistical significance out of noisy data”
As in the case of the Charney Climate Sensitivity values all over the map
https://ssrn.com/abstract=3162520
And a valid experiment must not only be reproducible, it must be consistently reproducible. If it is reproducible only a fraction of the time then it is not actually validating the hypothesis.
“Half the results published in peer-reviewed scientific journals are probably wrong.”
Half? Only Half? I think that makes it better than we thought not worse.
Irreproducibility is a certain hallmark of bad science. But there are many others equally certain to be indicative. In statistics, autocorrelated ‘statistical validity’ without a Bonferroni correction. In clisci, reliance on unvalidated (or worse, proven wrong) models. In renewable energy, failure to address the system effects of intermittency and lack of grid inertia. In regulation, modification of or ignoring contrary findings (recent example EU glyphosate cancer warning). Wrote a whole book with hundreds of examples in different categories, The Arts of Truth. Used clisci as the penultimate chapter because so rich with e amples from each of the previous chapter categories.
In the social sciences and in climate science, there is also mis (or dis)-applied statistical methods, which allow the author to claim significance where there is none.