![Reproducibility-Initiative_resized[1]](http://wattsupwiththat.files.wordpress.com/2014/06/reproducibility-initiative_resized1.png?w=150&resize=150%2C150)
Reproducibility — the ability to redo an experiment and get the same results — is a cornerstone of science, but it has been the subject of some troubling news lately. In recent years, researchers have reported that they could not reproduce the results from many studies, including research in oncology, drug-target validation, and sex differences in disease (and climate with Cook et al. ).
In response, journals such as Science have adopted new guidelines for certain types of studies, and members of the scientific community have published a flurry of articles and blog posts.
But, as of 2 May, a panel of experts that convened at the AAAS Forum on Science & Technology Policy was still troubled. Fortunately, while their tone was serious, the speakers also described some solutions that are moving forward.
A Far-Reaching Problem
There are many reasons that scientific results may not be reproducible, explained the speakers, who focused their remarks on the biological sciences. Sloppy research is one possible culprit, according to Story Landis, director of the National Institute for Neurological Disorders and Stroke (NINDS). Studies may be designed poorly or fail to use appropriate statistics, or the experiment’s details may be described inadequately in the published report. Researchers may also feel pressure to publish “cartoon biology” that overemphasizes the “exciting, big picture” and leaves out the more prosaic details, she said.
Brian Nosek, co-founder and director of the Center for Open Science agreed that pressure on authors contributes to a “gap between scientific values and scientific practices.” The more prestigious journals tend not to publish negative results — that is, studies in which a hypothesis is not borne out by the data — or studies whose chief aim is to replicate other findings. Researchers typically must publish in high-impact journals in order to advance their careers, and therefore have little incentive to conduct these types of studies despite their importance, Nosek said.
Studies that can’t be reproduced due to outright fraud are relatively rare, according to Robert Golub, deputy editor of the JAMA, the journal of the American Medical Association. But, even when researchers are not intentionally engaging in misconduct, non-reproducible results are troubling, the speakers agreed.
“There has been a confluence of concern from various sources within the scientific community and from outside the scientific community in the last few years that the scientific enterprise is not producing new knowledge of sufficiently high quality,” said Katrina Kelner, editor of Science Translational Medicine and organizer of the Forum session. “…This issue of reproducibility is a problem of increasingly great concern to the scientific community itself and it is, one could argue, legitimately of interest to the broader society because of the robust public support of scientific research.”
As an example of the serious consequences that non-reproducible studies can have, Landis cited a report that a drug called minocycline showed promise in mouse models of the neurodegenerative disease ALS. These findings led to a phase III clinical trial funded by the National Institutes of Health, which enrolled over 400 patients between 2003 and 2007. However, the disease actually progressed faster in patients who received the drug than in those who received a placebo. When scientists rescreened minocycline and many other compounds from 221 studies in mouse models of ALS, they found no statistically significant effects.
more: http://www.aaas.org/news/concerns-about-non-reproducible-data-mount-some-solutions-take-shape
Many in the climate field strive to be merely incompetent in non-reproduceability. They seem to take it as a badge of honor when they sequester their data from those who seek to check their results. Of course the most infamous are Mann refusing to release his code to Mcintyre and Cook lying about the availability of his data.
Just another reason that many in the field are not practicing science, but merely voodoo.
Well if is important enough, and if it saves the planet for our grandchildren , we dont need no stinking reproducibility !
People are noticing and acting on bad science, this is good.
Remember cold fusion? Deja vu all over again.
Reproducibility is important but there is a lot confusion about what is required. The most important thing to reproduce is the conclusion of a study. It is not necessary to reproduce it by the same route, in fact it is a stronger validation if you can reproduce someone’s finding by an independent route. Just think of some iconic experiments – could you reproduce the Michelson-Morley experiment? Well obviously not exactly as their apparatus no longer exists. Does this mean their result is not ‘science’? Well no, because their results have been reproduced dozens of times by people building newer and better apparatus and coming to the same conclusion. How about Watson and Crick’s double helix result for the structure of DNA? That has been reproduced many times, but you could not get their conclusion by the same route that they did because you do not have Rosalind Franklin to take the x-ray measurements for you. In the medical sciences you cannot repeat studies exactly – the patients have changed! You have to repeat with different samples and different statistics.
As far as Cook’s survey is concerned, you do not need his data. Getting the data and checking his arithmetic is a very weak form of ‘reproducibility’ , especially as the missing part of the ‘data’ is, as I understand it, the identities of the raters. If you want to challenge the results, do your own survey – with a bit of crowd sourcing you could easily read the abstracts of several thousand papers in a month.
Studies that can’t be reproduced due to outright fraud are relatively rare, according to Robert Golub, deputy editor of the JAMA, the journal of the American Medical Association.
Then he never heard of climate science (fiction) IPCC style or is it plain political korrectness?
What I find disturbing is the comment that out right fraud is rare. Yet, they say most researchers are not inclined to re-produce another researchers results. If that is the case, how can they say that fraud is rare? Would you publish knowing once passing “peer review”, no one will call you on it?
Anthony there is something wrong with your website. When trying to navigate though an article i am led to a gamers ad.
REPLY: wordpress.com which manages the ads (and hosts WUWT) has a rogue advertiser that has been hitting a few wordpress.com sites with javascript redirects. They told me earlier that they had nailed this advertiser and turned them off, but that some cached pages in your borwser may still have the code. Just clear your browser cache and cookies and that should solve it. – Anthony
The magnitude of the problem was quite shockingly highlighted in a note that appeared in the journal Nature in 2012:
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
According to the authors, who led a research lab at the pharmaceutical company Amgen, a full 47 out of 53 published experimental studies that suggested novel therapies for cancer could not be reproduced. The authors relay similar findings by researchers from other pharmaceutical companies. They also state: “Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications … [and] clinical studies — suggesting that many patients had subjected themselves to a trial of a regimen or agent that probably wouldn’t work.”
The authors, nevertheless, bravely proceed to put lipstick on this pig: “These results, although disturbing, do not mean that the entire system is flawed.” To me, with lipstick or without, this is an unmitigated disaster that is right up there with mainstream climate science. As in the latter, it was all sanctioned by peer review, and again as in climate science, the whistle blowers are mostly from outside academia. As a career academic myself, I have become convinced that the problems in academia are profound, entrenched, and pervasive, and that they will only be fixed from the outside, not from within.
jimmi-the-dalek writes “As far as Cook’s survey is concerned, you do not need his data. ”
Its not about reproducibility, its about audit. If Cook’s paper is found to be faulty (and it most certainly is) then sceptics can say, Cook’s paper does not support the 97% concensus idea.
Sceptics, however, dont have to write their own papers…sceptics have a specific role in science and that is to not believe findings and instead try to find fault in what others have done. Its little wonder they’re hated so much, but they’re arguably the most important gatekeepers to truth.
There are many journals chasing less many good studies. I’d bet need is greater than supply; they can’t afford fot deadline reasons to be too critical. Which is probably very much the case in the climate science industry.
TimTheToolMan,
If you want to say it about audit, then that is fine. However here it is common to find the mantra “if it’s not reproducible, it’s not science”. I was trying to say that “reproducible” means more than just getting the data and checking the arithmetic.
Valen E. Johnson, Revised standards for statistical evidence
http://www.pnas.org/cgi/doi/10.1073/pnas.1313476110 PNAS | November 26, 2013 | vol. 110 | no. 48 | 19313–19317
Nature is persistent, not random
Koutsoyiannis et al., detail how climate is NOT random, but rather shows persistence. Aka. Hurst-Kolmogorov dynamics. The natural HK standard deviation is about TWICE conventional statistics. E.g. see:
Markonis, Y., and D. Koutsoyiannis, Climatic variability over time scales spanning nine orders of magnitude: Connecting Milankovitch cycles with Hurst–Kolmogorov dynamics, Surveys in Geophysics, 34 (2), 181–207, 2013. Preprint
PS Before dismissing “cold fusion” out of hand, check out the rapidly increasing reports of “excess heat” due to “Low Energy Nuclear Reactions” (LENR).
When I was in grad school in England, a fellow American studying economics asked me what makes science science. I gave him the short answer that results are repeatable & falsifiable.
But both depend upon experiment testing hypotheses. Here’s a good test from another English institution of higher education:
http://www.youtube.com/embed/0Uqf71muwWc
Water in the brain, good. Water on the brain, not so much.
Jimmi_the_dalek says:
June 10, 2014 at 8:19 pm
TimTheToolMan,
If you want to say it about audit, then that is fine. However here it is common to find the mantra “if it’s not reproducible, it’s not science”. I was trying to say that “reproducible” means more than just getting the data and checking the arithmetic.
That presupposes that ‘data’ is being used in the first place.
AussieBear wrote;
“What I find disturbing is the comment that out right fraud is rare.”
I do believe that fraud is rare, but confirmation bias is rampant. The climate science community truly believes that they understand the most complex system anybody has ever tried to “model”. So they search for all the data nibbles that “confirm” their belief. And they turn away briskly from any “counter indications”. Just human nature really.
Engineering is a totally different beast, no matter how hard I want to believe that my design will work the laws of physics are the final arbiter. Models are a useful tool in engineering, but only just one tool in the toolbox. The final product performance is the judge/jury and sometimes the executioner as well. If it works well enough for the problem at hand we shoot the engineers and ship the product. If not, we shoot the product.
Don’t worry about the engineers we shoot, they get to be managers……
Cheers, Kevin
Predictive value.
The real problem is the AAAS and AGU. They published all the irreproducible papers, after all.
@Anthony:
You need to be careful accepting the one study unable to reproduce anti-cancer effects. The authors of that study claim they cannot legally disclose what they studied or how. It is complete heresay. Also, reproduction of any significant experiment from what I’ve seen requires cooperation between labs to get all the details right.
Right now, there’s a game of liar’s poker going on for funding in the biological sciences. Everyone’s work is tied to an imminant cure for any number of devastating intractable diseases. If you don’t make a big claim, you won’t work, or you will end up someone’s technician. There are almost no consequences. Some of the science is interesting, but there is a huge bias in preventing systematic investigation of basic principles because they’re not splashy enough.
The minocycline debacle is still ongoing. It’s spreading to other subspecialties. Let’s just say that some of the primary authors could stand to retract a few papers.
I was waiting for you to cover this topic. The most important thing is that there is a huge amount of non-reproducibility, but the articles I’ve seen claiming this are likewise typically irreproducible.
Science is more than a body of knowledge, it’s a way of thinking. A way of skeptically interrogating the universe with a fine understanding of human fallibility. If we are not able to ask skeptical questions, to interrogate those that tell us that something is true, to be skeptical of those in authority, then we’re up for grabs for the next charlatan, political or religious who comes ambling along. ~Carl Sagan~
o/t but this has just been posted online:
bipartisan bill goes to Congress:
11 June: WOWK TV: Mandi Cardosi: Lawmakers in Congress introduce bill to stop EPA carbon emission rules
It has been one week since President Barack Obama announced a new proposed rule capping carbon emissions for existing power plants – leaving West Virginia officials frantic.
On June 10, U.S. Reps. Nick J. Rahall, D-W.Va, and David McKinley, R-W.Va., introduced a bill to stop it.
“Last week, the EPA unleashed its latest assault on the jobs and livelihoods of our coal miners,” Rahall said in a news release. “The EPA needs to get their head out of the clouds and come back down to Earth where the rest of us must live and work. We don’t need more regulation to solve our energy challenges — we need more innovation.”…
Rahall and McKinley’s bill (H.R. 4813), which already has 68 cosponsors, would terminate the new rule for existing power plants, along with the proposed rule for future power plants. In addition, to prevent some sleight of hand maneuver by the EPA, the bill will aim to block the issuance of similar rules for at least the next 5 years without Congressional approval.
“I have fought with our coal miners for years, defending their jobs, promoting their health and safety, and protecting the pension and health care benefits they’ve worked so hard to earn,” Rahall added. “So when someone picks a fight with our coal miners, I put on the gloves. This may be one whale of a fight, but I am not slugging it out alone. …
http://www.wowktv.com/story/25738760/lawmakers-in-congress-introduce-bill-to-stop-epa-carbon-emission-rules
Anyone interested in this subject should go read the article “The Truth Wears Off” at http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all
It’s a remarkable piece that I reread from time to time.
Then read “Why Most Published Research Findings Are False” at http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124
The first is a New Yorker piece that is a very relaxing read. The second is a peer-reviewed paper that is heavy in statistics and rather dull, but informative. The statistics of that paper are a little beyond me though, so I don’t really pick up on everything in the paper.
Like David Ball’s comment at 9.25. When using automated endoscope reprocessors (AERs) to decontaminate used endoscopes in hospital, the quality of the final rinse water is paramount.
The endoscopes undergo a rigorous cleaning regime involving detergents and disinfectants and then receive a final rinse. However, depending on where your hospital is geographically situated, the water source will produce a supply of varying purity levels.
When you talk to the industry leaders in this field, they are adamant that when hospitals install their AERs they ALSO install an R/O plant next to them because it is the only way that you can guarantee a reproducible result for every endoscope going through the system.
That is the predictive value – in this case, of great importance to the clinical team and the patient who next come into contact with the scope – of ensuring that EXACTLY the same conditions apply when setting out to reproduce any “experiment”.
TimTheToolMan says:
June 10, 2014 at 8:08 pm
….
Sceptics, however, dont have to write their own papers…sceptics have a specific role in science and that is to not believe findings and instead try to find fault in what others have done. Its little wonder they’re hated so much, but they’re arguably the most important gatekeepers to truth.
It is not sceptics, but scepticism which has a role in science and every scientist has a duty to practice scepticism, particlarly about their own ideas. A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.”
There are several strong reasons a why bad idea (or shnappsidee) hangs on in science. The chief one is frequently “we have no alternative, and this idea looks right. It seems to have all the parts in place.” That kind of thinking is explicit in Kevin Trenberth’s “travesty” email where he complains the data has to be wrong. This also happens to be the exact problem with any discussion of the effects of CO2 on global climate. There are a modicum of facts, which are not in dispute, upon which the theory of GHG effects is built. (Even for the unmentionable crowd [overgrown lizard murderers], the facts aren’t in dispute, just the effects.) So far, there is no theory of climate that works even remotely as well as say the Newtonian theory of Gravity, and we actually know that Newtonian gravity fails at different scales: from predicting Mercury’s orbit, to accounting for galaxies hanging together rather than flying apart. Greenhouse theory at least has the benefit of including all the parts we know about, although we obviously don’t have the erector set assembled properly. No one at this time can say with confidence that “this is where climate was, this where it is, and this where it will be presently.”
KevinK What you say about engineer you shoot. “Don’t worry about the engineers we shoot, they get to be managers……” In education they do the same thing those who cannot teach get promoted to administrators.
Simple solution: don’t publish any original studies, only replications.
On the other hand, I have heard rumours of studies which cast doubt on the idea that red wine is good for you. I hope those rumours are false. That’s taking science too far.
No surprise in human biology. 9 out 10 cancer/tumor cell line drug results reported in leading journals can not be replicated by the Big Pharma scientists when they investigate the reported results in house.
But then again, hey, lets just take 4.5C as the CO2 doubling value. Published, peer-reviewed science is never badly wrong… is it?
Duster writes “A sceptic who offers no alternative is simply a philsopher”
In terms of climate science, there is no need to offer an alternative “to CO2” because the climate has varied in the past all by itself. Warming enthusiasts acknowledge that but say this time its different. This time its CO2.
a thought for the world of scientific journals — there ought to be journals devoted solely to articles which (1) seek to replicate or falsify past study results, and (2) detail studies of any kind which had only “negative” results, i.e., no significant causation found, and (3) publish lengthier comments and responses to existing articles than journals generally permit, etc.
Sure, such journals will never be the “prestige” journals leading a field, but it ought to be far more feasible for researchers to publish these kinds of scientific articles and get some degree of professional credit, C.V. items etc.
With the advent of online and open access journals, the logistics and expense of such journals ought to be far lower.
It may be extremely difficult to find organizations and editors, reviewers, etc. willing to run such journals, but perhaps teh various professional societies could be pressed to regard this as a fundamental service to the professionalism and quality of each field of speciality.
This all may be a huge long shot, but it is the sort of thing which ought to happen!!
jimmi_the_dalek the missing part of the data is NOT the ID of the reviewers some of which have already been give away , but the time stamps which would show how much effort each article took to review . In other words the chances of people doing a honest job when they spending seconds on this job are virtual nil and that is what is suspected that Cook and his little gang went all out to ‘prove ‘ the 97% claim and to do that that fired out results to fit that need without any real checking , hence the fact that included articles that could actual be used , if reviewed probably, to support the 97%
hence the fact that included articles that could NOT actual be used , if reviewed probably, to support the 97%
There is a strong aura of ‘fuzziness’ around the use of the term ‘scientific results’. In the context of discussing the validation step of the scientific method, the term should be restricted to *empirical* measurements.
All too often lately the term has been used in the broad sense of “the results of computer modelling by programs incorporating scientific cause-effect relationships”. In this latter use, computer modelling has been used to replicate the results of other similar computer models, but this does NOT qualify as ‘reproducing’ scientific results.
Scientific truth is reproducible. Errors can be replicated, but they cannot be anchored to empirical measurements.
Mother Nature does not make errors. Mother Nature is the ultimate arbiter of scientific truth, and valid experiments and measurements are those that interrogate her directly (to preserve the metaphor) in a way that yields a result that can decide whether hypothesis A or hypothesis B is more correct.
So managers are undead engineers? This explains much…
And that’s why some of them in the CAGW field refuse to release their data – because their results are unreproducable.
First of all, thank you for the interesting post.
Being in medical and biological science since more than 20 years, I have to admit that the problem is real and it is far larger than (i think) commonly perceived.
I have more than 300 publications among which at least 200 original papers: do you really think they are all containing “top science”? No way!
In order to be funded you need publications. On the other hand, while all agree that confirmatory studies are extremely important nothing facilitate their completion. Confirmatory studies are of poor interest even for low-ranked journals, they do not help careers. and funding agencies do not pay for something already done by others.
On the contrary the studies that contradict previous findings (that could also be very healthy thing) have often few chances to be published since encounter biased review processes.
There is an intrinsically masochistic relationship between science and economics. We do not have to forget that today science and scientists have to learn to survive in a society where only the emotionally breaking news count. The so-called “scientists” (much different subjects than the ones who produced the milestone discoveries in the past) are in a sense “forced” to prime-time findings to get their research going.
Today we are judged by number of publications, impact factor, h-index etc. Forget about the old times when science funding was independent from the science results. Economics in now dominating the scene and I suspect this is not gonna change soon.
So we have to learn to live in a science system in which the real discoveries are often hidden by an overwhelming level of noise.
I expected to see a learned discussion of the bankruptcy of the academic “publish or perish” mentality when I clicked on this article, and yet, nothing! Academe is creating this sciency diarrhea as is the (wrongheaded, IMO) cultural truism that all students must go to college. Although, in one sense, I suppose that the truism is accurate, as many, if not most kids graduating from high school seem to be dumber than posts, can’t read or write well, have read nothing of import, have no sense of history or perspective, and have been brainwashed by leftist progressivist BS. How do we correct this? I would make public sector and teachers unions illegal. I’d fund public education by vouchers and let the free market make new, competitive schools. I’d end subsidized college loans. I’d install some review and accountability at universities, and fire professors who crank out or advise their post-graduates to crank out garbage science. Of course, those very same garbage producers are helping to fund the colleges and universities through their grants. So we need to scale back the Federal government’s funding of science, too. Or end it completely, perhaps, at least for a time. Let’s let market forces and competition reassert themselves in the priority mechanism of scientific research.
I don’t think there’s fraud involved; The experimenters are just fooling themselves with
“data mining”. Someone runs a test believing “A” is true. The theory is worthless, the results don’t show anything, but “A1” shows significance at the 5% level, so the experimenters publish the results for “A1”, not realizing that the NEW results DON’T show “A1” significant at the 5% level. In reality, they’ve demonstrated that (assuming A1 and A are independent)
“Either A or A1 is true at the (1 – (0.95)^2) level = 9.75%- not significant and not worth publishing.
Here’s a real world example:
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1871503&http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1871503
Originally, they set out to test the hypothesis that “gullible believers” were more scientifically literate than “deniers”. THAT didn’t pan out. Publishing the actual result that DENIERS scored higher at the 5% level would have been an example matching my
(1 – (,95)^2) = 9.75% exactly, but they didn’t want to publish THAT. Instead they went into a spiel on CULTURE and how more scientific literacy increased one’s “pigheadedness” if you will.
The testers were actually throwing in a 3rd item in their data mine
1- (0.95^3))= 14.26% chance by dumb luck. Add to that the possibility that OTHER unmentioned variables may have bee tested and ruled out, but not addressed, and it’s easy to arrive at
a “5% or less probability of happening by chance alone” when the actual figure is closer to 50%.
Nicholas Schroeder says:
June 10, 2014 at 6:53 pm
Remember cold fusion? Deja vu all over again.
That is a poor example. Pons and Fleischmann have been replicated. MIT didn’t load the Palladium with sufficient Deuterium.
You might look at Elforsk’s report, expected later this month, on their six month trial of Rossi’s E-Cat HT.
Adrian Ashfield
milodonharlani says:
June 10, 2014 at 8:26 pm
“Water in the brain, good. Water on the brain, not so much.”
Try a tap on the head.
Jungle:
I had the same thing but it seems to have stopped now.
Journals need to publish research. If there isn’t enough good research out there to publish, then they publish research that isn’t so good. I once taught a basic research class for graduate students. There were about 20 in the class. Toward the end of the semester, each student chose 5 research studies from a variety of fields, analyzed them according to the principles taught in class and presented what they found. Out of more than 100 studies, only about 10 to 15 were evaluated as “clean” enough to deserve publication. Worst were found in health journals. Best were pharmaceutical studies.
Philjordan:
Do not forget Keith Briffa and his science scam involving tree rings which Steve McIntyre finally exposed through a FOI action.
“As in the latter, it was all sanctioned by peer review, and again as in climate science, the whistle blowers are mostly from outside academia. As a career academic myself, I have become convinced that the problems in academia are profound, entrenched, and pervasive, and that they will only be fixed from the outside, not from within.” ~ Michael Palmer
People who receive degrees from these colleges are also a considerable force in maintaining the paradigm under which they were educated. No one wants to admit that the degree they have is based on knowledge that has been debunked or is useless.For example, the previous claims of the health and nutrition experts, and the highly selective witch hunt for carcinogens, are all being falsified, but the Baby Boomers are exceedingly dogmatic about what they know, and conclusions they reached 40 years ago.
And so we have such preposterous legislation beginning to surface which regulates diets. Another problem resulting from the dogmatic clinging to false scientific findings is the criminalization of many neutral and even beneficial economic and social activities, such as commercial farming.
Jbird says:
… I once taught a basic research class for graduate students. …
—
What kind of topics did you teach?
@Palmer
Design, number of cases, independent ad dependent variables, control for extraneous variables, choice of statistic, use of null hypothesis, etc.
non reproducible “studies” are not studies they are experiments … that failed to validate the underlying theory …
I agree that skepticism is an important quality in a science. Without it, the reviewing scientist has
muted his critical faculties. In climate science, a skeptic is merely one who approaches the issue with his critical faculties in good order.
mpainter says:
June 11, 2014 at 9:47 am
I agree that skepticism is an important quality in a science. Without it, the reviewing scientist has muted his critical faculties. In climate science, a skeptic is merely one who approaches the issue with his critical faculties in good order.
Rather,
In climate science, a skeptic is
merelythe only one who approaches the issue with his critical faculties in good order.@Sandi says:June 10, 2014 at 9:39 pm
Sagan was soooo wrong “Science is more than a body of knowledge, it’s a way of thinking.”.
Science is a methodology, that’s all,nada nothing else, not a way of thinking (that’s philosophic) !!!
“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.”
yup.
if you dont have a replacement theory, the best one, no matter how flawed, rules.
Otteryd says:
June 11, 2014 at 7:21 am
As in the third tap in the Mozambique drill, after the double tap to the body?
Steven Mosher says:
June 11, 2014 at 10:26 am
The man-made GHG hypothesis has been falsified so thoroughly that it is worse than no theory at all.
OTOH, the hypothesis that solar activity (regulated by our planet’s orbital & rotational mechanics) & earth’s unusual liquid watery surface & atmosphere, plus plate tectonics, largely control climatic fluctuations has good observational support:
http://www.space.com/7195-sun-cycle-alters-earth-climate.html
Steven Mosher if your theory cannot stand up to actually being questioned , it not a theory at all your merely selling BS , peer review or not .
Plenty of theories in science and then all have to ‘put up with ‘ with ongoing critical review and accept that they may need to be modified thanks to this review , Climate ‘science’ is the first area that came up with a theory sold as ‘perfect form birth ‘ or scientific immaculate conception, that can never been wrong nor even questioned.
Reproducibility is only a requirement in the experimental sciences; in the observational sciences like astronomy or geology no flatly disconfirmatory experiments can be run. It’s more a matter of degree, with confirmatory or disconfirmatory observations adding weight or undermining a claim.
Thanks for your reply, Jbird.
—
Coincidentally, just today I received a request to review a research paper. The instructions contained the following clause:
One important concern in scientific publication right now is the apparent increase in data fabrication, data alteration, or duplicate publication. We ask that you pay attention to any signs of fraud in the text or figures, and that you inform us of any concerns about duplicate publication or unusual-looking results.
A recent opinion piece on this topic had an excellent suggestion – don’t review the results but the experimental method. That way, good papers are published whether or not the come the “correct” conclusions and the body of literature contains its own balance instead of being skewed to positive results.
http://www.genengnews.com/gen-articles/are-medical-articles-true-on-health-disease/5203/?kwrd=young
Sorry, the article had been free to view when I read it over the weekend, but is now pay-walled.
“A recent opinion piece on this topic had an excellent suggestion – don’t review the results but the experimental method”
That’s not a new suggestion, but a part of the standard review process. At least in the area I work in, the guide notes when reviewing a paper include a number of things to check, one of which would be “Are the methods used sufficient to establish the conclusion” , or words to that effect.
“if you dont have a replacement theory, the best one, no matter how flawed, rules.”
Why so? You could simply say, “We’ve got a bunch of phenomena, but we have no explanation. Every theory we’ve tried turns out to be bollocks. We just don’t know.”
Clear, honest, and justifies asking for a grant for further study.
Steven Mosher says:
June 11, 2014 at 10:26 am
“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.”
yup.
if you dont have a replacement theory, the best one, no matter how flawed, rules.
==============
Which leads us to BEST.
Same for air-to-air, or was that more exacting ?
@ Michael Palmer…..
Maybe the pendulum will swing back toward more rigor and objectivity. A lot of damage has been done to scientific integrity, especially with the politicization of climate science.
[“A sceptic who offers no alternative is simply a philsopher in Heinlein’s term, a “scientist with no thumbs.” if you dont have a replacement theory, the best one, no matter how flawed, rules.]
Such thinking led to the creation of the Greek Gods. Zeus was the theory for lightning and Hades was the theory of what happens when we die.
If I want to read a good science fiction book, I’ll read Heinlein. If I want to know if research is garbage, then I’ll listen to the skeptic who has taken the time to find the flaws in such research. Skeptics have done more to undo bad science. Skeptics who spot shoddy science deserve our praise.
KevinK says:
I do believe that fraud is rare, but confirmation bias is rampant. The climate science community truly believes that they understand the most complex system anybody has ever tried to “model”. So they search for all the data nibbles that “confirm” their belief. And they turn away briskly from any “counter indications”. Just human nature really.
With “peer review” possibly being poor at spotting confirmation bias. Because everyone involved knows how things should be. Possibly what’s needed is more a “proof reader” or some other “outsider”.
Engineering is a totally different beast, no matter how hard I want to believe that my design will work the laws of physics are the final arbiter. Models are a useful tool in engineering, but only just one tool in the toolbox.
Hence we still “crash test” cars and bend entire aircraft wings until they snap
Well, am late for the conversation, but am happy to see that finally the reproducibility question is being discussed. It was long overdue.
Science without reproducibility is not science.
I would naively think that papers could be simply classified like:
– reproduced
– not reproduced but pending – pending validation
– no data & methodology available for reproduction – not valid
Michael Palmer says:
June 10, 2014 at 8:07 pm
The magnitude of the problem was quite shockingly highlighted in a note that appeared in the journal Nature in 2012:
….
According to the authors, who led a research lab at the pharmaceutical company Amgen, a full 47 out of 53 published experimental studies that suggested novel therapies for cancer could not be reproduced.
….
The authors, nevertheless, bravely proceed to put lipstick on this pig: “These results, although disturbing, do not mean that the entire system is flawed.” To me, with lipstick or without, this is an unmitigated disaster that is right up there with mainstream climate science.
Agree. The system of promoting “science” based on non-reproducible papers is flawed.
jimmi_the_dalek says:
June 10, 2014 at 7:06 pm
…
As far as Cook’s survey is concerned, you do not need his data. Getting the data and checking his arithmetic is a very weak form of ‘reproducibility’ , especially as the missing part of the ‘data’ is, as I understand it, the identities of the raters. If you want to challenge the results, do your own survey – with a bit of crowd sourcing you could easily read the abstracts of several thousand papers in a month.
Without the data and methodology you come to different result and then what? How does one compare that? How does one understand where the difference is coming from?