From the Journal of Irreproducible Science – over 2/3s of researchers say they are unable to replicate study results
WUWT Reader “QQBoss” writes:
The BBC reports (shockingly), that the journal Nature is going to begin requiring a reproducibility checklist of authors, based on a survey performed last year where at least 70% of respondents (self-selected, of course) indicated that they were unable to reproduce expected results. As the ability to replicate studies is what allows science to demonstrate meaningfulness and continue moving the body of knowledge forward, it is surprising that it has taken this long for top of the line journals to more strongly encourage replication to establish validity.
“Replication is something scientists should be thinking about before they write the paper,” says Ritu Dhand, the editorial director at Nature.
But will they take the next step and more actively police published research and denote when it is not replicable? There needs to be an accessible list of papers that sits between valid, replicated studies and the full blown Retraction Watch . I highly doubt that most journals are willing to self-police themselves, so just as Retraction Watch has come into being, perhaps there needs to be be a web site that aggregates the list of all papers published each year and allows researchers who are able to replicate results to make some fanfare when they are able to remove a paper from the list, since replications are usually quiet affairs.
In the face of the hockey shticks, 97%s, and PAL reviews, combined with researchers refusing to release data “because you want to find fault with it” or just handing their hard drives over to their dogs to chew on, what percentage of AGW-related studies should be listed as unreplicable, perhaps even nonredeemable?
Has Nature thought through the implications of what they are suggesting for a significant amount of the papers they have pushed through that under tougher (aka more meaningful) standards would never have seen the light of day?
More: http://www.bbc.co.uk/news/science-environment-39054778?SThisFB
How? You require authors to submit raw data that has normal distribution. That what we do with our journals being doing it for 5 years now. It works.
Eliza- don’t you mean data with error measures that are normally distributed? Lots of things in the world display other than normal distributions. For instance personal income is log-normally distributed. Air pollutant concentrations are often best fit by Weibull distributions.
“Nature” is unlikely to hold global warming papers to the standard of reproducible results for two reasons:
1) “Nature” is 100% behind the concept of anthropogenic global warming; and,
2) if Pro-global warming papers were held to a standard of reproducibility, the number of such papers that could be published would be vanishingly small, while the number of pro-natural warming/cooling papers that could be published would be much higher. This would threaten the world view of the editors of “Nature”.
Politics had been at play in the science publishing business for a very long time.
What exactly do they mean by “reproducible?” Seems to me that this is somewhat akin to the principle of “falsifiability.” Do they mean reproducible in theory? It seems like an easy standard to insist upon in laboratory science. What does it mean for statistical studies? Is it enough to archive the data and explain the statistical methods adequately? Or what about earth sciences, or astronomy? How does one replicate the study claiming to have discerned “resonances” in the solar system that influences climate from sedimentary geology?
Isn’t the big deal over lack of reproducibility in laboratory sciences where papers pass peer review without anyone actually checking the methods? It sounds almost like they are saying they will not accept such papers unless the same results can be produced by two independent labs. Is that what this is all about?
In other news today – John Podesta, the founder of “Think Progress”, the employer of Joe Romm, the man who financed and organized, as the head of Hillary’s campaign last year, a constant drumbeat of Climate Alarmism last year, combined with constant vilification of anyone who took the skeptic position….
… has just been hired as a new senior political analyst at the Washington Post. Well, they were always very sympathetic to the Warmist position before, but this is a clear sign that they are ripping all the masks off, and from this day forward the most radical diatribes written by our most bitter enemies are going to be printed as “hard news” in the pages of the WaPo.
Disgusting, is all I can say.
WaPo has been hopeless for years. Bezos’ toy now.
The cause of climate change and explanation of why CO2 has no significant effect on climate are readily demonstrated using already existing publicly available data and understanding of science. http://globalclimatedrivers2.blogspot.com
Meh !
Just like how they changed what “peer review” is, they will change what “reproducibilty” means.
In the expectation that a serious discussion is possible here, what is meant by a test of reproducibility? Papers already have statistical analysis showing that the result was reproducible by the original researcher. It is clearly not possible for every published paper to have the results reproduced by an independent laboratory. That would be hugely expensive, and who would pay for that work?
I see the problem as a lack of interest in publishing negative results. Journals must be encouraged to publish when someone submits a paper showing that they could not reproduce a previous result. Maybe a publicly funded “journal of negative results” should be set up.
seaice1: As you have already said:
…you cannot possibly claim that you want a ‘serious discussion’ here. You have, by your own words, ruled yourself out of such discussion.
Harry, look up – I have posted the information.
Ha ha ha! Yes TWO hours after my comment here. But you had to add it to a different part of the thread to make it look like you’d already made the comment and I had missed it. Naughty step for you, my lad.
Harry, don’t get paranoid. I was not trying to make it look like you had missed it. I just posted it, then scrolled down and came across your comment here, so pointed you towards it.
The main criticisms of nearly all climate science papers is their shopping around for a statistical protocol will make a silk purse out of a sow’s ear. Only a few people, like Steve McIntyre have been working on this problem and his work has forced some retractions of papers and some admissions that the authors had stepped beyond the bounds of their results in their conclusions. McIntyre and McKittrick in deconstructing Michael Mann et al paper on the hockey stick showed that with Mann’s new methodology, white noise could be made to produce of hockey stick every time! How would you judge this central icon of the CAGW movement?
Gary “Steve McIntyre have been working on this problem and his work has forced some retractions of papers.”
I had a look for the papers he had forced to be retracted but could not find them. Can you tell me some of them please? Retractions are quite rare and I only found Said et al 2008 and Sidall et al 2009 and Lewandowsky 2013 none had anything to do with McIntyre.
here is another:
http://retractionwatch.com/2014/10/21/article-using-tin-foil-cling-wrap-to-debunk-ocean-warming-retracted-after-urgent-peer-review/#more-23347
Seaice, of course there will be no credit given to the dreaded S. McIntyre. He found terminal problems with a Karoly et al paper a few years ago. Karoly a few days later withdrew the paper saying HE himself had found errors, giving no credit to SM. I guess ‘technically’ you are correct, it was a forced withdrawal. Similarly, MBH correction recognizing the LIA which made it a scythe handle instead of a hockey stick handle was a halfway retraction as a ‘Sauve qui peut’ change to try to make the beleaguered hockey stick a little more respectable. The proof of McIntyre that red noise processed with Mann’s invented statistical method always gave a ‘hockey stick’ would, in more honest and ethical times, have caused a retraction. But hey, this was an icon of Climate science and, like the finding of a surging polar bear population, nothing will be acknowledged that kills icons.
Look, you are clearly a smart guy and some of your comments are thought provoking, but with all the shenanigans in this business, Climategate revelations, book cooking, cheerleading and the like, plus connections to NWO Champaign soshulizm, you must have had some doubts about this stuff. I believed it all myself as did Anthony about a decade ago. You can’t be this easy to fool. I’m a actually a geologist and engineer and I studied paleo climate as a regular part of the Geo curriculum. I also did work on Quaternary geology associated with the last glacial period and it’s melt back. I even discovered the extension of the old Missouri River when it flowed north through Saskatchewan and Manitoba to Hudson’s Bay during the Eemian.
“…Papers already have statistical analysis showing that the result was reproducible by the original researcher…
Actually, no. Maybe you’re referring to Wahl and Ammann trying to refute Steve McIntyre’s work but actually confirming it. Even if you were to accept that Mann et al’s non-standard PCA analysis were appropriate, ignore the dependence on bristlecone’s (inappropriately used as temperature proxies), etc, it’s a statistical failure. And even when it pretends to be statistically appropriate, it isn’t, such as citing that Preisendorfer’s Rule N was used to determine how many principal components were to be retained when it fact it wasn’t. Mann and RealClimate present some examples to explain how Rule N works, but this process does not match what was in the hockey stick code nor match the PCA’s retained in the hockey stick.
What you’re probably thinking of is the claim that many others have come to similar “independent” hockey stick results. What you/they are ignoring is the continued dependence on inappropriate proxies and a relatively apples-to-oranges comparison to thermometer records.
Michael, I was no thinking about climate in particular. I was thinking about papers in general. As I explained elsewhere on here, statements such as n=5, COV = 5% are an indication of how well the original author reproduced his own results.
This and requirement for a formal review by a professional statistician, will go a long way towards cleaning up the literature.
Then we can work on the inane global warming statements attached to every paper by the grant departments.
RAH, (https://wattsupwiththat.com/2017/02/25/the-journal-nature-is-going-to-begin-requiring-reproducibility-in-submitted-papers/#comment-2436751) I wasn’t thinking of government at all. I’m thinking of a new scientific organization that does this stuff. The workload is enormous at present, but if replication was hanging over the heads of would be cookers of books, there would be a couple of orders of magnitude fewer studies published and they would be incentivized to do their research correctly. Personally, I see no way that government can actually do science without corruption – except basic stuff like mapping the geology, collecting data on rain, snow and temperatures but no interpretation, or any discipline where you aren’t feeding a policy agenda. The policy question for research should be “What should we do?” Most should be contracted out. Replication would be an insurance policy, although you have to beware of “replications done by those on a political agenda.
I note commenters like seaice and other defenders of the faith accept that the temperature record is ‘replicated’ by several independent agencies. Those who know the inbreeding of these “independent’ researchers know that they have created one monster with 3 or 4 heads. After “Karlization” of the dreaded “Pause”, all the independents and one satellite temp keeper (RSS) suddenly also adjusted themselves out of the pause. They even created problems for themselves with other products that depended on them. I understand the deep concern of the spin doctors on such an article as this, who visit to badger sceptics (and we all have broad shoulders). They know that if over 70% of medical and other similar type sciences are spurious or fraudulent and their work is put to real world tests that can kill people, then climate science findings that won’t reach maturity until everyone alive today is dead, the percentage of worthless papers is undoubtedly much higher than for medical stuff.(Ironically I think 97% of papers add no honest knowledge to the science)
Question: how can you have a simplistic formula for CO2 affect on temperature that was promulgated ~ 50 years ago and yet have hundreds of thousands of papers on the subject that don’t change numbers of a couple of generations ago? The IPCC certainty on this anemic formula has reached 99% with half a dozen identical reports over 25 years. The most astounding contribution made by Cook of the 97% consensus was that he used 100,000 papers on climate that were produced over only 10 years!!! What novelties could they be regaling us with? That alone should make a sceptic out of everybody if they are being honest about it. Probably before 1950, there weren’t 100,000 papers in all of science for all time.
Gary, make some effort to get it right. There were 11 944 papers in Cook.
….and not all of them counted in the paper. Make and effort yourself.
seaice, You recheck the number of papers. They boiled it down to about 10% of them that they judged as alluding to the question. Their final ‘cut’ involved only about 75 papers and this doesn’t trouble you? Any reasonable and the majority of unreasonable people would be stunned that there were so many papers at all in all of climate science – that was my main point.
Here is a test of your sincerity: you argue against every finding by a sceptical author on the subject of warming and yet you defend everything that comes out of the vaunted consensus. What are the chances of that being right?
Please Mods, my comment is thoughtful and clean and it will just get buried under additional comments when it appears.
Reproducibility has long been a (supposed) pillar of the scientific method. So has falsifiability, as a matter of fact. While we’re at it, asking authors to set out the falsifiability conditions for empirical claims could be healthy.
Seaice1, “To my mind the refusal to allow the BBC entry reflects badly on Trump, not the BBC. The BBC may not be free from bias, but they are certainly not “fake news” peddlars.”
To my mind, the BBC are making up science. Highly qualified Scientists with similar thoughts on CAGW to myself are not permitted to speak on the BBC. Meantime, BBC Radio 4 broadcasts the Bishop of Stafford, claiming that Scientists with views similar to mine, “Are like Fritzl.” When I wrote to the Bishop of Stafford and requested a public debate with him concerning CAGW, I did not rceive a reply.
In this case Trump is right. The BBC has no impartiality and I resent the Government taking my hard earned money to support their propaganda machine.
Trump can’t drain the BBC swamp. That is up to the UK.
Andrew, which news organisations do you think are reliable?
Doy.
Why now and not 8 or 10 years ago when the basis of climate scam was being formulated? This is an overdue catastrophe for climate science, climate psychology, and climate political science.
” Papers already have statistical analysis showing that the result was reproducible by the original researcher. ” –seaice1
Utter nonsense.
I am no talking about climate science in this case, just papers generally. They will usually have a number of replicates and a measure of the precision (say n=5, coefficient of variation = 4% or something like that). This shows that the original researcher was able to reproduce his own results.
Goal-posts moving alert!!!
Not at all, I was making a general point about science papers in an attempt to discuss an important issue without the emotional baggage of climate.
Just a suggestion.
For any scientific work done for the government, all data, software, notes, etc. are placed in an archive completely open to anyone wishing to look at the basis of a study.
BTW, you don’t get paid for your work until everything is placed in the archive.
If it is a multi year study, place what you have at the time to receive your pro-rated grant amount.
If you work for a government agency, you place all your work in the archive on a semi-annual basis.
We seen many examples of individuals on this website able to critically exam several papers advocating different hypotheses with only the data supplied (sometimes unwillingly) testing the hypothesis advanced.
Imagine how much easier it would be to exam the basis of any report if everyone could exam the work.
In the end the only people who have refused data and code and when I requested it were skeptics
Nah Mosher until you have proven you can tell hot from colder your claim is worthless without proof. And if you can’t provide it then the claim’s based on your reputation which is that of a truth commode. Whatever goes into you, feces comes out.
The guy who released the climategate emails ADDRESSED YOU PERSONALLY for being such a SCIENTIFICALLY IGNORANT dunce.
Your word really is worse than worthless. If you said it then it means the opposite’s probably true.
You’re not escaping it, along with all the other kooks who endorsed the fraud.
Only on your planet, Mosher.
Jones not only didn’t provide data but claimed he couldn’t find the original data behind HadCRU’s alleged temperature reconstruction. Maybe he doesn’t count on Planet Mosh because you yourself didn’t ask him for them.
Steven Mosher makes a point from his own experience. E Mendes calls him a liar. Is that within site policy?
What are the chances of reading the next article in this ‘esteemed’ periodical NOT containing the words ‘may’, ‘might’, ‘possibly’, ‘suggests’, ‘perhaps’, ‘indicative’ ‘climate change’, ‘global warming’, and ‘man made/anthropogenic’ etc in the same sentence? Slim to no chance I would suggest. As for ‘natural variability’. Well, dealing properly with that baby will take a hell of an editorial mind shift. Still, overhauling reproducicity and pal review is a start, if they are serious…….Are they?
all science is conditional.. never settled.. if they dont say “may” grab your wallet
Reproducible and robust are two different vastly separate processes in research. The most important thing for readers to take away from this post is that reproducible science does not produce robust results. A lie can be reproduced.
“Reproducible and robust are two different vastly separate processes in research.”
=======
Adjectives aren’t processes.
I suppose you meant “reproduction” and “robustification”. 😉
Off-the-clock-close-enough…-ing…-tion…-ly…whatever.
What is the safeguard against pal reproduction, such as Wahl and Ammann’s garbage?
Sounds like this is just another “trust us” stamp.
Why Most Published Research Findings Are False
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Can we also get a statistician to check papers where relevant?
+1
So, does this mean that HadCRU’s temperature “data” are no longer allowed in Nature, since the original raw data have been “lost”?
Climate science typically used 95℅ as it’s significance level, so right out of the gate at least 1 in 20 papers are dead wrong. Now if 20 people try and replicate the result, 19 will fail but 1 will succeed by chance. Now if only that 1 positive replication is published, you have a replicated paper with a false result.
Yep. It’s easy to replicate false findings. It’s harder to put them to a robustness test. Mann’s removal of the medieval warm period can be replicated. However it has failed investigations of robustness. It did occur and could easily occur again without the slightest notice of the presence or not of cars on the road.
https://www.researchgate.net/profile/Jens_Zinke/publication/261404049_Inter-hemispheric_temperature_variability_over_the_last_millennium/links/00b7d534390bb2cdf2000000.pdf
Wrong- this one has Robustness in the title as a clue.
“Robustness of the Mann, Bradley, Hughes reconstruction
of Northern Hemisphere surface temperatures:
Examination of criticisms based on the nature and
processing of proxy climate evidence”
Eugene R. Wahl · Caspar M. Ammann
http://nldr.library.ucar.edu/repository/assets/osgc/OSGC-000-000-011-900.pdf
All climate change is regional in character and has strong chaotic behavior. That includes warming, cooling, storminess, drought, etc. Averaging and cherry picking temperature proxies and records is the least effective method of detecting weather pattern regime shifts such as the medieval warm period. To wit, I am impressed with the diverse outcomes of regional focused papers that describe probable conditions of the Eemian period. That warming period displayed itself in many interesting ways on a regional basis, some of which is surprising.
It is not argued that the medieval warm period did not occur. It is not argued that some proxies show higher temperatures than today, on a regional basis. It is also proposed that if it was warmer in one regional area, it was likely and reasonably colder in another. The jet streams do that you know. That is not contradictory evidence. In fact it may end up being confirming evidence as more is learned about the regime shift that was likely in place to bring about such a period.
So in summary, your paper does nothing toward elimination the medieval period. It only served to examine Mann’s statistical methods. That is not an examination of whether or not there was a medieval warm period of significance. Replication or examination of methods is not a measure of medieval warming period robustness. The totality of regional responses is. Which is why the MWP remains identified as a significant event among climate scientists even after the date of the paper you link to.
So Nature will no longer accept papers that had pixie dust sprinkled on the keyboard that produced the paper?
That’s good. Should have happened sooner.
Any paper that states in the beginning or ending that climate change is a problem without further evidence on how and why, should be examined.
Climate and its changes is so vast an area, that you have to specify what you mean if it should have any relevance. You could just as well have said weather change is a problem. And who says it is a problem?