Lance Wallace writes
Science magazine has instituted a new policy requiring authors of preclinical studies to state their statistical plans (sample size estimation, treatment of outliers, etc.). See editorial by the new Editor in chief, Marcia McNutt (p. 229, volume 343, 17 Jan 2014).
This reads as though it were written by McIntyre, Montford, Wegman….
“Because reviewers who are chosen for their expertise in subject matter may not be authorities in statistics as well, statistical errors in manuscripts may slip through. For that reason…we are adding new members to our Board of Reviewing Editors from the statistical community to ensure that manuscripts receive appropriate scrutiny in their methods of data analysis.”
That is, an audit!
Take a bow, gentlemen!
=============================================================
The article is here: http://www.sciencemagazinedigital.org/sciencemagazine/17_january_2014?pg=9#pg9
Now if we can just get publications like JGR to make the same demands of their authors, we’ll really be getting somewhere – Anthony
Bill Marsh,
Good on your daughter. When I was a reviewer for an NIH journal, I also looked at cited refs to see if they really said what was claimed in the submitted paper and was appalled at how things were twisted. I’ve also witnessed games editors play to bias and favor authors with whom they agree. It’s just appalling, the whole peer review system. Way too subject to gaming and is really meaningless.
I, personally, think it ought to be junked in favor of open publication and a renewed appreciation that replication, not publication, is what is important. Until it’s been replicated by others outside the initial circle, it’s just assertion, pure and simple, no matter how pompous and prestigious (or even rigorous) the reviewers.
In my opinion, all climate science should be published in the Journal of Irreproducible Results. The most prestigious scientific journal in the world.
Only time will tell …
When I was an educator I had ‘Science’ delivered as one of the school magazines. I dropped it in the 90’s for various increasingly lack of quality reasons (as I recall).
I expect the growing disenchantment from many sources (such as WUWT) has been a stimulus for this change.
Lets hope this change is good kindling!
George E. Smith:
Nearly snorted coffee through my nose over that one.
I write papers using statistics. There is endless numbers of ways — all of them at least somewhat justifiable — to statistically analyze a data set and reach conclusions about the underlyng population.
Will you be using parametric or non-parametric analysis? Bayesian or frequentists? Robust or non-robust? What tests will you invoke (there are endless numbers of them)? What numerical algorithms will you use, given that it’s not always easy to compute and different algorithms can give different answers? How about trying eigen methods like Principal Component or Singular Spectrum analysis to extract the dominant modes? The variations and options go on and on.
I may have mentioned this before. But when I was getting my MS, with a “heat transfer” emphasis, one assignment was to take one of about 30 sample “journal papers” which the professor handed out. NOW I know he was VERY selective on most of the papers. The one I took used a “variational method” to solve a transient conduction heat transfer problem for an arbitrary geometry. ONE example was given. The “basic” math was laid out in the paper. (About 6 pages long.) I showed up the night of our presentations (4 hour classes, once a week, extension grad school, U of Lincoln, NE) with 40 pages of “overheads” I took almost an hour, and expanded all the “math” into USABLE equations, with numbers. I DID NOT DO AN APPLIED EXAMPLE, but showed where the author got all his “numerical results”. (6 by 6 matrixes on a TI-59(??) helped.
I thought Dr. Lu would “ding” me for not doing a unique application. The mathematics were so compact, and took SO MUCH EXPOSITION (multiple uses of the fact that log (1.0000XXX) = .0000XXX pretty much, that sin(small angle in radians) = small angle in radians…and several other tricks) that it was enough in two weeks to work that out. Probably 20 to 30 man hours (I was young and foolish, single, etc.) ….Dr. Lu gave me an A for that presentation. AND he gave a 10 minute talk, pointing out: “Journal Papers, because of limited space…have to condense much information, and it is ALWAYS a struggle to apply the mathematics and figure out how the author did HIS work. Sometimes you have to contact them, get a copy of a graduate student’s thesis, or a textbook they are writing, and so on..”
Well, that was THEN, this is NOW. NOW the researchers CAN provide the DATA, the MATH and the CODEs, and well the DARNED SHOULD, if it’s PUBLIC MONEY????!!!!!!! If it’s PURE SCIENCE…If they work for a PUBLIC INSTITUTION (University, not private one…U of Chicago, Boston U, Brigahm Young, for example…could demand some OWNERSHIP and rightly so..) To quote Yoda: “Dragged, Kicking, SCREAMING they will be, 21st Century, if LIVE to see, they will..”
This is not on topic, but the bankruptcy of the German wind farm company Prokon is worthy of a thread.
[Reply: These suggestions should be posted in Tips & Notes. ~mod.]
In her editorial, Marcia McNutt writes, “For preclinical studies … we will be adopting the recommendation of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency.” However, she adds later that “we are adding new members to our Board of Reviewing Editors from the statistics community to ensure that manuscripts receive appropriate scrutiny in their methods of data analysis.”
So, as several earlier commenters have mentioned, the periodical Science will only be applying strict standards to preclinical medical studies. And I would argue that the inclusion of a few statisticians* in the Board of Reviewing Editors will likely have little impact on the quality of papers published outside the medical discipline. Also, nothing in McNutt’s editorial pertains directly on phenomenological or physics-based models. . . so Science’s new policy will do little, if anything, to improve the state of model validation or associated uncertainty quantification for papers that address climatology or other non-medical disciplines.
Dan
*Note: It has been my experience that statisticians who lack domain knowledge can actually be more harmful than helpful in the quest for truth.
One choice I have had, beginning as recently as two years ago, when I peer-review a paper, has been to note whether a statistician should be brought in to examine the statistical methods. So, one way to address this is to allow the peer-reviewers to nominate those papers where the stats knowledge of content experts is not sufficient.
So, a manuscript reporting the results of a trial is supposed to include the sample size analysis? This is often quite wordy, since it takes a lot to explain the argument for your sample size analysis plan, and explain the outcome from which you are extrapolating your expected clinically meaningful difference.
All fine and dandy. But I’ll believe it when I see its effects.
What if “climate science is special?”
As in “specially abled.”
Strong on the moneyed side, weak on the sciency side…
I’m skeptical of the motives of Science Magazine including the current Editor-n-Chief, Board and the Reviewers.
The posted change makes me think a (another) lawsuit (perhaps like the 2004 one or the current complaint against the Nobel Assembly at the Karolinska Institute in Sweden, or the lawsuit regarding a retraction of a paper in ‘Food and Chemical Toxicology – Elsevier’) is playing in the background and “money” i.e. cash, patents, IP and corporate donors to the clinic or institute are involved.
As for AGU, i.e. GRL and JGR et al., with Mann and Trenberth et al. as the “go to” reviewers and controlling the AGU (new fee increases to fund new propaganda awards, Climate Researchers Legal Defense Fund, bogus “journals” like ‘Earth’s Future’ and the shenanigans [Emperor Hansen Has A Cold] at the Fall Meeting [or was it tickets to the Seahawks vs 49’s game that moved the ice-breaker to Monday] and the HQ in DC with Wiley and Co. [for profit publisher needs more profit]), why bother submitting a paper.
Jeez and it is only Sunday.
Statistics in all the variations and esoteric maths can be used to prove just about anything you want. Truth lays only in original unfudged data, one only has to look at all the temperature series that have been manipulated to prove global warming.
Now with the eyes of the internet firmly looking over their shoulder, statistical manipulation and fudges are becoming harder. Now even with their best fudging the the temperature graphs show cooling. Warming through data fudging has now painted these keepers into a corner, as people are now finding old data series that show the extent of the manipulation.
Statistics is proving to be the weapon of choice of the charlatan. The chickens are coming home to roost.
One is speed cameras, and the other is drivers ed. Both can lower road deaths,
And even if they don’t lower deaths they can raise revenue.
There are so many retractions and irreproducible claims that are showing up in medical research. Read New Truths That Only One Can See http://www.nytimes.com/2014/01/21/science/new-truths-that-only-one-can-see.html?_r=1
One can only imagine how wrong many climate hypotheses are that cant be reproduced nor tested for decades. LIke the failure to predict growing Antarctic sea ice, instead of acknowledging their failures, they devise a new models to explain the failure away.
@Bill Marsh:
“She…..commented that most of the studies she reviews are atrocious.”
Many years ago I taught a graduate class in basic research. Each of the students was required to read 5 journal articles of their choosing and to evaluate the articles according to the information presented in class about research design, randomization, selection of n, control of extraneous variables, etc., etc. They then had to decide, according to their criteria, whether each article was worthy of publication. I expected that about half would be thrown out, but was shocked to discover along with the students that 90 percent should have been rejected.
That was the state of science several years ago. I’d hate to see what the same activity would now reveal.
What a bizarre confession from Science. If the new policy requires “reproducibility” and this will be out of reach for “expert reviewers”, what criteria have these “experts” been using all along when the authors could hide methodology and data at will?
In other words, if reviewers are incompetent with data/methodology, what on God’s green earth are they now? Mystics? Clairvoyants?
I am rereading Galileo’s Revenge: Junk Science In The Courtroom by Peter Huber. This is
outstanding news. Most of the “Expert witnesses” who make junk science never publish
their works, and are later proven frauds (See Bendectin case.)
Or, they only share their works with other like minded true believers (like AGW supporters.)
Even if Science Magazine is doing the right thing here, I noticed about 10 years ago that
Scientific American had already drank the AGW Kool-Aid. I loved this magazine as a kid,
If we cannot force Michael Mann to publish his works, maybe Mark Steyn’s defense lawyers
can force through discovery what used to be done by real scientists in the peer review
process.
My fear is that as long as Scientific American has been compromised, and that if other
journals are not as discriminating as Science, nothing will change.
I think that people may not have appreciated the significance of what Rabbit said (January 26, 2014 at 12:09 pm). It is nearly always possible to choose a statistical method which will give the result you want – if you torture the data enough (see Darrel Huff’s ‘How to Lie with Statistics’)
For example, “Eating chocolate is good for your teeth”
Who said so? — Independent Research
Who Commissioned it? – The British Sugar Corporation.
What did they find? – Children who eat chocolate have fewer cavities.
How many juries did you have to poll before you found that one?
Although it is not exactly what Rabbit meant, you can see that the Global Warming narrative is supported by something like this:
In the beginning, I show that temperatures are increasing year by year. No doubt about it, a global warming catastrophe looms. But around the turn of the century warming stops. So instead, I show temperatures filtered by a 21 year wide Gaussian function which has the effect of projecting the warming of the 1990s into the 21st century. Thus the running mean still gives the impression of increasing global temperatures (see the Met.Office charts) . – but eventually even that starts to show no warming (because there has been no warming for more than half of the filter width) So instead I show a chart of temperature levels for each decade, each decade being warmer than the other. This manages to give the impression that temperatures are still rising even though they are not ( see latest IPPC report for this latest piece of sophistry)
It is a small step forward….. One can hope that the rest of Science submittals will be subjected to the same criteria in the coming months. It is certainly worth sending them a short note to laud this step forward to more transparency and encourage them to apply it universally!
Be sure to reference the editorial Reproducibility by the new Editor in chief, Marcia McNutt (p. 229, volume 343, 17 Jan 2014)
http://www.sciencemag.org/feedback
“This reads as though it were written by McIntyre, Montford, Wegman….” etc.
==========
It was written and promoted by those above. Apparently it was read by the new Editor in chief, Marcia McNutt
There’s no way the Team will permit such apostasy to stand.
The ONLY question remaining here is whether or not top-level science even recognizes when it might very well exist……perhaps at the near end of the most recent interglacial? If not, please provide a cogent explanation as to how and why the Holocene may extend beyond about half a precession cycle. With or without anthropogenic influence. This is all I ask……
Otherwise this is all a silly buggers game, isn’t it?
Great news!
Now to next get JIR on board with proper article review processes.
I can’t believe what they print!
(The Journal of Irreproducible Results)
When Standard & Poor’s first started evaluating railroad bonds in the late 1860s, proprietors from Commodore Vanderbilt on down simply refused to supply valid data. Within 18 – 24 months, however, no operating railroad could maintain share prices without supplying verifiable statistics.
As Erie Gang scandals surfaced –Gould, Fisk, and Drew were printing railroad bonds in Wall Street basements– S&P’s nascent discipline of “securities analysis” spread to other industries, making honest women of many a brokerage house.
Why should Erie Gang successors –credentialed academic hustlers calling themselves “scientists”– be any different?