Secret Science Under Attack — Part 3

Opinion by Kip Hansen — 2 February 2020

The attacks continue in the Journals of Science against the EPA’s proposed “Strengthening Transparency in Regulatory Science” rule — commonly referred to as the Secret Science rule.

[WARNING:  This is a long Opinion Piece — only those particularly interested in this policy issue should invest the time to read it.  Others can click back to the home page and select another posting. — kh ]

The latest salvo comes from two scientists.  They are:  David B. Allison, PhD, Indiana University School of Public Health–Bloomington and Harvey V. Fineberg, MD, PhD of the Gordon and Betty Moore Foundation

David Allison is known as a long-term critic of faddish obesity research and “in 2008, Allison resigned as president-elect of the Obesity Society after signing an affidavit (expert report) stating that there was insufficient scientific evidence available to determine whether a proposed a law to require calorie counts to be listed on restaurant menus would be effective in reducing obesity levels.” [ source ] and “has been described as one of the leading skeptics regarding commonly issued nutrition advice”.  [ source ]   Further, “the National Institutes of Health is currently funding Allison to explore statistical tools to improve research reproducibility, replicability, and generalizability so as to contribute broadly to fostering fundamental creative discoveries, innovative research strategies, and promoting the highest level of scientific integrity in the conduct of science.^”  [ source ].

Harvey V. Fineberg is the president of the Gordon and Betty Moore Foundation.  Moore was a co-founder of INTEL.  Fineberg was also previously a Dean of the Harvard School of Public Health (now Harvard Chan School of Public Health). He has been involved with the National Academies of Science committee on reproducibility and replicability in scientific research.

These credentials are very solid for speaking up on the subject of what science should be used to set public policy. Both of these men have been active in combating what is known today as the Replication Crisis.  This crisis in science is not a figment of the imaginations of a few skeptics and science-deniers.  It is a very mainstream problem, both specific science fields, like psychology, and for science in general.

You would think then that both of these scientists would support the “Strengthening Transparency in Regulatory Science” rule whose purpose is to:

“EPA should ensure that the data and models underlying scientific studies that are pivotal to the regulatory action are available for review and reanalysis. The “Strengthening Transparency in Regulatory Science” rulemaking is designed to increase transparency in the preparation, identification and use of science in rulemaking. When final, this action will ensure that the regulatory science underlying EPA’s actions are made available in a manner sufficient for independent validation.“ …. “…the science transparency rule will ensure that all important studies underlying significant regulatory actions at the EPA, regardless of their source, are subject to a transparent review by qualified scientists.”

Yet, we find that Allison and Fineberg have joined forces, as co-authors, of two “shotgun” opinion/editorial pieces (simultaneous pieces in different journals) each being lead author on one and co-author on the other.

EPA’s proposed transparency rule: Factors to consider,many; planets to live on, one — by David B. Allison and Harvey V. Fineberg [published in PNAS as an Editorial]

and

The Use and Misuse of Transparency in Research — Science and Rulemaking at the Environmental Protection Agency — by Harvey V. Fineberg and David B. Allison [published in JAMA as a Viewpoint/Opinion piece]

Each of these articles uses rather peculiar logic to oppose the EPA’s proposed rule that would require that pivotal studies upon which public policy is based should be transparent and have their data and  methods made available for independent validation by qualified scientists. 

 Allison, in his PNAS piece, uses the following:

“All other things being equal, absent the opportunity to fully inspect the data, methods, and logical connections of a study, scientists are less able to judge the validity of conclusions or the truth of propositions drawn from a study. …. Generating and evaluating the scientific evidence to form conclusions about the truth of a proposition is fundamental to the work of science. Notably, the Environmental Protection Agency (EPA) is not a scientist; it is a regulatory agency. EPA employs scientists and uses science to aid in its mission, but its primary mission is regulation and the protection of the environment and the public health, rather than simply drawing scientific conclusions. Regulatory decisions can, are, and should be informed by science. But science alone is not dispositive of regulatory decisions, and one should not conflate the role of scientists qua scientists with the role of scientists working in a regulatory process. Scientists working in a regulatory process must utilize the best information available to fulfill their charge of making decisions that benefit society, often under conditions of uncertainty.”

Somehow, scientists working at the EPA, “working in a regulatory process”, are not qualified “to form conclusions about the truth of a proposition” presented in scientific research – even though that is the very purpose of allowing scientists the “the opportunity to fully inspect the data, methods, and logical connections of a study”.   Still Allison says that  “scientists working in a regulatory process must utilize the best information available” — but states that they may, or even should,  be denied access to the evidence necessary to form conclusions about validity of findings presented to them.

Allison seems to intentionally obfuscate the issue by conflating replication with EPA’s stated desire to be able to have “independent validation”. 

“Certainly, reproducibility and replicability play an important role in achieving rigor and transparency, and for some lines of scientific inquiry, replication is one way to gain confidence in scientific knowledge. For other lines of inquiry, however, direct replications may be impossible due to the characteristics of the phenomena being studied. The robustness of science is less well represented by the replications between two individual studies [or reproduction of one or more studies] than by a more holistic web of knowledge reinforced through multiple lines of examination and inquiry”

Independent Validation of a study’s findings, or of the findings of a series of studies, is not the same thing as replication and not the same as reproducibility.

The National Academies offer these definitions:

Reproducibility means computational reproducibility—obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis.

Replicability means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. 

Reproducibility and Replication are two similar terms that are often mistakenly conflated.   It is difficult to talk about these terms without exact definitions — but many readers may, like myself, not quite agree with the above definitions.  Neither covers the most basic action considered to be replicating a study — doing exactly the same thing over again and seeing if the results are the same — not just the computational steps, but everything.  We see complaints when there are replication failures (when studies are found not to reproduce/replicate, as in the Ocean acidification fish behavior studies) that the reproduction/replication team did not follow exactly exactly the same procedures — “that’s why it failed to replicate”, they say.

The National Academies has produced this poster on the subject:

[ click to download full-sized .pdf ]

I generally think of this whole topic as two separate scientific activities:  one is an attempt recreate exactly all the steps and procedures of a previously reported experiment to see if one gets the same results and the other is doing a similar experiment(s) meant to elucidate the same natural phenomena [phenomenon] — to see if one can ascertain if a reported effect really exists in the real world.

I fear when Allison speaks of “a more holistic web of knowledge reinforced through multiple lines of examination and inquiry” he means “lots of studies done by groups of like-thinking researchers who all support the same bias.”  Ioannidis referred to this as “research findings may often be simply accurate measures of the prevailing bias.”

Allison quotes “Importantly, confidence in results can be obtained in other ways. These include peer review, replication [defined as “obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data”], demonstration of generalizability, and yet other procedures.”   I am afraid that Allison slips off the tracks when he makes the claim that Peer Review can give us confidence in results.  Allison is active in the Replication Crisis arena and knows full well that despite peer review, most research findings, across nearly all fields,  are probably false, despite the peer review process of all important journals.   Note that Ioannidis is not just talking about epidemiological, psychology or nutritional research, he is talking medical and clinical research as well.

Recent findings in the Ocean Acidification field demonstrate Allison’s hope that “confidence in results can be obtained” by peer review and “consistent results across studies” is a false hope.  Psychology discovered this sad-but-true fact years ago.  What is required to obtain true confidence in results is Independent Validation.

The tendency in many fields of research that are “popular”, such as nutritional epidemiology,  environmental health issues and climate science,  is to claim that if one has many studies that all show the same small and tentative associational results that this then represents “proof” that has been “reinforced through multiple lines of examination and inquiry”.   That concept is not true and is not scientific.  Having lots of studies with very small associational results mean that there might be something that deserves further rigorous study — and this comes with the caveat that there might well be nothing there at all. 

Allison actually tries to make the point that EPA not only can,  but should,  make policy based on science that is not definitive.

“Given this, many studies which might be uniquely informative and offer sound scientific evidence on which to base policy decisions might be excluded from the process. Again, this might be fine for a scientist qua scientist drawing conclusions about the truth of a proposition who might justifiably state that he or she is unwilling to declare a proposition to be demonstrated unless some rigorous standard of science has been met, but it is not appropriate for a decision-making entity which has the goal of making prudent decisions. Such a decision-making entity should base its decisions on the best available information, even when that best available information may not support definitive scientific conclusions.”

In other words, Allison promotes making regulations based not only on strong evidence, but on prevailing bias.

Allison raises the false flag of “patient confidentiality” as almost all detractors of the EPA proposed rule do, without any specific-to-EPA examples — always vague hand-waving about some other science somewhere else.   There is a valid question regarding patient confidentiality — but it does not exclude serious scientists from reviewing data (for which permission has already been granted by the study subjects) for the purposes of validation.  Obviously, these subsequent scientists would be under the same obligation of confidentiality as the original researchers.  It has never been EPA’s position that any and all data from all studies must be made available to the general public.

After arguing that: “Should EPA trust all reported conclusions from scientific papers without probing further and, where reasonable, requiring that the data and studies be made reproducible and transparent? In our opinion, no.” he then falls back on “As we have argued here and elsewhere, reproducibility and public availability of data, while valuable, are neither necessary nor sufficient markers of the soundness of science and are not the only indicators of the soundness of science. Therefore, relying only on reproducible studies and publicly available data cannot be taken as equivalent to using the best available science, and adopting such a restrictive policy would be contrary to EPA’s responsibility.”

Allison raises the issue of trust — and concludes that EPA and its administrators cannot be trusted to identify the best available science under the proposed new rule.  Yet, he apparently trusts them to have done so in the past where they have selected what science upon which to base regulations.  It is apparent that Allison fears that some preferred science in the past is in danger of being invalidated under the proposed rule.   Which science?  Secret Science — science used to formulate regulations in the past for which the data and methods are still being hidden from EPA and from independent review by other qualified scientists.

In all of Allison’s long and wandering dissertation about trust and replication, he neglects the simple and obvious fact that EPA has always been trusted to have and to hold the scientific data necessary to fulfill its functions.  Someone has always had to decide what science is valid and applicable to every regulatory decision.  The fear that bias might be injected into the process does not only apply to the future under the proposed rule, but has always applied equally in the past — past biases may have driven policy making.    This is the point of requiring that data and methods be available for independent validation by disinterested, qualified scientists.

I have demonstrated, in my recent essay, Secret Science Under Attack — Part 2, how weak and uncertain the science of the Harvard Six Cities Study and the American Cancer Society study known as ACS II was.  James Enstrom, of UCLA and the Scientific Integrity Institute, in a recent letter to Allison, includes a link to his 22-page document in support of the EPA’s proposed “Strengthening Transparency in Regulatory Science” rule, which includes a massive amount of information concerning the weaknesses and faults of the past EPA’s scientific justification of air pollution regulations, especially those concerning PM2.5.   Readers with deep interest in the subject should download Enstrom’s document and use the links and references as a topical guide to the literature that does not support the current PM2.5 regulations.

The JAMA opinion piece by Harvey Fineberg is easier to discuss, though just as logically odd.  Fineberg freely admits:

“Transparency in science is a laudable goal. By describing with sufficient clarity, detail, and completeness the methods they use, and by making available the raw data that underlie their analyses, scientists can help ensure the reproducibility of their results and thus increase the trustworthiness of their findings and conclusions.”

but he then falls back on one of the same arguments presented by Allison:

“At the same time, transparency is not in and of itself a definitive standard for the usefulness of science in policy making.”

And, that is true — simply because some piece of science is transparent and its data freely available, does not mean that it is suitable as a basis for policy making.  However, if it is transparent and data is available, then other scientists, both at EPA and elsewhere, can easily determine if it is suitable, or not.

Raising the same false flag of “patient confidentiality”, Fineberg launches into the oh-so-typical spiel of:

While sometimes falling short in its use of science,  the EPA has traditionally strived to base regulations on the best available scientific evidence. For example, in 1997 the EPA adopted new air pollution regulations based mainly on 2 large epidemiological studies. The Harvard Six Cities study had begun in the 1970s to monitor the health of more than 8000 adults and children in 6 cities over 15 years while simultaneously tracking levels of air pollution, mainly related to burning of fossil fuels to generate electricity. Published in December 1993, the study found a strong gradient of mortality associated with increasing levels of airborne small particulates (diameter <2.5 μm).   A second, independent study by the American Cancer Society followed 500 000 people in 154 cities for 8 years and reached similar conclusions in 1995.

Out of the thousands of research studies that the EPA must have used over the last 25 years to support regulations of differing kinds, Fineberg, like nearly all other detractors of the EPA proposed Secret Science rule, is concerned about ONLY TWO studies — the Harvard Six Cities study and the American Cancer Society study, known as CPS II.  These two studies are the scientific backbone used to support the EPAs current regulation on PM2.5 air pollution and their [I believe “entirely unsupported”] claims of “thousands of lives lost very year”.

The National Academies poster on Reproducibilty and Replicability above [repeat link] does not call for use of non-definitive science — science which does not really answer a question — but warns:

“One type of scientific research tool, statistical inference, has an outsized role in replicability discussions due to the frequent misuse of statistics and the use of a p-value threshold for determining “statistical significance.” Biases in published research can occur due to the excess reliance on and misunderstanding of statistical significance.”

and

“Beyond reproducibility and replicability, systematic reviews and syntheses of scientific evidence are among the important ways to gain confidence in scientific results.”

Neither opinion piece by Allison/Fineberg and Fineberg/Allison makes a strong case for opposing the EPA’s proposed “Strengthening Transparency in Regulatory Science” rule.

Bottom Line:

If we want governmental regulations based on strong science, then that science must be fully available for review and validation by qualified, disinterested (not involved in the policy fight) scientists.  That’s transparency.

Yes, we do want EPA’s internal scientists to be able to review, re-analyze and validate any science that is going to be used to make policy and regulations.

Yes, we do want other qualified statisticians and epidemiologists and clinical researchers to be able to see all the pertinent data, all the methods, all the computer code, all the statistical assumptions — everything.  We want them to find the strengths and weaknesses of research results so that follow-up research can be done and prevent costly and destructive policy from being based on research fads prevailing biases in fields of research.

No, I do not want to take your word for it.   I don’t care how many letters you have behind your name or what school you went to, who your mentor was or who your research pals are.   I don’t care how many important names you can get to sign on to your paper as co-authors.  If your research is important enough to have weight with policy makers and regulators, then I want that research independently validated.  I don’t want to personally pay, nor do I want society to pay, for your hubris and pride.

# # # # #

Author’s Comment:

One way in today’s world to tell if something is a good idea, is to gauges how much certain segments of society protest against the idea.  The better the idea, the more the outcry.

EPA’s Secret Science rule is a fine example of this.   In calling for transparency, a broad cadre of scientists, researchers and journal editors have simultaneously realized, I think, that Independent Validation based on all-the-data-and-code-and-methods transparency will reveal that their favorite bedrock environmental studies are like the Emperor’s-New-Clothes:  not much there.

The National Academies poster on reproducibility [pdf] contains this odd [to me] point as it #6:  “Not all studies can be replicated. While scientists are able to test for replicability of most studies, it is impossible to do so for studies of ephemeral phenomena.”   They only have ten points on the poster and they include this one?  Readers, please, can you give important examples of this principle?  You can buy the NAS book “Reproducibility and Replicability in Science as an eBook for about US$ 55  — it may explain ephemeral phenomena.

Addressing your comment to “Kip…” will bring it to my attention.

# # # # #