Analyzing Studies

Just Facts

For issues that are really important, you need to learn how to separate credible studies from unreliable ones.

Script / Documentation

---

Do you want to live in a world of self-delusion where you make decisions that hurt yourself and others? Have we got a plan for you!

Step 1: Blindly accept any study that confirms your preexisting views.

Step 2: Reject any study that challenges what you already believe.

Or you can watch and apply the new “Analyzing Studies” video series from Just Facts Academy to help you separate the credible studies from the trashy ones.

We know that no one has the time to truly analyze every study they hear about. But for issues that are really important, you need to know how to sort through the junk studies that infest your media feeds.

So let’s build on the Standards of Credibility and lift the curtain on studies.

First and foremost, don’t be fooled by trite propaganda masquerading as studies.

Journalists,[1] commentators,[2] and so-called fact-checkers[3] often treat the opinions of selected “experts” as if they were scientific studies, and politicians use the phrase “science says” like it magically turns mere claims into facts.[4]

Such misuse of the word “science” has been a longstanding problem. As the renowned physicist Richard Feynman remarked more than half a century ago, “When someone says, ‘Science teaches such and such,’ he is using the word incorrectly.” When people discuss actual science, they don’t “say science has shown”—but “this experiment, this effect, has shown.”[5]

Another good way to be misled is to place too much trust in your own common sense. That’s why Einstein defined common sense as a collection of prejudices acquired by age eighteen.[6]

Another trap to avoid is basing your views on personal anecdotes. People sometimes defy the odds, like your great uncle who lived to the age of 95 even though he smoked like he was on fire and drank like he was trying to put it out.

And if you think you can understand the issue of climate change by poking your head outside your window,[7] you might want to consider there are “huge differences” in climate trends around the world,[8] and the entire United States covers only 1.9% of the world’s surface area.[9]

Likewise, don’t let your go-to news source fool you. There are nearly 8 billion people in the world,[10] and the stories chosen by your favorite media outlets may not accurately represent the big picture.[11] As one academic book explains:

People like stories—math and numbers and statistics not so much. Our preference for stories over numbers means that we can often be convinced of something by a string of stories, even if they aren’t representative of the whole.[12]

Another great way to be a sucker is to believe that the peer-review process used by most scholarly journals “ensures” studies “are sound.”[13] This naive notion is exploded by reams of facts and candid statements from people involved in this process.[14] Let’s look at a few.

The journal Nature published a study that attempted to confirm the findings of 53 prominent peer-reviewed papers about cancer drugs. Scientists were unable to reproduce 89% of these results, despite the fact that “when findings could not be reproduced,” an “attempt was made to contact the original authors” and “repeat experiments under the authors’ direction.”[15]

The Proceedings of the National Academy of Sciences published a “detailed review“ of “2,047 biomedical and life-science research articles” that have been retracted. It found that “21% of retractions were attributable to error,” and 67% were due to misconduct.[16]

Wait a second. How can this happen? Well, one main reason is explained by Phil Hurst, a publisher for the Royal Society.[17] He wrote that:

traditional peer review is confidential, with research papers scrutinized by a small number of anonymous experts. Although publishers are vigilant, this secrecy provides the opportunity for fraud.[18]

Likewise, Austin L. Hughes, a professor of biological sciences, wrote that:

the high confidence in funding and peer-review panels should seem misplaced to anyone who has served on these panels and witnessed the extent to which preconceived notions, personal vendettas, and the like can torpedo even the best proposals.[19]

Does this sound like the kind of system you want to stake your life on?

So, how can you determine a study’s credibility? I’m glad you asked.

In the upcoming videos, we’re going to give you the tools you need. Here’s just some of the topics we’ll be covering:

• Differentiating correlation from causation
• Omitted variable bias
• Included variable bias
• Randomized controlled trials
• Observational studies
• Laboratory & simulation studies
• Confidence intervals
• Cherry-picking & overgeneralization

Don’t worry. This is not as hard as it might seem. In fact, the strongest studies are often the most straightforward.

So stay tuned to Just Facts Academy, and while you’re waiting, review our previous videos so you can research like a genius.

---

Footnotes

[1] Article: “Scientists Believe Coronavirus Originated in Wild Animal Markets — and They Want the Chinese Government to Shut Them Down.” By Sophie Lewis. CBS News, February 7, 2020. <www.cbsnews.com>

Experts say meat sold at wild animal markets in Wuhan is likely the culprit for the outbreak of the deadly coronavirus, which has so far killed more than 630 people and infected more than 31,000 globally. Similar viruses will continue to pop up regularly unless the root of the problem is addressed, scientists say.

“I want the wild animal markets closed,” infectious disease expert Dr. Ian Lipkin told CBS News on Wednesday.

Lipkin is an epidemiologist from Columbia University, currently advising authorities in China. He said the coronavirus epidemic is “very close” to becoming a pandemic. …

“We have to shut these wild animal markets,” Lipkin stressed. “If we don’t do that, we will see one of these emerging infectious diseases every couple of years.”

[2] Commentary: “Marco Rubio Demanded People Look at the Science on Abortion. So We Did.” By Philip Bump. Washington Post, May 15, 2014. <www.washingtonpost.com>

On the question of when life begins, then, the scientific experts we spoke with didn’t offer any consensus.

“Life” is something of a philosophical question, making Rubio’s dependence on a scientific argument — which, it hardly bears mentioning, is an argument about abortion — politically tricky.

NOTE: For the actual facts of this matter, see Just Facts’ research.

[3] Article: “Clinton: Undocumented Workers Pay More Than Trump in Federal Income Taxes.” By Lauren Carroll. October 27, 2016. <www.politifact.com>

There is no official count of undocumented workers who pay federal income taxes. But Clinton’s claim — that half of all undocumented workers pay federal income taxes — is an educated assumption that many experts use, said Kim Rueben, senior fellow at the Urban-Brookings Tax Policy Center.

“As far as we can vet it, it is true,” Rueben said. …

While there is no official figure, experts estimate that about half of all undocumented workers pay federal income taxes, if not more.

NOTE: For the actual facts of this matter, see Just Facts’ research.

[4] Webpage: “Environment and Climate Change.” Congressman Jared Huffman (D–CA). Accessed November 3, 2023 at <huffman.house.gov>

Science says that we need to keep fossil fuels in the ground to protect the planet for future generations, which is why I introduced the ‘Keep It in the Ground Act’ to reduce carbon emissions and break our nation’s addiction to fossil fuels by permanently barring new fossil fuel leases on all federal public lands and in federal waters.

NOTE: For the actual facts of this matter, see Just Facts’ research.

[5] Lecture: “What is Science?” By Richard Feynman. Presented at the Fifteenth Annual Meeting of the National Science Teachers Association, 1966. <www.feynman.com>

When someone says, “Science teaches such and such,” he is using the word incorrectly. Science doesn’t teach anything; experience teaches it. If they say to you, “Science has shown such and such,” you might ask, “How does science show it? How did the scientists find out? How? What? Where?”

It should not be “science has shown” but “this experiment, this effect, has shown.” And you have as much right as anyone else, upon hearing about the experiments—but be patient and listen to all the evidence—to judge whether a sensible conclusion has been arrived at.

[6] Book: The Universe and Dr. Einstein. By Lincoln Barnett. Forward written by Albert Einstein. Dover Publications, 1948.

Pages 57–58:

At first meeting these facts are difficult to digest but that is simply because classical physics assumed, unjustifiably, that an object preserves the same dimensions whether it is in motion or at rest and that a clock keeps the same rhythm in motion and at rest. Common sense dictates that this must be so. But as Einstein has pointed out, common sense is actually nothing more than a deposit of prejudices laid down in the mind prior to the age of eighteen. Every new idea one encounters in later years must combat this accretion of “self-evident” concepts. And it is because of Einstein’s unwillingness ever to accept any unproven principle as self-evident that he was able to penetrate closer to the underlying realities of nature than any scientist before him.

NOTE: The above is commonly quoted and paraphrased as, “Common sense is the collection of prejudices acquired by age eighteen.”

[7] Article: “Virginians’ Attitudes About Global Warming Hinge on Local Weather.” By Brevy Cannon. University of Virginia, October 22, 2008. <news.virginia.edu>

The survey asked Virginians to identify the primary factor underlying their beliefs about climate change. Among the 75 percent of Virginians who do believe the earth is warming, one in four cited personal experience as the top reason. The next most popular reasons were melting glaciers and polar ice (21 percent), media coverage (14 percent) and changing weather patterns or strong storms (13 percent)—another type of personal experience of the weather.

Among the 13 percent of Virginians who do not believe the Earth is warming, the top reason given was also personal experience of the weather, suggesting that weather is in the eye of the beholder.

Tied for the top answer among Virginia’s global warming disbelievers was the notion that natural patterns explain any fluctuations in temperature.

[8] Climategate Document 988466058.

From: tom crowley† <tom@…>

Subject: Re: Low Frequency signals in Proxy temperatures:

Date: Sat, 28 Apr 2001 09:54:18 -0500 …

look at the instrumental record! there are huge differences between different regions—Alaska has warmed substantially while eastern North America cooled after the 1950s. locking onto local records, no matter how beautiful, can lead to serious errors.

NOTES:

• † Crowley was a reviewer of the chapter about proxies in the 2007 IPCC report. [Curriculum Vitae: Thomas John Crowley, January 2009. <www.ae-info.org>. “Reviewer, Intergovernmental Panel for Climate Change (IPCC) Chapter on Paleoclimatology (April, 2005)”]
• For more facts about differing climate trends around the world, see Just Facts’ research on this issue.

[9] Calculated with data from:

a) Webpage: “Area Country Comparison to the World.” The World Factbook, U.S. Central Intelligence Agency. Accessed February 03, 2016 at <www.cia.gov>

b) Webpage: “State Area Measurements and Internal Point Coordinates.” U.S. Census Bureau. Accessed February 03, 2016 at <www.census.gov>

NOTE: An Excel file containing the data and calculations is available upon request.

[10] Report: “World Population Prospects 2022: Summary of Results.” United Nations, Department of Economic and Social Affairs, 2022. <desapublications.un.org>

Page 3: “The global human population will reach 8.0 billion in mid-November 2022 from an estimated 2.5 billion people in 1950, adding 1 billion people since 2010 and 2 billion since 1998.”

[11] For a prime example of how media outlets create misleading impressions by reporting selected stories, see this research from Just Facts.

[12] Book: Navigating the News: A Political Media User’s Guide. By Michael K. Baranowski. Praeger, 2013.

Pages 70–71:

People like stories—math and numbers and statistics not so much. Partly, this is because we use language all the time, and so it’s easier for us to grasp stories (at least simple stories) than it is to understand numerical data. But it’s also because numbers, by themselves, generally lack the emotional content that draws us in and helps us to remember. Our preference for stories is why both politicians and the media covering them think in terms of crafting a narrative far more often than they consider analyzing the data (at least for public consumption). …

There are often numbers behind the political stories we’re told, but very few of us bother to take a good hard look at them. One reason for this is that figuring out whether the numbers support a claim isn’t always an easy thing to do. …

What this means is that politicians and pundits can more easily mislead us because they know how difficult it can be to check the numbers. …

But outright lies are problematic because getting caught in one can lead to a lot of negative publicity. A more common way our distaste for numbers is used against us is through data manipulation. Manipulating data isn’t exactly lying—it’s more like finding ways of highlighting things that support your viewpoint. And so the worst you can usually say about someone who does it is that they’re basing their conclusions on questionable assumptions or using an inappropriate method of analysis. This isn’t exactly the sort of stirring denunciation that will grab headlines or make much of an impact on voters.

Our preference for stories over numbers means that we can often be convinced of something by a string of stories, even if they aren’t representative of the whole.

[13] Commentary: “Scott Pruitt’s Attack on Science Would Paralyze the E.P.A.” By Gina McCarthy and Janet McCabe. New York Times, March 26, 2018. <www.nytimes.com>

“Opponents of the agency and of mainstream climate science call these studies ‘secret science.’ But that’s simply not true. Peer review ensures that the analytic methodologies underlying studies funded by the agency are sound.”

[14] See Just Facts’ research on this topic here.

[15] Paper: “Drug Development: Raise Standards for Preclinical Cancer Research.” by C. Glenn Begley and Lee M. Ellis. Nature, March 28, 2012. <www.nature.com>

The scientific community assumes that the claims in a preclinical study can be taken at face value — that although there might be some errors in detail, the main message of the paper can be relied on and the data will, for the most part, stand the test of time. Unfortunately, this is not always the case. …

Over the past decade, before pursuing a particular line of research, scientists (including C.G.B.) in the haematology and oncology department at the biotechnology firm Amgen in Thousand Oaks, California, tried to confirm published findings related to that work. Fifty-three papers were deemed ‘landmark’ studies…. Nevertheless, scientific findings were confirmed in only 6 (11%) cases. …

To address these concerns, when findings could not be reproduced, an attempt was made to contact the original authors, discuss the discrepant findings, exchange reagents and repeat experiments under the authors’ direction, occasionally even in the laboratory of the original investigator. ….

Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis. More troubling, some of the research has triggered a series of clinical studies — suggesting that many patients had subjected themselves to a trial of a regimen or agent that probably wouldn’t work.

[16] Paper: “Misconduct Accounts for the Majority of Retracted Scientific Publications.” By Ferric C. Fanga, R. Grant Steen, and Arturo Casadevall. Proceedings of the National Academy of Sciences, October 16, 2012. <www.pnas.org>

A detailed review of all 2,047 biomedical and life-science research articles indexed by PubMed as retracted on May 3, 2012 revealed that only 21.3% of retractions were attributable to error. In contrast, 67.4% of retractions were attributable to misconduct, including fraud or suspected fraud (43.4%), duplicate publication (14.2%), and plagiarism (9.8%). Incomplete, uninformative or misleading retraction announcements have led to a previous underestimation of the role of fraud in the ongoing retraction epidemic. The percentage of scientific articles retracted because of fraud has increased ∼10-fold since 1975.

[17] Interview: “What Is a Publisher Now? Phil Hurst, the Royal Society.” Ingenta, September 9, 2014. <www.ingenta.com>

In the week of the ALPSP Conference in London our latest What is a publisher now? interview is with Phil Hurst, Publisher at The Royal Society. In recent years The Royal Society, which is the world’s oldest scientific publisher has made multiple innovations in the way it publishes and distributes scientific content and Phil has been at the heart of that process.

[18] Article: “Transparency in Peer Review.” By Phil Hurst. Phys.org, September 8, 2017. <phys.org>

Finally, transparency has the potential to improve the quality of research and reduce research misconduct. Traditional peer review is confidential, with research papers scrutinised by a small number of anonymous experts. Although publishers are vigilant, this secrecy provides the opportunity for fraud.

[19] Article: “The Folly of Scientism.” By Austin L. Hughes. The New Atlantis, Fall 2012. <www.thenewatlantis.com>

Austin L. Hughes is Carolina Distinguished Professor of Biological Sciences at the University of South Carolina. …

Second, the high confidence in funding and peer-review panels should seem misplaced to anyone who has served on these panels and witnessed the extent to which preconceived notions, personal vendettas, and the like can torpedo even the best proposals.