The Decline Effect – Part 2:  How Does This Happen?

Guest Essay by Kip Hansen – 29 February 2022

What exactly is the decline effect?  Is it the fact that certain scientifically discovered effects decline over time the more they are studied and researched? Almost, but not really.  The Wiki has this definition for us:

“The decline effect may occur when scientific claims receive decreasing support over time. The term was first described by parapsychologist Joseph Banks Rhine in the 1930s to describe the disappearing of extrasensory perception (ESP) of psychic experiments conducted by Rhine over the course of study or time. In its more general term, Cronbach, in his review article of science “Beyond the two disciplines of scientific psychology” [ also .pdf here ] referred to the phenomenon as “generalizations decay.”^[1] The term was once again used in a 2010 article by Jonah Lehrer published in The New Yorker.”

Some hold that the decline effect is not just a decrease of support over time but rather that it refers to a decrease in effect size over time – or, according to some, both because of one or the other.  That is, the support decreases because the effect sizes found decrease, or, because of decreasing support, reported effect sizes decrease.  The oft cited cause of the decline effect are: publication bias, citation bias, methodological bias, and investigator effects.  Part 1 of this series was an example of investigator effects.

Let’s be perfectly clear:  In no case does the decline effect refer to an actual decline in real world effects of some physical phenomena, but only to effect sizes found and/or reported in research reports over time.

One of the best discussions of the decline effect was published in The New Yorker over a decade ago.  In an article titled: “The Truth Wears Off — Is there something wrong with the scientific method?” by Jonah Lehrer.  At 2100 words, it is about a 10 minute read – and worth every minute.

Lehrer’s piece starts with this:

“The craziness of the hypothesis was the point: [Jonathan] Schooler knows that precognition [ think ESP –kh ] lacks a scientific explanation. But he wasn’t testing extrasensory powers; he was testing the decline effect. “At first, the data looked amazing, just as we’d expected,” Schooler says. “I couldn’t believe the amount of precognition we were finding. But then, as we kept on running subjects, the effect size”—a standard statistical measure—“kept on getting smaller and smaller.” The scientists eventually tested more than two thousand undergraduates. “In the end, our results looked just like Rhine’s,” Schooler said. “We found this strong paranormal effect, but it disappeared on us.”

It is worrisome that the decline effect might mean that something is wrong with the scientific method.  Some group did a careful study two years ago and found truly impressive, strong effects, but since then, additional studies have shown less and less-strong effects, putting their originally supported hypothesis into doubt.  What’s going on?

Many readers are statistically knowledgeable, and can recognize the possibility that this effect is nothing more than regression to the mean. As the experiment is repeated and more data points are collected, early statistical flukes, at first some unusually high or low scores, then as more and more results come in the average of the whole set of results tends to regress to the real statistical mean.   I won’t spend too much time on this, but this image will help with the concept:

But we have a problem with our ESP results – they were finding only high values of ESP ability, not some really high and some really low, unlike our randomly generated numbers in the graph above.   Without outliers both high and low, it is hard  to blame the ESP results on regression.   The most probable and the most usual cause of this is that the investigators — the researchers — were looking for ESP, not looking for the absence of ESP.  This is the principle that one generally finds what one is looking for, either intentionally or through some psychological effect.   

In practice, I have a relative whose family tradition favors the number 13 – a great-grandfather’s lucky number.   So my relative is surprised by how often the number 13 or combinations containing that number (313, 1313, 3131, 1:13 etc.) appear in daily life – and is absolutely convinced that it appears more often than is statistically supportable.  They see what they are looking for. Or, notice occurrences of the number because they are sensitized to do so. 

In science and research, this often is unintentional – it may result from poor study design, biased data collection methods or subjective observation biased by expectation.  Studies that depend on “eye-balled” data — a human with a stopwatch, counting the number of times a chimpanzee uses its right or left hands, how quickly a spider reacts to stimuli, etc.–  can easily go awry. These types of causes would be methodological problems. 

Or, if Jonathan Schooler is right, in a larger field, it could be caused by publication bias and particularly unpublished results.  Journals like big splashy results.  Journals don’t like null, negative or “nothing found” studies.  That means when a meta-analysis is done on published papers, it is usually the studies with big results which are found – and not many which found tiny or no effects.  These null or tiny effect papers may have appeared in journals with no real influence: non-English, small or obscure journals. Schooler points out than many studies with null findings or small effects are rejected and not published or, worse, never written by researchers who know their chances of publication are remote.

In new areas of research, scientists tend to look for findings similar to the large results that brought fame and success to those first to report the new phenomena.  When they find them, they rush to publish. On the other hand, if they don’t find big impressive results, they may think they have erred in some way  and be unwilling to buck the new trend.  Publication bias affects meta-analyses. In most fields, those that do not become politicized or subject to enforced consensus, eventually come right, more and more realistic effects are found, we see the decline effect and science moves on.

Another way in which meta-analyses can be biased is:

Citation bias

The citation or non-citation of research findings, depending on the nature and direction of the results. Authors tend to cite positive results over negative or null results, and this has been established over a broad cross section of topics. Differential citation may lead to a perception in the community that an intervention is effective when it is not, and it may lead to over-representation of positive findings in systematic reviews if those left un-cited are difficult to locate.

Selective pooling of results in a meta-analysis is a form of citation bias that is particularly insidious in its potential to influence knowledge. “ — Wiki

For citation bias and publication bias, think of the effect of that these two have on subjects whose public perception depends on meta-analysis, like the IPCC Assessment Reports or NOAA and NASA reports on sea levels, climate or extreme weather events – which use only what the authors of those reports consider “approved“ studies and “authoritative” sources.  One of the first steps of a meta-anaysis is collecting what papers to consider and, all too often, approved and authoritative in practice simply mean “agrees with us”. 

Methodological bias

This is a broad category of problems but one that has the simplest solution.  In the earliest stages of ocean acidification science, there was a lot of bias:  the largest of which was the a priori assumption that lowering pH of ocean water was bad – that it led inevitably to adverse effects.  But to prove this, as “it was so obvious”, researchers not expert in ocean water chemistry did experiments of “lowered pH’ using the easiest method, the one they learned in high school, they just added acid to sea water.  Chris Cornwall and Catriona Hurd published an important paper that set some of that nonsense right (see here and here).  Once the methods were corrected, OA experiments started to find far less adverse effects in general.  And now, in that field, with fish behavior effect papers found to be highly questionable, the field is back-burnered among the climate crisis crowd.  In defense of the OA researchers, standards were developed for proper methods of OA research and published by the European Project on OCean Acidification (EPOCA) – which produced the booklet “Guide to best practices for ocean acidification research and data reporting”. 

In other fields, methodological bias is rampant.  In the sea level field, there are still many papers that simply average tide gauge station data (tide gauges which each measure relative sea level at a single location) and then  paste the satellite sea level measurements of eustatic sea level to the tide gauge observational series of local relative sea level (a very apples, oranges and bananas fruit salad mistake) or the latest methodological madness of hybrid sea level reconstructions (which commit all of the above errors combined).

There are and have been many solutions proposed to what is becoming to be known as the Saving Science movement which includes serious detailed replication of important findings and pre-registration of studies (including hypothesis, data collection, methods, data analysis methods, everything).  Pre-registration allows peer review of the proposed study, before the effort and money are spent on a poorly conceived plan.   

In my opinion, research fields need pull their brightest minds together and layout real research goals for strengthening their foundations and pointing out where they have knowledge gaps – setting goals to replicate and verify foundational knowledge and fill in knowledge gaps.   The whole point of Saving Science is to separate out the good from the bad, the truths from the myths, foundational principles discovered from current science fads. 

In other words, we need to quit fooling around.  We need some researchers doing crazy blue sky research.  But we are in dire need of correctional science studies – science done to correct science errors of the past.  There are far too many fields like OA research, sea level research, coral reef research that have wandered off along dangerous paths to Science Nowhere, following faddish Just So story versions of reality, memes turned into Facts™, themselves created ex nihilo, to forward social and political agendas. 

# # # # #

Author’s Comment:

The decline effect doesn’t mean necessarily someone is cheating.  It can just mean that what looked important at first glance isn’t a big deal.  Unfortunately, some young researcher that makes a hit with a Big Find often wants to ride that pony to a tenured position, fortune and fame.  When subsequent researchers cannot replicate and call into question the original Big Find(s), trouble ensues.  Those trying to correct the scientific records are vilified for “attacking science” even though they themselves are scientists. 

Science can’t be self-correcting if those trying to make needed corrections are attacked for doing so. 

If making anything other than a general comment, address your comments to a specific individual by name/handle.  This makes conversations much easier to follow.  For me, start with “Kip…”

Thanks for reading.

# # # # #