Guest Essay by Kip Hansen – 29 February 2022
What exactly is the decline effect? Is it the fact that certain scientifically discovered effects decline over time the more they are studied and researched? Almost, but not really. The Wiki has this definition for us:
“The decline effect may occur when scientific claims receive decreasing support over time. The term was first described by parapsychologist Joseph Banks Rhine in the 1930s to describe the disappearing of extrasensory perception (ESP) of psychic experiments conducted by Rhine over the course of study or time. In its more general term, Cronbach, in his review article of science “Beyond the two disciplines of scientific psychology” [ also .pdf here ] referred to the phenomenon as “generalizations decay.” The term was once again used in a 2010 article by Jonah Lehrer published in The New Yorker.”
Some hold that the decline effect is not just a decrease of support over time but rather that it refers to a decrease in effect size over time – or, according to some, both because of one or the other. That is, the support decreases because the effect sizes found decrease, or, because of decreasing support, reported effect sizes decrease. The oft cited cause of the decline effect are: publication bias, citation bias, methodological bias, and investigator effects. Part 1 of this series was an example of investigator effects.
Let’s be perfectly clear: In no case does the decline effect refer to an actual decline in real world effects of some physical phenomena, but only to effect sizes found and/or reported in research reports over time.
One of the best discussions of the decline effect was published in The New Yorker over a decade ago. In an article titled: “The Truth Wears Off — Is there something wrong with the scientific method?” by Jonah Lehrer. At 2100 words, it is about a 10 minute read – and worth every minute.
Lehrer’s piece starts with this:
“The craziness of the hypothesis was the point: [Jonathan] Schooler knows that precognition [ think ESP –kh ] lacks a scientific explanation. But he wasn’t testing extrasensory powers; he was testing the decline effect. “At first, the data looked amazing, just as we’d expected,” Schooler says. “I couldn’t believe the amount of precognition we were finding. But then, as we kept on running subjects, the effect size”—a standard statistical measure—“kept on getting smaller and smaller.” The scientists eventually tested more than two thousand undergraduates. “In the end, our results looked just like Rhine’s,” Schooler said. “We found this strong paranormal effect, but it disappeared on us.”
It is worrisome that the decline effect might mean that something is wrong with the scientific method. Some group did a careful study two years ago and found truly impressive, strong effects, but since then, additional studies have shown less and less-strong effects, putting their originally supported hypothesis into doubt. What’s going on?
Many readers are statistically knowledgeable, and can recognize the possibility that this effect is nothing more than regression to the mean. As the experiment is repeated and more data points are collected, early statistical flukes, at first some unusually high or low scores, then as more and more results come in the average of the whole set of results tends to regress to the real statistical mean. I won’t spend too much time on this, but this image will help with the concept:
But we have a problem with our ESP results – they were finding only high values of ESP ability, not some really high and some really low, unlike our randomly generated numbers in the graph above. Without outliers both high and low, it is hard to blame the ESP results on regression. The most probable and the most usual cause of this is that the investigators — the researchers — were looking for ESP, not looking for the absence of ESP. This is the principle that one generally finds what one is looking for, either intentionally or through some psychological effect.
In practice, I have a relative whose family tradition favors the number 13 – a great-grandfather’s lucky number. So my relative is surprised by how often the number 13 or combinations containing that number (313, 1313, 3131, 1:13 etc.) appear in daily life – and is absolutely convinced that it appears more often than is statistically supportable. They see what they are looking for. Or, notice occurrences of the number because they are sensitized to do so.
In science and research, this often is unintentional – it may result from poor study design, biased data collection methods or subjective observation biased by expectation. Studies that depend on “eye-balled” data — a human with a stopwatch, counting the number of times a chimpanzee uses its right or left hands, how quickly a spider reacts to stimuli, etc.– can easily go awry. These types of causes would be methodological problems.
Or, if Jonathan Schooler is right, in a larger field, it could be caused by publication bias and particularly unpublished results. Journals like big splashy results. Journals don’t like null, negative or “nothing found” studies. That means when a meta-analysis is done on published papers, it is usually the studies with big results which are found – and not many which found tiny or no effects. These null or tiny effect papers may have appeared in journals with no real influence: non-English, small or obscure journals. Schooler points out than many studies with null findings or small effects are rejected and not published or, worse, never written by researchers who know their chances of publication are remote.
In new areas of research, scientists tend to look for findings similar to the large results that brought fame and success to those first to report the new phenomena. When they find them, they rush to publish. On the other hand, if they don’t find big impressive results, they may think they have erred in some way and be unwilling to buck the new trend. Publication bias affects meta-analyses. In most fields, those that do not become politicized or subject to enforced consensus, eventually come right, more and more realistic effects are found, we see the decline effect and science moves on.
Another way in which meta-analyses can be biased is:
The citation or non-citation of research findings, depending on the nature and direction of the results. Authors tend to cite positive results over negative or null results, and this has been established over a broad cross section of topics. Differential citation may lead to a perception in the community that an intervention is effective when it is not, and it may lead to over-representation of positive findings in systematic reviews if those left un-cited are difficult to locate.
Selective pooling of results in a meta-analysis is a form of citation bias that is particularly insidious in its potential to influence knowledge. “ — Wiki
For citation bias and publication bias, think of the effect of that these two have on subjects whose public perception depends on meta-analysis, like the IPCC Assessment Reports or NOAA and NASA reports on sea levels, climate or extreme weather events – which use only what the authors of those reports consider “approved“ studies and “authoritative” sources. One of the first steps of a meta-anaysis is collecting what papers to consider and, all too often, approved and authoritative in practice simply mean “agrees with us”.
This is a broad category of problems but one that has the simplest solution. In the earliest stages of ocean acidification science, there was a lot of bias: the largest of which was the a priori assumption that lowering pH of ocean water was bad – that it led inevitably to adverse effects. But to prove this, as “it was so obvious”, researchers not expert in ocean water chemistry did experiments of “lowered pH’ using the easiest method, the one they learned in high school, they just added acid to sea water. Chris Cornwall and Catriona Hurd published an important paper that set some of that nonsense right (see here and here). Once the methods were corrected, OA experiments started to find far less adverse effects in general. And now, in that field, with fish behavior effect papers found to be highly questionable, the field is back-burnered among the climate crisis crowd. In defense of the OA researchers, standards were developed for proper methods of OA research and published by the European Project on OCean Acidification (EPOCA) – which produced the booklet “Guide to best practices for ocean acidification research and data reporting”.
In other fields, methodological bias is rampant. In the sea level field, there are still many papers that simply average tide gauge station data (tide gauges which each measure relative sea level at a single location) and then paste the satellite sea level measurements of eustatic sea level to the tide gauge observational series of local relative sea level (a very apples, oranges and bananas fruit salad mistake) or the latest methodological madness of hybrid sea level reconstructions (which commit all of the above errors combined).
There are and have been many solutions proposed to what is becoming to be known as the Saving Science movement which includes serious detailed replication of important findings and pre-registration of studies (including hypothesis, data collection, methods, data analysis methods, everything). Pre-registration allows peer review of the proposed study, before the effort and money are spent on a poorly conceived plan.
In my opinion, research fields need pull their brightest minds together and layout real research goals for strengthening their foundations and pointing out where they have knowledge gaps – setting goals to replicate and verify foundational knowledge and fill in knowledge gaps. The whole point of Saving Science is to separate out the good from the bad, the truths from the myths, foundational principles discovered from current science fads.
In other words, we need to quit fooling around. We need some researchers doing crazy blue sky research. But we are in dire need of correctional science studies – science done to correct science errors of the past. There are far too many fields like OA research, sea level research, coral reef research that have wandered off along dangerous paths to Science Nowhere, following faddish Just So story versions of reality, memes turned into Facts™, themselves created ex nihilo, to forward social and political agendas.
# # # # #
The decline effect doesn’t mean necessarily someone is cheating. It can just mean that what looked important at first glance isn’t a big deal. Unfortunately, some young researcher that makes a hit with a Big Find often wants to ride that pony to a tenured position, fortune and fame. When subsequent researchers cannot replicate and call into question the original Big Find(s), trouble ensues. Those trying to correct the scientific records are vilified for “attacking science” even though they themselves are scientists.
Science can’t be self-correcting if those trying to make needed corrections are attacked for doing so.
If making anything other than a general comment, address your comments to a specific individual by name/handle. This makes conversations much easier to follow. For me, start with “Kip…”
Thanks for reading.
# # # # #
Kip: I think the random integer plot caption has two ‘100’ typos that should be ‘1000’—they were picking integers from 1 to 1000, not 100.
Monte ==> Gads — you have the best eye ever. I’m gonna put you in my holiday card list!
I Have a strong FEELING That THE ORIGINAL HOCKEY STICK HYPOTHESIS WAS ONE OF THOSE BLOWN UP EVENTS THAT COULD DO WITH A “DECLINE”
Dudley ==> And you would be right — but as CliSci is an enforced consensus subject — the early part of that hockey stick was declined — but the consensus enforcement team fought back and they have gone back to the original sticky stick.
Kip: Excellent article.
“[S]ome young researcher that makes a hit with a Big Find often wants to ride that pony to a tenured position, fortune and fame.”
Correction: “For example, a young PhD student name Michael Mann created a Big Lie, and road that pony to a tenured position, fortune and fame.”
Sometimes (often?) the problem with science is just hubris and cupidity.
Thomas ==> There is a lot of hubris and cupidity going around…….
It’s a pandemic. Masks are necessary to cover the greedy, arrogant grins.
I use mine to hide my runny nose.
I did a parapsychology experiment as a class project in the mid 1970’s. The only subject who claimed to be psychic was the only spoiled run, as he was the only one who misunderstood the instructions.
The Rhine card precognition v. Telepathy study had null results.
Or the subject WAS psychic, but the evaluators were cognitively biased and threw out his test results……/s
I guess he couldn’t foresee what the instructions would be.
I don’t think the decline effect applies to anthropogenic global warming, AGW. Because the people promoting it have vast sums of money behind them given by non-tax paying foundations; mostly in the USA. AGW is really part of a culture war financed mostly by billionaires who run these foundations.
Mark ==> Preaching to the choir with that here…..
The UNIPCC’s very existence depends on continually finding evidence of something catastrophic and weather related that can be pinned on humans.
The catastrophe used to be Global Warming caused by burning fossil fuels and releasing CO2.
The warming part sort of stopped around 1998 so the idea morphed to Climate Change – still catastrophic and where every significant weather event of any nature gets labelled Climate Change.
Climate Change has become whatever the carpet baggers say it is as long as governments keep funding the existence of the nonsense.
So Kip, this is not the Decline Effect where science prevails but rather a struggle for the very existence of would-be dictators desperate to get governments to agree to untethered “climate ambition” so they can sit atop their autocratic pile and enjoy the spoils while the Hoi Pollio do their bidding to give them the life they would like to become accustomed to without grovelling to upstarts like Donald Trump.
Possibly the most unscientific ruse in the whole unphysical fairy tale that is Climate Change is the idea that deep oceans can be radiatively heated from the surface in 50 years – “so silly” as Trump would say.
RickWill ==> I do not think CliSci suffers from the decline effect — but the opposite — wherein enforced consensus keeps what should be an whole basket of effects that decline as more science is done — the effects keep getting more and more exaggerated.
And that BS about 1.5 °C or 2 °C of warming being disastrous. If the room you were sitting as you read the article got 1.5 °C or 2 °C warmer, or cooler, over the period of time that it took you to read the article, you probably would not have noticed. No human or other living organism can detect a change of 2 °C that happens over 200 years. That fact that they could morph such a minuscule amount of warming into an existential crises is astonishing. If it’s not a conspiracy, it’s mass hysteria. Or both.
I would suggest that an alternative viewpoint may also be valid; The same way we suspect that some wars are started by politicians to divert their constituent’s attention from the mess they have created, the Global Warming / Climate Change campaign also serves the purpose of creating an enemy that must be defeated at all costs, with no large body count to explain to grieving families, and they can be seen to be proactive.
Tony ==> Everyone should re-read 1984 every couple of years.
I believe the left does just that – only they look at it as an instruction manual rather than a warning.
Owen ==> Guess maybe it should be banned then?
No, but we need to put an intro on the front about how it was a socialist’s warning that what he was seeing his dream turn into was a nightmare.
I think the problem is too many people see themselves in the role of “Big Brother” and not enough identify with Winston Smith and the psychological torture he endured. It’s like they have never looked at the history of the Soviet Union and noted how all the people who thought they would be in charge – except one – wound up dead long before they got near the top. But they all couldn’t be Stalin, only the most immoral and ruthless can get the cheese.
Owen ==> Actually re-read it myself last year and watched the original movie too….
Owen: Personally, I feel little for Winston. My (not so) favourite character is his friend, the one who bragged about diminishing the vocabulary so people will not have the terms needed for complex thought and discourse…
I strongly feel ya’ll (not youse guys there) don’t appreciate the horror hiding in “contemporary” language.
“The warming part sort of stopped around 1998 so the idea morphed to Climate Change “
Not really. The IPCC (remember what the CC stands for?) was created about 10 years earlier. They knew then that any change would be “bad” and blamed on humans.
Couldn’t you have just called the article ” The problem with Griff is…..”
Waza made me chuckle.
Why do you think griff deserves any attention?
Deserves attention, maybe not
Needs attention, I would say so
Attention or detention?
Kip: What a fantastic article.
Science is never, in fact, settled. However, some facts are so accepted that an experiment disagreeing with the current consensus may never be published due to inappropriate peer review. Rejection because the reviewers themselves have published papers disagreeing with the young buck’s paper.
Good peer review is like finding typos in a caption in a graph. Finding a mistake in an equation that is clearly a mere typo of using ‘+’ where ‘-‘ was correct.
Most splashy results — the kind reported by journalists — are wrong.
The funding of science by politicians — governments — is fundamentally wrong. Politics is always about opinion, and getting elected. Science should never, ever be funded by a politician. Opinions in science must be based on facts, never how the political wind blows.
How true that is — on both sides of the argument!
George ==> You must refuse to be old — that’s what I do.
Thanks for your comment — so right. CliSci has become a branch of PoliSci.
Where have I heard that before? Oh yeah Eisenhower warned about it in his farewell. Everyone remembers the military industrial complex part of his speech and overlooks the part about academia being corrupted by the fetters of funding by those beholden to the politics of the day (politicians and bureaucrats.)
I have just finished reading chapter 13 of Iain McGilchrist’s new book The Matter with Things.
Among a great many other things, McGilchrist describes Leo Szilard’s satire The Mark Gable Foundation. “The story describes the creation of an endowed, not-for-profit foundation for the specific purpose of slowing the pace of scientific progress.” Needless to say, the methods the foundation espouses are pretty much what we find in today’s institutional science.
Even without a pile of corruption, which McGilchrist documents, the nature of institutional science, as it is currently run, guarantees that original insights and scientific breakthroughs will never see the light of day.
The Matter with Things
commie ==> Yeah…great link by the way, Mark Gable Fdn.
The IPCC’s AR6 report isn’t final yet, but I think the Global Warming Potential numbers have been given less prominence. The press still goes on about methane being 85 times more powerful than CO2, but it seems to be hard to find in the AR6. After several minutes of scanning down through the AR6 I find Table table 7.15 that says that Methane’s GWP number is 82.5 over 20 years which is a reduction since the AR5.
Steve ==. AR6? see my Opinion piece.
Kip. Is your stated essay date of 29 February just a typo or am I missing a joke somewhere?
He was leaping to conclusions. 🙂
Very good Cylde
Malrob ==> Just trying to see which readers are really “woke” — in its old sense meaning awake. There may be some techie gizmo that updates dates….
The problem certainly needs analysis and this sounds very reasonable– “But we are in dire need of correctional science studies – science done to correct science errors of the past.” I’m not at all sure about this–“There are and have been many solutions proposed to what is becoming to be known as the Saving Science movement which includes serious detailed replication of important findings and pre-registration of studies (including hypothesis, data collection, methods, data analysis methods, everything). Pre-registration allows peer review of the proposed study, before the effort and money are spent on a poorly conceived plan. ”
We are skeptical about peer review, but this would seem just to be pre–peer review, again a group deciding about research priorities. Already happens? Consensus? One of the early criticisms of the grant system was that it favored ‘easy’ subjects. I saw evidence that funding government programs were, and still are with strong pork-barrel qualities, not sure it started out that much or not.
As to “blue-skies” research, you get into the basic–applied conundrum.
As to OA, have to wonder if we had insisted on proper chemical language along with lack of hyperbole would the science have been better. I may be too idealistic but am old enough to remember the beginnings of ‘publish or perish’ with numbers required increasing. At least some of us of that era thought that we were fortunate to have been in the best of times with more control over our own research than developed. Spoiled before overproduction of researchers with various priorities.
With inadequate study of the subject I currently would prefer less top-down control. Might not be possible in all fields, might need reworking a number of requirements. I posted recently on new papers on “Sleeping beauties in science.” Can’t predict future about them, but I know unanswered several marine biology questions, even one in physical oceanography that go back decades, rarely centuries but older literature gets less attention. All done before stats dominated publishing.
If your experiment needs statistics, you ought to have done a better experiment. Rutherford
H.D. ==> There is a reason that the best advice on Science is coming from those retired from the current system. Your insights are helpful.
An example of the value of pre-peer review is that the mistakes of early OA science would have been avoided altogether if the methods had been posted to a field specific pre-registration site. Some sea water chemist would have pointed out the errors.
Professional organizations in specific fields of science should and could lay out known areas of knowledge gaps and foundational knowledge that may need strong replication. Obviously they would not dictate what science could be done, but they could supply direction for all those post-grads needing PhD topics.
That pre-registration idea only works if the site it is on is fairly public, at least within the broader scientific community. If it gets limited to a self-selected group of activists, it will simply become another avenue to suppress studies that might find the emperor has no clothes.
Owen ==> Yes, quite agree. Acceptance of the idea requires accepting that almost all scientists are acting in good faith, even if they have ideas we don’t agree with.
We need more articles like this but what we need even more is a way to get information like this to the average Joe in words anyone can understand. What makes this matter so important is I have regular conversation with a guy who has never visited a site like WUWT but watches the news religiously. When we disagree l tell him there is precious little evidence to support what he is saying. He disagrees and states he can prove to me that he is right. He whips out his phone, he has saved news article after news article to prove to me that he is right. I tell him not to trust everything he sees on the news, so he asks where I get my information. When I mention sites like WUWT or Junk Science or C-FACT or Heartland or any of the sites I visit he pretty much writes me off. Why? Because they aren’t ABC, NBC, CBS,CNN or PBS. I don’t visit the real news sites. He suggested I crawl out of my hole and broaden my horizons by watching the trusted sites that he and the rest of America rely on. This is a big problem and sooner or later it will have to be addressed.
I agree. This is the biggest problem of all and I believe it will be self correcting as in the old saying ”The truth will out” but how long that will take is anyone’s guess.
There is a need for a master plan to get the truth out. It must knock down the pillars of “the consensus”.
For 60 years common knowledge said dietary cholesterol caused heart disease, carbs good, fat bad.
KETO may not be the answer, but the general population understands that they have been lied to. Bad science and government overreach has done incalculable damage to our health.
This may be the paradigm for a change in perception.
Energy is civilization. CO2 is not pollution.
David ==> Yes, dietary cholesterol does not cause heart disease generally. Carbs and fats are simply dietary components and neither good nor bad.
“Energy is civilization. CO2 is not pollution.” A perfect 10!
Bob ==> There are two truths here. 1) You should read/view more widely. 2) Your friend should read/view more widely.
If you’ve followed my essay here for long, you will have noticed that I write about Science Journalism — which I can only do by reading “everything”. Can’t write about medical issues if you don’t read JAMA, Lancet, and the NEJM.
Gotta read both sides to have any real understanding at all.
Kip, the thesis behind Ketogenic diet for heart health is that carbs and vegetable (seed) oils are inflammatory (ie. Bad!) Cholesterol accumulation at the site of inflammation damage is the body’s repair response, but prevention requires reducing inflammation rather than cholesterol. The body is complex and sensitive to more than just calories in calories out. Similarly the earth’s climate is far more sensitive to other influences.
The Laws of physics behind greenhouse effect are not understood by the man in the street, or the typical politician.
Common knowledge is that CO2 “traps heat”. People are surprised to learn that water is the predominate greenhouse gas, or that the hypothesis pivots on unproven H2O “feedback”.
Like Bob, I wish there was an article or video clearly explaining the certainties and uncertainties of AGW. The truly alarming damage to civilization is being done right now, not 100 years in the future.
David ==> Send me an email at my first name at i4.net. I’ll try to send you a link to true climate facts — there is a good booklet, not sure on a simple, general public video.
READERS: Can you help here? Know of a good simple video explaining the basics of climate and why CO2 is good?
Kip, I am with you and I do read more than the sources I listed but there is no need for me to bore WUWT readers with that. I think you missed my point. To make progress in the debate on CAGW information needs to get to the average Joe, that isn’t happening now. The average Joe gets lots of information but unfortunately for the vast majority it is from the mainstream media. We Americans are over loaded with information but just because we take in all this information doesn’t mean we are well informed. My point is that the average Joe who primarily relies on the mainstream media for their information is grossly misinformed. We have to find a way to get good information into the mainstream media.
Bob ==> What you demand is the very purpose of WUWT, the CO2 Coalition, and dozens of blogs and non-for-profit web sites.
We are trying…..
Bob, you sound like you flap a good lip.Do you start casual,friendly conversations in the checkout queue? Captive audience and all that…
I find if one pokes fun at one of the oh-so-many virtue signals, pointing out in precise but hilarious detail the hypocrisy, the stupid assumptions and wasteful engineering in these ‘solutions’, people spend less time thinking up counter arguments,and more on thinking quietly.
Remember,brother, change starts with you!
I do not have the exact details to hand and my memory may be at fault, but I recall seeing a study that showed that adding carbon to the soil improved the production of, I believe, onions. There was also a study that showed that adding carbon to the soil did not improve the production of onions. These studies were cited in one that I did read (I cannot justify subscriptions to expensive paywalled sites) but I can surmise what the real problem here is, namely that they are treating the soil as a constant, with similar composition everywhere. Since this is obviously incorrect, and the nutrients that do reach the plant are dependent on the contents of the soil it passes through as well as those of the atmosphere, in order to determine what nutrition the plant is actually receiving the sap should be tested, and only then can a determination be made as to cause and effect. If this is indeed the case, It would call into question the validity of results from many experiments, in my view.
Tony ==> Your example speaks to the need to actually read the study and not the press release or the news article. You spot a weakness in the opposite results but don’t know why the studies differed. You won;t know until you take the time to dig in and find out.
Curiosity is it’s own rewards.
Errata to my previous comment; I should have mentioned that the water carrying nutrients to the plant will take up nutrients and other substances via it’s passage through the atmosphere as rain and then through its passage through the soil en route to the roots of the plant, and the composition of the soil will determine what is taken up by the water before it becomes saturated.
“Publication bias effects meta-analyses.”
Jeff ==> danged auto-spell correction!
Thanks, I’ll edit secretly so no one knows….sssh!
“The truth wears off – is there something wrong with the scientific method?”
As so often in climate related matters, reality is the precise opposite.
Lies and pseudoscience wear off;
there is nothing wrong with the scientific method.
Thanks Kip for these excellent articles.
Phil ==> Science is, eventually, self-correcting….but bad science today can created great harm while we wait for it to come around.
Regression to the mean – is exploited by local authorities to impose traffic controls. A spike in Road Traffic Accidents, deaths and injury means imposition of speed limits and traffic calming. When things go back to Normal or fall below historic levels it was because of the controls, not because it would have fallen anyway. Climate Science works on the same principle
Ben ==> One needs to be aware of regression to the mean to keep from being fooled by outliers presented as shocking new findings….
If one were to take 2 identical populations and sample them with a 20 question survey you would expect around 1 of the questions to show a statistically significant difference at p .05. If you look at all 20 questions with 1 showing significance you would NOT conclude that there was a difference in the populations. (The p value needs to be adjusted because of the 20 questions)
Now take the 20 survey questions and separate them and give them to 20 different researchers. 19 would throw the survey question away and 1 would publish there significant result because he/she was not aware of the other 19 questions.
Similarly if 20 scientists each do a study (doesn’t have to be related in any way) and 1 of them gets a statistically significant result at the p .05 level 19 will toss there study and move on the other will publish. But just like the survey example there is really no actual effect. The p value did not take into account all the studies done with null results.
thus you will get a declining effect even without resorting to fraud, confirmation bias, etc. this really isn’t exactly publication bias as the null results are not reported to anyone. It’s just the result of lots of scientists doing lots of studies.
The p value for a positive result should be adjusted to take into account all experiments being done by all scientists.
John ==> or, if your team statistician is Wm Briggs, you could ignore the p-values altogether and use a better metric.
When I moved into a small Sussex village many years ago I learnt that the neighbour two houses down had just won big in the pools, a kind of lottery. I have landed in the right street, I thought, but in spite of many attempts in that lottery never won a penny. The effect size for the street as a whole was continuously decreasing.
when research is initiated after such a fluke, the effect will always decline if the original fluke is included in subsequent analysis. Which is the case in meta studies, which are therefore fundamentally flawed.
Ed ==> Understanding regression to the mean is important for those trying to get a grasp on research results. Or betting on sports games or batting averages.
Meta-analysis is hard and depends on there being enough good enough research to use as a basis.
This notion of the decline effect is representative of a system with positive feedbacks.
Journals – whether scientific or popular – like controversy and shock because it sells … in the old days it was selling newspapers or magazines, today it’s mouse clicks and social media likes. So naturally journals reward “shock science” by publishing it over “boring science”.
So scientists, in need of publishing (“publish or perish” is the old saw in academia), and seeing that journals favor shock science, therefore produce shock science. Ipso facto, a positive feedback.
But shock science rarely benefits from the passage of time and more information, more data, more analysis. The shock effect naturally declines. So in a neverending search for more shock science, new shocks must be invented and published, while the old shocks gradually dissipate.
The scientific method is not at fault. What is at fault is that scientists are humans who naturally respond to positive reinforcement. A dog who gets a treat for behaving properly on the leash is going to tend to behave properly on the leash. Scientists are no better than dogs, as are virtually all humans, when it comes to positive reinforcement.
And the scientific journals, who really are at fault? They’re human too, and they respond to the positive reinforcement of more mouse clicks and likes (i.e., more money) whenever they focus on shock science.
It’s a self reinforcing system with positive feedbacks that is never going to improve. It’s human nature.
Duane ==> Agree with everything except “never going to improve.”. sure, it will get better the more exposure the problem gets.
I wish I could believe that … but humans never really change.
Duane ==> Human Nature (if there is such a thing) never really changes much, but things do get better. Even tough stuff, like race relations….I attended a predominately black high school s a “white, California, surfer-boy”. Tough time, but things are better now.
The whole AGW thing is becoming a bigger joke than it was when it was first pronounced. Really should be listed as a “sitcom” by now, along with the profoundly over-educated experts on the subject being touted as comedians. Except that to ignore them is to just ask for them to bring more harm to their fellow man with their useless bans on this and that.
RevJay ==> The report I shouldn’t have but did quote was prepared by a room full of politicians bickering about exactly what words to use to mis-represent reality.
Where is Jack Nicholson when you need him?
The effect is created by the fact that most of the real world has 1/f type noise aka flicker noise. And, there is no “regression to the mean” … that is a fallacy of those who don’t understand the real world.
In the real world, data doesn’t regress to a mean … instead it goes away from the mean for a variety of reasons that increase with time. A simple model for this is “ageing” or the assumption that things change over time. Yes increasing the samples can get a figure that appears to head toward the CURRENT mean, but the real world changes, and so does the CURRENT mean to which data is heading.
This is one of the biggest failures in the way science is taught. Student never get taught the more advanced concepts of noise and natural variation. So, when they see data like the climate, they have no concept that natural variation and noise can be occurred over time periods much longer than the human experience. There is always a change occurring in climate, always a change occurring in temperature, just as there is a change occurring in the average height of people, and a change occurring in IQ and even potentially changes occurring in fundamental constants like the gravitational constant etc.
In contrast, science is taught as if long term variation doesn’t exist, and this is how we get the fallacy of the regression to the mean … in real life (in almost all real experiments) the mean is changing.
Scottish ==> Ah, you have conflated a mathematical concept with a real world physical facts.
I tried to avoid reader’s making that error when I said
“Let’s be perfectly clear: In no case does the decline effect refer to an actual decline in real world effects of some physical phenomena, but only to effect sizes found and/or reported in research reports over time.”
So, you are quite right.
Propaganda is the one area where the ‘pressure’ is kept up. There is no decline in the messaging no matter what is happening in the world of science. After all, the science is… settled. And the clock is ticking.
“Impact of climate crisis much worse than predicted, says Alok Sharma
US supreme court signals it may restrict EPA’s ability to fight climate crisis
IPCC issues ‘bleakest warning yet’ on impacts of climate breakdown”
Climate crisis | The Guardian
And there’s no decline in deceit, either
“The UK has no gas supply issues, but what Kwarteng carefully leaves out is that the UK certainly does have a gas storage issue. “
Is the UK government finally seeing sense on renewables? | Energy | The Guardian
We have no gas supply issues. But rather than frack it from under our soil, the Grauniad prefers us to buy it from Qatar and Norway etc at the highest market rates. That’s solidarity with the poor from the post-modern left.
We do have a storage problem but that could be fixed relatively easily. In Guardian land increased (locally cheaper) supplies do not lower the price. Just ask our expert, griff, how that works.
Anyone remember James Randi? I followed him for years, and learned about such biases concerning parapsychology and its research, amongst other topics.
John ==> Oh yes, Randi really got people thinking.
There is more to esp than meets the public eye….
like the old XKCD comic where they test 20 different flavors of jellybeans at 95% significance to see if they cure cancer
only the interesting result is kept
TallDave ==> Me and that ex-President really appreciate the best jellybeans. Substitutes need not apply.
“So my relative is surprised by how often the number 13 or combinations containing that number (313, 1313, 3131, 1:13 etc.) appear in daily life – and is absolutely convinced that it appears more often than is statistically supportable. They see what they are looking for. Or, notice occurrences of the number because they are sensitized to do so.”
When once doing some recreational mathematics on triangular numbers, 9 out of every 10 car number plates that my glance fell on when out and about, contained triangular numbers. Something senses it and directs the glance. My ESP for detecting police cars around the corner ahead is red hot,
Ulric ==> Exactly!
Admitting to “recreational mathematics” can get you in trouble in some circles. Never went down big with the chicks when I was a lad….
Horses for courses.
Ulric ==> Rings a bell and pushes a button for me — had an older brother who spent years on recreational maths.
Confirmation bias is a universal human trait. We all do it.
Tom.1 ==> It is common in everyday life. We like things that agree with us and our current opinions.
Confirmation Bias, however, should not take place in science and needs to be watched for and circumvented. Proper science methods do just that. CliSci does something else.
I certainly agree. However, I think it does some harm even outside of science because many people hold to beliefs about life, the economy, and politics that are contrary to their own best interest and the interests of others. A lot of people seem to crave having their opinions and ideas fed back to them. People should try to be more suspicious of their own ideas and always be on guard against fooling oneself. It may be a shortcoming of our educational system that we don’t teach people how to think more critically about everything in life. That said, people are entitled to their own beliefs; I respect that.
Tom.1 ==> We certainly have to do far better at Critical Thinking skills, individually and in our educational systems.
Kip: Regarding your grtgrandfathers lucky number and the frequency of the occurrence of one and other small digits being the first numerals in amounts in income tax returns and some other types of numerical data, there is a frequency distribution giving such small numbers. I’ve been unable to find the link but it was my understanding that forensic accountants use it to see if books might be cooked. Hmm. The thought just came to me it is the Bentford distribution
Gary ==> Yes, I have a personal friend who is a “forensic accountant”. Catches cheaters and tax dodgers. Uses
Simply put, low numbers are more likely as leading numerals in real-life data sets. Faked number sets however, fail to exhibit this feature.
Further research on Benford distribution.
“Benford’s law, which tells us about expected distribution of significant digits in a diverse set of naturally occurring datasets and how this can be used for anomaly or fraud detection in scientific or technical publications.”
Is this a potential silver bullet? Here is the distribution diagram for individual numerals making up the significant digits of numbers in datasets – tax returns, populations of countries, areas of rivers (watersheds?) to molecular weights of chemical compounds, cost data, address numbers, population sizes and physical constants.
Gary ==> A friend used this to expose a Mafia boss for the FBI….
Your article made me think of Cook’s 11944 papers. Most of the scientists who don’t think AGW is CAGW won’t write papers to say that so Cook’s 97% is to be expected. Let’s guess that there were 2000 authors who contributed to the 11944 papers. There may be 20,000 scientists who don’t think there is CAGW because the evidence they have seen does not lead them to CAGW conclusions. This probably is not a decline effect but it shows a weakness in the publication system that does not encourage rebuttals. I’d guess that this situation is related to money and profitability i.e. the gravy train.