"It’s as if our facts were losing their truth"

Below is an excerpt from an excellent article in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:

Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it’s known, is the foundation of modern research. Replicability is how the community enforces itself. It’s a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.

For many scientists, the effect is especially troubling because of what it exposes about the scientific process. If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved? Which results should we believe? Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.

Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BYjefYnF

h/t to WUWT reader Edward Lowe

=======================================================================

This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.

If I may, I propose the name for this could be: confirmation entropy

Advertisements

170 thoughts on “"It’s as if our facts were losing their truth"

  1. This is the result of permitting corruption be rationalized as post-normal science.
    It has been very very lucrative: Climate science is perhaps the most extreme example.
    If we can work up the backbone to reject post-normal science and return to real science, I will be pleasantly surprised.

  2. Schooler demonstrated that subjects shown a face and asked to describe it were much less likely to recognize the face when shown it later than those who had simply looked at it. Schooler called the phenomenon “verbal overshadowing.” … But while Schooler was publishing these results in highly reputable journals, a secret worry gnawed at him: it was proving difficult to replicate his earlier findings.
    The problem is that he put his research into words!

  3. “Below is an excerpt from an excellent artic in The New Yorker which describes a recognition of curious phenomenon spanning many different fields of science:”
    My guess is that it should read
    “….excellent article in The New Yorker which describes a recognition of a curious phenomenon spanning many different fields of science:”
    (presumably not)
    “….excellent article in The New Yorker which describes a recognition of curious phenomena spanning many different fields of science:”

  4. Science and science fiction are interwoven. This blurrs the truthful respected science from coming to the surface.

  5. The most relevant section in the article is later on:

    Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for.

    Researchers may attempt to be objective, but editors and peer-reviewers certainly aren’t. This might be termed “Black is SO last year” or “New Shoes” science.

  6. “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.

  7. An excellent article, worth reading in full.
    This statement: “… the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found.”, combined with the pal review prevalent in climate science, seems to me to describe why cAGW still isn’t dead yet in scientific journals.

  8. Kevin Trenberth is inciting hate speech against those who question the science of global warming, by calling them “deniers”. Trenberth is promoting hate speech by attempting to label those skeptical of global warming as “holocaust deniers”.
    Calling someone a “denier” just because he/she questions the science of global warming is deeply offensive, especially to Jews. Since when has hate speech been ok? Calling a skeptic a “denier” is just like calling a gay person a “faggot”, or calling a Black person a “nigger”. But this is exactly what Trenberth is doing and getting away with, because no one in the climate science community has the cojones to stand up to him.
    And note the timing of the release of his speech–just a few days after the Jared Loughner shootings! I think Trenberth did this on purpose: to encourage unstable individuals to physically harm global warming skeptics–people like the eco-terrorist that took people hostage at the Discovery Channel office.
    So here is my request: please shame KEVIN TRENBERTH for inciting hate speech against global warming skeptics (which is 2/3 of the US population, according to recent polls). In whatever way you can. Stop this guy before his hate speech causes physical violence against global warming skeptics.
    Thank you,
    Reply: Yo CJ, your email does not appear to be valid. Please fix or be deleted per blog policy. ~ ctm

  9. Does an explanation lie, unexamined, in the phrase ” ..across a wide range of fields, from psychology to ecology”. That’s not a wide range. I expect that no experimental result of mine has diminished in accuracy, save for any where (dear God, I hope not) I simply made an unrecognised blunder. But then my field is a “hard” science not a “soft” one. Heavens, isn’t that a large part of the distinction between hard and soft?

  10. Grab yourself a cuppa tea before reading the whole article, as it’s well worth it.
    I’m not a stat’s guy but would the mean, mod, whatever of society now be corrupt, meaning there is no part of society now where corruption is not the mean, mod, whatever?

  11. With these comments here “This phenomenon doesn’t yet have an official name, and If replication is what separates the rigor of science from the squishiness of pseudoscience, where do we put all these rigorously validated findings that can no longer be proved?”
    The answer to both is what we used to call Magic Numbers, and both deserves to go under that name.

  12. “But it appears that nature often gives us different answers.”
    I disagree.
    Nature has only one answer.
    It is our interpretations that are different.

  13. Green friends tell me the science can be trusted because science is self-correcting.
    But how long does it take to self-correct? And how do you know when you’ve reached that point?
    They boldly apply the “it is self correcting” mantra to stuff that’s only a couple of years old!!
    The truth is in the system pipeline somewhere, deep down, and will take centuries for it to come to the surface.

  14. “In the late nineteen-nineties, John Crabbe, a neuroscientist at the Oregon Health and Science University, conducted an experiment that showed how unknowable chance events can skew tests of replicability. He performed a series of experiments on mouse behavior in three different science labs: in Albany, New York; Edmonton, Alberta; and Portland, Oregon”

    “The premise of this test of replicability, of course, is that each of the labs should have generated the same pattern of results.”
    (the results didn’t follow any detectable pattern.)

    “The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand. The problem, of course, is that such dramatic findings are also the most likely to get published in prestigious journals, since the data are both statistically significant and entirely unexpected. Grants get written, follow-up studies are conducted. The end result is a scientific accident that can take years to unravel.”
    This is very interesting and very disturbing. If I understand correctly, we have a big problem here. What we now consider “statistically significant data” may have to change.

  15. In my experience, this happens when the empirical ‘facts’ are expressed in terms of a simplistic model, which is subsequently found to be naive. When the simplistic model finally collapses, the textbooks must be rewritten.
    This is not to say that old ideas don’t go out without a fight.
    There are typically some folks who are heavily invested in the old orthodoxy, and they tend to scoff mightily at the new framework.
    RR

  16. Hmmm… This passage reminds me of something…
    “It feels good to validate a hypothesis. It feels even better when you’ve got a financial interest in the idea or your career depends upon it. And that’s why, even after a claim has been systematically disproven, you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”
    – John Ioannidis, epidemiologist, Stanford University
    http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1BZnK5QtS

  17. So that is what they meant by hide the decline!
    This was an extemely well written and interesting piece. I would have found it more disturbing if I had not suspected it already.
    My guess is that the likelihood of false results increases proportionally to the amount of money at stake and the degree of polarisation of competing theories.
    However I won’t try to prove it because the effect will surely wear off in time.

  18. It is not surprising that research produces results that reflect the bias of the scientists doing the research. Especially in climate science.

  19. Very thoughtful article. I can see only one thing he missed: Failure to replicate shouldn’t tell us to abandon the line of research. If scientists were operating out of curiosity instead of the need to justify grants, a failure to replicate would guide us toward a better theory. His example of the cocaine-fed mice is perfect. The Edmonton mice ran a lot farther despite every known factor being held identical. The difference was dramatically beyond expectable variance; it should lead to seeking the unknown factor that isn’t identical.
    But nobody [or rather, nobody who gets published!] is operating out of curiosity. Nobody has the time or money. All science is based on the need to satisfy grantors within an annual budget goal, so all published science is stuck on bad theories.

  20. When politics trumps what the scientists find
    You get what you pay for – more of the same kind.
    The peer review process
    Abounds with sheer excess.
    Who paid for it anyhow? – Ah, never mind.

  21. “It’s as if our facts were losing their truth”. Well, there lies the problem. In what sense were they ever ‘facts’ and in what sense were they ever ‘truth’? Hasn’t this shift been going on for centuries? It’s hardly going to stop now. We can argue about whether it is gradual or paradigmatic (after Kuhn), but there has been no time in the last 500 years at the very least when this has not been happening. I was amazed to find that (apart from the Aristotle worship) science books that were written in the Middle Ages were still being used as textbooks, seemingly without need for revision, for hundreds of years. With mathematics it was millenia – Euclid was still on the curriculum in the twentieth century.
    Just see how science changes. Very little of what was considered science orthodoxy, whether in physics, astronomy, medicine etc in 1900 is accepted as orthodoxy today. It has always been thus if you care to study history. I feel quite confident in asserting that little of what is considered orthodoxy today will be considered orthodox by the end of the century. As Richard Lindzen and others have pointed out, folk will look back at today’s climate science and hysteria the way we look back and sneer and guffaw at the cranky ideas that scientists held not that long ago.
    And as for science itself, there is no consensus either currently or through history as to what the objects of scientific investigation are, nor a scientific method, nor what relationship empirical results have with reality. The whole question of what scientist can know, and how they can know it has never been settled. Many have given up on the philosophical, epistemological and metaphysical issues and just got on with science in an instrumental way. There is far too much scientific practice based on fallacious reasoning, induction and design of experiments to find the thing you are looking for. Popper’s falsification criteria, while logically true and powerful, is considered by many to be an impediment to scientific progress.
    Anyone who says ‘the science is settled’ is either mad or moronic.

  22. This phenomenon doesn’t yet have an official name

    Well, it kinda does. It’s “Postnormalism”, and it’s rife. It’s like a cancer spreading through scientific discovery – very much including climatology – and it has to be stopped.

  23. Just for fun, let’s project this idea to the “applied science” arena–engineering, then on to consumer goods. Say I build an MP3 player that doesn’t “reproduce” the music contained on it because as an engineer I didn’t think reproducibility was a critical factor how the unit worked–would that be a fair item to sell to customers? Don’t you think they’d get ticked off in short order? Don’t you think they’d at least demand their money back? Nobody tolerates pseudoengineering or pseudomarketing. Nobody!
    So it is with the climsci people–we demand reproducibility in their work. The consuming public just isn’t buying their pseudoscience.

  24. Replication has gone from laboratories to computers, from reality to virtuality, from hard work and hardware, to light work and software, a kind of a “sophisticated” move. It´s so “cool” now!. The analysis and peer reviewing has turned into something resembling to the discourse of a “Critic of Art” , a collection of words of inextricable meaning filled up to the rim with the purest of vacuums. The more difficult to understand, for those nasty commoners laymen, the more “intelligent” and better.
    It doesn´t matter if it is wrong, we are the wise ones and…anyway we can always fix it by an elegant adjusting or kindly massaging. Ya know, those fools will never realize it!

  25. I dont think that there is anything new about science being potentially ‘corrupted’. There are certian things which cannot be easily proved or disproved, and so there will alwalys be the potential for certain research fields to become hopelessly tainted. Such a thing has been around a while, but I would be more inclined to say that it is academia itself that has the potential to become corrupt, rather than ‘science’ per say.
    Academia by nature has a certain freedom of expression, because of the need to obtain knowledge through free enquiry. But this ‘expression’ has to be consistent with, and be conducted within, the context of broader community values. The requirement of obtaining consent before conducting research on individuals is just one example of this.
    One can give several historical examples where academia has gone of the rails. Social Darwinism in the late 19th century and early 20th is a good example. Eugenics was one result (Michael Crichton was keen on pointing this out), Hitler and Nazism was another. Withouth going into details here, suffice to say that Hitler derived many of his more radical ideas, (as did the Nazis in general), from academic culture and beliefs prevalent in German academia from the late 19th to early 20th centuries. (Source: Weikart-“from Darwin to Hitler”). This is part of the reason the Nazis got into power and got away with so much, the ‘soil was prepared’ by academic ‘research’ for many decades prior. (Note that Darwinism doesnt actually support radical Social Darwinism, it is radical intellectuals within academica who corrupted the data to make it fit into their ideology. Hitler simply took many ideas already matured within academia, and put them in practice). I bring this sort of unsavoury history up up because we dont want the same sort of thing to happen in the 21st century, academic culture ‘preparing the soil’ for radical changes to energy policy through corruption of data, and by research ‘values’ which are inconsistent with broader community values.
    Richard Pipes of Harvard also points out that Bolshevism-Communism has its origins in what he calls ‘radical intellectualism’ within academica of the mid 19th to early 20th century. So I would argue, and others have also done, that 2 of the greatest ideological evils of the 20th century-Nazism and Bolshevism-Communism, had their origins within corrupted academic culture.
    If we keep academic culture and values consistent with generally agreed community values, we have at least one way of stop radical intellectuals from springing more nasty surprises on humanity. Parts of the modern green movement are no doubt being driven by radical academic culture, and it is important that somebidy keeps the system honest and consistent with broader community values. A better internally regulated academic culture will also go a long way. (Keep those FOI requests coming).
    I dont think corruption in ‘science’ is really the problem, I think its the general lack of regulation within academic culture.

  26. It’s the “Red Queen” at work… The “facts” that no longer hold true about humans and reactions to various drugs and chemicals (presented examples of “facts”) may very well be due to genetic adaptation to the drugs. Evolution is driving the train, and the “Red Queen” ensures that the genome doesn’t get too far before equalizing genetic changes enforce the status quo. All drugs will become ineffective once the affected cells make changes to the DNA…

  27. Corporate, political, sponsored science is in denial of a serious problem. They get the results that they pay for regardless of the science. Truth be damned!
    This is bad enough in climate research, but when entire studies are falsified to get dangerous, but profitable drugs onto the market, people die in large numbers.
    I still have faith in science. I do not have faith in the money motivated political b’stards who pervert science to falsify reality and make money and gain power and influence over others.
    We need a careful true scientific audit of the methods scientists have actually been using to get their results.

  28. Must have drunk from the post modern fountain.
    The experimenters in a group think obviated replicability, and it is the fault of nature, not the experimenters!

  29. Great article that proceeds from an incorrect assumption. That being that the science that was performed was correct in the first place.
    One of the most intriguing books I have ever read was Dan Ariely’s ‘Predictably Irrational’. In that book, he takes a good deal of time to address the medical and pharmaceutical industries and basically turns them on their heads. In particular, was the glaring lack of proper control and placebo comparison on both drugs AND operations.
    For real, if people even knew the half of the fraud by way of non-scientific “scientists”…
    I am reminded of this quote from “Around the World in 80 days” (2004):
    Monique La Roche: Where’s your proof?
    Lord Kelvin: This is the Royal Academy of Science! We don’t have to prove anything!

  30. The complete article in ‘New Yorker’ is superb and illustrates just how silly, inconsequential or just plain dangerous much peer-reviewed and widely-accepted science may possibly be.
    The article remindded me of a behavioural psychologist I worked with many years ago who was the absolute ‘kiss of death’ for any simple textbook experiment he attempted to replicate in his lecture or labs for his students. No matter how exactly the poor guy wrote up his experimental specifications, his fruit flies, his mice, his pigeons and his rats would behave in extraordinary ways under experimental conditions which were totally contrary to the behaviours carefully documented in the text books.
    But a clue to this trail of disaster occurred when he designed and commissioned a device made from a 20-gallon steel drum, that had been used to ship peanut oil, to barbecue a suckling pig as the centrepiece for his leaving party.
    The drum was split longwise into halves complete with hinges and a lock to keep the drum closed, then welded to an axle which was turned by a small geared electric motor. I attempted to warn him that his design had a major flaw, but he insisted he was merely following information from some of the world’s most successful cooks of suckling pigs.
    On the night, a charcoal barbecue fire was lit under the drum which rotated slowly as the psychologist and his guests toasted each other with increasing hilarity. The time duly arrived for the fire to be extinguished, the drum was stopped and opened, the guests crowded around to see the beautifully-cooked suckling pig and…it had, as I had warned him, become a pig-shaped block of charcoal, complete with charcoal apple in its mouth!
    As I had warned him, he had designed and commissioned a very effective charcoal-producing kettle!

  31. Whatever the ultimate “cause” of this phenomenon turns out to be, my advice to those seeking to uncover it is to follow the money. If there’s corruption of any sort to be found, it’s usually found with a tenner in it’s back pocket, eh?

  32. I think you would want to call it confirmation bias and think a number of people already use that term. However, I don’t really care what you call it I’d like to comment on the following statement at the end of the italiziced paragraph.
    “Francis Bacon, the early-modern philosopher and pioneer of the scientific method, once declared that experiments were essential, because they allowed us to “put nature to the question.” But it appears that nature often gives us different answers.”
    As a scientist, I must say a bigger issue is that the different answers that nature gives us are often more interesting than the ones we were looking for. We may find that the limits of our thinking are actually limiting our opportunity. The thing you thought you understood were not possible suddenly become feasible. That’s the real loss going forward.

  33. The writer doesn’t seem to “get” that the “sciences” he is referring to (psychology, ecology, medecine) deal with very complex phenomena and that “replication” — in the true sense of the word — is virtually impossible. High energy physics is a piece of cake by comparison. He’s also failing to “get” that the conclusions too many scientists claim as the result of an experiment frequently go far beyond what the experiment actually demonstrates. The chains of logic verge on fantasy. The core corrupting mechanism is the mechanism for funding. Pure and simple. The science bureaucracies of the Federal Government are the root cause.

  34. A couple of thoughts-
    -The concept of unrecognized bias is very important. I teach my students that scientists are just people, and subject to biases they don’t even notice. How many of us would submit for publication every ‘what-if’ hypothesis we have that doesn’t pan out? None, of course. We submit only the ones that survive the tests with promising results. But sometimes, those results are statistical accidents. And sometimes we overestimate the significance, due to hidden variables. All entirely honest.
    -I remember reading or hearing a comment once that if you have to rely on statistics to ferret out some result, it’s probably not real. Statistics, the comment continued, are best mainly for proving what’s plainly obvious to the eye. The article concentrated on studies based on statistical analyses of complicated phenomena. I suggest that the ‘failure’ of science, the ‘nontruth’ of ‘facts,’ is probably concentrated in such studies. Clearly, science continues to work awfully well in general- my smartphone is sufficient proof to me that we have an awfully good understanding of electronics and materials, for example.

  35. This is not a phenomenon of only or even mostly of post-modernism, but it seems a genuine human trait that has affected scientific research from the beginning. There never was a golden period. It is a “perception bias” (to use a term from the full article) and a sample bias, because the history of science is told through the successful pioneers, disregarding the failures, and those who stubbornly persisted in their wrong beliefs due to personal gain, hanging to fame and pride.
    The full article is a great read.

  36. Concerned Jew,
    Trenberth intends to give his speech to the American Meteorological Society, 23-27 January 2011, Seattle, Washington.
    Perhaps a number of those who will be present are readers of this blog; perhaps even our host will be present.
    Would it not show Dr. T what people think of him if as many people as possible could attend his talk, and simply get up and leave if/when he uses the term “denier”?

  37. The first thing that struck me in the article is that the “decay” was always in experiments involving biological systems. As a “hard” science guy myself (BS Geology), I’d say that research bias and measurement error is a gazillion times worse (to use a highly technical term) than in chemistry or physics. Even the physics experiment the article did mention was regarding a highly dynamic system, the Earth. Taking gravity measurements in deep bore holes is going to be complicated by the density of the rock, the possibility of unknown gravitational anomalies (which the experiment may have disclosed), and many other possible factors.
    As in the case of the Mars mission that crashed because a NASA engineer used metrics instead of English (or vice-versa, I forget which it was), is it a matter of “the science is getting harder,” or “we’re not doing science as well as we used to”?

  38. Go to the New Yorker and read the entire article. It is worth it. Again, read the whole thing.
    Thanks Anthony and Mr. Lowe

  39. But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain.
    This seems like more post-normal philosophical ‘science’.
    The article does not expand on this with but one example, which was probably a sampling problem, too small, at the onset. The example of psycho-tropic pharmica does not really count as science, considering the confounding of a subjective field, psychology, with subjective/mutable diagnosis’, and big pharmacological dollars at play. IIRC, Lilly buried 30% of the original prozac studies that showed it was ineffective without talk therapy.

  40. Having spent some time working in pharmaceutical research, I am reminded of the reason why I curtailed that segment of my career!

  41. Okay, I only read the first page, I now see there are five. I think Girma captures the gist of the perceptual error in this article.
    Girma says: January 20, 2011 at 4:01 am
    “But it appears that nature often gives us different answers.”
    I disagree.
    Nature has only one answer.
    It is our interpretations that are different.
    and I might add, ‘limited’ by what we do not know.

  42. The problem is lack of critical/sceptical attitude, which should be required of every scientist. The more the better. This problem has existed since there is science, but it increased with time. Meanwhile it is so huge that all established or consensus science is BS. All of it. Just try to be sceptical of any established science/theories and watch the attacks, dogma and suppression rear their ugly heads.
    It can be solved, but it needs first to be recognised/acknowledged/admitted widely.
    One of my favorite scientists of all time, Thomas Gold:
    “The inertia of scientific thought”
    http://www.suppressedscience.net/inertiaofscientificthought.html

  43. In the scientific community, it’s widely known and practiced, de facto SOP really, that results of testing often deliver different results than expected, so the parameters of the test are altered to meet the result.
    Color me surprised.

  44. Girma says:
    January 20, 2011 at 4:01 am
    “But it appears that nature often gives us different answers.”
    I disagree.
    Nature has only one answer.
    It is our interpretations that are different.
    =============================================================
    Of course, but this sentence is very precise because it underscores the fact that results decline over time using methods thought to “eliminate” subjective interpretation. That’s why I actually love that particular sentence.
    BTW, thanks for posting this Anthony. I’ve read it a good while ago as it actually applies to my field of research in judgment/decision making, heuristics and biases in psychology. Solving this problem of, up until recently, unnoticed area of human logical fallacy could be one of the greatest contributions to modern science. It would surely revolutionize many, but not all, fields of science. I guess you could call me a skeptic of climate science BECAUSE of my preoccupation with cognitive biases. Research methodology is THE most important factor in making sense of data. Currently I am sadened to say that the climate science movement are making incredible errors in basic epistemology. History will not be kind to those who broke the most basic principles.

  45. Could it be related at all to the fact that some “scientists” are not releasing their data in order to confirm their findings? Something about “confidentiality” .
    Post-normal = lying is ok.

  46. In the 60s an international study into lifestyle and longevity included a detailed study in a remote mountain area in Ireland, close to where I live.Their health and longevity stats were impressive. The diet was largely full dairy, fat meat, vegetables and potatoes, the lifestyle was a physical, spartan one. Their data never became part of the published study. The agenda drives the result in too many cases.

  47. This issue simply reflects the damage generated by the pervasive misuse of basic statistics.
    Most of the soft sciences use relatively modest 3 sigma deviations from the expected norm as a standard of proof. This creates lots of false positive findings which are published as gospel.
    Lubos Motl in his excellent Blog The reference Frame addressed this issue very well here:
    http://motls.blogspot.com/2010/03/defending-statistical-methods.html
    His suggestion was that researchers should raise the bar to 5 sigma before drawing conclusions.
    The difficulty is that small sample sizes and problems in controlling the extraneous variables combine to make that level of proof nearly impossible to attain. Even in very well financed sectors such as the drug industry the efficacy of most drugs is statistically not hugely different from a placebo.
    That suggests that skepticism should be the default choice for any research finding, with grudging acceptance perhaps over time if it is shown to have predictive value.

  48. W Abbott says:
    January 20, 2011 at 3:43 am
    “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.
    I think that was Rutherford.

  49. I think another thing at work here is simply the complexity of what current science is bumping up against. It seems to me that the advance of science inevitably leads to attempts to conquer ever more complex conundrums. As that process proceeds through time it gets increasingly complex to design and execute a good experiment.
    Climate science is a wonderful example of this taken to extremes. The ‘experiments’ have moved into simulations because our incomplete knowledge of climate in the real world has not yielded easily to experimentation. Contrast the complexity of a climate experiment with one meant to confirm something from Newton.
    I would submit that climate science has outstripped its experimental underpinnings because the grant writers can create far sexier submissions for simulation than for those that would be aimed at the root mechanisms that drive the system. It might well be true that this is a factor at work in every modern (complex) scientific endeavor.

  50. I tend to agree with those who find the problem to be essentially one of definition. Specifically, the definition of what constitutes science, as many fields have adopted that monicker (social sciences comes most immediately to mind) to give their statistical fudgings an air of authority and certainty on par with Newtonian mechanics. I say this as an economist (a field that seems to have fallen under the rubric of social scientiest).
    From the article: “Other studies claimed that females had more orgasms when their partners were symmetrical, while a paper by anthropologists at Rutgers analyzed forty Jamaican dance routines and discovered that symmetrical men were consistently rated as better dancers.”
    If a ‘study’ of So You Think You Can Dance Jamaica is considered science, no wonder science is in trouble.

  51. The Crabbe experiments deserve more careful thinking. His model, at least as described in the above article, make numerous ceteris paribus assumptions. Sure he controlled for many of the obvious possible variables – but not for the obvious one of individual differences. Just think of the variability among humans in any number of attributes – even amongst identical twins. Moreover, unless we have some sense of the baseline activity level for these mice, it is hard to understand the real import of the 10X difference in distances. Perhaps this is made clear in the original Crabbe article – I do not see a reference to the original article.

  52. The article itself is excellent read. It is not a mistake to consider its relevance to climate science, although the article doesn’t discuss it explicitly. Interesting that the author, Jonah Lehrer, mentions a finding that challenges the Law of Gravity:

    Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings… The law of gravity remains the same.

    Read more http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1Ba9jMcIC
    But here is the oddity. When Judith Curry first discussed to this article back in December, she also mentioned a follow-up comment by Jonah Lehrer on another web page.
    http://judithcurry.com/2010/12/14/lies-damned-lies-and-science/
    And here is Jonah Lehrer talking explicitly about climate science :

    Question #1: Does this mean I don’t have to believe in climate change?
    Me [Lehrer]: I’m afraid not. One of the sad ironies of scientific denialism is that we tend to be skeptical of precisely the wrong kind of scientific claims. In poll after poll, Americans have dismissed two of the most robust and widely tested theories of modern science: evolution by natural selection and climate change. These are theories that have been verified in thousands of different ways by thousands of different scientists working in many different fields. (This doesn’t mean, of course, that such theories won’t change or get modified – the strength of science is that nothing is settled.) Instead of wasting public debate on creationism or the rhetoric of Senator Inhofe, I wish we’d spend more time considering the value of spinal fusion surgery, or second generation antipsychotics, or the verity of the latest gene association study.
    http://www.wired.com/wiredscience/2010/12/the-mysterious-decline-effect/

    Interesting that Lehrer doesn’t name as ‘denialist’ even those experimenters who cast doubt on the Laws of Gravity.
    So before we pat the author on the back for this article -and we should because it is an excellent read-, let’s make note: the author is not a Skeptic but a Believer.
    With regard to climate science, Jonah Lehrer is a CAGW cultist.

  53. Steve Keohane says:
    January 20, 2011 at 5:25 am
    IIRC, Lilly buried 30% of the original prozac studies that showed it was ineffective without talk therapy.
    =============================================================
    You have perfect recollection 😉 (although I am not familiar with the percentage)
    This, and many other findings like this, led to an enormously influential meta-study published in the british medical journal “the Lancet”. It gathered all research (published AND not published findings on newer generation anti-depressants) and found NO effect when put together, except for a minor effect for people with very severe depression. Remember then that placebo worked just as well, but with NO side effects which can be very harmful in some SSRIs (anti-depressants).
    The time is perhaps ripe for a big meta-study conducted of all research on climate data to establish if there is an active confirmation bias in the publishing part of the field. I think there is. Even though it might still prove CO2 effects, they are likely to be seen as less severe if viewed as a whole. A problem with such a study is that much research is concluding what computers are saying. This is an enormous problem, and could perhaps lead to a fall of the “modelling paradigm” in the future if found to be very influenced by biases.

  54. In reply to W Abbott:
    “If your experiment needs statistics, you ought to have done a better experiment.”
    Ernest Rutherford quotes (Baron Rutherford of Nelson. New Zealander born British Chemist who laid the groundwork for the development of nuclear physics by investigating radioactivity. Nobel Prize in 1908. 1871-1937)

  55. If the facts are changing and losing their truth, then it wasn’t a real truth to begin with. A little over thirty years ago, climate scientists were debating the coming ice age, based on a series of facts. Today, they debate how badly we are going to cook ourselves with AGW. The facts are all still the same, but the truth behind them is still hidden. No amount of claiming a consensus and moving to a “solution” will help to uncover the truths as to how climate operates and what drives it. The target of climate science is probably not changing as fast (or at all) as the targets of drug manufacturers–climate science is very complex, and has more moving parts than we can count, but those parts stay more or less the same parts. Drug manufacturers, on the other hand, are chasing a moving target–the Red Queen effect, mentioned by another commenter.
    Bottom line–science is science and the scientific method, if applied correctly, works. I have much more respect for the people who follow their curiosity instead of the money. I think the curious folks get better results.

  56. Another comment:
    If the review process in climate science in any way resembles the “bore hole” standard of realclimate.org, I am very confident there is a fundamental and rotten problem to be concerned about. It would then, in my view, be a case of serious corruption to be dealt with as a serious legal problem.

  57. People in the UK should watch the BBC’s “Horizon” program about the “attack on science” (or rather, the credibility of scientists) next week.
    [Monday, 21:00 on BBC Two (except Northern Ireland (Analogue), Wales (Analogue))]
    According to the “Time Out” TV reviewer, the new head of the Royal Society, Sir Paul Nurse, will effortlessly demolish all the “deniers” claims that AGW isn’t happening in an interview with the Daily Telegraph’s “denier-in-chief” which left the journo “spluttering that he is an interpreter of interpreters”.
    This is what the BBC have to say about the program:
    “Nobel Prize winner Sir Paul Nurse examines why science appears to be under attack, and why public trust in key scientific theories has been eroded – from the theory that man-made climate change is warming our planet, to the safety of GM food, or that HIV causes AIDS.
    “He interviews scientists and campaigners from both sides of the climate change debate, and travels to New York to meet Tony, who has HIV but doesn’t believe that that the virus is responsible for AIDS.
    “This is a passionate defence of the importance of scientific evidence and the power of experiment, and a look at what scientists themselves need to do to earn trust in controversial areas of science in the 21st century.”
    http://www.bbc.co.uk/programmes/b00y4yql

    So there you have it; AGW exists because the Royal Society, the BBC and the “Time Out” TV reviewer said so.
    Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?

  58. The best article I’ve ever read in any magazine. And I would say of all the articles I’ve read in magazines, less than 5 are any good at all. I now think the other less than 4 articles sucked compared to this one.
    When I was in graduate school, we did not take a single full year course on the history of research and experimental design. I took a semester to study statistical analysis. Later, I took a semester of research interpretation (similar to taking a course in literature). That was it. That I am able to sniff out poor research design and question results cannot be credited to a stellar graduate course. Worse than that, my study, published in a major journal, was chasing the lab director’s beliefs. Had I not found a positive result, there is no question in my mind that I would have been sent back to the lab with orders to look harder.

  59. Grade inflation, too many “scientists” for academic positions or research institutions, and pal reviewers. Stop it all now: a) the subject matter must be mastered or you do not advance or graduate; b) no more government grants for “scientific” research; c) peer review process open to the public, at the very least. Transparency and accountability along with replicability. A free market system in “science”.

  60. Page 5: “Many scientific theories continue to be considered true even after failing numerous experimental tests.”
    They always said global warming was going to lead to snowier winters. Right?
    The problem with global warming theory is that it never fails. Rather, the true implications of it only become obvious in hindsight. Then the theory is adjusted as if it was always thus. The science becomes settled. And then, when the next mysterious unpredicted phenomenon pops up, it gets settled again. So, at all times, it’s always settled. It’s just not the same.

  61. “So there you have it; AGW exists because the Royal Society, the BBC and the “Time Out” TV reviewer said so.
    Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?”
    Horizon ceased being an objective and rational science programme a long time ago. Sadly they are tied in to the same vested interests who abuse and pervert the scientific process to achieve profits and power.
    I will be willing to bet that this episode will not be interviewing scientists who have admitted to falsifying “double blind” tests and experiments in order to get the results that pharmaceutical corporations want in order to licence the selling of highly toxic chemicals in the guise of medicines.
    There is nothing wrong with science and it is highly decietful of the AGW alarmists to even claim that sceptics are attacking the science of climate change.
    On the contrary, most climate realists attack the perversion of the scientific method and the sloppy and dishonest deviation from the scientific method which Alarmists have become well known for.
    It is the climate realists and not the alarmists which uphold and support real science. It is the Alarmists who do not.

  62. At the risk of mis-quoting something I am not sure I understand myself, I think this is an example of the problems with frequentist stats giving over-confidence in results. A much better explanation can be found here:
    http://wmbriggs.com/blog/?p=3388&cpage=1#comment-34338
    One could say that Dr Briggs is on a bit of a campaign for bayesisn stats methods, but I think he is completely correct that we “fall” for a low ‘p’ value far too quickly. There are very few experimental systems where there there are no sample selection issues and little or no thought is given to that when publishing a “significant” finding from a study. As the New Yorker article notes, when scientists know what they are expecting (or trying?) to find, it is easy to select a sample that will maximize the effect. This can happen without any ulterior motives or even without a conscious decision on the part of the scientist.
    Nice to see such a good article in the New Yorker as well as a story on a real scientist telling people when follow-up results don’t gel with the original findings!

  63. kcom says:
    January 20, 2011 at 6:49 am
    Page 5: “Many scientific theories continue to be considered true even after failing numerous experimental tests.”
    The problem with global warming theory is that it never fails. Rather, the true implications of it only become obvious in hindsight. Then the theory is adjusted as if it was always thus. The science becomes settled. And then, when the next mysterious unpredicted phenomenon pops up, it gets settled again. So, at all times, it’s always settled. It’s just not the same.
    =============================================================
    Yes. It has all the flaws of what is called retrospective studies. They are fundamentally flawed as there is too much openness for biases to infect the conclusions. It can be useful, but it is widely held to be very poor at finding causes to observed effects.
    Prospective studies are more in line with good scientific philosophy. You make a hyopothesis (e.g. this level of CO2 will cause these effects by this time) and discard the hypothesis if it is not matching the real world data in a meaningful way (i.e. statistical significance).
    Now, to be fair, the climate is extremely tough to study, but knowing this shold make scientists extremely humble when presenting predictions due to uncertainty. This is not what we see. We see the URGE for a consensus. A trend towards labelling the wholy grail of science (‘skepticism) as “denial”. And we sometimes see absurd claims such as: 95% certainty such and such, and claims such as: “the world will warm by 6degrees celsius by April 3rd in the year 2087 with sea level rises of 787.9 mm.
    I’m glad this article came up, bc this is (IMO) the biggest Achilles heel of climate science, and it is a crime against the science.

  64. Very interesting, very little can be said to add to the article. The heart is near the end..
    According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. “The scientists are so eager to pass this magical test that they start playing around with the numbers…”
    In other words, decide what you’re looking for and find it in the data. Where else have we seen this sort of bias?

  65. “Nevertheless, the data Ioannidis found were disturbing: of the thirty-four claims that had been subject to replication, forty-one per cent had either been directly contradicted or had their effect sizes significantly downgraded.
    The situation is even worse when a subject is fashionable. In recent years, for instance, there have been hundreds of studies on the various genes that control the differences in disease risk between men and women.” …. “But the most troubling fact emerged when he looked at the test of replication: out of four hundred and thirty-two claims, only a single one was consistently replicable. “This doesn’t mean that none of these claims will turn out to be true,” he says. “But, given that most of them were done badly, I wouldn’t hold my breath.””
    And then – According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher.[b]{ “The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,”}[/b]
    Does this remind you of Dr. Hanson?

  66. Kate says: January 20, 2011 at 6:39 am
    “travels to New York to meet Tony, who has HIV but doesn’t believe that that the virus is responsible for AIDS”
    “Of course, the main question everyone here will be asking is why did Sir Paul Nurse interview a journalist for the AGW bit of the program and real scientists for the rest of it?”
    ————–
    Any idea if “Tony from New York” was a real scientist? Wonder why Sir Paul Nurse didn’t travel to Berkeley to talk to Dr. Peter Duesberg if he really wanted a discussion on HIV/AIDS? Thanks for the pointer, I’ll let Dr. Duesberg know about the program, if he didn’t already.

  67. Great article.
    While I can see that those in the “hard sciences” (as I was) would feel somewhat immune to the problems brought out in the article, I think it should be clear that the more complicated the system under research (yeah, like climate), the greater the probability of being subject to these flaws and biases. Even that doesn’t mean some simple experiments aren’t capable of going awry. Remember cold fusion (in it’s simplest form as originally reported, not what it has morphed into today)? The original results were significant and got published. At least one group (GA Tech, I believe) replicated the results. It was only because the research WAS so simple that it was quickly shown to be false. If for some reason replicating the experiment were difficult, required specialized instrumentation, or collecting data from across the globe (ahem), cold fusion power plants might be under construction today.
    Cold fusion segues into one last thought: at the atomic/sub-atomic levels, nature gets as squimish as the subjects in the soft-sciences. I think that area is ripe for the same problems detailed in the article.

  68. “Palmer emphasizes that selective reporting is not the same as scientific fraud”.
    “But the worst part was that when I submitted these null results I had difficulty getting them published”.
    “scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness”.
    “The situation is even worse when a subject is fashionable”.
    This explains about 11.9% of Climate Change science. Which leaves the other 89.1% as likely the opposite of the first quote.

  69. @dearieme January 20, 2011 at 3:49 am
    Kudos for seeing the distinction between hard and soft science. That is why each of the soft sciences had a tough time being accepted in the first place – quantification was difficult and ambiguous. They put in equivalents of proxies to try to make them look and sound like proxies, and the biggest one was statistics. I say “equivalent of proxies” because they do things like assign one answer in a questionnaire a numeric value and another answer another value (based on vague and often hidden reasoning), so when the stats are done, what can it really mean? If the assigned values are incorrectly assigned (no matter how unintentionally), is that science really a science?
    I recently read (sorry I have no idea of the source now) a statistician argue that the 95% confidence thing is misunderstood by almost everyone who uses it. If that abuse (mostly unintentional) is their main claim to quantification, then is that science really a science?
    Archeology. Read any archeology article and 80-90% of it is about interpretations. That is the equivalent of Olympic ice skating judging, except they don’t throw out the high and low scores – or do they? I also long ago (again the source is long since out of reach) read where archeologists toss out 85% of the C14 dates, because they don’t fit into the expected range, on the (arbitrary and spurious) premise that the samples must have been contaminated. This weeding out is bias, pure and simple, the equivalent of fudging the data or cherry picking the data. I have been arguing for some time now that archeology is not a science, even though they use a few scientific procedures. But C14 testing isn’t one of them. Just because they send off samples to be tested by a lab doesn’t make the purchasers of that service scientists. It only makes them customers. Same thing goes for the various other dating tests done in labs – it is not the archeologists who do it; they are only customers. Take that away and almost (not quite all) of archeology is left with very few claims to being a science. Comparing ceramic vase types or paleographic types or building types doesn’t make it a science, especially when those are used against C14 dating or even in the early days to calibrate C14 dates. I see them as historians, little more. They say that put 500 archeologists in a room with evidence and you come out with 1000 different opinions. How scientific is that, when they can’t look at the same real-world evidence and not come to the same conclusion about it?
    Climatology is seen here as little more than guessing. The more time I spend here, the more that seems so. Being thermometologists – does that make them scientists? Anyone can read a thermometer. And statisticians can process data. The (admittedly necessary) use of proxies, that really rest on theoretical bases but cannot be tested in the real world – is that science?
    Yet, when the real world evidence conflicts with the theory, and they choose to say the theory is wrong, let’s fudge the real world numbers is that science? That is in reference to Keith Briffa’s tree ring studies (see C’gate emails) in post-1960 recent years, the ones that conflicted with the instrument record. They beat him down and forced him to accept the proxies as more real than the instruments, when they should have been stepping back and deciding to use that discrepancy to learn something about proxies.
    Soft sciences are ripe for being influenced by bias. Their assignation of quantified values is also a critical area of distortion. And distortion is not science.

  70. When I was a kid, hot weather would cause me to feel light-headed (eventually becoming a bad headache), weak & listless.
    Eventually at the doctor’s office, it was determined to be low blood-pressure in hot conditions (my blood-pressure was normally average). He said there wasn’t anything to do (even then, salt was already politically-incorrect).
    In high-school football, they offered salt-tablets during practice in hot weather. Somehow, the listlessness & headaches disappeared! Didn’t take long to figure out the connection. Not suggesting that anyone take salt in hot weather or otherwise, but in my situation, it was an extraordinarily simple “cure”. It still works for me in hot weather.
    I bet those salt tablets have long been removed from HS football practice fields. I know, Gatorade has “replaced” it.

  71. It is just as important to not only know, but but know HOW we know. I realized this when I studied Buddhism. I realized this again when I read “Good Calories / Bad Calories” by Taubbs. No where has science been more perverted to conform to an agenda than in the Food & Drug market. (Sorry climate people, but the food science industry has you beat in years and billions) Saying red meat causes heart disease in one animal (it does when you feed it to rabbits, exclusive herbivores) and extending that to another (human, omnivore) is bad science. Also, when heads of the FDA go to Pepsi Cola after their public jobs, you have to raise an eyebrow.
    I’m sure when I say to climate skeptics to consider the source, that I am preaching to the choir, but what is happening here is nothing new.
    Everything is a business model. Everything.

  72. Yes, very interesting article.
    Also interesting that Lehrer has his own bias, the “consensus” bias, with regard to CAGW. The CAGW industry probably has the greatest set of built-in biases of all, including:
    Perception bias
    Confirmation bias
    Selective reporting bias
    Pal review bias
    Publication bias
    Funding/Career bias
    Fame/Ego bias
    Herd Mentality bias
    Consensus bias
    Academia Bias
    MSM bias
    I’m sure there are more.
    Not all of these biases are innocent, either, as evidenced by Climategate, and with the Hockey schtick, the stated need to “get rid of” the MWP.

  73. Palmer noted ”
    “The funnel graph visually captures the distortions of selective reporting. For instance, after Palmer plotted every study of fluctuating asymmetry, he noticed that the distribution of results with smaller sample sizes wasn’t random at all but instead skewed heavily toward positive results.”
    Looks very much like cherry picking from larger sample sizes, to present the smaller subset of the total sample that gave the desired results.

  74. Now look at the problem from the layman’s point of view as a recipient of the mainstream media’s outpourings:
    Breast milk good, breast milk bad, breast milk good …….etc
    MMR good, MMR bad, MMR good ……….etc
    Statins good, statins bad, statins good ……etc
    London will be under water in ten years time, make that twenty, make that next year ….etc.
    The problem of the lack of consistency in science is the ‘Ego Loop’
    Nowadays, scientists have lost objectivity and scientific rigour caused by the urge to publish something/anything to satisfy their overinflated egos. They will grasp for funds from any source but the government is the best one because the pot is bottomless.
    Journalists, another ego-driven ‘profession’, compound the problem of poor science by simply regurgitating the scientists’ press releases without analysis. This appears to be because they no longer have the education needed to determine pertinent questions nor to understand the answers if they had been asked. Newspapers and TV broadcasters (some of whom, despite being publicly funded) also have political agendas that influence their reporting and determines their servility.
    Finally the politicians. Politicians, are amongst the leading examples of ego-mania. Unfortunately because of a low IQ problem, they haven’t got a clue about the science; any science. Their scientific advisers are hand picked ‘yes men’ who are paid handsomely to advise as required. The politicians, therefore, compound the ‘poor science’ problem further by throwing money at the purveyors of that aspect of the science they think is likely garner the most votes; thereby closing the ego-loop. And we all know what positive feedback leads to.
    And the public is confused. As far as they are concerned they would rather that the scientists, journalists and politicians grew backbones and followed that well known aphorism:
    “Oh Lord help me to keep my big mouth shut until I know what I am talking about”

  75. Part of this (I would imagine) is because our engineers in this world have been very very good for the last 30-50 years. They have given scientists a huge wealth of new instrumentation to investigate the universe that we didn’t have and couldn’t dream of before now. Most of this new instrumentation is quite expensive and requires that scientists get funding from someone else to do their work. Those who have the equipment to confirm the findings of one scientist probably get their funding from the same places the original writer does. Hence, if there’s something very beneficial to those giving the funding that is being confirmed by replication, the confirmation bias can spread.

  76. “Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.”
    Paradigm blindness or paradigm paralysis has been well documented, and is very common. Scientists with a cherished belief that has a lot of ego integrity invested in it have been shown data/studies that invalidate the belief, and only days later cannot recall seeing the data/study.
    Read Joel Barker

  77. dearieme says:
    January 20, 2011 at 3:49 am
    Does an explanation lie, unexamined, in the phrase ” ..across a wide range of fields, from psychology to ecology”. That’s not a wide range. I expect that no experimental result of mine has diminished in accuracy, save for any where (dear God, I hope not) I simply made an unrecognised blunder. But then my field is a “hard” science not a “soft” one. Heavens, isn’t that a large part of the distinction between hard and soft?”
    I do agree this could be the case. Does climate science fall in the category of “soft” science even though the principle components under it are clearly from “hard” disciplines.

  78. Related articles:
    Lies, Damned Lies, and Medical Science

    That question [Can any medical research studies can be trusted] has been central to Ioannidis’s career. He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.

    Is Peer Review Broken?

    Some critics argue that peer review is inherently biased, because reviewers favor studies with statistically significant results. Research also suggests that statistical results published in many top journals aren’t even correct, again highlighting what reviewers often miss. “There’s a lot of evidence to (peer review’s) downside,” says Smith. “Even the very best journals have published rubbish they wish they’d never published at all. Peer review doesn’t stop that.” Moreover, peer review can also err in the other direction, passing on promising work: Some of the most highly cited papers were rejected by the first journals to see them.

    Problems with Scientific Research

  79. This is what you get with government funded science. Something Eisenhour warned us about. It sounded like a good idea at the time….

  80. AGW science is unique in that those who contest it are beat about the head by a large organized gang extending far beyond the core group of researchers.
    Climate science is fiat science.

  81. Having read the whole article, I agree Pamela, this is excellently thought out and written. If, as others have mentioned above, the author is a true believer of CAGW, this article is a perfect example of the underlying problem of human nature, and our perception of what we divine as reality. A mind, obviously capable of rational perceptive thinking won’t apply the same to itself, i.e. cognitive dissonance. Further, I would proffer that this is a necessary survival mechanism. In order to have a sense of self, and to be able to function moment-to-moment, we can’t be questioning whether gravity works every time, or if fire burns the finger each and every time the two meet. The intellect will arrive at some conclusion, even lacking enough evidence, unless we are conscious enough to notice that it does that. Basically we gloss over a lot and go on.

  82. “So here is my request: please shame KEVIN TRENBERTH for inciting hate speech”
    I agree 100% with this author. Scientific intolerance of opposing views is at the heart of the current problems in science. It is the same problem that afflicted the church centuries ago when it was the repository of knowledge.
    Scientific intolerance is no different than religious intolerance is no different than racial intolerance. History repeatedly shows how, when left unchallenged, this escalates over time with great harm to all society.
    The is no evil that cannot be justified in the name of good.

  83. Fortunately for everyone, now that this phenomenon has been described and discussed, it will gradually happen less in the future…

  84. Nature can be cruel. This may be why AGW “proof” is based entirely on computer models. With models, all of the input can be defined and there is no chance of empirical data getting in the way.

  85. Has anyone considered the possibility that in at least some cases, this is a real effect and not the result of confirmation or publication bias?
    I’m reminded of something that might be its complement, namely morphic resonance. If you can spare around 30 minutes of your life to listen to an interesting lecture on this topic, check out:

  86. Very thoughtful article. I can see only one thing he missed: Failure to replicate shouldn’t tell us to abandon the line of research. If scientists were operating out of curiosity instead of the need to justify grants, a failure to replicate would guide us toward a better theory. His example of the cocaine-fed mice is perfect. The Edmonton mice ran a lot farther despite every known factor being held identical. The difference was dramatically beyond expectable variance; it should lead to seeking the unknown factor that isn’t identical.

    With respect, I think you missed the main point of the article. You are assuming that an unknown factor exists, when it’s also quite possible that all that happened is that the Edmonton result was in the far tail of the distribution of possible results. For that matter, the other two could also have been in the far tail, but in the other direction. The norm of the unknown, but actual, distribution of results might be, say, 2,000.
    That’s the point of regression to the mean. If the mean of the actual population of results is 2,000 then, over time, replication will drive away the original “discovery” of an “Edmonton Effect,” whatever that might be.
    And to take the point of the article further, if you could happen to identify one tiny little difference in the layout of the Edmonton experiment, the publication bias toward finding unexpected results might even get you published. Then others would do the same sort of experiment, with the null results not getting published, while the few supporting an “Edmonton Effect” do, though showing a diminished “effect.”
    Meanwhile Edmonton fills up with coke heads looking for that special “Edmonton Buzz” and junk science destroys yet another economy.
    All in all, a very interesting article.

  87. “But in the Edmonton lab they moved more than five thousand additional centimetres. Similar deviations were observed in a test of anxiety.”
    This goes to show the lengths test subjects will go to to get out of Edmonton.

  88. Re Abbott’s comment:
    ““….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.”
    I have heard this attributed to Rutherford, but I haven’t seen a citation to support that.
    [Reply: It was the late British physicist Ernest Rutherford. ~dbs]

  89. @MattN January 20, 2011 at 7:44 am:

    This is what you get with government funded science. Something Eisenhower [spelling corrected] warned us about. It sounded like a good idea at the time….

    Yes, scientific research’s history can be divided into two periods – pre-Manhattan Project and post-Manhattan Project.

  90. In comment to Pamela’s comment…
    The intolerance in science of other views was birthed by the response to creationists. There has long been an absurd fight between Biologists who are certain that random processes in ordinary organic chemicals billions of years ago led to consciousness and religious crazies who are certain that a magical hand from the sky shaped dirt into the first human being. It is a stupid fight, it’s always been a stupid fight, but neither side sees the reasoning that BOTH can be correct. Science can only tell us what we can prove, and we cannot prove/disprove the existence of magical intelligent creator, so there truly is no conflict. The whole argument would have died immediately 200 years ago if some scientist had simply kept his mouth shut and said, “Yes, uh-huh, I see how you could see it that way.” and moved on. But those of us who use our brains too much tend to impose our views on others when they appear to be wrong. Yes, human nature causes problems; sun rises in east; water is wet.
    What is unforgivable is the degree to which this fight has been ratcheted up. The religious people feel like they’re being attacked, and I don’t blame them so much because they’re the unarmed people in this fight. They’re the ones who insist on believing in Santa Claus, so it’s best to just let them have their harmless beliefs rather than force a certain viewpoint on them. The scientists feel like science is being attacked, and I don’t blame as school boards have held actual hearings to determine whether an unprovable “creationism” should be taught alongside calculus.
    The religious people responded to this fight how they always do, they threw money at clever people to make their case for them in a political theater.
    The scientist response, however, is what should have been different. The scientists turned into activists. Regardless of how wrong someone else is, you cease being a scientist when you tell someone else what to think, you become an activist altering perception in others. As a scientist you should be trying to make the obvious plain to someone, while continuing to let them determine their own reality. The truth has a way of sneaking up on people, or smacking them in the face; but the perception of truth does not obey the whim of anyone even the smartest communicator on the planet. Scientists were choosing a horrific battle when they decided to directly battle overt human stupidity, and Einstein warned them a long time ago about it. Science *will never* win the battle against creationists or any other battle in the political arena. Let me repeat that, Science *will never* win any battle in the political arena. The reason is simple, in order for scientists to win a political battle, they must CEASE being scientists and become activists. It has happened to countless scientists thus far, and many more are seduced daily into becoming activists, rather than simple investigators of the universe.
    What’s worse, is that it is now clear that this fight against creationism has made the idea of being a scientist/activist the trendy thing to do. Suddenly all these pompous PhD’s with desires for fame have a way to make a name for themselves without writing a single worthwhile paper. All they have to do is make a “principled stand” against the “Darkness and danger of belief in _____ (Insert cause here),” and the fame can roll in. You can even pick up funding this way, but who cares if you don’t? With fame you can simply write a book, throw PhD after your author name, and make money that way. It used to be the case that scientists remained locked away from the media because the only thing less palatable to a scientist than being misquoted, was being misquoted publicly by a liberal arts major.
    And this is why we arrive here, ready to move on to… something. It is now OK for scientists to become activists and treat opposing scientific views in the media as if they are of the same ilk as creationism. This is because it is now OK to be a scientist and activist and never hold two opposing viewpoints in your mind at any time (something that good scientists do REGULARLY). It is all a response to trying to poke the eye of creationists. It’s stupid.., and now I’m rambling off the rails..

  91. W Abbott says:
    “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this

    It was said by Ernest Rutherford.

  92. 1. ScientistForTruth:
    Quit picking on Euclid. What is taught is a couple of thousand years of revision of Euclid. The Muslim mathematicians did a lot of work on the 5 postulates. Much has been discovered since Euclid himself. Get the 3-volume Dover Euclid set and enlighten yourself. No one teaches Euclid’s formulation of the Fifth Postulate anymore.
    2. etudiant:
    Three sigmas? I’m sorry, but 95% confidence is not 3 sigmas.
    95% is two sigmas. 3 sigmas is 99.7%.
    And all data sets are not necessarily normal, either.
    Please get your stats straight.
    3. General observation: the lurking variable. A lurking variable is an input to the study which is not documented, not studied, and not included. In the social sciences, such lurking variables are very difficult to identify.
    It is bad enough in the hard physical sciences to control all but one variable. Just ask a structural engineer about high winds and earthquakes.
    The New Yorker article vividly illustrates the problem with lurking variables, chief of which is the expecation of the person establishing the study and setting forth the parameters for data collection.
    In the alleged field of climate science, the lurking variable is, of course, the consistency and reliability of recorded temperatures. What happened to all of those sites which are no longer included in GISS? What, precisely, are the details of the inferential statistics used to populate the unsurveyed locations with temperatures? What, precisely, is the relation between max/min and average temperature? How is it even conceivable to infer temperatures from proxies?
    What the *** is going on with the Sun? I know She is variable, but the dearth of radio emanations is quite striking. My Sky and Telescope is suspiciously silent on the question. Galileo was accused of seeing what was not there during the Maunder Minimum. His observations could not be replicated, as the Sun went very quiet. It got really cold from 1660 to 1715, too.
    While I am on the subject, what is the whole carbon dioxide budget of the Earth? How does the absorption/release of carbon dioxide by the oceans depend on the PVT of the bordering atmosphere and the ocean temperature? What are the time frames for absorption/release of heat energy (all frequencies) for all gases in the atmosphere, and how do these vary with altitude and latitude?
    And while I am on the subject, where is that computer model which will predict cloud cover?
    And we are unlikely to learn much, due to the substantial lack of data. Just do a quick calculation of the surface area of the Earth, and figure out how many data points one would need for one each square mile or even each square kilometer. And then figure that one would need vertical data, up to 100,000 m. The cost of such a grid is out of sight!

  93. A part of the problem, maybe a large part, is the constant attempt to reduce chaotic systems to predictable systems using simplistic, reductionist model.
    Whether looking at the behavior of climates, nervous systems, or computer software, one needs to respect the unknowability of the state of the system and the immeasurability of the interactions of the many subsystems (each poorly predictable). Some things cannot be known. Quantum Mechanics has been able to quantify this “unknowability” in the realm of the very small.
    In the realm of the large there is no corresponding precise mathematics. Instead, proponents of various “theories” try to glosss over the simple truth that often they do not know what they are talking about. Skepticism should be proportional to complexity and the chaotic nature of the subject.

  94. I would venture to suggest that this is not a new phenomenon. Researchers have always interpreted their results in ways that favour their view of life and how the world works. They may also have interpreted them in ways that favour their own researches and careers. That’s the whole point of replication that the same work is done by different people who have a different take on things.
    It’s more obvious now for three reasons. One, the internet and the more rapid dissemination of science and the easier access to previously published work. Two, increasingly research focuses on smaller and smaller changes in more and more complex systems. Proving to everyone’s satisfaction that an apple, when released, always falls towards the centre of earth is not complex and there are few factors involved. Proving to anybody’s satisfaction that anthropogenic CO2 emissions have warmed the earth is extremely difficult given the complex nature of climate influences, temperature measurement over the globe and doubt over earlier temperatures.
    Three, the rewards for making exciting findings is much greater than it ever was.

  95. Jeremy certainly made the best comment on this discussion. What annoys me is that most folk don’t seem to understand the philosophy of science. Even something as simple as distinguishing between a theory and a hypothesis. Both are explanations but theories can be tested unlike hypothesises. A prime example is Darwin’s explanation of the diversty of species. It is a hypothesis. It cannot be tested. This doesn’t mean it’s wrong. It’s simply an explanation that seems to work for the moment. The same is true of many theories (eg Einsteins relativity theory). They have passed the test and seem to work but this doesn’t mean they are correct. There’s always a better theory waiting to come along which will pass more tests. AGW is a hypothesis. There is no way to test it. We don’t have the evidence in any reliable form. Even our current readings of the Earth’s temperature are open to debate. We even lack a sensible theory about how the Earth’s climate actually operates. As such, predictions are merely guesses in the dark. I’m a pessimist so my money is on the next ice age arriving pretty soon.

  96. If you give a 100 question survey to a random sample to see if they are different from the population at large, you expect to have 5 of the questions show a significant difference (at the 95% level). Thus if only 5 out of the 100 show a difference you shoud conclude that they are spurious and conclude that none of the questions truly show a difference.
    We have so many people doing so many experiments that as a society we are in the position of the person doing the aforementioned survey. We expect a large number of studies to show siginificance at a 95% (or whatever level you choose) even if no real effects are there. This will result in many spurious false positives from the studies.
    We cannot even measure the effect of this as the many studies that do not show a positive wil not be published, and may not even be written up.
    It’s like the person doing the 100 question study just published the 5 that showed a difference and neglected to mention the 95 questions that showed no difference.
    Normally, we would expect that with replication the spurious effects would be weeded out. But 5 of 100 spurious results will again show positive results when retested. and with the number of studies done, 5% of 5% of 5% still leaves a lot of spurious results out there accepted as fact.
    Then when you add confirmation bias and friendly peer review, which waters down the effect of replication, It’s no wonder that many findings are spurious.

  97. @Feet2theFire says:
    January 20, 2011 at 7:26 am
    Thanks for the comment. It helped crystallise some of my own views on the matter?
    Archeology is definitely not a science. It is certainly a perfectly legitimate field of intellectual inquiry, but Archeology along with its cousin, History, are not sciences like Physics and Chemistry. They may follow scientific procedures, but only to a limited extent. In Archeology and History, one cannot run experiments. It is not possible to re-create the past events in controlled fashion to see how the outcomes respond to changed conditions.
    This is not just common sense, it is also taught in junior high school (where I first learned of this).
    Now, one of the most common defensive arguments of the CAGW science is that we do not have another Earth where we can run controlled climate experiments. The fact that there is no other Earth to use as a test subject means that climate predictions for the one and only Earth may not be as precise as those predictions based on sciences that benefit from lab experiments. Hence, 95% confidence levels, which is supposed to prove sufficient for Climate Science.
    There are two problems with this. Firstly, if we do not have a second Earth to run a climate experiment, then that counts against Climate Science. Just like the impossibility of replicating the Trojan Wars count against Archeology and History as proper sciences. Some aspects of Climate Science are lab-tested and are indeed basic physics, but the model as a whole has not been tested. We still don’t, for example, what the climate sensitivity for per doubling of CO2 is, or whether feedbacks are positive or negative. Hence we have a vast level of uncertainty that can be interpreted either way depending on one’s political views, just like in Archeology and History.
    Secondly, 95% confidence levels is not always a good enough level for scientific certainty. Evidently, 95% is huge level of certainty for some sciences such as Medicine. But, to my knowledge, -I’d be happy to be corrected on this- certain areas of Physics, eg, Particle Physics, seeks statistical certainty at 99.98% level while running experiments.
    One day perhaps, technology will enable Climate Scientists to re-create the dynamics of the Earth’s atmosphere in a chamber where they can experiment and learn from it. Like the much maligned, much delayed, and much awaited CLOUD experiment that will hopefully help us understand how clouds are formed.
    Until that happens Climate Science is not basic physics and is as hampered as Archeology and History in terms of scientific standing.

  98. The article first discusses the disappearing efficacy of second generation anti-psychotic drugs. That problem is well known.
    All studies have a baseline, and some commentators believe the problem results from comparing every new generation of drugs to placebo. The first generation drugs were proven to be more effective than placebo in double blinded trials, and so were the second generation drugs. So how does one determine whether the second generation is more effective than the first, when the two have not been tested head to head? There doesn’t seem to be much of a basis on which to make claims of superiority, beyond researcher enthusiasm and drug company advertising.
    When a drug exists that is generally considered to be safe and effective, some commentators believe that newer drugs should be tested against the older, not against placebo. Only new drugs that are safer and/or more effective than their predecessors would be approved. Other commentators strongly condemn this research design because it leads to a shifting baseline over time.

  99. dave38 says:
    January 20, 2011 at 8:47 am
    W Abbott says:
    “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this
    It was said by Ernest Rutherford.

    Wasn’t it also Rutherford who said “All science is physics. The rest is speculation”?

  100. Most of the time, scientists know what results they want, and that can influence the results they get.
    Total crock. Written by someone who has never done an experiment.

  101. I’m sure The New Yorker didn’t intend this as an indictment of Climate Science. But their emphasis on replicability has been at odds with the way Climate Scientists do their work, as evidenced by their resistance to publishing their raw data and correction methods, even when faced with legal ramifications.

  102. I’m going to say something that will please my dad no end…
    Scientific Method is a lot like learning to play golf.
    When you first get some sort of swing you go out and try to use your driver on every tee you can within reason. Sometimes even without reason.
    You tee up, swing (or do something that looks like a swing) and hit the ball more times than not. You’re having a good day.
    Trouble is you don’t hit each shot very far or straight. In fact the worst ones are the 90 percenters..the ones that fly straight for 90% then curve off at the end into the trees…to which you learn about the truly versatile nature of a Wedge.
    But then if you do hit a semi-decent shot..you continue along that path, taking out the driver and hitting more duff shots than not…
    And it’s hard to stop getting out the driver. You believe you can hit it right. Just the next time.
    Once you see reason and especially if you start to BET on your games with your mates…you magically start to increase your good shot percentage..largely because you AREN’T getting the Big Dog out every single tee shot…
    Lo and behold humility on the golf course breeds results and consistency…and it is then that you realise all along what an ego-driven tool you have been from day one.
    And that the course never changed while you did.
    And you then learn to respect practice, consistency and how to swing properly so you can play ANY of your clubs.
    Strangely this is exactly the same thing that happens in scientific investigation. It is so much better when real “reputation” is on the line to produce conservative and good science than when money is thrown around willy-nilly and ANY result is celebrated and hyped…then you start to believe you can drive successfully each time.
    Trouble is you often didn’t actually hit the fairway the first time…but no-one calls you on it until much later. And then there’s all this self-examination. Oh the scientific method is failing.
    It’s not. There are just a lot of tools out there who should learn about humility, calling themselves scientists when they are really just theorists without evidence.
    And it’s only a bit depressing if you think just HOW many of them are out there.

  103. sHx says:
    January 20, 2011 at 10:02 am
    dave38 says:
    January 20, 2011 at 8:47 am
    W Abbott says:
    “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this
    It was said by Ernest Rutherford.
    Wasn’t it also Rutherford who said “All science is physics. The rest is speculation”?
    _________________________
    It does seem to come down to that…

  104. climatebeagle says: Any idea if “Tony from New York” was a real scientist?
    …No, I don’t, but it will be interesting to find out… especially if he is!
    The BBC is continuing its tradition on this issue of never letting any sceptical scientist anywhere near a camera or microphone. Rules of impartiality be damned! Let the AGW shill have the entire program to himself!

  105. JDN,
    You take the hate-filled emanations of PZ Myers as rational? That article was unconvincing.
    Pharyngula caters to the disgruntled [and there are a lot of them], so it was probably fodder to that unhappy subset of the population.
    In the most recent Weblog Awards for the internet’s Best Science site, it became more and more clear that WUWT was winning, so the climate alarmist blogs started piling their votes into Pharyngula, because even that blog was ahead of them.
    Pharyngula lost, but it would not have done nearly as well without all the alarmist blogs monkey-piling their votes onto Pharyngula, to try and keep WUWT from winning. They openly discussed their strategy on RC when it was apparent they were going to lose badly. Which they did, as the link shows.
    Normal folks can’t take much of PZ Myers. It’s hard to believe, but Myers is often more wacked out than Joe Romm.
    Well… almost.☺

  106. Look up Michelson and the speed of light. It slowly drifted to the correct value over time. Why? Well Michelson was thought to be a very careful experimenter. Confirmation entropy indeed.

  107. Murray Duffin says:
    January 20, 2011 at 7:38 am
    “Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits. As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.”
    This wide discrepancy suggests to me that acupuncturists in the US, Sweden and the UK are no good at it.

  108. A couple of Feynman quotes:
    “The first principle is that you must not fool yourself – and you are the easiest to fool.”
    OT, but relevant to the lack of a falsifiable global warming hypothesis:
    “I believe that a scientist looking at a nonscientific problem is just as dumb as the next guy.”
    I miss Feynman more and more. And Ike, too.

  109. —I recommend a new book by MELANIE PHILLIPS, who writes for the Daily Mail and the Spectator, on this woolly subject: THE WORLD TURNED UPSIDE DOWN, published by Encounter Books last year.
    She covers all the scares which have had us in their grip over the past few years, pointing out that we are really not thinking straight these days and far from being rational as we all believe, we’re headin back to superstition. One chapter, for instance is ‘How Enlightenment Unravelled’

  110. JPMiller hit the nail on the head. There are a lot of individual scientific phenomena that we understand quite well. Gravitational effects near the surface of the earth, for example. Galileo showed that a feather and an iron ball are both accelerated at the same rate. However, wher you try to do an experiment by dropping them from the same tower at the same time, the results contradict the fundamental scientific principle that the “consensus” accepts, because there are more phenomena at work here than just acceleration due to gravity.
    In the case of medicine, ecology, and psychology, and many other fields, it is not possible to isolate one phenomenon and then make a broad conclusion based on your discovery about that one phemonenon, because there are usually many phenomena at play in determining an outcome. We are a long way away from understanding enough about those fields to really call them “science”. Rather, they might best be called proto-sciences, where the facts have not yet come together enough to be able to make real, testable predictions. In some cases (e.g., the social “sciences”) I suspect that this time is far in the future, and these fields should just be called speculation.

  111. This does turn out to be a fine article, though it nearly lost me at the beginning by using the alleged superiority of the second generation of psychiatric drugs as a prime example. Those particular drug trials were simply rigged — not merely my opinion, but the FDA’s, which forbade the drug companies to advertise them as superior to the first generation. But the drug companies have bought so much of the psychiatric establishment, they didn’t have to say it in ads, their paid doctors said it in interviews with all the big publications, and the false message got spread. Now that the second generation is getting old enough to lose patent protection, we are allowed to “discover” it wasn’t ever that good, and rigged tests are being prepared for a third generation’s patents and profits. (See Robert Whitaker’s Anatomy of an Epidemic.)
    So the author does under-estimate the influence of lying and corruption. Yet by the end, he does persuade me that even without those factors, we would see problems.

  112. Pat Frank says:
    January 20, 2011 at 10:05 am
    “Most of the time, scientists know what results they want, and that can influence the results they get.”
    Total crock. Written by someone who has never done an experiment.
    Written by someone who has never done a grant applications Pat??
    Perhaps the most important implication of this excellent article is the failure of the self-correction peer review feedback loop to act early enough to cancel the influence of faulty research.
    That assumes there is any impulse at all to question fashionable results. The fact is that some researchers are able to insulate themselves from any real attempt to probe their findings by pronouncing “ex cathedra” that their conclusions are too urgent and significant to be put to the test.
    With the enviro reporters covering this being urged to “forget objectivity and go with your gut” the results, I theorize, are predictable.

  113. Micky H Corbett says: “…Scientific Method is a lot like learning to play golf…”
    Yeah. Looks like Michael Mann has been spending too much time working on his putts.

  114. Michael Larkin January 20, 2011 at 8:19 am :
    Has anyone considered the …

    When does the ‘pitch’ for (healing) ‘crystals’ take place in that video?
    ‘morphic resonance’? turns out to be a term coined by Rupert Sheldrake in his 1981 book “A New Science of Life” … the expression to refer[ring] to what [Sheldrake] he thinks is “the basis of memory in nature….the idea of mysterious telepathy-type interconnections between organisms and of collective memories within species.”
    Really? Believe in teleconnection much?
    Mann et al 2007 Precipitation Teleconnections
    .

  115. “The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand. ”
    If Crabbe had read Dougleas Adams he would realise that it was the white mice that are conducting the experiment , not the humans. They do this sort of thing and then analyse how we deal with it.

  116. As I see it the problem is as it always has been. Humans expect their enquiries, whether scientific or not, to provide them with answers. The reality is that the best we can hope for from them is the ability to ask better questions.

  117. As a scientist, I absolutely agree with that article. It is completely true. Repeatability is one of the founding pillars of science, and anything unreproducible is not science; just a phenomenon or observation (the lowest level on the scientific method, before hypothesis).
    The truth is, a lot of papers are published and only YEARS LATER does another lab come along and reproduce the results. It is when this occurs that suddenly it’s found out that things weren’t what they seemed as of the first paper. The antidepressants are a perfect example. More research was done only when it became apparent the original work wasn’t living up to expectations. And low and behold, it was true.
    But there’s another disconnect. Sometimes something in the lab is completely reproducible, but ONLY IN A SPECIFIC INSTANCE AND CASE. Only when the variables are exact, only when the absolutely same buffers and concentrations and test organism are used do those results occur. Once you transition from the lab to the real world, suddenly things no longer work like you thought. Even clinical trials, which are what test this transition from lab to reality, can be biased by picking a unique subject pool, and only that subject pool gets the results, while the majority of everyone else differs.
    This is the hard fact of science, and why medicine is still such a black box. In science we control the variables very exactly, but in the real world the variables have high variability and even worst are often changing on the fly–rather than being held steady. And there is where so much of the disconnect creeps into the scientific literature, and when going back with new machines and testing old procedures can suddenly grant you completely different results.
    Heck, in computer modeling, even the very processor you are running the model on will change the results, sometimes significantly so.
    So, I believe in the end, this dichotomy of lab and variables verses nature and reality, coupled with science sometimes being lazy (you don’t get funding for doing the same experiment as someone else! You only get funding for new experiments!) about reproduction of results, is what is leading to this current shake up of all our “facts”.
    The world is just so much more complex than a particular lab…

  118. One of the most perceptive observations about empirical science that I’ve ever encountered was the following: “Science is not like going to the supermarket”. (Unfortunately, it was in an article in a popular magazine that I’ve long lost track of – in a discussion on a dubious esp experiment done with a radio audience [!] – so I can’t cite the reference.) What the author meant (which I only appreciated after a time) was that the world does not come neatly packaged like goods in a supermarket; so the image of Science as a kind of consumerism where one wanders down the aisles and selects items of knowledge from the shelves is a travesty of the actual process of gaining empirical knowledge. Unfortunately, (and there are many reasons for this, including the great success of Science and the widespread learning of causal explanations without an appreciation of what goes into their derivation) too many people have this view of science. The problem becomes worse when empirical methods are extended to areas (like epidemiology and climatology) where really adequate control is simply impossible, without appreciating how difficult it becomes to establish anything. This is certainly part of the reason why “facts are losing their truth”; our shallowness is catching up with us.

  119. rw says:
    January 20, 2011 at 1:25 pm
    One of the most perceptive observations about empirical science that I’ve ever encountered was the following: “Science is not like going to the supermarket”… What the author meant (which I only appreciated after a time) was that the world does not come neatly packaged like goods in a supermarket; so the image of Science as a kind of consumerism where one wanders down the aisles and selects items of knowledge from the shelves is a travesty of the actual process of gaining empirical knowledge.

    This is also a very important point. The public is all too eager to accept prepackaged sound-bytes from experts on how to handle their daily lives. Very few actually dive into the details where the real understanding comes from. It would easily appear to the layperson that “our facts are losing their truthiness” when in fact the truth they had subscribed to was simply the label on the package. They had been considering a bazooka-joe gum wrapper of factoids to be gospel and they’re only now coming into contact with reality.

  120. As the article indicates, replication of results, by independent investigators, is important to generate confidence in ‘scientific’ findings but there can be pitfalls.
    Some while ago, I was asked to ‘teach’ Design of Experiments in a two-hour slot to 1-year Masters Courses in Design, Manufacturing and the like. I thought those in receipt of this might have the best chance of remembering something for their future careers if I made most of it ‘hands-on’.
    Thus I ran a quick experiment based on an 8-run, 7-factors at two levels, orthogonal array. It is by the way but this also gave the opportunity, if not the time, to debate the pros and cons of Taguchi Methods.
    The 7 factors, related to a spinning top, were disc diameter, disc thickness, height of disc underside above spinning surface, radius of spin point, length of grip shaft above disc surface, radius of grip, whether grip knurled or smooth. The response was the spin time until the edge of the top first touched the surface. At the end, I summarised the individual experimental sub-groups’ results to show the size of the effect (increased response time) that each had obtained for each factor. After all, they had all done the same experiment.
    In general, although the size of the effects (in seconds) often varied “between sub-groups within occasions (lecture slots)” and also between “sets of sub-groups between occasions”, the ranking of the relative importance of the design factors of the top was reasonably consistent over time. Until one day……
    At this point, I should say that I don’t have the results immediately to hand but, from memory, disc diameter had the largest influence on spin time, disc thickness very little with length of the grip shaft also negligible. [Cue discussion on how to optimise material costs to maximise response time.]
    As I was saying, one day amongst the usual pattern of results, two young ladies (females I think is the politically correct term) to their consternation posted the finding that a long spin shaft gave the longest spin time with most of the other factors negligible.
    To their everlasting credit, they were concerned about this (though perhaps they wouldn’t have been if they didn’t know about the other replications and only had their own results to go on – a not uncommon occurrence methinks). They borrowed the equipment to research the matter further in their own time and reported back to me that they both had long fingernails which prevented them getting much spin-time at all out of tops with a short spin shaft.
    Obvious, isn’t it! ?
    P.S. The experiment was done as an empirical one like so many in industry have to be, though I appreciate that there is a body of theoretical knowledge in the mathematics of gyroscopes.

  121. Anthony,
    Thanks for brining this study up. I keep forgetting that many individuals feel that data mining and correlations prove causation- (at 95%CI)- vs association. The comments reminded me of the thoughts of Peter Checkland in his seminal (for me anyway) book copyrighted in 1981 entitled “Systems Thinking, Systems Practice”
    http://www.amazon.com/Systems-Thinking-Practice-30-Year-Retrospective/dp/0471986062
    I highly recommend a review of his work as it covers the hard and soft sciences in the context of model building.

  122. “This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology.”
    Could the origins of the phenomenon be blamed on cheap Hindu Kush, I wonder. o_O

  123. W Abbott says:
    January 20, 2011 at 3:43 am
    “….if your experiment needs statistics, you need a better experiment” I wish I could remember who said this. It’s a big problem. We need better experiments.
    think it was Dr Michael Kelly at one the parliamentary inquirys

  124. Jeremy says (January 20, 2011 at 8:45 am): [on creationism v. evolution] “The whole argument would have died immediately 200 years ago if some scientist had simply kept his mouth shut and said, ‘Yes, uh-huh, I see how you could see it that way.'”
    I’ve always liked this way of putting it: “The past and present diversity of life on earth may have been intelligently designed, but if so it was done in a way indistiguishable from evolution.”

  125. This is a good description of how government money, which is always poisoned by politics and corrupted because it was stolen from the people at the point of a gun, has corrupted science to our great detriment.

  126. A couple of posters commented on the on again, off again nature of scientific media. Too wit:
    “Breast milk good, breast milk bad, breast milk good …….etc
    MMR good, MMR bad, MMR good ……….etc
    Statins good, statins bad, statins good ……etc
    London will be under water in ten years time, make that twenty, make that next year ….etc.”
    As a mother of three children, I can put to rest the first one. Breast feeding has got to be the funniest thing I’ve ever done. Laughed till I cried. First of all, the stuff comes out in seven different directions through tiny nozzles that are capable of squirting it into your own nose as well as that of your baby. Hell, I even had the stuff squirt across the ceiling in the maternity ward. Second, it is the best tasting stuff ever made. I love honey nut cheerios right out of the box. I love even more the taste of honey nut cheerios milk. That’s what human breast milk tastes like. And every mother I have talked to will admit that they have either tasted it on accident (I refer the reader to the multiple spigots in each nipple capable of taking paint off a house), or on purpose. No wonder kids love it. I don’t care whether or not the stuff is good or bad for you. It tastes good. Too bad I’m too damned old for that kind of thing anymore. It would be right convenient and quite tasty in my morning coffee.

  127. Smokey says:
    January 20, 2011 at 11:04 am
    You take this whole blogging thing *way* too seriously. PZ Meyers is another opinionated atheist who happens to have a decent opinion in this case. Lehrer isn’t *completely* wrong, just highly derivative and wrong about a number of important issues. He goes for a “big picture” view, as if he had actually done science and was now retiring and writing his memoirs so that we lesser scientists might profit from them. I thought Meyers did a nice job criticizing the article and that his take would be non-controversial. I was rather hoping not to write a line-by-line refutation of our lad Lehrer. Here’s an article more critical of Lehrer that hopefully hasn’t pissed off anyone at WUWT: http://www.psychologytoday.com/blog/neuro-atheism/201101/jonah-lehrers-decline-effect-now-in-decline. And here’s the person that Lehrer is deriving his “effect” from: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 although this paper too tends to take a bureaucratic “big picture” approach.
    I have no idea what Pharyngula’s opinion of WUWT or global warming is; I don’t go there for climate news.
    @Murray Duffin
    Has not. You need to read down to the end of the article where Meyers says:
    “But those last few sentences, where Lehrer dribbles off into a delusion of subjectivity and essentially throws up his hands and surrenders himself to ignorance, is unjustifiable. Early in any scientific career, one should learn a couple of general rules: science is never about absolute certainty, and the absence of black & white binary results is not evidence against it; you don’t get to choose what you want to believe, but instead only accept provisionally a result; and when you’ve got a positive result, the proper response is not to claim that you’ve proved something, but instead to focus more tightly, scrutinize more strictly, and test, test, test ever more deeply. It’s unfortunate that Lehrer has tainted his story with all that unwarranted breast-beating, because as a summary of why science can be hard to do, and of the institutional flaws in doing science, it’s quite good.”
    Obviously, I’m a lot more critical of Lehrer than Meyers is, but, Meyers is an amusing writer.

  128. Could the origins of the phenomenon be blamed on cheap Hindu Kush, I wonder. o_O
    Interesting hypothesis. Have you also considered the interference of the blond Lebanese guy in the experiments?

  129. Gary Hladik says:
    January 20, 2011 at 4:00 pm
    ‘I’ve always liked this way of putting it: “The past and present diversity of life on earth may have been intelligently designed, but if so it was done in a way indistiguishable from evolution.”’
    Sure, I do not disagree. But we have made a big mistake by accepting the formulation that Darwinian Evolution is true if and only if Creationism is false. It prevents us from making critical claims about Darwin’s account. For example, Darwin’s highest level hypothesis is “All species evolved from some other species, except the first one.” In all of science, there is no other high level hypothesis with an exception clause. We should take this seriously, but we cannot because our PC culture will not permit criticism of Darwin, unless you are Gould or Lewontin.

  130. In medicine, we have a cynical saying about this. “Quick, prescribe it while it still works!”
    In climate science, they could say about carbon dioxide production, “Quick, tax it while it still causes climate change!”

  131. Withouth going into details here, suffice to say that Hitler derived many of his more radical ideas, (as did the Nazis in general), from academic culture and beliefs prevalent in German academia from the late 19th to early 20th centuries. (Source: Weikart-”from Darwin to Hitler”). This is part of the reason the Nazis got into power and got away with so much, the ‘soil was prepared’ by academic ‘research’ for many decades prior.
    We are still at in in the 21st Century. The hypothesis from the 20th Century was that Drugs Cause Addiction. There was never any proof of this (look up the original heroin trials). But it was sold to the public and politicians. Comes the 21st Century and we are getting fairly unpoliticized science on the matter (contra monkey asphyxiation by Heath et. al.). And the NIDA now says addiction is about 50% genetic and 50% environmental factors. (Of course if they came out and told you what the environmental factors were – trauma – the persecution of users would be very difficult. Evidently there are still limits to how much truth you can tell on a government grant.) And yet no effort has been made to educate the public and lawmakers on this new understanding: you do not catch addiction from drugs. Drug use is a response not a cause. But ask the psychological intake nurse at the local hospital about it. The answer will come back – self medication. So at least the medical protocols have changed.
    So we are not over acting like Nazis (3 AM no knock raids) based on ideas known (by some) to be false. I’m going to add a couple of links to start those of you who wish to look into the matter further. They should be a good springboard for further research.
    Heroin
    PTSD and the Endocannabinoid System

  132. No wonder kids love it. I don’t care whether or not the stuff is good or bad for you. It tastes good. Too bad I’m too damned old for that kind of thing anymore. It would be right convenient and quite tasty in my morning coffee.
    I blame it on the anamides.

  133. M. Simon, I learned something today. I didn’t know that breast milk can give you a kind of pot hit. Could be why I laughed a lot as a young mother. I coulda fed an Army base hospital maternity ward back then. It even affected my kids. My first born toddler was wanting to wonder around while having his morning cup of milk, so I filled a tippy cup and gave it to him just to see what he would do. I sat him down with the cup in his two hands. He took a swig, slammed the cup down on the floor, and laughed till tears rolled down his chubby little cheeks.

  134. Wxcellent piece, although I find the title somewhat misleading. The science isn’t changing, it is merely the perception of the scientists examining it. Whether the delusion is deliberate or accidental confirmation bias is also worth examining. I would argue that the journals being biased towards publishing papers with positive results introduces an incentive for scientists to “find” positive results. I am sure the likes of Stephen Levitt (of Freakonomics fame) would have a field day with this stuff.
    In a nutshell, if so many of the rewards in academia/research revolve around being published, and publishers are biased towards publishing positive results, then there is sufficient incentive for scientists to introduce confirmation bias (aka cheat). Whether it is deliberate or largely sub-conscious is debatable, but I think the case for CAGW papers is clear. The incentives are huge, therefore the likelihood of dubious results being published is particularly high.
    I wonder if we are nearing a point where it will become “trendy” to publish counter-CAGW papers or whether the financial incentives will hold that at bay for many years yet.
    “Hide the decline” could take on a whole new meaning 😀 OK, just being facetious there …

  135. FRUTH. plu Fruths. Origin. faith and truth.
    Meaning. What you believe to be true.
    Examples: Tooth fairy, Father Christmas, Anthropogenic Global Warming.

  136. As somebody who considers the AGW theory to be broadly proven (subject to a lot of unanswered questions such as the extent – and sign – of cloud feedbacks) I would not generally post here but your pointer to this article is the single most useful thing I have read here – ever!
    What a brilliant piece, and what a great service to science generally it would be if it were compulsory reading for every scientist and also for anybody reading scientific papers.
    I think there may be two forces at work here, cognitive biases (which is what the article focusses on) and just the general weirdness of the universe – in particular the utter incomprehensibility of quantum mechanics.
    The classic demonstration of this is the fact that the result of an experiment changes simply by observing it or – more accurately – by trying to remove uncertainty from the result. The classic demonstration of this is the “two slit” experiment. In that experiment, if you do not know which of two slits photons pass through the resulting pattern they make on the detecting screen is a wave interference pattern. If you then put a detector in that will definitively tell you which of the two slits the photons pass through the pattern is indicative of a random particles passing through two slits. There is no change in the randomness of the photons, just on how they are observed.
    Maybe this quantum uncertainty effect has more subtle effects on the results of experimentation generally? Observing the world changes it. How weird is that!?

  137. Not just the soft ‘sciences’ which are affected…
    “The same holds for any number of phenomena, from the disappearing benefits of second-generation anti-psychotics to the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation anti-psychotics are still widely prescribed, and our model of the neutron hasn’t changed. The law of gravity remains the same.”
    My take is that the human ego is an almost insurmountable obstacle to making progress on the BIG questions in science.
    Our perception of reality is all through various proxies (our senses), which are not reliable in all circumstances.
    Much scientific data is just noise – a by-product of unknown/perceived unknowns we we can’t account for.
    Our beliefs are a form of blindness which prevents objective observation.
    The observer is an integral part of the system being observed – a small shift in position (and/or belief) can change the observed result.
    Statistical correlations and trends have no worth when dealing with deterministically chaotic systems and produce misleading results. They have little worth when used for analysing linear systems, unless sample size is enormous.
    So where does this leave science?….
    “We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.”

  138. UnfrozenCavemanMD:
    “Quick, prescribe it while it still works!”
    Now that’s funny right there.
    And a perfect synopsis of the New Yorker article.

  139. i was just reading about how the astrological signs have shifted. it’s God’s way of pointing out how assumptions and beliefs can change without negating the divine.

  140. Any day that I learn something new is a good day. Lehrer’s article crystallises some very important concepts — what I might have vaguely suspected to be the case I can now think and talk about with a new measure of clarity. Many thanks! It’s a great credit to Anthony and WUWT that such things abound here. And a pretty severe indictment of science as a whole that I get it from the New Yorker via WUWT instead of from those people and publications who purport to represent ‘science’ to the world.

  141. Pamela Gray says:
    January 20, 2011 at 9:13 pm
    I coulda fed an Army base hospital maternity ward back then.”
    Mamma Mia!

  142. What are the specific cases? Is it the new “research” that is bad or the old?
    (Health claims reported by the media tend to flip-flop, sometimes in short order, sometimes longer (remember saccharine vs cyclamate artificial sweeteners?). Some recent promotions may come from an agenda (one attack on canola oil looked to me like it came from an advocate of olive oil, others were just stupid). Considering all factors is another – a recent MSM article on coconut oil pointed to its much higher temperature threshold, because overheating oils can create carcinogenic compounds.
    I once was being given advice by a relative about dietary fat. While waiting in the living room while she was preparing dinner I skimmed through two books she had been praising – and noticed that they contradicted each other on a key issue. (She was counselling what she had wanted to believe because of her up-bringing – foods from her pre-teen years were magical. Years later her very bad behaviour convinced me that she was a skilled rationalist – using logic to justify a conclusion that had already been determined by her emotions (so pseudo-logic I suppose, often based on false premises). She had the skill in persuasion (good talker and pushy) to get further than many people do, so is more dangerous than average.)
    “polistra’s point about seeking the unknown factor is excellent”. David Harrimans’ book “The Logical Leap” includes that. It also covers whether science is superceded or just amplified and expanded (good science that is, I used to naiively think that phrase was internally redundant but words are so cheap to so many people). He also provides examples of scientists being slowed down or led off track by personal beliefs. “thingadonta”, I recommend Leonard Peikoff’s book “The Ominous Parallels” for backup to what you are saying about the path to National Socialism (“Nazi” in short) in Germany.

Comments are closed.