Sound familiar? A study revealing a stunning lack of reproducibility in psychological science triggers another instance of reluctance to share data with any but friends, and an “adjustment” of data to fit a theory.
Frank Lee MeiDere writes:
By now most have probably heard about the paper published in the August 28 edition of Science magazine entitled “Estimating the reproducibility of psychological science.” Coordinated by the Center for Open Science, and headed by its executive director, Dr. Brian Nosek, the project examined 100 psychological studies mostly from three sources: Psychological Science, the Journal of Personality and Social Psychology, and the Journal of Experimental Psychology: Learning, Memory, and Cognition.
According to the The New York Times’ article, “Many psychology findings not as strong as claimed, study says,” written by Benedict Carey: “The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions.”
In the end, 60 of the 100 studies did not hold up well when reproduced.
This, of course, is disturbing in itself. As Carey points out, “the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.”
Of more concern, however, is a quote from Dr. Norbert Schwarz, a professor of psychology at the University of Southern California: “There’s no doubt replication is important, but it’s often just an attack, a vigilante exercise” (italics added).
The problem is, that’s exactly what a replication is supposed to be: “an attack, a vigilante exercise.” It’s not a waltz with both partners in perfect step with each other; it’s a battle in which the invaders try every trick in the book to break through the castle walls. Schwarz’s attitude would seem to suggest a rephrasing of that famous justification for withholding data: “Why should I make the data available to you, when your aim is to try and find something wrong with it?”
Has this become the cry of all science? Hardly. In the physics world, for instance, new results are flung into the air for target practice like so many skeet. But then, the reproducibility rate in physics is pretty high — in fact, it makes up our entire technological world, virtually all of which is simply practical reproductions of laboratory experiments.
Other scientific foundations, however, are much shakier, and all too often their response is to ask everyone to sit quietly rather than examining the cause.
Psychology is especially prone to this shakiness. No sooner does one theory get established than it’s uprooted for another. In fact, there are numerous theories in play at any one time, each backed by its own body of “scientific” evidence. As Jelte Wicherts, associated professor of methodology and statistics at Tilburg University, Netherlands, said, “I think we knew or suspected that the literature had problems, but to see it so clearly, on such a large scale — it’s unprecedented.”
There are, of course, those who disagree with the findings of this study. The New York Times reports an email from Paola Bressan, a psychologist at the University of Padua who criticized the project for its reproduction of her study “Female preference for single versus attached males depends on conception risk.” Her complaint was that they used female psychology students as subjects whereas she had used female Italians. She is quoted as saying “I show that, with some theory-required adjustments, my original findings were in fact replicated.”
Any time the phrase “theory-required adjustments” is uttered there should be cause for alarm. This isn’t to say that it can’t be valid, but in this case we have to remember that Bressan’s original study was assumed to hold true universally (it wasn’t titled “Italian female preference” after all), and any adjustments made to “correct” the new study’s results in order to match her own could well be open to bias.
All scientific research should be considered “guilty until proven innocent” and requires an extremely strong defense to stand up to a justifiably hostile prosecutor. The idea of “settled science” is wrong in virtually every field, although there might be vast areas of strong replication and, therefore, confidence.
But at no point should we be under the old army edict of “Don’t ask: Don’t tell.”
Related stories where Stephan Lewandowsky, Naomi Oreskes, and John Cook produce some questionable, and perhaps irreproducible psychological “science”:
Lewandowsky and Cook – back from the dead with another smear paper
And then there’s Rasmus Benstad’s recent laughable paper that five journals rejected, before he and his psyops crew found a journal that would publish what other the five journals rejected. Yeah, that’s what you might call “robusted” science.
It’s called making stuff up. Or wishful thinking.
…or post-modern science.
…or post-modern science.
aka, lying about reality, because your political progress was retarded at the level of the starry-eyed left-wing schoolboy.
(…. and yes, I do know Jerome Ravetz was on here the other day).
There is no such thing as “Science.” There are only individual fields of study, some of which deserve being called a science, and others not. They don’t all really fit into one overarching category because the methodologies and criteria for what count as valid findings vary so greatly among them.
The panorama can be taxonomized as follows. First divide the fields of study into: A-the natural or physical sciences; and B-the social sciences. Then divide the natural sciences to separate: A1-those concerned with homogeneous entities and deterministic (at least in the aggregate) processes; from A2-the ones that deal with chaotic processes (like climatology).
Most of the progress in knowledge and technology comes from the A1 category. Although researchers in the other categories would like you to think they are making comparable contributions to society, they are not.
But you can take this even further. Throughout history much of the progress initially came from the tinkerers, inventors and engineers. The relevant sciences were discovered or substantially elaborated after the fact to understand why the things they created actually worked. The Romans built great aqueducts and the church produced grand cathedrals in the Middle Ages before materials science was developed. The Wright Brothers ran a bicycle shop before producing their airplane.
And unlike science, replication is not an issue in engineering. You may get away with “scientific findings” that can’t be reproduced, but not with a product that fails.
@RalphDaveWestfall,
“Throughout history much of the progress initially came from the tinkerers, inventors and engineers.”
Even in the Engineering which I am in, you have those who think you can just calculate everything out with a bunch of numbers and formulas. Unfortunately there is a the big problem called the “real world” where things just don’t add up to all those numbers.
…..or post-modern science.
Back in my day, they called it Natural or Pure science. Presumably to distinguish it from the other branches which were un-natural or impure….? but then we wrote our papers on clay tablets which took for ever to bake so the peer reviewers couldn’t ‘doctor’ our work! sorry about the nostalgia.
And many thousands go into medical internment with less statutory rights than a convicted child molester and undergo forced medication or “electrotherapy” based on the “expert” testimony of a psychologist.
Psychological Science is an oxymoron.
Lewandowsky’s “Fist of fury” paper is just the tip of the iceberg.
“Fist of fury”? What? Is he some kind of wannabe Bruce Lee fanbois?
Your comments suggest you are confusing psychology and psychiatry.
“Psychological Science is an oxymoron.”
Nonsense. There are plenty of replicable results – during the 20th century there were probably tens of thousands of student replications of some the basic response-reinforcement relations first demonstrated by Skinner with his Skinner box. That’s just one example. (I don’t know if college students still do such experiments, but many once did.) Or demonstrations of imprinting in certain birds. Or classical reaction time results. (It really does take 80 msec longer on average to classify “a” and “A” as the same letter versus “A” and “A”. Trust me, it’s replicable.) An endless number of other examples could be given. In fact, it’s of psychological interest to me that people often make such fatuous comments, statements that can be so easily refuted. What is the basis for the surface plausibility of such propositions? What is their appeal?
@rw “Trust me, it’s replicable.”
The issue isn’t whether psychology can do replicable studies or not, at least in some of it’s sub-fields such as learning theory and what used to be called physiological psychology. It’s why there are so few replications of studies, and why attempts to replicate so often show lesser effects.
And genuine sciences don’t work on the idea of “trust me.” That’s why verification is such an important issue. And why Phil Jones’s statement, “Why should I make the data available to you, when your aim is to try and find something wrong with it,” was so outrageous.
“…why attempts to replicate so often show lesser effects”
This has been deemed “The Decline Effect.” See my reply at 8/30 7:52 for an exploration of it’s causes… http://wattsupwiththat.com/2015/08/30/shades-of-lewandowsky-and-cook-when-psychological-science-isnt-so-sciency/#comment-2017923
Its not psychological science, its neurophysiology. That is why the results of the test are reproducible, you measure the speed of conduction of a nerve impulse. And that is the same for all healthy humans. No psychology involved.
Mike says: “And many thousands go into medical internment with less statutory rights than a convicted child molester and undergo forced medication or ‘electrotherapy’ based on the ‘expert’ testimony of a psychologist.”
Psychological Science is an oxymoron.
Mike, it would be helpful if you posted some source for your rant. This would be quite a bizarre story. Well, maybe you are no in the U.S.
In the U.S., someone may, BY DUE PROCESS, lose their statutory rights temporarily. Temporarily.
Why? Because the best “experts” available, combined with the best executers of law available, following established, Constitutionally vetted/tested processes, have found that person to be in imminent risk of harm of himself/herself or others.
This is nearly all regular POLITICAL process. If you don’t like it, you have to change state law, case law, or constitutional law. Mental health clinicians only play an expert role in this political issue of taking away someone’s civil rights, temporarily. For the purposes of 1 safety and 2 restoring that person to the point where they can AGAIN assume to be ad libitum, as thousands are, as are the rest of us.
In yet another exercise of our political system, along with someone being detained, they can have “treatment” forced on them involuntarily. Temporarily.
Electroconvulsive therapy is extremely rare in these cases. And, Mike, it is NEVER ordered or overseen by a psychologist. Never.
In your ignorant rant, you confuse “psychologist” with “psychiatrist.”
THEN, because our political system errs on the side of liberty, we release these individuals at the point when they BARELY are back in range to be walking around with the rest of us.
Then, a small portion go on to wreak havoc.
Now, regarding the “expert” issue: google “daubert” and “expert witness” and see what our LEGAL system is trying to do to maximally depend upon the use of “experts” in the legal system.
It’s political or economic based science?
“It’s called making stuff up. Or wishful thinking.”
AKA LYING.
And when grant money is involved, or products being hawked, it is AKA “Ripping Us Off”!
As I have said before a friend who I worked with would say “I always tell the truth…..as I see it” and it sounds like he could have used it in scientific work as well.
James Bull
Female preference for single versus attached males depends on conception risk.
====================
it has nothing to do with conception risk.
a guy without a girlfriend is seen as a loser and no girl is interested. show up at a party with a girl on each arm and all the other girls will want your phone number.
I was taught in grade school when we looked at pond water under a cheapy microscope that replication of results was the bases of ALL science, it wasn’t put in those words but the basic concept was there and even as a 10 or 11 year old I could grasp the basics.
As my schooling continued the concept was only reinforced. And in practical situations, like operating a nuclear submarine, my life depended on the science being replicated.
The real world also taught me that computer models are OK for initial testing of an idea, and a way to do so with less cost. But that before acceptance the idea had to pass actual replicated experimentation.
Lately replication of results doesn’t seem to be held in high esteem. And computer models are accepted even when the real world data contradicts the computer output.
I feel sad for those currently in school. And if this trend isn’t reversed I fear that the human race is headed off a cliff like a bunch of lemmings. (and yes I know lemmings don’t actually run off cliffs).
In my experience, we used experimental results and computer models together. When computer models based on the theory agreed with (reproducible) observations, the theory gained “supporting evidence.” If not, the models or theory were suspect, and the observation was king. Now it seems there are some seeking observations that confirm the models.
And if they can’t find any observations that fit the model, they just adjust the data until it does.
Hopefully you keeping model training and testing data completely separate and never debug test data results, otherwise this becomes, well, like climate science…
Peter
Ah, Scarlette, no, I don’t.
I taught at a city college for almost 10 years, and virtually every student started my class believing that a couple of references from second or third-level sources constituted proof. More disturbing, however, was their attitude that the references didn’t really matter; what counted was what they (the students) believed was right.
You are describing a common affliction of the human psyche. “I believe it, that settles it”.
That’s it in a nutshell, Alan, AMEN
(!)
the new students are even worse and why they bother with college must be just to get the ticket:
https://www.washingtonpost.com/posteverything/wp/2015/08/25/im-a-duke-freshman-heres-why-i-refused-to-read-fun-home/
As Feynman said, it doesn’t matter how beautiful the theory is.
It doesn’t matter how many scientists support it.
It doesn’t matter how eminent the scientists are.
If it disagrees with observed reality, it’s wrong.
That’s it. It’s wrong.
The linked article is a student’s explanation of why he refused to read a graphic novel that featured female masturbation.
When I was a student, the professor who prescribed the book would have been jailed.
While I agree that most university students should not be there, I withhold judgment on this particular student. A truly inclusive university would provide the student with an alternate assignment. I strongly suspect that the purpose of the assignment is indoctrination rather than teaching the student how to think for himself; another example of value-free education.
Frank Lee MeiDere do you give a damn? 😀
It’s beginning to become very evident that what constitutes “scientific” for one person or group, may not be the same thing another person or group accepts as scientific! It’s as if every college in the world defines it differently and they spew out graduates that don’t even agree on the overall parameters and principles of how “scientific results” are to be obtained!
I even read one poster at RealClimate call the newest Benstad et al paper AND the ones it critiques mere “rhetorical exercises”! As if what constitutes good, solid science work is how articulate and convincing you can be about what you did! It’s no longer about facts, experiments, results etc…it’s about how appealing your argument is.
No wonder so many of Cook’s and Lews and Benstads papers are filled with logical fallacies! They don’t understand that rhetoric can be ten times more poisonous than it can be helpful, and when it’s used by the unskilled, it reveals hubris and irrationality faster than anything else could.
Aphan, you make a very good point. While it is oft thought that a well-stated political argument is ‘convincing’ there is nothing so convincing as a reproducible result.
Even if a cause cannot be discerned, a reproducible effect is a guide to understanding. I have seen many interesting ‘abilities’ to predict performance or outcomes accompanied by frankly speculative causal mechanisms dismissed out of hand because there was no clearly articulately stated mechanism. Just because we don’t know how something works doesn’t mean we can’t rely on predictions made using it.
Most people have no idea how a calculator works but they have a lot of confidence in its outputs. The result is not invalid just because the user can’t explain how it works.
“I have seen many interesting ‘abilities’ to predict performance or outcomes accompanied by frankly speculative causal mechanisms dismissed out of hand because there was no clearly articulately stated mechanism. Just because we don’t know how something works doesn’t mean we can’t rely on predictions made using it.”
I think you have to be extremely careful in those cases. By statistical analysis you may be able to find correlation between variables even if there is no causal relationship. This correlation can then be used in a theoretical model. The theoretical model can then be used to predict an output which is reasonably close to a independent measurement of the output variable – for a while, or for a very limited range of conditions. Obviously the theoretical model may seems to have predictive skills, without actually having any. Or the predictive skills can have very limited validity. Also if you don´t understand the mechanism it is hard to formulate proper hypothesis, deduce necessary consequences and design clever tests.
Regarding the calculator example, there are people who understands how it works. Both the theory and the device has already been exposed to rigorous testing over many many years. A theory, hypothesis, idea, device or what you will is merited by the severity of the tests it has been exposed to and survived.
it is logical that heavier objects fall faster, therefore it must be true.
There you go Fred…What happens these days is that we don’t see so many testable hypothesis as we do “plausible arguments”. It is really apparent if one reads the warmist commenters in news feeds that are clearly smart people that have adopted a whole series of “plausible” justifications for their views without actually investigating veracity of the source of the “plausible story”. Of course, in case you stated, there is no way to test it until you invent a way to conduct the experiment in a vacuum to discover that, indeed, the feather falls just as fast as the stone!
Psychology you say?
Stephan Lewandowsky, Winthrop Professor School of Psychology University of Western Australia
http://websites.psychology.uwa.edu.au/labs/cogscience/Stephan_Lewandowsky.html
How about we start by reminding everyone what peer review really is?
…and halfassed spell check
Psychology, mostly bullshit.
Probably applied most to the folks who thought it up…
Like here…
Bob Newhart psychologist:
I have to agree with Paola Bressan. It’s difficult enough to reproduce one study because every detail counts. Reproducing 50 studies without the help of the original investigators seems likely to be useless garbage. I’ve noticed this trend recently where it’s now cool to dump on scientific reliability. The main stream media will now run with the story.
Hasn’t anyone noticed that the “reproduced” studies are not open? They usually don’t say what studies they tried to reproduce, or what equipment/reagents they used. In the case of Paola Bressan, the use of a different self-selecting group should have ended the discussion. Paola is right and they are wrong. But, it looks like a new anti-science agenda is brewing. Watch for more political hacks to jump on this bandwagon.
” In the case of Paola Bressan, the use of a different self-selecting group should have ended the discussion. Paola is right and they are wrong.”
So, you really think that only Italian women can be used to prove a universal about all women? Really? The study did not claim to be only about Italian women.
And then there is the issue that self-selected subject groups prove damn little anyway most of the time other than about self-selected groups.
I am reminded of Mann’s few pine trees being used to “prove” a universal.
“So, you really think that only Italian women can be used to prove a universal about all women? Really? The study did not claim to be only about Italian women. ”
So, nobody should be allowed to conclude anything about human behavior unless every human has been examined? Students are usually self-selecting. Any conclusion needs to add that caveat. So, I side with Paola, that her study was not adequately reproduced. It’s a given that the study can only conclude about Italian women, and awaits further testing on other groups of women.
“It’s a given that the study can only conclude about Italian women, and awaits further testing on other groups of women.”
And so, she will be changing the title of the study to minimize the falsehood?
JDN: No, I would say that nobody should be allowed to conclude anything about human behaviour unless numerous subgroups have been examined. That seems pretty much like a given to me. Any study on standards of beauty, for instance, that only considered the views of white, Anglo-Saxon males from California would be dismissed out of hand as being totally biased if it made any claims concerning universality.
JDN I have to side with markstoval. You say
“So, nobody should be allowed to conclude anything about human behavior unless every human has been examined? ”
Well not all. But a good cross section on cultures so that its shown that the Behavior is not deviant or unique.
I have no problem with accommodating publish or perish, but don’t set yourself up as a target by pushing junk. People tend to use these studies in education Bad Bad bad. No need to warp young minds and give them complexes.
michael
JDN, if a study only is valid for a small cross section of the human population, then by definition, you have learned nothing about human psychology.
That should be simple enough for even a psychology student to understand.
JDN is a perfect example of what is being discussed here.
Blind to his own blindness.
Dont forget that behaviour is strongly influenced by cultural values at a time in history. You may as well say the study has relevance to a group of Italian women. And name them.
Ahh what you all don’t understand is those lusty, sultry, italian women responded only to the most virile multiple partner italian stallion studs. Clearly indicating their desire to copulate in the middle of of their menstrual cycles when conception is most likely. Thereby confirming every single cultural stereotype that Poala has cultivated in her life. Of course they were “self selecting!”
All of this discussion goes especially true for psychology studies, as there is a huge confounding issue: culture. You cannot conclude anything about humanity’s psychology as a whole unless you control for as many aspects of individual cultures as possible.
The studies were done with the hep of the original investigators. The New York Times article says:
“Dr. Nosek’s team addressed this complaint in part by requiring the researchers attempting to replicate the findings to collaborate closely with the original authors, asking for guidance on design, methodology and materials. Most of the replications also included more subjects than the original studies, giving them more statistical power.” To me, however, this also means more of a chance of including methodological mistakes and experimenter bias.
More to the point, however, is that if a study is intrinsically difficult to reproduce, then it is correspondingly useless as a source of information.
Needless to say, you can’t supply any evidence that you’re correct. I was going to silently ignore you, but I thought that I might say for the record that your arguments fail and that my statements here were properly qualified.
This is a constant problem with scientific literature. Scientists know how to read it, and we know the game. People such as yourselves who think you are scoring points are using rhetorical tricks are going to be ignored.
Replication requires a really close collaboration and back and forth. Details that are significant to the original investigator (because he/she has invested a lot of time adjusting them) are not going to be significant to someone outside the field attempting to replicate a result for an article that, let’s be honest, expects to fail to replicate things and dump crap all over science.
That’s another problem. If a scientist working in their own field attempts to replicate one or two results he/she finds troublesome, such a person will take care to get it right and will be open to the idea their colleague might be completely correct (albeit with some caveats). People attempting bulk replication because they feel that the experimental method has failed have no story to tell if everything works out as published. In other words, they have experimenter bias to not replicate the original studies.
Our discussion up to this point has been myself pointing this out in terms of the one researcher that objected and the rest of you trying to trip me up with semantics. The last replication of an experiment I read (that was successful) was 8 pages in print. If these jokers replicated 50 experiment, then at least 250 pages should be dedicated to their results. It should be a small book. It’s not, so I don’t believe they are giving us enough information to evaluate their work. I keep trying to tell you guys that this sort of opaque science is the kiss of death for replication failure. People are going to start counterpunching along these lines. As far as I’m concerned, this movement to dump crap all over probably crappy science is also full of crap. Sometimes, the enemy of my stupid & corrupt enemy is an even dumber, more corrupt enemy.
Time will tell, but your comments do nothing to change my mind. I remain hugely skeptical of these reforming skeptics.
JDN: I don’t understand what you’re objecting to in my comments. As I’ve said before, I’m neither endorsing nor condemning the project itself; my issue is completely different and, I think fairly clear.
@ JDN
You are saying that you believe that most studies can not be replicated and are not generally replicated. What field of “science” do you work in? No paper that reputes to demonstrate important results is worth a darn if it can not be replicated and in fact is replicated. The methods and data to do so should be in the paper. The scientific method is not based on “just trust me”.
Gee, I guess if we all agreed with Paola Bressan (“…without the help of the original investigators…”), that pretty much means we’re done testing Einstein’s relativity (for the psychologists in the audience, he’s dead…) relativity. Won’t even bother mentioning Newton, Et al.
This is the problem with psychology and other “observational sciences’: too much “trust me” and “secret sauce” compound by an “observational science” cultural distaste for reproducibility.
The good news is we have Einstein and Feynman;
the bad news is they have Lewandowsky and Cook (and, apparently, Bressan).
If you don’t include enough details in your paper that others can reproduce your findings, then you aren’t doing science, you are doing propaganda.
I love the way you assume that when others can’t reproduce the original study, it’s because the others just aren’t doing it right, not that the original study was faulty.
And from there you jump straight to the notion that anyone who demands reproducibility is anti-science.
That’s pretty close to troll territory there bud.
I don’t what is so complicated in the criticism of the Bressan study. I am sure the study could be replicated by using Italian women, in equal proportion according to age, wealth, family history, health, build, social status and career choices.
Whats funny is that even if the participants were replicated exactly as per the above demographics and characteristics, it would not be surprising if there were different results. But no problem throw in a bit of “theory-required adjustments” and you can get any result you want based on your personal preferences.
There is no anti-science bandwagon except in your fevered imagination or people who believe in Noah’s arc and men at one time having dinosaurs as pets. The issue is the idiots who proclaim to be practicing science when in fact they skip the most basic tenants of scientific practice and jump right to biased conclusions. It becomes worse when poorly derived delusions then became the basis for political or social activism. Then we get what in the most scientific terms can be called a festival of crap.
Here is something easily reproducible even if data are not exactly the ‘real thing’.
http://www.vukcevic.talktalk.net/CT4-pause.gif
As it can be seen during 160 years of the ‘data’ global temperature was static for more than 100 and rising for only about 50 years.
In Crutem4 quasi-data the current pause is only 10 years, many would think that’s far too short and I agree. I went through whole span and looked for pauses longer than 10 years, than extended in both direction until pause can not be re-established any longer; result is two pauses around 50 years of duration.
If that is the nature of the climate data, I extended current pause into the to 50 years in total (shown in green) by integrating data from the two preceding pauses + constant.
So what is the purpose of this exercise?
It is a preliminary warning that the CAGW fraternity may keep extending their limit for ‘models failure’ from the original 15, to 20, to 30, 40 or even 50 years and still claim that models are working.
You have been warned!
Thanks for the warning.
The seasoned climate scientists finished their dissertations 25-30 years ago, so around 1985-1990. Not much global temperature rise in their direct experience. I wonder if the next generation, writing their dissertations now, will push on like their advisors, or will they really look critically at the body of climate science.
Oh, and vucevic, “pause” and “hiatus” are passé. Never happened.
…But at no point should we be under the old army edict of “Don’t ask: Don’t tell.”
==============
You do realize you just posted on the internet, which by definition means anything goes, right ?
Ain’t no edicts here.
My first psychology textbook back in 1972 had a chapter on deviant sexual behavior. Homosexuality and lesbianism each had a sub-chapter. The following year a vote was taken among the membership of The American Psychiatric Association that declared both were now perfectly fine. (45% were against that). No new studies. No new research. It was a political decision to remove them from “the list”. (Pressure from Stonewall Union and all that.) As far as I know, what Brucette Jenner did to himself is still on “the list”. Yet, I could get in trouble at work (Government job) if I said that.
“Reproducible Science”. Not in these fields.
I don’t think that “perfectly fine” is the opposite of deviant. Generally, deviant (meaning deviating from the norm), is the opposite of normal (statistically normal).
Psychological science? What?
It must have been a typo, I thing it was supposed to be “pathological science”.
“Stephan Lewandowsky, Naomi Oreskes, and John CooK”
They affirmed and reconfirmed findings. Original findings were replicated. Why bring them up? The study you report on is on findings that were not replicated and that replication shows many effects to be doubtful. Stephan Lewandowsky, Naomi Oreskes, and John Cook are not part of that as far as I can see.
Lewandowsky’s original findings were replicated by whom? He refused to provide his data to researchers, so how could anyone replicate his findings unless he did it himself?
“Stephan Lewandowsky, Naomi Oreskes, and John Cook”
All got similar results. Replication done.
When someone outside of their camp is allowed to use the data, it will be much more ethical science.
Wagen my my
“Stephan Lewandowsky, Naomi Oreskes, and John Cook”
All got similar results. Replication done.
Do you feel clever now? Is it heh, heh I showed them?
Did you not think someone, might have studied just a little bit of psychological and education to see the scam.
Lets play, lets talk of first “Maslow’s hierarchy of needs”; then we shall go to Vygotsky’s sociocultural theory, And finally Howard Gardner’s Theory of Multiple Intelligences.
Oh and everyone who knows what I speak of, come play too.
How does Lewandowsky and Cook writings reflect on these educational theories,
michael
mods just pitch my comment from 3:21 PM, moot by now.
miky
Cook – conflict of interest. They reviewed each others papers, not so?
Wow, three people get similar results, none of which provide details of their methods to others. And in your “mind”, this constitutes proof that they all are correct?
They all agreed on the desired conclusion. “Replication done”.
(Will they do it again?)
Because they chose not to cooperate with the study? These were not randomly selected studies, they were recruited and required cooperation of the original authors.
Wagen
You plainly do not understand the issue.
ROFL
Wow, you really do live in an alternate dimension?
Lewandowsky et. al’s results have never been replicated and have been refuted multiple times.
But don’t let reality get in the way of a religious conviction.
I think I know who you really are Wagen.
So do several others here.
Ergo, you are a troll.
Several agree.
Case closed.
I’m sure that Lysenko had a bunch of cronies who could replicate his results, too. When evaluating research results it’s also worth subjecting them to the cold light of common sense (if one has any). When Oreskes says that all 928 of the studies she examined supported AGW or Cook et al. say that 97% of climate scientists (or was it scientists?) support AGW, doesn’t that sound a mite suspicious? In a field where there is no hope of controlled observations? And when one actually looks at their methods, I can’t see how any sane, intelligent reader can come away convinced. I don’t know about the Bressan work (and others that were checked – maybe they were just wrong; that happens even in physics), but the Oreskes, Lewandowsky, and Cook stuff is genuine faux science.
Alex Berezow: ‘… But to claim it [psychology] is “science” is inaccurate. Actually, it’s worse than that. It’s an attempt to redefine science. Science, redefined, is no longer the empirical analysis of the natural world; instead, it is any topic that sprinkles a few numbers around. This is dangerous because, under such a loose definition, anything can qualify as science. And when anything qualifies as science, science can no longer claim to have a unique grasp on secular truth’.
http://articles.latimes.com/2012/jul/13/news/la-ol-blowback-pscyhology-science-20120713
Psychology (and economics and sociology) is a “field of observation”, not to be confused with science.
Nothing wrong with that…unless you happen to be jealous of the academic respect shown physicists (among other sciences).
Good observation. Nothing of substance to observe.
“Theory-required adjustments.” Psychology isn’t the only “science” where “theory-required adjustments” are made to data. In climate science records of temperature seem to require adjustment quite often and we all know what the theory is that requires those adjustments.
Perhaps “Conclusion-required adjustments” would be the better term?
That’s why it is called “psychobabble.”
There’s that word “stunning” again…
Those were the top 100 psychology papers for all of 2008 in the top 3 psychology journals. And they found only about 40% were reproducible, in the other 60% the major findings/conclusion couldn’t be reproduced.
In 2008 there were of course many thousands of peer-reviewed psychology papers published in 2nd and 3rd tier journals as well. Doubtful the replication rate would be any higher, and much more likely to much worse.
Litterally the entire field of scholarly academic psychology publshes mostly garbage. It comes from the intense pressure to acquire tenure via publishing papers, any papers, no mater how bad or shoddy the underlying work was/is.
But before hard scientists gets on a high horse, it doesn’t fare well in reality either. Remember the BICEP2 retraction fiasco in 2014? And of course the on-going fraud of the century, catastrophic anthropogenic global warming. CAGW is now hanging on by the thinnest of threads of dubiously manipulated and cooked surface temperature datasets.
Climate science is not one of the hard sciences. Never has been. Hard sciences go out into the field and collect and analyze actual data. They don’t use statistical “tricks” to compensate for missing data, or data that has been polluted by confounding affects.
And they certainly don’t use computer models for “proof” of anything.
Wait, it’s worse than we thought! One of the criteria for selection of the reviewed studies was cooperation by the original author(s). How many fine upstanding psycho researchers chose to take a pass because they knew that the original work was .. a) not the most rigorous, b) just plain shoddy, or c) an outright fraud?
The world of physics is sadly no better. If it were you wouldn’t be hearing about absurd magic like Dark Matter, Dark Energy, Black Holes, Snowball Comets or Back Radiation!
You take issue with the idea of black holes…? I’ve heard of people doubting pretty much all of the rest of those, but… black holes? What about those is remotely controversial?
The concept of infinite density makes no logical sense. What really happens is the relevant theory breaks down. No one can say if they exist.
I have no reason to doubt their existence, other that the absurdity of the concept…which means nothing.
After all, basically all of quantum mechanics is seemingly absurd from a macroscopic perspective.
That the principles have been confirmed experimentally does not really change that.
Menicholas:
Indeed. Which is why there is no such thing. What you have is a singularity from which no information can be retrieved, thus no physical laws can be written to describe. The nonsensical idea of “infinite density” just arises from the singularity being a case where the universe would be dividing by zero. Complaining that it can’t do that, and thus the laws of physics as we know them can’t apply to the singularity inside a black hole, misses the point of a singularity. Singularities, by definition, are points where the laws of physics as we know them are accepted to cease to apply (the same is true for the Big Bang).
But it also misses the point that a black hole isn’t a singularity. Black holes have singularities inside them, but singularities are basically single point entities with zero volume while black holes can have significant amounts of volume. So you’re not even complaining about black holes themselves.
That we can’t explain how the state at which the Big Bang happened appeared doesn’t prevent us from being able to say that state existed. Similarly, not being able to understand how the singularities inside black holes can exist does not prevent us from being able to acknowledge that they do exist, much less that the black holes which contain them do exist.
In no way does lacking an explanation for something prevent us from acknowledging its existence.
Black holes can’t exist http://www.dailymail.co.uk/sciencetech/article-2769156/Black-holes-NOT-exist-Big-Bang-Theory-wrong-claims-scientist-maths-prove-it.html
Brandon. Yes yes.
I know all of that. I am sure you know it better than I do too.
But the possibility remains that there are other explanations for the data that have led us to these conclusions…just no one has thought of them yet.
Or maybe someone has, but no one is paying attention to them.
Or maybe we simply cannot understand what is truly the case.
Or it may all be exactly as presently understood by mainstream cosmologists and physicists.
We could even give odds on which on how likely each of the above scenarios is…but I believe that would only reflect the bias, or imagination, or lack thereof, of whoever the odds maker was/is.
Anyway, you asked what was remotely controversial. I was not saying I was leading a controversy.
And I did not mean to imply that I was offering an expert analysis of the state knowledge and any controversies or lack thereof.
Plus it was very late and I was very tired.
So…just sayin’.
🙂
wickedwenchfan
In so far as I know no PhD physicist claims Dark Matter, Dark Energy, Snowball Comets or Back Radiation to be anything other than interesting but explicitly unproven theories. Black holes are much further along the path of having testable theory and supporting observational data.
Nothing wrong with (even psychologists) having unproven theories. The problem comes when parties (i.e.: Lewandowsky and Cook) claim their crap is universal law simply because of their innate moral superiority and skeptics innate badness.
With a few goofball exceptions, PhD physicists would not claim or recognize “moral superiority” as reasonable proof (no theory is ever “proved”; I’m using proof in the common usage for the term).
True. Physicists generally make no bones about the fact that each of these is an ad hoc explanation.
How about cosmic inflation?
Talk about ad hoc!
That Alan Guth sure knows how to talk some people into some stuff.
Hear tell he dreamed it up at a party one night.
With the exception of dark matter and dark energy, what’s wrong with any of those?
Even dark energy and matter are still considered theories, nobody has ever claimed them to be proven.
It takes time. Most people still think gravity is a force of nature, while there is supporting evidence otherwise.
Science is a frame-based philosophy, designed specifically to constrain secular excess. Today, it has lost its perspective.
We can’t even agree when human life begins, even though the scientific evidence and self-evident knowledge is unambiguous, and the consequences have been brutal.
That’s one giant leap for mysticism. Several thousand years of progress down the toilet.
n.n you are confirming that a lot of what passes for science is philosophy. Philosophical conclusions are quite permissible of course as long as they are presented as such. The Platonic ideal of an absolute truth or perfect example, a prototype, is not accessible to us even if we were to seek it. We are too flawed.
It is philosophical arrogance to think that anything we accept as ‘proven’ can never be overturned by new evidence. Thus the true scientist is humble. They dare merely to push the light a little deeper into the darkness.
Climate science is most noted for the arrogance of its exponents and also it’s philosophy: that puny Man has gained control of the climate accidentally in the night, as if by a wand’s wave. We understand nature enough to make it do our bidding. We are all-powerful, a last. Anything bad that happens is, ergo, caused by the misuse of knowledge or misuse of the resources. We are all-powerful yet fatally flawed.
It sounds like a Greek tragedy which during the final scene, everyone dies. Climatism is just another aspect of materialism. Only what is material exists, all else does not. It is a vacuous philosophy. One divided by a neutron star.
Modern science (if that is what it is) is driven by funding and publicity and is further compromised by ideas of the conscience we call “political science”. Rather than sceptically approaching results, researchers now produce results that reflect the ‘consensus view’ that has either been formed or which they are trying to form.
As a result, we have pharmaceutical studies which are used as proof of efficacies which don’t exist, medical doctors who don’t examine the evidence presented by the patient, psychological studies which can’t be reproduced and global warming proofs that have time lines beyond the range of normal human lives. All these are performed to ensure steady funding streams and headlines.
In short, we have entered a modern dark age in which true scientific method is forgotten and crowd-sourcing is seen as a valid measure of whether a result is correct.
Reminds me of “Max Headroom” the TV show.
Anyone who thinks replication of results doesn’t matter (such as most so-called “climate science”) needs to go back and re-read the story of “Cold Fusion.” Good intentions, crappy and false results.
Psychology I know. And the issue is not what you think it is. At issue here is the unreproducible unknown or immeasurable variables involved in underfunded research, not to mention the politically incorrect nature of hot-button-issue areas of research.
Let’s just take factitious syndrome in children (look it up). This is when humans (or lower animals) fake a debilitating conditions in order to garner sympathy and attention. Killdeers do this perfectly, instinctively knowing that if they fake a broken wing, a predator will follow them, thus without their awareness, saving their DNA in the lives of the chicks not consumed by predation (else you have to believe in parental altruism in the breed). This is so ingrained in the species that no learning is required for the species to exhibit this behavior.
However, in humans faking illness or condition is a political hot button. Take for example children and adolescent adults who fake a learning disability. Can you imagine the uproar in the educational and parent community if it was discovered that not a small percentage of young humans fake an educationally relevant condition for the sole purpose of attention? And can you imagine how difficult it would be to measure such a phenomenon? The individual variables, known and unknown, make it a wickedly complex condition to study with any amount of validity or reliability. Which means that loads of coinage would be needed to fund a study large enough to result in repeatable results.
Since we are spending loads and loads of tax payer dollars on (cough cough gag) human caused climate change and weather, any left over dollars spent on psychologically and moniterially important (think of all the money spent on one student who might be faking a condition) research means that we can at most study maybe, on a good day, 20 to 50 kids. This means that, guaranteed, the results will not be reproducible.
So instead, the one or two (or more) students per school in a district are pulled out of regular education for loads of specially designed instruction that they don’t need, resulting in SEVERE detriment to the student and fewer dollars available to students with real educationally relevant disabilities.
For this educator, psychological research is extremely important but underfunded to the point that studies cannot be replicated, not because they are poorly done, but because not enough money is available to provide for the number of subjects needed to overcome the issue of multiple variables inherent in such research.
There are two sides to every issue, and this one has a side related to underfunding, not poor research methods.
Pamela there is a further complication when there is a suggestion that a pharmaceutical solution is available to “cure” the perceived problem and loads of clinical study looking for a cause finds one!
Cogsys
Nice rant, but most of it simply is eye-wash.
Political science, psychology and “global warming” do not use the scientific method, and, thus, are not sciences (they are “fields of observation”).
The scientific method is not forgotten (popular internet material, written by non-scientifically trained journalists IS NOT SCIENCE). I would refer you to the recent search for the Higgs Boson – that definitely was not decided by a “consensus” vote; nor was the technology in your smart phone, PC, GPS…
If you choose to further argue this point, knowing your educational background would be interesting (mine: Ga Tech physics).
This column misrepresents the effort of the Reproducibility Project to some extent.
Those wishing to really understand this issue need to go to the original sources.
The NY Times article:
http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html
about this:
Reproducibility Project: Psychology https://osf.io/ezcuj/
The original report summary is here:
Estimating the Reproducibility of Psychological Science https://osf.io/phtye/
The online resources include virtually all of the data on all of the attempts at replication. Note that in most cases, the authors of the original studies, those being replicated, were involved in the planning of the replication attempts.
This effort in Psychology is an example for other fields….and there are many….in which one off studies, iffy at best, are leading whole fields astray wasting research money and efforts chasing down ideas that should have been discarded early on but that have been buoyed up by unsupportable, unreproducible findings.
(This project has no relation at all to the nonsense perpetrated by Lew et al. The project is a serious attempt to understand what has been going wrong in psychological research.)
[Please reserve square brackets for the mods. .mod]
(Why? Where is that rule written? –snr. mod)
This article wasn’t meant as a criticism of the project, but rather a focus on the quote regarding attempts at replication as being an attack.
Reply to Frank ==> I’m sorry, that was not all that clear to me. There is a movement afoot in the greater Science world that is trying to rein in the worst abuses in many fields. The Reproducibility Project is one of them, and focuses on Psychology. There are other psychology professors in Europe trying to teach responsible psychological experiential design. Armstrong is making some progress in Forecasting science. I have an essay in the pipe on a CliSci related field trying to sort itself out.
These are all good things.
Here at WUWT we have seen many (way too many) essay’s that are really nothing than slash-and-hack so-called refutations or debunkings “but it’s often just an attack,” nothing more than “a vigilante exercise”. If you are a regular reader here, you will recognize this fact, and can probably list a dozen or so of the more obvious examples of this.
You have been misled by a statement in the NY Times….the full section is this:
The quoted Dr. Schwarz was incorrect if he was referring to the replication attempts by the Reproducibility Project — possibly he was entirely ignorant of it — he certainly can not have read the paper, the methods, or the structure of the project. The NY Times journalist let the public down on this one, as he failed to ask Schwarz if he was familiar with the project, its methods, and findings before asking for a quote.
The NY Times today carries this:
Psychologists Welcome Analysis Casting Doubt on Their Work
Take home quote:
Kip: I actually had the information about Dr. Schwartz not being part of the project, but in reducing and condensing the article in edit I appear to have deleted it. Regardless of whether he was talking about this project or not (and it’s still possible he knew of the project or had been told of it by the NYT columnist who got the quote), his attitude still strikes me as dangerously close to that of Phil Jones’s reaction to being asked for data.
And your criticism of the NYT article hardly goes far enough. It’s disjointed and you have to search through the entire thing just to get the information that should be given up front, such as who is doing the project and so on.
Kip: Also, yes, I’ve seen the kind of knee-jerk attack you’re talking about, but these attacks never bring anything useful to the table, nor do they attempt to replicate anything. But it doesn’t matter whether a replication is done in a friendly spirit of scientific inquiry or as a personal grudge against the original experimenters, if it is methodologically sound, then it is valid.
Reply to Frank ==> You are right and right again.
The NY Times article was, like yours, probably edited down to its deficient published length (we’ll give the journalist the benefit of the doubt), taking out the qualifiers.
If Professor Schwarz was familiar with the RP, then his answer is disingenuous as well as incorrect. The RP went to great lengths to make sure that the replication attempts were as close to the original work as possible, involving original researchers in the planned replications when possible, and increasing the cohort sizes to increase significance of the findings. The RP was not trying to falsify studies, not trying to debunk studies, not trying to refute studies — it was trying to replicate them, to validate them. The difference in these approaches is almost 180°.
In the field of medicine, this list of reversals in clinical practice after re-examination or failed replication of previous studies is a hair-raising example of the work needed in all fields of science.
Speaking of Lewandowsky et al., Dana Nuccitelli writes in the Guardian that those papers outside of the official 97% consensus have major problems:
“This new study was authored by Rasmus Benestad, myself (Dana Nuccitelli), Stephan Lewandowsky, Katharine Hayhoe, Hans Olav Hygen, Rob van Dorland, and John Cook.”
http://www.theguardian.com/environment/climate-consensus-97-per-cent/2015/aug/25/heres-what-happens-when-you-try-to-replicate-climate-contrarian-papers
“Instead, as our paper shows, the contrarians have presented a variety of contradictory alternatives based on methodological flaws, which therefore have failed to convince scientific experts.”
Hold on to your hats when reading that article, it’s disturbing and amusing at the same time.
Green energy, climate, sociology, anthropology, psychology. What do they all have in common? They are controlled by third rate scientists who are really left wing ideologues.
They didn’t start out as left wing ideologues. It was inculcated over a period of about ten years.
Obama remarked that a trades education might produce better financial results than a four year degree in Art History. Professor Ann Collins Johns took him to task and he quickly apologized.
I heard the good professor interviewed on CBC radio. She was really proud that her students could produce a reasoned argument based on scanty evidence. That’s nothing to be proud of. It’s mere pedantry.
In Voltaire’s Bastards, John Ralston Saul makes the point that this rationality, divorced from reality, is having a seriously pernicious effect on our society. In The Master and his Emissary, Iain McGilchrist provides the neurological basis to explain the science behind Saul’s observations.
We’re in deep serious doodoo thanks to the efforts of folks like Prof. Johns.
@commieBob, why are you still listening to CBC is it just the “know thy enemy” thing?
There is still some excellent stuff, which is amazing given how their budget has been hacked. To get me by the painful crap, I have Scarlatti on autorotate on the headphones.
The Saturday and Sunday morning shows usually keep me from my chores. A large part of the over-night programming has shows from foreign broadcasters: ABC, BBC, Deutsche Welle. The ABC science programs are well worth listening to, as opposed to Quirks and Quarks which is mostly a snooze fest.
The good stuff is very good. The painful stuff is easily avoided.
“According to the The New York Times’ article, “Many psychology findings …””
I found that NYT article with a Google search, but should there not have been a direct link to it?
http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html?_r=0
I agree. Post authors: Please put a link to the article (if there is one) instead of making your readers do that for you. Yes, the search is not as easy as it used to be just a year ago, but do it anyway.
There was when I submitted it. I guess it got lost somehow. (I also didn’t connect this to Lewadowsky et. al.)
Unhappily, it’s not only psychology and the social sciences, but also medicine, cancer, genetics, etc.
@RD, you are right, just look at what happened to “dr” Suzuki.
It’s really widespread across multiple disciplines. Not to mention exaggerated press releases that are not backed by the papers they publicize.
Judging by the papers on climate change, and the idiot findings of all kinds one hears on the radio today, it seems that universities are churning the stuff out at break neck speeds. I suspect most will never be read or heard about again. They’ll become expensive door stops for graduates buried under student loans.
Most folks take the easy road when it’s available and scientists are no different. It tasks a special, dedicated person to find truth.
Have you found the truth about Frank Lee MeiDere? Or don’t you give a damn? 😀
Examining the cause is exactly what Jonas Lehrer details in “The Truth Wears Off, a New Yorker piece from December, 2010: http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off
It chronicles the efforts of Jonathan Schooler to explain “the decline effect” in his own studies, which he first noticed in 1995 when attempting to replicate a 1990 study he did on “verbal overshadowing.” The effect of “verbal overshadowing” was 30% less in 1995 than in 1990. In 1996 the effect shrunk another 30%.
Lehrer provides an easy read that isn’t limited to Schooler’s efforts or psychological studies but includes the modern ineffectiveness of second-generation antipsychotics as well as irreproducible results in other branches of science.
And so begins an interesting story that reveals problems not so much in “The Scientific Method” as in “The Scientific Process,” a process distorted by tenure, funding, peer-review, consensus and more but mostly, by human nature. And while it doesn’t once mention the climate, it goes a long way to explaining the rise of the failed CAGW hypothesis. Encouragingly, it also explains how and why we may be ont he threshold of seeing it supplanted with a different hypothesis.
Lewandowsky has finally realised that he’s a charlatan – he’s giving a talk on Suspect Science in Cambridge at the end of September. http://www.crassh.cam.ac.uk
Oh, you mean he isn’t applying that label to his own drivel? He’s giving real psychologists a bad name!
“The Romans built great aqueducts and the church produced grand cathedrals in the Middle Ages before materials science was developed.” — RalphDaveWestfall
A concrete formulary IS science. Architecture that doesn’t fall down IS science. Finding what works and what doesn’t IS science. Finding why it works is also science, but we don’t have to know why concrete works to be able to use the underlying knowledge.
“The Wright Brothers ran a bicycle shop before producing their airplane.” –RalphDaveWestfall
No, they ran a bicycle manufacturing company, including development of leading edge improvements to the craft. But the Wright Brothers were among the world’s foremost aerodynamic scientists when they designed their first successful plane. They were not a couple of bicycle mechanics who one day threw random parts together and found that they flew. They knew what they were doing and went about it in a scientific manner.
“Education: Both were good students in school and their favorite subjects were math and science. The both excelled in math. Each attended high school but neither received a diploma. Wilbur had enough credit to graduate but the family moved to Dayton from Indiana. High school was not challenging for Orville so he dropped out in the 11th grade to start his own printing business.”
You might enjoy the story of the US government’s plan to invent the airplane using a hand-picked expert. It is not in many history books but Burt Fulsome includes it in his lectures and books.
Uncle Sam Can’t Count (Burt Folsom – Acton Institute)
https://youtu.be/i5fONzEwmfU?t=27m57s
@jorge “A concrete formulary IS science.”
How about a recipe for pineapple upside-down cake?
Were the prehistoric peoples who who developed stone chipping techniques to create arrowheads scientists? And those who learned to use fire for cooking and heating? And who first used the wheel in wheelbarrows and oxcarts?
You might want to consider the implications of the prominent role of Francis Bacon, who lived from 1561 to 1626, in the development of the scientific method. And also look at the discussion of the differences between science and technology at http://www.diffen.com/difference/Science_vs_Technology
Karl Popper wrote his treatise on the philosophy of science largely because he was concerned with the irrefutability of the ‘soft ‘ sciences – and indeed specifically mentions psychology.
Sop soft sciences are riddled with what a friend of mind calls ‘physics envy’ – the desire to achieve a concrete and irrefutable – or art least very strong view – on a given subject, where none such is possible.
Leo Smith says, “soft sciences are riddled with what a friend of mind calls ‘physics envy’”
Karl Popper objected to what he called “the aping of the physical sciences by the social sciences.” Physics envy works too.
He also was not a positivist and had to write a whole book to correct that mistaken reputation he was given by positivist colleagues. He also was inspired to write because of the obscurity of the language and jargon in science. Something about Frankfurt School. (:
He fortunately also wrote a book which debunked Kuhn, called “The Myth of the Framework.” His name today is used as synonymous with “falsifiability” as a test of a good theory, but he was a very very busy man.
I suggest pseudo-scientific psychologists should publish their papers in the “Journal of Irreproducible Results” and be done with it.
Science has some fundamental problems. On the one hand, you have a strong push from one side at minimizing waste and redundancy by forcing scientists to predict the minimal number of replicates needed to test your hypothesis, including power calculations, and then to try to use those minimal numbers. This is especially true in biomedical studies involving animals. You literally cannot tell the IACUC committee that you plan to replicate a previous study, because it will be denied for being wasteful and unnecessary.
On the other side you have a herd mentality in science where the next logical step is obvious to most people, so you end up with a dozen laboratories all running the same experiments in parallel, and only one gets the credit by publishing first. I was at a big scientific conference where someone was receiving a lifetime achievement award. As part of her remarks she talked about how she was one of 15 labs trying to make a particular knock-out mouse for the same gene at the same time. Most of them succeeded, so there were a dozen different variants of the same mouse. Very wasteful, yet a symptom of the nature of modern science. To get funded you need to propose something that is of interest to your peers but also very likely to succeed. More unusual or risky projects don’t get funded. It almost guarantees redundancy and herd mentality science by the way it’s set up.
This quote by Schwarz classically illustrates the lack of the most basic understanding of the scientific method. To add further to this tragedy of ignorance, criticism of such disastrously poor understanding of the scientific method is responded to by accusations of being “anti-science”. Not even Shakespeare could write such a tragic comedy.
It’s worse then being proved wrong. Recall Mike’s nature trick.
Reply to Alx ==> That is not a quote from Professor Schwarz. It is a line from a now-famous ClimateGate email, if I recall correctly. Someone more familiar with the Climate Wars can fill in the who said it when and where.
Frank (the author of this column today) is likening Schwarz’s quote to that famous one…..
I have a master’s degree in computer science from a reputable department at a reputable university (one of the top 10 in the world at the time I received my degree decades ago) — and I can tell you absolutely that computer science is not science. It is a mixture of mathematics, engineering, logic, and craftsman techniques. Not that there’s anything wrong with that, but it ain’t science.
Journalists and editors are also to blame.
http://www.bloombergview.com/articles/2015-08-31/why-we-fall-for-bogus-research
Speaking as someone who has spent time in that industry you’re damned right they are! I’ve been disgusted for a long time how little real research journalists do and how much they rely purely on self-serving press releases for their articles. In a discussion with a friend of mine who is a CBC news editor a couple of months ago I was complaining about how journalists seem to get most of their information purely from the media itself and gave a couple of examples of actual research. His reply was (and I’m not making this up), “Well, that’s not what I’ve been reading in the newspapers.”
Yeah — exactly.
It seems to me that this study ended up with what I would consider the expected results. It didn’t purport to find fraud, but rather it found that the original studies’ impacts were reduced somewhat in, what, 60% of the cases.
I would expect something like that for the simple reason that it’s the outliers that tend to get published. Do 20 studies and one shows an unusual result outside the expected and it’s published because it’s unique. Of course, if one did 20 similar studies altogether, the one outlier would be recognized as the outlier, i.e., in the tail of the distribution of expected study results.
But we don’t do 20 at once, we do one here and one there instead. But even so, if we do 20 here and there, one of them is, statistically speaking, going to yield results in the tail of the distribution of expected results. And that’s exactly the one that gets published.
This study demonstrated that outliers tend to get published and that replication is important, but it also demonstrated that most of the published studies are showing a real effect, just perhaps not as strong as the original study indicated.