Readers may recall this survey: A poll to test the Lewandowsky methodology The results are in, which is why we can’t say global warming proponents support pedophilia.
Guest essay by Brandon Schollenberger
They don’t. The fact there is a correlation (0.14) between believing global warming is a serious threat and saying pedophilia is good is meaningless. The fact this correlation is “statistically significant” (at the 99.99% level) is meaningless. Anyone who looks at the data can immediately see the results are bogus (a small jitter value was added to allow us to see the density of responses):
There are only 20 or so respondents (out of over 5,000) who claim to believe global warming is a threat and claim to support pedophilia. It’s likely those responses were false. Nobody can seriously claim that proves global warming proponents are pedophiles.
The issue of false responses received a lot of attention with Stephan Lewandowsky’s paper, “NASA faked the moon landing—Therefore (Climate) Science is a Hoax. In that paper, similarly false responses created spurious correlations. Unfortunately, the focus on false responses meant a more fundamental issue got missed. Namely, the entire idea behind this approach is nonsensical. The approach is like taking the data displayed above and drawing this line:
The line fits okay in the bottom left corner where most of the data lies. That means there is positive correlation between the two data sets. However, that corner clearly shows a correlation between thinking pedophilia is bad and being a skeptic. It tells us nothing about global warming proponents or pedophiles. Similarly, when I said global warming proponents support genocide, I was doing this:
If we put this in words, the argument is:
Skeptics believe genocide and pedophilia are bad. Global warming proponents are the opposite of skeptics so they must believe genocide and pedophilia are good.
Change a few words, and you have Lewandowsky’s argument for why we should believe skeptics are conspiracy theorists:
Global warming proponents believe the moon landing was real. Skeptics are the opposite of them so they must believe the moon landing was faked.
With this corresponding image:
All of these results are “statistically significant.” However, all of these results assume skeptics must hold the opposite view of global warming proponents on all things. Assuming that guarantees the results. We can do that to criticize any group we want. Just follow these simple steps:
1) Ask group X if they think the moon landing was real. They’ll say yes.
2) Assume group Y would answer the opposite way.
3) Conclude group Y believes the moon landing was faked.
You can replace “the moon landing was real” with anything you want. I showed this by doing it with genocide and pedophilia. Had Lewandowsky asked about those, he could have concluded skeptics are pedophiles. He could have probably got it published too. After all, he didn’t do this just once. He published a second paper using the same approach (with a slightly less skewed sample).
And he’s not the only one who uses it. Lewandowsky’s recent paper cites the paper, Dead and alive: Beliefs in contradictory conspiracy theories. That paper argues conspiracy theorists are so loony they’ll accept multiple, contradictory conspiracies. It’s namesake comes from the “statistically significant” correlation between believing Princess Diana and/or Osama Bin Laden was killed in a conspiracy and believing he/she is still alive. The image for this claim would be:
The scale in that image is correct. Let it sink in.
There is no justification for this methodology. Even so, three scientific journals have approved of it. Dozens of scientific articles approvingly cite its results. Half a dozen people have been paid papers using it. It has been promoted hundreds of times in the media. It is widely accepted in the global warming debate. It is complete and utter nonsense, but people like the results so they don’t mind.
And if my suspicions are correct, it’s probably been used in many other papers.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
What was the basis for the political decided UNFCCC?
Was it not claims from environmentalists(Greenpeace, WWF etc)? And they still are very central in the whole UNFCCC scheme.
Bjørn Lomborg’s book “The Sceptical Environmentalist” raised some serious questions about the scientific basis for most of the scary stories the environmentalist have given us the last 40 years.
So we might call us after 40 years of doom, sceptical of environmentalist doom?
Merovign,
Yes, there is a lot of unthinking tribalism. Supporting those on ‘your side’ without really knowing what it is you are actually supporting. Good point. I see that a lot here as well 😉
A number of people have wondered if Stephan Lewandowsky could have been stupid enough to have believed his methodology was appropriate. It’s an interesting question. Lewandowsky might argue it shows conspiratorial ideation, especially when people go beyond just asking like BoyfromTottenham did:
I believe the answer is unquestionably yes. It is possible for someone to be stupid enough to use this methodology and not see any problems. It is also possible for a person to be so close-minded they blind themselves to problems with this methodology. There are few limits to stupidity or self-delusion. What I find more interesting is, should we assume one of those explanations?
Is it kinder to assume stupidity and the like, or is it kinder to assume dishonesty? I think the former is far more likely, but it seems more offensive.
Brandon Shollenberger says:
January 23, 2014 at 6:33 pm
I sent an e-mail to both Stephan Lewandowsky and Michael Wood to notify them of this post. Based on my previous exchanges with them, I don’t expect a response, but it’ll be interesting to see if they say or do anything.
=========================
Why would anyone touch this with a 10 foot pole ??
Unbelievable, that it got posted.
Ian Schumacher, a remark I made when I first began discussing the results of my survey may help:
The terminology you’re uncertain about is basically irrelevant. Anything that is true for basic correlation matrices will be true for his paper as a whole.
As for whether or not “a proper approach” would get the same results, it is easy to check whether or not one would not with the data used. I did that with Michael Wood’s data. The same test could be done with Lewandowsky’s (one of my links shows Steve McIntyre doing it with the moon landing result). It’s easy to check what data is responsible for any correlations that are found. It is theoretically possible one could collect data which showed the same conclusions, but the data these people used definitely does not justify their conclusions.
And if you’re only talking about fringe elements, it’s highly unlikely one could get a large enough response rate from them to actually identify them.
u.k.(us), I understand you may think the use of the word pedophilia makes a post unworthy of being published. I don’t think you’ll find many people agree. I think most people judge posts by the material in them, not whether or not they use a particular word.
If you have a problem with me pointing out a methodology commonly accepted by global warming proponents is such that we could use it to label literally anybody a pedophile, please tell me what it is. If you don’t, but instead are just bothered by the word pedophile, I think you’ve made your point and can leave.
Brandon –
Let me point out that global warming alarmists DO, in effect, believe mass murder (genocide) is good, on several counts, and I am not just talking about those who explicitly want to drastically reduce the population of the Earth.
Policies touted by global warming alarmists have been demonstrated to result in millkions of needless deaths and untold suffering. Carbon taxes made some 33,000 Britons unable to afford to heat their homes in the winter of 2013, with the result that they died of hypothermia. The ethanol program diverted enough corn from food supplies that food grains became unaffordable or unavailable in Third World countries, with some 2,000,000 dead from starvation in sub-Saharan Africa, and even some in Mexico. And this is A-OK with alarmists – I dare them to deny it. They will just say it’s “for the cause.”
Not only is there 100 percent correlation between global warming alarmism and mass murder, there is positive, unarguable, specific, direct causation. Global warming alarmism is MASS MURDER, period.
These people are like my former history professor colleague who was forever singing the praises of the Soviet system, bleating about how it was so much more efficient and humane than the US. When I confronted him about the 80 million people murdered by Stalin, his response was, “Well, that was a necessary step in reforming society.”
Global warming alarmists are saying the same thing: mass murder is a “necessary step” in “saving the planet”.
Brandon Schollenberger, congratulations on a brilliant set of scatter plot visualization of the dataset. A first class use of jitter. One I’ll remember.
I suggest this improvement in your article and future paper:
Let’s assume that researchers have fooled themselves by looking at the correlation coefficient matrix from the raw data.
So, before you get to the scatter plots, play it straight and show the same tables in the same format as Lewandowski used. Show how you can generate bizarre correlations with a few lines of R code.
Then hit your readers between the eyes with the scatter plots.
The goal will be to build a list of papers whose authors committed the blunder and editors who accepted the papers.
Then end with, “Don’t let this happen to you.”
———————
I wish I could come up with a few bullet points of how to avoid correlation pitfalls short of eschewing correlation on ordinal data entirely. Hopefully some survey statisticians will chime in with some sage advice.
One pitfall to avoid in ordinal correlation:
It would seem that any survey violates this assumption in spades. One person’s correlation between X and Y might be monotonic, but every person in the survey has a different correlation, possibly with different sign.
Nice work Brandon. Thanks.
Chad Wozniak, I don’t agree with your comment, but I don’t think that’s important for this post. However, you do show the danger of this approach. We can use it to get results which are utterly absurd (warmists think they’ve seen Bigfoot), but they wouldn’t be accepted by anyone.
The danger is this approach lets us get results which aren’t necessarily crazy. If we genuinely believe something to be true, we can use this approach to “prove” it. That’s what likely happened with Stephan Lewandowsky. He came up with an idea that wasn’t obviously wrong then tested it with a methodology guaranteed to confirm his beliefs.
I intentionally chose the topic of genocide because I knew some people would believe the results unless explicitly warned not to. The hope was it would be a lesson for all sides.
“And if you’re only talking about fringe elements, it’s highly unlikely one could get a large enough response rate from them to actually identify them.”
I’m not as optimistic. We’ve all seen the stories how 30% (or some similar high number) of Americans think dinosaurs and humans walked the Earth at the same time, or 30% (pick you select source high number) believe 9/11 was an inside job and so on. With that large base to draw from I think they could definitely be a factor. Just as there is a not-insignificant fringe of environmentalist that support genocide (didn’t you write an article about that), there is also a not-insignificant fringe of skeptics that believes 9/11 was an inside job. Ca La Vie.
A non-mathematical problem with this survey and all like it though is that “people lie”. How many people would actually admit to supporting genocide, or pedophilia! If you are a conspiracy theorist aren’t you going to be at least a little paranoid about getting a knock on your door from men in black suits? 😉 Or if you support CAGW and you can sense the real purpose for this poll (and I think you’d have to be an idiot not to know it was destined for use as propaganda), might you lie in other areas to support your team? I would think so. If anything I suspect the ‘outrageous’ question responses are biased towards socially acceptable answers. Being a conspiracy theorist is one of the less outrageous labels on the survey.
Actually now that I think about it. Wouldn’t some people be quite likely to do a ‘false flag’ operations and answer outrageously as if from the ‘other team’. Sounds like something I might even do (in the opposite direction of course).
They’re pop scientists that live in a pop culture. There’s nothing else to say.
Stephen Rasey, I don’t deserve much credit for the use of jitter. I was looking for a way to just create circles whose size was proportional to number of responses. I just stumbled across this idea when using Google to try to figure out the other.
As for your recommendation, I actually toyed with the idea. I generated a fairly crude rough draft of a document showing skeptics are racists. I even managed to relate the recent “Dark Money” argument into it. I decided to abandon it because I figured nobody would believe it.
It’s not that they couldn’t have been fooled by it. I just would have needed to hide my association with it to do so. Since my name was tied directly to the survey, that wasn’t possible. Plus it would have involved waiting around a lot. I didn’t want that. Lewandowsky’s work has already been accepted for long enough.
As for doing surveys properly, you highlight the major concern. To address that, one should check the distribution of the data to make sure it matches that required by the tests being used (this step includes looking for outliers, skew and other potential issues). In most cases, if it does, you can be confident your tests gave meaningful results.
Mark Bofill, thanks!
Ian Schumaker, I did write a post about people supporting genocide. It was facetious. I did as part of my effort to show what this post shows – this methodology generates bogus results.
As for your thoughts about untruthful responses, that’s a major problem with surveys. I suspect it’s a major factor in the stories you mention. A group of researchers recently announced their estimation of the homosexuality rate in high school students was likely greatly overstated because of people giving false answers. The same is probably true of those. I know I’ve given bogus responses like that to surveys before.
Anyway, checking for correlation between answers to questions pretty much doesn’t work unless the effect you’re examining is strong. There are too many confounding factors otherwise.
Dan in Nevada says:
January 23, 2014 at 3:49 pm
So some warmists aren’t genocidal pedophiles?
======================================
Some, but, they insist that we exercise tolerance and inclusiveness towards the ones that are.
“Eric Worral said “Is it possible Lewandowski is so stupid he actually believes in the validity of his technique?”. I say “given what we know about how the “survey” was constructed, undertaken, and “analysed”, how the heck can anyone answer “Yes” to Eric’s question? Given the number of steps taken to achieve the result, surely this has to be the work of one or more persons acting with a purpose (e.g. to create a powerful piece of “disinformation”), rather than honest, scientific research?”
I see it more, in a time climate is not conform to the UNFCCC, as provocation, distractions, establishing political correctness around UNFCCC, give the dog a bad name and let it hang, etc…
Don’t waste your time trying to understand policy based studies?
Brandon Shollenberger says:
January 23, 2014 at 7:31 pm
……”If you have a problem with me pointing out a methodology commonly accepted by global warming proponents is such that we could use it to label literally anybody a pedophile, please tell me what it is. If you don’t, but instead are just bothered by the word pedophile, I think you’ve made your point and can leave.”
============
I’ve been here for years now, not going anywhere soon.
I’ve seen your likes, come and go many times.
It is a bother, but, it usually doesn’t last long.
I think I’ll stay.
I wonder if Jerry Sandusky was a fervent believer in Global Warming?
@Brandon Shollenberger at 8:08 pm
As for your recommendation, I actually toyed with the idea.
I think I need to clarify my suggestion. I did not mean for you to change the questions. What I thought I was suggesting was that you first show the correlation matrix of YOUR question responses in the same way and same format as Lewandowski did.
In the main post above, you went straight to the scatter plots which makes clear the nonsense of any meaningful correlation. Correlation designed in the question
The lesson to be delivered to the reader is the correlation matrix out of your favorite statistical program might deliver a corelation coefficient with a sign confirming your hunch while the scatter plot says bupkis. So you need to show the misleading matrix first. Better to show a plausible correlation matrix first. Then show a absurd correlation matrix. The lift the veil on the scatter plots.
Brandon Shollenberger says:
January 23, 2014 at 4:05 pm
richardscourtney, I’ve considered that, but I believe I’d have to pay a fee if Psychological Science was publishing it. I don’t like the idea of spending my money that way. Part of it is simple finances. I’m not getting paid to do this. Paying to publish would mean I’m losing not just time, but money.
But the biggest part is they screwed up, badly. They shouldn’t get to make money off forcing people to do work to correct their screwups. The only way I could see wanting to publish with them is if they gave a waiver of any and all fees. Otherwise, I’d want to publish somewhere else.
Brandon,
Perhaps you might reconsider publishing in Psych Science? If money to pay the fees is an issue, this is a forum that can readily generate modest sums for worthy projects that clearly refute crapstack science.
Mac
Good morning, Brandon. I’m back late because of time zones. Let me reiterate a point I already made. A basic shortcoming of this research is that you do not know whether there is a relationship between what people think and their reponses to your questionaire. In better research each question is at least asked twice in different formats. Next, reliabilities of the items can be estimated. Because it is so important, I would ask the genocide question at least four times in other wordings and formats.
I am one of your subjects but I am sure you do not know that I often think that my left hand is my right hand. So it may happen that I mark ‘agree very much’ at the left whereas I should have marked the opposite at the right. It is possible that I am one of your outliers. You also can get extreme outliers because of jokers who do this on purpose.
You don’t have evidence in your study, I appreciate very much, that you have measured any opinions at least reliably.
“He came up with an idea that wasn’t obviously wrong then tested it with a methodology guaranteed to confirm his beliefs.” That particular modus operandi seems to be endemic in post-normal science, from the Hockey Stick onwards.
An object lesson that while non-correlation always implies non-causation, correlation does not of itself imply causation. There’s just no way of getting around this!
Perhaps of more interest: I rather agree with the Lew Paper to the extent it notes a correlation between a “laissez-faire conception of free-market economics” (which I take to mean right-ish wing political views) and “rejection of climate science” (in the Lew Paper sense).
Why is it this – is there actually some causal process here?
Re my last: perhaps my question can be better formulated “Why is it this – is there actually some causal process here (other than just aspiration to achieve political ends)?