Replication of Lewandowsky Survey

Guest post by A. Scott

There has been considerable discussion about the methodology and data regarding the recent paper “Lewandowsky, S., Oberauer, K., & Gignac, C. E. (in press). NASA faked the moon landing – therefore (climate) science is a hoax: An anatomy of the motivated rejection of science” (copy here)

This allegedly peer reviewed paper claims their survey data show climate skeptics are supporters of wild conspiracy theories, such as “NASA faked the moon landing.” The author admits, however, no climate skeptic sites were involved in the survey, that essentially all survey results were obtained thru posting the survey on pro-global warming sites. 

Due to the serious and legitimate questions raised, I have recreated the Lewendowsky Survey in an attempt to replicate and create a more robust set of replies, including from skeptic users.

Please click on the Lewandosky Survey Page above and you’ll be presented the survey. This survey replicates the questions, both the paper, and several sites have indicated were in the original survey, including those questions deleted from the survey results.

The only change was to use a 1 to 5 ranking vs. Lewandowsky’s 1 to 4, which several people with experience have noted should improve the overall responses.

Each visit to the survey is tracked. Access is password protected for an additional layer of tracking.

THE PASSWORD FOR THE SURVEY IS “REPLICATE” (case sensitive)

Please only complete and submit once. Also, please respond to each question with the answer that best reflects your position, even though the question may not be perfectly worded.

This survey is built on the Google Doc’s open access platform. Results are collected automatically. As no significant randomization or counterbalancing was performed on the original survey none is applied here. Data collected will be provided upon request.

A. Scott

Advertisements

  Subscribe  
newest oldest most voted
Notify of
En Passant

It’s a conspiracy against the biased and incompetent!

I would love to see a cumulative response vs time curve on this survey.

Alvin

omg, what a leftist poll. Many questions lead to catch-22 answers. The questions refuse to acknowledge that environment can be valued in a free market system. The questions pit free market against environment, and involve Marxist social justice.

Mike

Submit button? What submit button?

JonasM

38. Out of 100 climate scientists how many do you think believe that human CO2 emissions cause climate change?

Why are they asking us? – there’s a peer reviewed paper with the answer provided!
:rolleyes:
LOL

Dan Evans

To see the SUBMIT button you need to move the scroll bar to move the text up.

Otter

En Passant says
It’s a conspiracy against the biased and incompetent!
-0–
So lewandowski launched it against himself?

Andrew Newberg

Mann, I hope you grade on a curve…

cui bono

I’ve completed the survey, but what is it supposed to prove? That we aren’t all nutters?
Why not put up the “When did you stop beating your wife” survey?
Can’t help feeling we’re playing so far in the crazy alarmist ballpark that we’ll get lost.

Pamela Gray

To replicate a study, you need to use the original (poorly designed) 1 through 4 choices, not 1 through 5. You have not replicated the study, you have improved on it.
However, you have made the same mistake the original author did, by not obtaining a random population sample. The original results, plus yours will be biased, thus making the results useless.

DaveA

I added the following feedback…
Bad phrasing:
“An economic system based on free markets unrestrained by government interference automatically works best to meet human needs. *”
No not always automatically. If phrased as “tends to” then would get strong agreement from me. As is I disagree.
“4. The preservation of the free market system is more important than localized environmental concerns *”
Define “local”. We’re trashing the Amazon or an empty field at the edge of a city where frogs live?
“Smoking causes lung cancer *”
It doesn’t cause lung cancer, but it does increase the risk. You can smoke and not get lung cancer; you can not smoke and get lung cancer.
“Human CO2 emissions cause climate change *”
In sufficient quantity CAN, that doesn’t mean always will.

FWIW, I’ve submitted my response. But some of the questions are a bit nutty. Like what is my opinion of what is done at Area 51? It’s a top secret facility, I can have no agreement nor disagreement with any assertion of what is done there as I have no knowledge of the place. Or the “Kennedy lone gunman” question: There’s a load of evidence that L.H.Oswald was not alone; but that doesn’t make it a ‘grand conspiracy’. Again, an ‘unknowable’ answer on which I’m expected to have an opinion? How about “I don’t know and neither does just about everyone; so I reserve judgement.”? But other than the “have you stopped beating your wife” nature of some of the ‘nutty’ questions, it didn’t take long and isn’t very hard.

Yep. Done.
Lewandowsky gets a chance?
“The quality of mercy is not strain’d .. “

AnonyMoose

Did the original survey invite reposting the invitation? Should more be invited elsewhere?

wayne

Have to agree with Alvin. Wouldn’t play this sick game with a rather sick man (Lewandowsky) if I were you. Sick men make sick claims as he apparently has made of the science oriented minds that frequent this site. Just by making his claim on inadequate data he has proven he is no scientist, that’s for sure!

Absurdly biased set of questions. Did the authors of those questions believe they were writing unbiased questions? I would not call the authors of that survey “scientists” on a dare.

statgoblin

Anthony, I’ve been reading your site for some time now and finally decided I wanted to be involved in the discussions. It seems that anyone who is actually honest with themselves would answer ‘i don’t know’ for many of these questions. I realise it’s about opinions, but I really can’t form a worthwhile opinion about free market economies if I don’t know anything about free market economies. Seriously, any opinion I have is going to be coloured only by popular opinion and not from any real knowledge of whether something is good or bad.
I think far too many people in this debate (the climate science debate) seem to have lost the ability to simply say “I don’t know”.
Anyway, looking forward to being involved in future discussions.

cui bono

Given this survey is now infamous, aren’t we going to get every troll making their way here to say they don’t believe in AGW but do believe the Illuminati run the world? Just for fun?

Let’s do another suvery.
Same questions. But the responses are:
1 – Great well phrased, meaningful question
2 – poorly phrased question but still useful.
3 – badly phrased question. Conflates issues. Could be saved with rephrasing.
4 – assumptions bad, results not usable.
5 – Hopeless bad question. Irredeamable.

Bill H

Looks like climate scientists are using the Saul Alinsky method of data collection…
OY!

Bill H

3. The free-market system may be efficient for resource allocation, but it is limited in its capacity to promote social justice
yep! got to hand it to them… this is your opinion and you will like it…

A. Scott

Mike … scroll down the page for more questions … the SUBMIT button is at the end

A. Scott

To the several that have commented on the questions themselves, this is a recreation of the original survey.
The questions in the survey here are as created by the original authors. Yes, many are poorly written, hard to understand, and/or otherwise flawed. However, for the purposes here retaining the original questions, warts and all, is important.

RockyRoad

The test will prove, like thermometer sitings, that biased questions give biased results. No more, no less.

Latimer Alder

7. The Iraq War in 2003 was launched for reasons other than to remove WMD from Iraq.
Umm, It was launched for many reasons. One of them was to remove WMD. But there were others too. OTTOOMH: Hubris, completion (Bush), ‘legacy, doing the right thing’ (Blair), Do I strongly agree with the proposition because others existed as well? Or strongly disagree because WMD was indeed one of the causes?
Very badly phrased and ambiguous question..typical of many.
I already had severe doubts about the methodology of Lewandowsky’s paper. Now I have seen the very poor quality of the questions he asked, I have had those doubts confirmed (1 = strongly agree).
And if this is typical of the level of rigour of ‘work’ produced in climatology as well as in sociology, then the last 30 years of the climateers has all been pretty much a waste of time. Hardly rises above a science project by Lisa Simpson (age 8). And I may be doing a disservice to Lisa 🙁

Brian H

I’m sure my responses will not compute. 😉

@E.M.Smith:
“FWIW, I’ve submitted my response. But some of the questions are a bit nutty”
As a “cognitive scientist” he’s running with a hell of a handicap.
” … the “Kennedy lone gunman” question: There’s a load of evidence that L.H.Oswald was not alone; but that doesn’t make it a ‘grand conspiracy’ .. ”
I’m almost tempted to try to discuss this one with him 🙂
The event happened when I was 15. First thing my old man noticed on the newsreels was people hanging out of upstairs windows waiving. BIG failure of SOP.
In that year, I had scored 95/100 on a 400 yard .303 range, Lee Enfield with conventional sight. I knew I couldn’t have done what Oswald was claimed to have done with a bolt-action rifle, down-grade, receding target, plus telescopic sight (= slower action). So I was skeptical. Then there were the “Lucien Sarti” revelations. Quite recently a full reconstruction was done, and yes, it was possible, so I changed my view …

How does one answer this one? “20. The claim that the climate is changing due to emissions from fossil fuels is a hoax perpetrated by corrupt scientists who wish to spend more taxpayer money on climate “research”. *
The answer is these are incompetent boobs who make crap up. Heck, half of them believe the nonsense they spew! Look at Steig!

DocMartyn
Ally E.

Didn’t join in on this one. With warped questions you can get warped answers, meaning they can read what they like out of it. Not interested in Lew’s methodology, he’s shown himself as biased and slippery in his thinking. His dodge and weave and his “See? See? That proves it!” approach only proves to me that he is completely untrustworthy. I wouldn’t touch any of ’em with a barge pole.

Nick in vancouver

The original survey is bogus and a stooge, do not complete this version as it gives credibility to the unmitigated bias of first laughable piece of junk.
Here’s a better survey question.
What qualifies one to be an Australian Professor of Psychology?
We can have more fun and learn more by
coming up with 4 alternative answers.

banjo

Mike says:
September 8, 2012 at 6:30 pm
Submit button? What submit button?
Have you still got the box your computer came in?

highflight56433

Motivated rejection of science
Abstract
“Although nearly all domain experts agree that human CO2 emissions are altering the
world’s climate, segments of the public remain unconvinced by the scientific evidence.”
Nice opening leap of abstraction. It receives a failing for the poorly designed survey and complete bias in citations that support the survey purpose; to mock others.

reno

When i read social justice i gave up

Gvii

I almost gave up there too, but I really wanted to get through it. It was another fine example of a question that just doesn’t fit the agree/disagree scaled answer. And the last few questions are a bit creepy, to be honest. The majority of the survey seems much more political than scientific, to be honest.

A. Scott

Pamela Gray says:
To replicate a study, you need to use the original (poorly designed) 1 through 4 choices, not 1 through 5. You have not replicated the study, you have improved on it.However, you have made the same mistake the original author did, by not obtaining a random population sample. The original results, plus yours will be biased, thus making the results useless.

Pamela – you are correct, technically a “recreation” rather than a strict “replication.”
I gave fair amount of thought to this decision. My conclusion was that as this survey is not re-polling the same people who took the original poll, and there is no attempt to validate those responses, it was well worthwhile to increase the quality of the data collected by using the 5 point system.
In my opinion, by using the original questions, in as close to their exact original wording as possible, the basis for responses is essentially identical. The change does not affect the response to the question as written, but the quality, or granularity, of the response is improved.
Because this survey is public, and I assume will be well disseminated – including to pro-science sites who provided most of the original respondents, their users can take the poll again if they choose and their answers will be validated on the same scale as all others.
Also as no significant randomization or counterbalancing was performed on the original, and considering the relatively simple nature of this survey, none is applied here.
This effort isn’t perfect.
I researched the best way to recreate this survey considering accuracy, economics, ability to track responses etc. Based on that research the Google Docs “Forms” platform, coupled with an independent stat counter that tracks responses provided in my opinion the necessary features and robustness, along with tracking, to accomplish this work with a sufficient degree of accuracy. Results and stats are collected automatically, but there is some manual work reconciling the two.
A perfect solution would have incorporated all features in one package. However, the cost for that was quoted at $5,000.00 – not suitable or necessary here.
The original survey was collected using KwikSurvey.com. As part of my research I set up a paid premium acct. there (and set up trial accounts at 6 other online survey providers) and found they offer less robustness than the platform here.
Hopefully what we do here is provide data from respondents across the entire spectrum.
That said, even if the respondents are primarily from “skeptic” sites, I do not see an issue. The original study conclusion (and its clear it’s intent) was to show what/how climate skeptics think. Thus a collection of data skewed towards climate skeptics will provide accurate data on climate skeptics.
At least that’s how I see it. YMMV.

Gvii

I have to agree with DaveA above. Too many questions are badly phrased for an Agree/Disagree scale answer.

Bill H

I completed it… without throwing up I might add… this is a huge piece of garbage..

Tom in Texas

A 2nd survey should be done, with Stephen’s 1/2/3 questions rephrased, and compared.

Mostly Harmless

Pamela Gray:
“However, you have made the same mistake the original author did, by not obtaining a random population sample. The original results, plus yours will be biased, thus making the results useless.”
I think it will be interesting to discover just how different the results will be. My answers to the survey would have been the same if I had completed it at another site. I agree that a bias will exist but I suspect a larger percentage of pro-AGWers visit this site than skeptics visit pro-AGW sites, especially those which stifle dissenting opinion.

A. Scott

I noted this to Pamela – but bears repeating in its own comment.
Technically, with the improvement in granularity by using 5 response options vs. 4, this would be a “re-creation” rather than a “replication” of the survey.
And since we cannot retest and match to the same people that originally took the survey we cannot technically “replicate” anyway.
However, it was my opinion that by retaining the same questions and format, we can obtain a valid re-creation, with better data quality, here. And that allows us to replicate the process, and subsequently test the results and conclusions of the original survey.

A. Scott

Doc Martyn … working for me – try this:
http://ascottblog.wordpress.com/

Rick Bradford

I’m reminded of Bernard Woolley, the perfect balanced sample.
I can’t find the clip, but scroll halfway down this page and you’ll find the survey questions asked.
http://openlearn.open.ac.uk/mod/oucontent/view.php?id=403748&section=1.1.1

Really? That was the survey? And that was used in some sort of science journal? The questions say a lot more about the author than they would any test taker.
Clearly the author is a warmist, but he’s also a leftist. And, he lacks critical thinking skills.
The survey did get me to pause a bit. Do leftists still dwell on some of these things? I can’t remember, but I thought there was a 3rd person involved in the OKC bombing. As E.M. said, if you don’t know, you don’t know. Do warmists have a litany of things they go through each day and ponder? Like is area 51 alien infested or not?

rogerknights

cui bono says:
September 8, 2012 at 7:09 pm
Given this survey is now infamous, aren’t we going to get every troll making their way here to say they don’t believe in AGW but do believe the Illuminati run the world? Just for fun?

Maybe there’s a way for the survey to accept only people with over 25 (say) posts here.

Edohiguma

I’m looking through the questions and fail to see the point. I will not take this survey. Some questions require deeper explanation and can’t be answered with a simple yes or no.
Best example for me is this: “10. The US government had foreknowledge about the Japanese attack on Pearl Harbor, but allowed the attack to take place to as to be able to enter the Second World War.”
The answer to that depends on what you consider foreknowledge. Did they know the fleet was coming? Or did they know that Japan would eventually attack? Because that Japan would eventually attack was pretty clear to anyone who had paid attention to what was going on in Japan (with all the talk about the “ABCD encirclement” that was “strangling” Japan, the war in China and the rhetoric by Japan’s leaders which became more and more aiming at expanding the war into the Pacific, etc, things that were taught in elementary schools even after the surrender and before the occupation), so from that point of view, yes, they knew and the US was caught with the pants down. Of course people like Lewandowsky seem to define it by FDR actually having a report about the fleet breathing down Pearl Harbor, which is completely laughable.
So I have to say that Lewandowsky’s survey is still completely worthless, since he clearly lacks any knowledge to define his questions properly.
A psychologist should never dabble with things like this because he lacks training and knowledge for it. It’ll lead to just another worthless statistic.
Science and statistics are not synonymous.
Discovery of a numerical discrepancy is not science. Accounting for that discrepancy in a reproducible manner is science.

Bill H

reno says:
September 8, 2012 at 7:48 pm
When i read social justice i gave up
_________________________________________
screams of justifying wealth redistribution by any means…which is not scientific in any sense of the word… just power hungry fools..

Dr Burns

I agree Bill H. Science has nothing to do with beliefs, attitudes, income or whatever.

highflight56433

Bill H says:
September 8, 2012 at 7:57 pm
I completed it… without throwing up I might add… this is a huge piece of garbage..
Universities are eventually going to all fail by their own hand. Too much cost for students both monetarily and socially. Condescending elite intellectual destruction of young scholars will be their demise. They eat their own.

Toto

Dr. Loo has no sense of humor. He left out the most fun questions, such as “Obama’s birth certificate is a fake”, “At least one of Obama’s books was ghost written by an ex-terrorist”, “skeptics of CAGW believe the earth is flat”, “skeptics are funded by Big Oil”, “this survey was funded by the Koch brothers”, “Michael Mann’s real name is Dr. Evil”, “CAGW is a world takeover plot”. I’m sure others can think of better questions.

Mac the Knife

Q: What do crappy statistics generated from a pathetically biased survey produce?
A: What ever you want them to………