Guest post by A. Scott
There has been considerable discussion about the methodology and data regarding the recent paper “Lewandowsky, S., Oberauer, K., & Gignac, C. E. (in press). NASA faked the moon landing – therefore (climate) science is a hoax: An anatomy of the motivated rejection of science” (copy here)
This allegedly peer reviewed paper claims their survey data show climate skeptics are supporters of wild conspiracy theories, such as “NASA faked the moon landing.” The author admits, however, no climate skeptic sites were involved in the survey, that essentially all survey results were obtained thru posting the survey on pro-global warming sites.
Due to the serious and legitimate questions raised, I have recreated the Lewendowsky Survey in an attempt to replicate and create a more robust set of replies, including from skeptic users.
Please click on the Lewandosky Survey Page above and you’ll be presented the survey. This survey replicates the questions, both the paper, and several sites have indicated were in the original survey, including those questions deleted from the survey results.
The only change was to use a 1 to 5 ranking vs. Lewandowsky’s 1 to 4, which several people with experience have noted should improve the overall responses.
Each visit to the survey is tracked. Access is password protected for an additional layer of tracking.
THE PASSWORD FOR THE SURVEY IS “REPLICATE” (case sensitive)
Please only complete and submit once. Also, please respond to each question with the answer that best reflects your position, even though the question may not be perfectly worded.
This survey is built on the Google Doc’s open access platform. Results are collected automatically. As no significant randomization or counterbalancing was performed on the original survey none is applied here. Data collected will be provided upon request.
A. Scott
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
‘The author admits, however, no climate skeptic sites were involved in the survey, that essentially all survey results were obtained thru posting the survey on pro-global warming sites.’
So actually then, this was a study aimed at his own bigotry, about how he thinks all of us think. Perfect.
Ok, Lazy Teenager (et al), here’s your chance to skew the survey by answering it as stupidly as you possible can… never mind, forget I said that, just answer the questions.
One of the best ways to introduce graduate students to poor versus high quality surveys is to have them take such surveys and then discuss their experiences. Both response points and question design can and often do destroy the validity and reliability of surveys. Anyone thinking they can just willy nilly create a high-quality survey without having studied both valid and reliable survey development and statistical analysis of such research tools at the graduate and post-graduate level is engaging in useless work.
I took the “recreated” survey and it reminded me instantly of the poor quality designs we panned in class. This post would have been far better had we been encouraged to spend time discussing the validity and reliability weaknesses this survey, whichever version, is filled with.
However, I am not too concerned that we have not focused our attention on that kind of discussion. I have no doubt that this survey will land on the desks of graduate-level sociology class discussions on poor survey design. That includes the improved version.
Sorry A. Scott, but you need to be taken to task on this one, though I wonder if you might be reluctant to learn from such a discussion. Be forwarned. If you come back to post your results, and I hope not, you will be on the hotseat.
A. Scott says:
September 8, 2012 at 8:35 pm
Again, yes the original survey is flawed – significantly.
But understand that the point of this new survey is to collect a more transparent data set of responses to that survey as written, and from a broader spectrum – including from “skeptics” – in order to compare the findings of a more robust data set with the biased and highly questionable data and conclusions claimed by the original author.
This is a pipe-dream and an invalid conclusion. Whether N=10 or 10 million is irrelevant to the conclusions of a biased, manipulating survey. What you get is a survey of responses to the survey questions, not a survey of the subject matter of the survey questions. What you are testing is the distribution of interpretation of the questions, again, not the subject content of the questions. This is Interviewing 101.
There is a much greater danger in replicating/recreating the flawed survey over a greater N, and publishing the results. A clear and present danger that any entry level PR hack can school you on. You can put wheels to the bias contained in the questions that you have no ability to stop. To consider this, all you need to do is reflect on the global effect of climate science by press release. Accuracy and truth are not the concern of either the PR rep or the scientist who plays the game, only the agenda, and its ALL built to feed the agenda.
The result of this re-survey, can and will be used against you. Almost none of the community at large will know of the controversy over the questions – they will only know of the selected interpretations pulled from the results. The survey is already skewed and biased – making it bigger doesn’t do anything but shout the skew and the bias through a megaphone.
Many of you may well and truly be good, even great, scientists, but the majority who post here are utterly naive to the science of propaganda and “herd” management. I sincerely hope you bury the results of this so deep even the geologists can’t find it…
The survey is unbalanced.
The “free market” questions are badly framed. 3 are Free Market v. Environmentalism, so a positive FM response might be anti-green and a pro-green response might be anti-FM
Most pro free-market people are first and foremost Libertarian. So there should be questions about relating Environmentalism to authoritarianism.
A lot of environmentalists are also to the left of centre. So a balanced survey should investigate their preferences. For instance, are green policies worthwhile if they make the poor poorer – through raising domestic fuel prices, or making motoring unaffordable to the majority.
And where’s the question on the biggest conspiracy of them all – “Climate denial only exists as a serious force due to significant funding by oil and tobacco interests.”
Problem is though, there is plenty of incentive for fake responses. So I think you will find a much higher belief in conspiracy theories on both sides as a result than Lewandowsky found.
What’s the difference between “True” and “Absolutely true”? And between “False” and “Absolutely False”? Is that like “Free” and “Absolutely free” in advertising that still requires you to pay something to get the “absolutely free” item?
So is there any way to get this in front of respected polling groups? I know there are question standards out there. I suspect this would be easily and quickly panned by Gallup or Rasmussen or any of the major pollsters.
Surely, social science is enough of a joke already, without Lewandowsky and Cook doing their best to make things worse.
The problem with the vast majority of surveys is self-selection bias. Only those people who believe surveys are a worthwhile activity take part it them.
“Many of you may well and truly be good, even great, scientists, but the majority who post here are utterly naive to the science of propaganda and “herd” management. I sincerely hope you bury the results of this so deep even the geologists can’t find it…”
One of my master degrees include a mandatory psychology course. I learned that personalities who engage in the study are psychologically looking for their own self generated questions of their own psyche. The group who post here who lend any credibility to this are suspect. The survey is written for a specific agenda around being mean spirited and accusatory condescension. Herd management exploits those who are incapable of their own opinions and certainly lack critical thinking around hard science. Pretty sad.
Doh! … take part in them.
Robert of Ottawa says:
September 9, 2012 at 5:31 am
———————————————–
“how many climate scientists want a paycheck?”
@Robert. You made the same serious mistake I did. You stopped and thought about the question. At first I thought “how many climate scientists actually do believe in GW?”. Not 100, but probably most of them. So I entered 90. Then I thought “what if the survey was confidential? I bet a lot would fess up or at least admit they can’t be sure with the evidence at hand.” So I changed my answer to 60. But then I thought “how could a scientist live their whole career not believing in what they were studying?”. So I entered 100. But then I thought that maybe climate science is like the equivalent of physical education in teaching. And then I remembered that 97% of all climate scientists do believe in global warming and – not being home schooled – I did the right thing and entered 97. So I know I got one answer right!
The questions should be like:
“The freemarket system can save the economy” or “Private land owners can protect the environment by protecting their property rights through the courts”
I can give them capitalist’s answer to these……”Strongly Agree”
or
“Unsustainable spending by ‘The State’ can save the economy” or “Land ownership by ‘The State’ can protect the environment”
Again, the capitalist’s answer……..”Strongly Disagree”
Do you feel like Charlie Brown, the State is Lucy and she keeps moving the football while we fall for it over and over and over again.
Who OWNED Lake Erie when it burned? The State.
Who OWNED the forests when they burned? The State.
Who WASTED hundreds of billions of taxpayer dollars on bad mortgages? The State.
Who WASTED hundreds of billions of taxpayer dollars on bad science? The State.
Who CAUSED thousands of innocent people to die by drug cartels? The State.
Who CAUSED California’s excellent agricultural land to die without water? The State.
I see a correlation here, so……
I’m looking at Gary Johnson for POTUS. The R’s and D’s are trying to keep him off the ballot and out for the debates….I see why? To give congress a chance to fix things he would:
Present a balanced budget for 2013. He vetoed 750 bills as Governor of NM.
End “The Income Tax System” for revenue. Propose a Fair Tax.
End the unnecessary wars. Bring troops home.
End corporate welfare. GE and GM are on their own.
End the nanny state. It’s time for the takers to get into the game. Everybody needs to contribute something in return for living here.
IMHO it’s the job of our congress to pass laws to protect the US Constitution and keep the president and his minions off the people’s backs. Major FAIL. Did AG Holder give congress what they demanded for ‘fast and furious’ or are they just going to let it die with Agent Brian Terry? Holder works for the POTUS. Major FAIL. The State kills citizens and congress and the president just ‘let it slide.’
cn
” I sincerely hope you bury the results of this so deep even the geologists can’t find it…”
Yeah, but it’s the other side who bury adverse results in case they spoil the integrity of the story, even at the expense of their own. Not that I think there will be adverse results or that some on the other side wouldn’t make accusations out of any imaginable result. All part of the rich patina of life at the blogface.
I ended up answering in the middle for most questions. I assume the middle option means “I don’t know” or “No opinion”. I think a similar option, since you were “improving” the quality of the survey, should have been provided for the questions requiring a numerical answer (number of so and so…)
cui bono says:
September 8, 2012 at 7:09 pm
Given this survey is now infamous, aren’t we going to get every troll making their way here to say they don’t believe in AGW but do believe the Illuminati run the world? Just for fun?
Yep.
I think, judging by one of the posted questions I got a request on my suggestions page from the same pollster, going under the name ‘Wermet’, stating they wanted to remain anonymous. I don’t see how this can be expected to gain an unbiased sample, given the tendency for people to crowd source respondents via the net.
Pamela Gray and others have commented on the use of a 5-point rather than a 4-point scoring system and seem to be of the opinion that this improves the survey. In my opinion it is an improvement, however it is possible that the professor chose a four point system specifically because it forces respondents to polarise into one of two camps rather than remain neutral, which I am sure would assist him achieve his desired outcome.
Since i am a sceptic with regard to CAGW I am neither a believer nor a denier – I remain unconvinced by the evidence and am naturally suspicious of anyone who claims certainty – so the use of a 4-point scale would not enable me to state my true view on many of the questions..
Alvin says:
September 8, 2012 at 6:29 pm
omg, what a leftist poll. Many questions lead to catch-22 answers. The questions refuse to acknowledge that environment can be valued in a free market system. The questions pit free market against environment, and involve Marxist social justice.
====================================================
I agree !!
Those questions are worded to get a leftist/AGW answers
Wow is all that comes to mind. A “real science journal” actually published a “survey” based on THESE questions. Out right FAIL The editor of this ‘journal” has a great deal to answer for.
I suppose I need not point out that what the author has demonstrated is that Warmists are basically delusional.
It’s a stupid and annoying survey, but I answered the questions as I understood they were intended, i.e. to make skeptics look bad, so I answered to make us look good.
The phrase “I believe that. . .” is anathema to me and to science. I make my decisions based on evidence, not belief.
And I have no idea how much my neighbors make.
/Mr Lynn
A Scott
Perhaps more surprisingly, the majority of respondents have voluntarily left contact information including emails, which I think pays strong testament to the validity and quality of the responses.
>>>>>>>>>>>>>>>>>
Would be most interested in dividing the data between the ones with contact information and the ones without and comparing them.
Who in is right mind would take a survey like that? What propels the human to fill out a totally useless survey that produces totally useless answers to totally useless questions? Is this a late saturday funny? I will not participate in this farce.
@BA
“This allegedly peer reviewed paper…”
I suppose that the most important finding this paper presents is that peer review is no guarantee of correctness, academic rigour, or anything, really…
I completed the survey because I thought it was a joke, anyway. Some of the questions revealed ignorance, like the one about Area 51. Who knows anything about Area 51 that has not come from bad programs on the History Channel? Plus, how should I know how many scientists believe CO2 is a world ending gas? Who cares?
The big question is how can anyone think they do a decent survey on the web? It is a fool’s game.
Jeff Alberts says:
September 9, 2012 at 8:19 am
What’s the difference between “True” and “Absolutely true”? And between “False” and “Absolutely False”? Is that like “Free” and “Absolutely free” in advertising that still requires you to pay something to get the “absolutely free” item?
It’s like being a little pregnant.