Guest analysis by Shub Niggurath
In 2012, Stephan Lewandowsky and co-authors submitted a paper to the journal Psychological Science, generating widespread publicity. Here, I address a simple issue/question that has hovered around the paper from the time it made its appearance. The issue is at the heart of Lewandowsky’s first ‘Moon Hoax’ paper and the now in limbo second paper in Frontiers in Psychology.
The ‘Moon Hoax’ paper (a.k.a LOG12, LOG13 etc) draws a number of conclusions about climate skeptics (called ‘deniers’). A major portion of the data and analysis is devoted to ‘rejection of climate science’. The paper’s title advertises its findings about ‘deniers’.
So the question is: how did Lewandowsky and co-authors actually study climate skeptics?
The answer may surprise you.
The paper draft (pdf) stated simply that authors ‘approached’ 5 skeptic blogs to post a survey, but ‘none did’. This lead to a hunt to find who exactly these bloggers were (Lewandowsky wouldn’t tell). Lewandowsky spread significant amounts of distraction and smoke on the matter, raising hue and cry that he did email skeptical bloggers:
First out of the gate was the accusation that I might not have contacted the 5 “skeptic” bloggers, none of whom posted links to my survey. Astute readers might wonder why I would mention this in the Method section, if I hadn’t contacted anyone.
What matters however, is not whether or not Lewandowsky contacted skeptics but what came of such contact. The whole point of contacting the bloggers was to get surveys posted on their websites to ensure skeptic participation. This never took place. Through the noise, the question of non-sampling of skeptics remained unresolved‡.
As a way of providing answer, the paper itself appeared in final form about a month back. When examined, the authors appear to have settled on a remarkable method of addressing the defect. In the supplementary information, Lewandowsky et al (LOG13) make a startling claim. They state the blogs that did carry their survey have a broad readership ‘as evidenced by the comment streams’:
All of the blogs that carried the link to the survey broadly endorsed the scientific consensus on climate change. As evidenced by the comment streams, however, their readership was broad and encompassed a wide range of view on climate change.
The authors claim to have analysed reader comments at one venue to determine this. They state:
To illustrate, a content analysis of 1067 comments from unique visitors to www.skepticalscience.com, conducted by the proprietor of the blog, revealed that around 20% (N = 222) held clearly “skeptical” views, with the remainder (N = 845) endorsing the scientific consensus.
Extrapolating, the authors infer further that close to eighty-thousand skeptics saw Lewandowsky’s survey on Skepticalscience alone (see below). Owing to such broad readership, enough skeptics are said to have been exposed to the survey.
Readers of climate blogs will at once see several things that are off. However, these are the assertions forming the basis on which Lewandowsky et al 2013 rests.
Analysis
To start, the authors’ premises are accepted. It is deemed that comment streams can be analysed to determine whether a blog has a broad readership, or a more polarized one.
Comments on six blogs where Lewandowsky et al’s survey was posted were analysed. Commenter names and comment counts were obtained from web pages using R scripts. Following the authors’ method, this was carried out for the entire month the survey was posted. For each blog, duplicates were removed.
Commenters were classified as (a) skeptic, (b) ‘warmist’ (c) ‘non-skeptic’ (d) lukewarmer, (e) neutral, or (e) indeterminate. Regulars whose orientations are familiar (e.g., dana1981 – ‘warmist’) were tagged first. Those with insufficient information to classify, and infrequent posters with singleton comments were tagged ‘indeterminate’†.
The results are presented below. A total of 614 commenters contributed 4976 comments to six blogs in the month the survey was posted (range: 2 – 2387 comments/blog). An estimated 111 commenters posted across blogs, with 504 unique commenter aliases from all blogs.
The results show a skewed commenter profile. As a whole, there are 59 skeptical commenters, amounting to about 9.5% of total. Individually, skeptics range from 5-11% of commenters between blogs, with one venue (Hot Topic) showing 19% skeptics. Closer examination shows this to be made up by just 10 commenters. Non-skeptics are close to 80%, i.e., 480 of 614. Neutral posters are 9%, and indeterminate 3%. Of the 59, more than half are from comments posted at one blog (Deltoid).
The same pattern can been seen to repeat by blog:
The marked difference in comment number between the blogs obscures underlying similarities. When commenter proportions are made equal, these become plain:
From the data above it is evident these blogs are not places where readership is “broad” or encompasses a wide range of views on climate. To the contrary, these are highly polarized, partisan blogs serving their cliques. One half of the blogs hosted comments from all of 6 skeptical commenters in total (Scott Mandia, A Few Things Ill Considered, and Bickmore’s Climate Asylum).
The non-surveyed Skepticalscience.com
What about Skepticalscience’s comment stream? Lewandowsky et al state that John Cook at his website analyzed 1067 comments to identify 222 skeptics and use this to buttress claims of broad readership in survey blogs. One wonders how Cook got the fantastic figures! When commenters for Sept 2010 are analysed, there are 36 skeptical voices of a total 286. Cook’s estimates are inflated six times over. In reality skeptics form 12.58% of commenters for that month, and a mere 0.03 fraction of John Cook’s 1067 unique commenters. These results verify with independent analysis performed by A.Scott.
Furthermore, close to 90% of commenting viewers are not skeptics. Contrary to Lewandowsky et al, Skepticalscience is not a place where readership is “broad and encompasses a wide range of view on climate”. In fact Skepticalscience exactly matches Deltoid, a virulently anti-skeptic website, in commenter profile.
Importantly however, John Cook never posted the survey at Skepticalscience (see here and here). In the face of this false claim, the authors’ post-hoc exercise of computing skeptic exposure becomes counterfeit.
How would the picture have been had Lewandowsky et al actually obtained survey exposure with a skeptical audience? As a comparative exercise, I pulled comment counts from widely read skeptical blogs, Wattsupwiththat (WUWT), Bishop Hill, Joanne Nova and Climate Audit for the same period. Traffic figures provided by Anthony Watts indicate close to 3 million views in August 2010. The results ought to be eye-opening:
Conclusion:
A number of things can now be confirmed. The authors of Lewandowsky et al 2013 did not survey skeptical blogs. The websites that carried the survey have neither a broad readership, nor represented skeptical readers and commenters. The authors did not survey any readers at the website Skepticalscience, but represent their data and findings as though they did. Lastly, the authors’ calculations in assessing survey exposure, which they base on the same Skepticalscience, are shown to be wrong.
With the above, conclusions drawn about skeptics by Lewandowsky et al by sampling a population of readers and commenters who are not skeptic can be termed invalid. At best the study’s skeptic-related analysis is meaningless, arising from non-representative sampling. At worst the possibility of false conclusions owing to flawed survey exposure arises. The above data combined with Lewandowsky et al 2013 survey results, in fact, show one possible outcome of displaying loaded questions relating to climate skeptics to a non-skeptical audience. Conclusions about non-skeptical ‘pro-science’ commenters and their psychology are probably more appropriate.
Notes:
‡ The list of surveyed blogs (from Lewandowsky et al 2013 SI):
Skepticalscience – http://www.skepticalscience.com
Tamino – Open Mind http://tamino.wordpress.com
Climate Asylum – http://bbickmore.wordpress.com
Climate change task force – http://www.trunity.net/uuuno/blogs/
A few things ill considered – http://scienceblogs.com/illconsidered/
Global Warming: Man or Myth? – http://profmandia.wordpress.com/
Deltoid – http://scienceblogs.com/deltoid/
Hot Topic – http://hot-topic.co.nz/
Note that (a) there is no record of Skepticalscience having posted the survey, and (b) the Climate Change Task Force entry is available on the Waybackmachine (for e.g., here)
† Batch Google searches (e.g., http://google.siliconglobe.co.uk/) and keyword searches on scraped HTML blog posts were used to search for commenter output. Multiple entries were frequently required for each commenter to be satisfactorily classified. Wherever possible (which was so in almost all instances), results during August and Sept 2010 were employed. Comments supportive of consensus, critical of ‘deniers’ and ‘skeptics’ and/or unequivocally appreciative of article (e.g., “great post, now I can use this in my arguments with deniers”) were classified as coming from ‘warmists’. Comments approving of main thrust of a ‘warmist’ blog post, but with no further information available were classified as ‘ns’ – not skeptic. Commenters questioning basic premises of blog post, being addressed to by ‘denier’, ‘denial’ etc, whose stance could be verified by similar mode of behaviour in other threads, were classified as ‘skeptics’. In most instances they were easily recognized. Those, in whom no determination could be made, owing to various factors, were classified as ‘indeterminate’. Commenters explicitly professing acceptance of consensus but posing relatively minor question, etc – classified as lukewamers. Entries required reading at least two different comments for almost every commenter, except in instances commenter orientation was known from prior experience. Certainly there will be errors to a degree, and subjectivity is involved. It is unavoidable that infrequent (and singleton) commenters, and those with non-unique names (‘tom’, ‘john’) are resistant to classification. Validation of method was available when blogger A.Scott arrived at similar results working independently on portions of the data.
Related articles
- Lewandowsky, Cook claim 78,000 skeptics could see conspiracy survey at Cooks site where there is no link (climatedaily.wordpress.com)
- Lewandowsky Doubles Down (climateaudit.org)
- John Cook’s new survey – lots of questions, no answers (wattsupwiththat.com)
- Why publishers should explain why papers disappear: The complicated Lewandowsky study saga (retractionwatch.wordpress.com)
- IPCC Lead Author calls Lewandowsky “deluded” (climatedaily.wordpress.com)
- The Moon Hoax has Landed (geoffchambers.wordpress.com)
- Dear John [Cook], you want “deniers” to help you do a fallacious survey eh? (JoNova)
The uninformed will always proffer an uninformed opinion, regardless of the subject. No wonder the CAGW meme continues unabated.
Shub, Your analysis clearly needs to be forwarded to any editors of Journals considering any Lewandowsky article that is built of this foundation of sand.
You may want to mail a copy of the article to the Royal Society that so generously supplements Lewandowsky’s salary for his move to the UK. While I doubt that they’ll make a face-saving turnabout it should not be said that they have not been warned about the quality of his work and the extent of his scientific misconduct.
It really is quite alarming this whole pattern of deceit coming from the “consensus”. Mind you the attempts to homogenise a whole section of society into a close set of parameters would be considered racist if applied to an ethnic group or elitist if applied to a social strata.
Nasty piece of work this Lewandowsky fellow.
Well, duh. Lew is a psychologist, not a scientist. Of course he uses tons of data that fits his agenda rather than actually doing research. This is not scientific work, it’s statistic work. You can’t even nail him for scientific misconduct because science and statistics are not synonymous. Psychology is not a science. It doesn’t deal with tangible evidence, but rather exclusively with railroaded interpretation of extremely limited data.
This is only one thing: “social sciences”
Social science is neither social nor science. In fact, when used by the political Left it is profoundly antisocial. It is for marginalizing, dividing, and conquering all opposition to progressive… oops, “progressive” ends. “Ha-ha, AGW deniers are ignorant and stupid. Here’s proof!”
Science and statistics are not synonymous.
Discovery of a numerical discrepancy is not science. Accounting for that discrepancy in a reproducible manner is science.
It’s no wonder I’m a skeptical bloke!
Surely this is fraud?, why are we not hearing that criminal investigations are taking place into the fraudulent spending of taxpayer money?
It all seems so childish. Infantile.
Someone needs to conduct a study….a real one, no con-tricks…investigating the level of correlation between CAGW belief and belief in conspiracy theories that 9/11 was an inside-Bush and Cheney Did It production. i would expect virtually every “Troofer” that draws air believes fervently in CAGW. Its this intensity of correlation on that side of the equation that would be interesting, not how many CAGW believers as a whole are “Troofers”.
It still wont be science…correlation is not evidence of anything by itself in any case.
But thats not the point.
So now the skeptics are being adjusted to fit a preconceived notion of who they think we are..
If they had courage to post a survey on any reputable skeptic site and if the skeptic would even bother to participate knowing what they know about the warmists.. It would not have given them their preconceived notion..
A impossible task that would have to be adjusted / smoothed to fit their intended message..
I wonder if they would allow skeptic surveys on warmists sites..
Wild eyed climate alarmism is mainly due to..
A) Strong eye muscles.
B) Community theater training.
C) Father / Mother issues.
D) Inability to get a real job.
E) All of the above.
This is fun, and I can guarantee the results!!!
Edohigmuma…get a sensible name will you:
“Psychology is not a science. It doesn’t deal with tangible evidence, but rather exclusively with railroaded interpretation of extremely limited data.~”
Well thats just dumb ignorant crap. I guess you think Sigmund Freud represents what psychology is.
Psychology can be every bit as real a science as any other. The transmision speed of a nerve signal or the re-potentiation frequency of a synapse are every bit as much real data as anything.
The problem wirth Lew-another silly name -guy is that he has obtained credibility through psychology and is now exploiting his position through abuse of that status to promote pure shite of the most highly refined first order. He has a post as cognitive psychologist but this shite he retails is social sychology…not only not his province or what hes being paid to do but….here I would agree, not in the least a proper scientific undertaking.
And yes, he IS bound professionally by very stringent codes of ethics (I know from experience, Ive had to jump the hurdles posed by ethical reviews) …and he I~S in breach of those codes of ethics…and it MAY be possible for dilligent folk to pursue this.
You can call Lew-guy an idiot, an arse-wipe or any other simple statement of fact, but bellowing “psychology aint no science boy” is only going to count against yourcase, hamper his critics and help him immensely.
And Cook is surprised to find skeptics unwilling to participate in a rigged survey.
When the entire foundation of your opinion is based on skewed and cherry-picked data, finding more support for your bias must necessarily entail more of the same.
Did he really not think that someone would attempt to verify his data? Thank you Shub.
Psychology is not a science. It doesn’t deal with tangible evidence, but rather exclusively with railroaded interpretation of extremely limited data.
In which case, I suppose it’s perfectly OK for Stephan Lewandowsky to conduct this kind of study, to report it in the manner he did—even for him to publish it in a journal called…. Psychological Science.
We psychologists don’t have the answers to all of the questions in our field. We don’t even have answers to most of them.
But we have learned a few things over the years about how to conduct and analyze empirical research with human subjects.
Lewandowsky may have learned some of these things, once upon a time, but since he got on the CAGW bandwagon he’s acted in complete disregard of them.
Shub’s analysis is further evidence of how just poorly Lewandowsky designed his study, recruited his participants, and collected his data—then gave the results a misleading presentation.
All of which ought to matter. But to someone who’s decided in advance that psychology isn’t a science, none of it will.
I have my own social analysis.
http://geosciencebigpicture.com/2013/01/03/global-warming-and-prohibition/
No one can be this incompetent. The only value that any “data”, analysis, or conclusions these people have produced can have is in potential criminal proceedings.
Bit the overall numbers provided in the post are very interesting. They confirm what has recently been becoming a concrete reality to me: in the world of Believers, there is no-one there.
In the above graph for comments in August 2010, proper sites: junk is 5.95:1. And this was in August 2010! When this still had the momentum of hysteria behind it! Six to one!
It would be really really interesting, for those who know how to twiddle the dials, to do an up-to-date survey across all known sites. Even if just in gross comments, without attempting to break them down into distinct commentators or positions. I’d put money on it being more like ten to one now. And a hell of a lot of those junk site comments will be from people pulling money out of this.
What this really means, is that after having labored under the burden of being a beleaguered minority of dissenters, which at all times has been a manufactured deceit in any case, people can start dealing with this in the full knowledge that the Believers, no matter how much noise they make, and how much money they can splash, and how many people in positions of influence have been suborned, are virtually non-existent.
As soon as this dawns on the average adherent, they will back off and start to explain any embarrassing social exchanges away. This is the heartland of the social conformist. They would rather die than be seen as part of a dwindling and in particular, increasingly uninfluential minority.
This is 5 minutes away from being reduced to the core beings jerking around in their evaporating pond.
On a link from another web site, I actually went out and looked at the paper.
I couldn’t believe it. My first reaction was that it wasn’t intended as a real scientific paper. I thought it was a paper designed to trigger a reaction on the Internet (pro and con) and the real goal of the study was to investigate how stories, rumors, and theories spread on the Internet.
Shub,
Excellent analysis.
Please send it to Erich Eich, the editor in chief at Psychological Science.
Sorry, *Eric* Eich.
It appears that John Cook’s numbers for commenters at Skeptical Science (included in the supplemental materials to LOG13) are completely fantastical.
One way to see this is that Cook claimed to be able to classify every commenter at his site as warmist or skeptic.
As Shub has noted, some commenters say so little, or are so cryptic in what they do say, that they will have to be classified as “indeterminate.”
as we know the 8 blogs were described by Lewandowsky et al as having a ‘diverse audience’
Interestingly, in another paper (also in press, concurrently with ‘moon papaer, at the time)) Lewandowsky/Cook recognise that blogs can become ‘cyber-ghettoes’ of likeminded (unrepresentative) people that become increasingly polarised in readership.
Misinformation and Its Correction: Continued Influence and Successful Debiasing
Lewandowsky, Ecker, Cook et al (in press Psychological Science in the Public Interest)
“Blog readers employ selective exposure to source information from blogs that support their existing views. More than half of blog users seek out blogs that support their views,
whereas only 22% seek out opposing blogs, leading to the creation of “cyber-ghettos” (T.J.Johnson, Bichard, & Zang, 2009).
These cyber-ghettos, in turn, have been identified as one reason for the increasing polarisation of political discourse (McCright, 2011; Stroud,2010)”
http://websites.psychology.uwa.edu.au/labs/cogscience/documents/LewandowskyEcker.IP2012.PSPI.pdf
Yet Lewandowsky et al surveys of 8 ‘anti-sceptic’ blogs were the very worst sort of these ‘cyber-ghettoes’ (censorious, abusive,etc)
which gives a lie to the papers claim that blogs with a diverse readership were surveyed, they know this to be false.
I asked for Prof Stephan Lewandowsky for the Skeptical Science URL(early August 2012), he had only provided domain names to me, when I requested them, he replied he told me it had been posted, he had made a note of it.
“I worked with John Cook directly at the time and he posted it (and I made a note of it), but I don’t have the actual URL to the survey dating back to the time when he posted it” – Stephan Lewandowsky – August 01, 2012
I to had never heard of Prof Lewandowsky as far as I recall at the time, I had merely read Dr Adam Corners article in the Guardian and sought to ask the lead author for the identity of the 8 ‘pro-science’ blogs and 5 ‘sceptic blog) blogs. Prof Lewandowsky had sent me the domains names, but not urls’ to the surveys. I had quickly located 6 of them, but could not find 2, I thought that the most high traffic website Skeptical Science was important, especially to see th ecomments! so that I might discuss this with Dr Adam Corner at his blog, where he had reproduced the Guardian article in full (less comments to wade through, 1300+, vs 42)
http://talkingclimate.org/are-climate-sceptics-more-likely-to-be-conspiracy-theorists/
This email exchange is below (all emails between us here:)
http://climateaudit.org/2013/03/28/lewandowsky-doubles-down/#comment-407927
Also, as you can see, I did not doubt that he ha contacted ‘sceptic’ blogs.
———————-
From: barry.woods
Sent: Tuesday, July 31, 2012 5:00 AM
To: Stephan Lewandowsky
Subject: Links to surverys – Skeptical Science – Guardian Article about you recent paper.
Hi Stephan
sorry to approach you one more time.
I cannot find the link to Skeptical Science survey, this is probably the most high profile blog with the most media/public recognition (i.e. won awards) of the ‘pro-science’ vs. the “Skeptical” blogs
(I’m guessing Climate Audit, WUWT, Bishop Hill & maybe The Air Vent (ie Condon) and Jo Nova )
I’ve found six of the links to the opinion surveys, and the range of comments on the blogs are quite interesting as well, did you consider this feedback in the research?
but, I would expect that Skeptical Science would have the most comments and opinions and probably the largest readership.
Can you send me the link to the Skeptical Science blog article/comments?
And was the survey able to capture the referring blog, as this might also give indicators of relative popularity of the blog,
does the survey break down by referring blog and are these figures available?
Best Regards
Barry
rather than lots of questions, if you have the supporting data, etc in an easily accessible package (without too much trouble for yourself) could you send that as well.
If not quickly to hand, that’s fine please don’t waste any time, as I’m mainly just curious on the couple of point above.
there were the links I found:
http://scienceblogs.com/illconsidered/2010/08/counting-your-attitudes/
http://profmandia.wordpress.com/2010/08/29/opinion-survey-regarding-climate-change/
http://scienceblogs.com/deltoid/2010/08/29/survey-on-attitudes-towards-cl/
http://hot-topic.co.nz/questionnaire/
http://tamino.wordpress.com/2010/08/28/survey-says/
http://bbickmore.wordpress.com/2010/08/28/take-a-survey/
I’m missing this blog survey link as well.
http://www.trunity.net/uuuno/blogs/
From: Stephan Lewandowsky
Sent: Wednesday, August 01, 2012 11:00 AM
To: barry.woods
Subject: RE: Links to surverys – Skeptical Science – Guardian Article about you recent paper.
Hi Barry, the survey was done about 2 years ago, and I don’t have the link to SkS: I worked with John Cook directly at the time and he posted it (and I made a note of it), but I don’t have the actual URL to the survey dating back to the time when he posted it. I suspect he removed it when the survey was closed because then the link would have been dead.
Regards Steve
———————
Prof Lewandowsky, based on the above, clearly lied about, the existence of the survey at Skeptical Science, to me personally, and then I find myself named in the ‘Fury’ paper……..
Punitive Psychology?
John Cook (Sceptical Science – & Fury paper, co author, with Prof Lewandowsky) seems to have lied to Geoff Chambers as well.. Geoff ALSO ended up, I the data of the Recursive Fury paper….
——————————–
JC: Hi Geoff, you can email me via this email address if you have any direct questions, although there’s not much more that I can add other than what I’ve mentioned in the comment threads.
GC: Thanks John
My interest comes from the fact that, of the eight “pro-science” blogs contacted by Lewandowsky, SkS is by far the most important. One might therefore expect that the majority of respondents to the survey came from SkS (depending on the coverage you gave it, and the date at which you posted it, etc.)
At two of the six (Tamino’s and Deltoid) there was significant discussion of the survey, with people criticising and taking up positions. This, too, is interesting when it comes to interpreting the survey. So here are my questions:
– The date the survey was posted
– The date the post was deleted
– Were there comments to the post? If so, how many, and are they still available, or were they deleted along with the original post?
JC: Hi Geoff, sorry for the delay in replying, very behind in my email correspondence at the moment plus for this email, had to fire up the old machine that I was using back in 2010 to find any email correspondence back then. All I can find is an email from Steve on 28 August 2010 asking for me to link to his survey.
GC: Hi John
Thanks for the reply. So did you in fact link to his survey? It looks to me that you just forgot and didn’t post the link. So Stephan just assumed you had posted, and put in his paper the reference to eight blogs he’d contacted, including yours and the dormant NZ one. A silly mistake easily corrected. All he has to do is correct the “eight blogs” in his paper to six. Can you confirm that his survey was not in fact linked from Skeptical Science?
JC: I did provide a link to the survey.
GC: Hi John
Any chance of telling us when you put up the link? Sorry to keep pestering you but you are being a bit coy.
JC: I’ve given you everything I’ve got – I have no records in the blog archives (I searched the database for kwiksurvey, came up empty) so I must’ve either deleted the text link or deleted the blog post once the survey had closed. The only forensic evidence I could find was the email from Stephan asking for me to post a link and my reply that I posted it on the same day.
——————————–
Lewandowsky was ridiculing sceptics for not keeping 2year old unsolicited emails from somebody unknown to them (his assistant Hanich, not even a co-author), yet he can’t even prove the claim in his paper that Skeptical Science hosted this survey.
In fact this is clearly a lie….
Takes only a moment to webcite a url, yet of course you can’t if it does not exist
There are some darned fine psychologists that do very credible scientific work. On the other hand, it is quite easy to get away with shoddy work in the social sciences. Imagine the trouble with trying to vet a study of this sort. Doing so explains why editors of journals, reviewers, and colleagues rarely recognize lame efforts like this, especially when the results, sort-of, conform to one’s preconceptions. In every respect this present work is the equivalent of Jost’s attempt to “prove” Republicans as being authoritarian personalities through surveys and analyses that were every bit this bad.
The problem with all of this is lack of skepticism–Lew tries to give skepticism a bad reputation, the succesful end effect of which is to amplify group-think and wreck science.
Cook and Lewandowsky correspondence, reproduced here:
http://climateaudit.org/2013/04/03/tom-curtis-writes/