Guest analysis by Shub Niggurath
In 2012, Stephan Lewandowsky and co-authors submitted a paper to the journal Psychological Science, generating widespread publicity. Here, I address a simple issue/question that has hovered around the paper from the time it made its appearance. The issue is at the heart of Lewandowsky’s first ‘Moon Hoax’ paper and the now in limbo second paper in Frontiers in Psychology.
The ‘Moon Hoax’ paper (a.k.a LOG12, LOG13 etc) draws a number of conclusions about climate skeptics (called ‘deniers’). A major portion of the data and analysis is devoted to ‘rejection of climate science’. The paper’s title advertises its findings about ‘deniers’.
So the question is: how did Lewandowsky and co-authors actually study climate skeptics?
The answer may surprise you.
The paper draft (pdf) stated simply that authors ‘approached’ 5 skeptic blogs to post a survey, but ‘none did’. This lead to a hunt to find who exactly these bloggers were (Lewandowsky wouldn’t tell). Lewandowsky spread significant amounts of distraction and smoke on the matter, raising hue and cry that he did email skeptical bloggers:
First out of the gate was the accusation that I might not have contacted the 5 “skeptic” bloggers, none of whom posted links to my survey. Astute readers might wonder why I would mention this in the Method section, if I hadn’t contacted anyone.
What matters however, is not whether or not Lewandowsky contacted skeptics but what came of such contact. The whole point of contacting the bloggers was to get surveys posted on their websites to ensure skeptic participation. This never took place. Through the noise, the question of non-sampling of skeptics remained unresolved‡.
As a way of providing answer, the paper itself appeared in final form about a month back. When examined, the authors appear to have settled on a remarkable method of addressing the defect. In the supplementary information, Lewandowsky et al (LOG13) make a startling claim. They state the blogs that did carry their survey have a broad readership ‘as evidenced by the comment streams’:
All of the blogs that carried the link to the survey broadly endorsed the scientific consensus on climate change. As evidenced by the comment streams, however, their readership was broad and encompassed a wide range of view on climate change.
The authors claim to have analysed reader comments at one venue to determine this. They state:
To illustrate, a content analysis of 1067 comments from unique visitors to www.skepticalscience.com, conducted by the proprietor of the blog, revealed that around 20% (N = 222) held clearly “skeptical” views, with the remainder (N = 845) endorsing the scientific consensus.
Extrapolating, the authors infer further that close to eighty-thousand skeptics saw Lewandowsky’s survey on Skepticalscience alone (see below). Owing to such broad readership, enough skeptics are said to have been exposed to the survey.
Readers of climate blogs will at once see several things that are off. However, these are the assertions forming the basis on which Lewandowsky et al 2013 rests.
Analysis
To start, the authors’ premises are accepted. It is deemed that comment streams can be analysed to determine whether a blog has a broad readership, or a more polarized one.
Comments on six blogs where Lewandowsky et al’s survey was posted were analysed. Commenter names and comment counts were obtained from web pages using R scripts. Following the authors’ method, this was carried out for the entire month the survey was posted. For each blog, duplicates were removed.
Commenters were classified as (a) skeptic, (b) ‘warmist’ (c) ‘non-skeptic’ (d) lukewarmer, (e) neutral, or (e) indeterminate. Regulars whose orientations are familiar (e.g., dana1981 – ‘warmist’) were tagged first. Those with insufficient information to classify, and infrequent posters with singleton comments were tagged ‘indeterminate’†.
The results are presented below. A total of 614 commenters contributed 4976 comments to six blogs in the month the survey was posted (range: 2 – 2387 comments/blog). An estimated 111 commenters posted across blogs, with 504 unique commenter aliases from all blogs.
The results show a skewed commenter profile. As a whole, there are 59 skeptical commenters, amounting to about 9.5% of total. Individually, skeptics range from 5-11% of commenters between blogs, with one venue (Hot Topic) showing 19% skeptics. Closer examination shows this to be made up by just 10 commenters. Non-skeptics are close to 80%, i.e., 480 of 614. Neutral posters are 9%, and indeterminate 3%. Of the 59, more than half are from comments posted at one blog (Deltoid).
The same pattern can been seen to repeat by blog:
The marked difference in comment number between the blogs obscures underlying similarities. When commenter proportions are made equal, these become plain:
From the data above it is evident these blogs are not places where readership is “broad” or encompasses a wide range of views on climate. To the contrary, these are highly polarized, partisan blogs serving their cliques. One half of the blogs hosted comments from all of 6 skeptical commenters in total (Scott Mandia, A Few Things Ill Considered, and Bickmore’s Climate Asylum).
The non-surveyed Skepticalscience.com
What about Skepticalscience’s comment stream? Lewandowsky et al state that John Cook at his website analyzed 1067 comments to identify 222 skeptics and use this to buttress claims of broad readership in survey blogs. One wonders how Cook got the fantastic figures! When commenters for Sept 2010 are analysed, there are 36 skeptical voices of a total 286. Cook’s estimates are inflated six times over. In reality skeptics form 12.58% of commenters for that month, and a mere 0.03 fraction of John Cook’s 1067 unique commenters. These results verify with independent analysis performed by A.Scott.
Furthermore, close to 90% of commenting viewers are not skeptics. Contrary to Lewandowsky et al, Skepticalscience is not a place where readership is “broad and encompasses a wide range of view on climate”. In fact Skepticalscience exactly matches Deltoid, a virulently anti-skeptic website, in commenter profile.
Importantly however, John Cook never posted the survey at Skepticalscience (see here and here). In the face of this false claim, the authors’ post-hoc exercise of computing skeptic exposure becomes counterfeit.
How would the picture have been had Lewandowsky et al actually obtained survey exposure with a skeptical audience? As a comparative exercise, I pulled comment counts from widely read skeptical blogs, Wattsupwiththat (WUWT), Bishop Hill, Joanne Nova and Climate Audit for the same period. Traffic figures provided by Anthony Watts indicate close to 3 million views in August 2010. The results ought to be eye-opening:
Conclusion:
A number of things can now be confirmed. The authors of Lewandowsky et al 2013 did not survey skeptical blogs. The websites that carried the survey have neither a broad readership, nor represented skeptical readers and commenters. The authors did not survey any readers at the website Skepticalscience, but represent their data and findings as though they did. Lastly, the authors’ calculations in assessing survey exposure, which they base on the same Skepticalscience, are shown to be wrong.
With the above, conclusions drawn about skeptics by Lewandowsky et al by sampling a population of readers and commenters who are not skeptic can be termed invalid. At best the study’s skeptic-related analysis is meaningless, arising from non-representative sampling. At worst the possibility of false conclusions owing to flawed survey exposure arises. The above data combined with Lewandowsky et al 2013 survey results, in fact, show one possible outcome of displaying loaded questions relating to climate skeptics to a non-skeptical audience. Conclusions about non-skeptical ‘pro-science’ commenters and their psychology are probably more appropriate.
Notes:
‡ The list of surveyed blogs (from Lewandowsky et al 2013 SI):
Skepticalscience – http://www.skepticalscience.com
Tamino – Open Mind http://tamino.wordpress.com
Climate Asylum – http://bbickmore.wordpress.com
Climate change task force – http://www.trunity.net/uuuno/blogs/
A few things ill considered – http://scienceblogs.com/illconsidered/
Global Warming: Man or Myth? – http://profmandia.wordpress.com/
Deltoid – http://scienceblogs.com/deltoid/
Hot Topic – http://hot-topic.co.nz/
Note that (a) there is no record of Skepticalscience having posted the survey, and (b) the Climate Change Task Force entry is available on the Waybackmachine (for e.g., here)
† Batch Google searches (e.g., http://google.siliconglobe.co.uk/) and keyword searches on scraped HTML blog posts were used to search for commenter output. Multiple entries were frequently required for each commenter to be satisfactorily classified. Wherever possible (which was so in almost all instances), results during August and Sept 2010 were employed. Comments supportive of consensus, critical of ‘deniers’ and ‘skeptics’ and/or unequivocally appreciative of article (e.g., “great post, now I can use this in my arguments with deniers”) were classified as coming from ‘warmists’. Comments approving of main thrust of a ‘warmist’ blog post, but with no further information available were classified as ‘ns’ – not skeptic. Commenters questioning basic premises of blog post, being addressed to by ‘denier’, ‘denial’ etc, whose stance could be verified by similar mode of behaviour in other threads, were classified as ‘skeptics’. In most instances they were easily recognized. Those, in whom no determination could be made, owing to various factors, were classified as ‘indeterminate’. Commenters explicitly professing acceptance of consensus but posing relatively minor question, etc – classified as lukewamers. Entries required reading at least two different comments for almost every commenter, except in instances commenter orientation was known from prior experience. Certainly there will be errors to a degree, and subjectivity is involved. It is unavoidable that infrequent (and singleton) commenters, and those with non-unique names (‘tom’, ‘john’) are resistant to classification. Validation of method was available when blogger A.Scott arrived at similar results working independently on portions of the data.
Related articles
- Lewandowsky, Cook claim 78,000 skeptics could see conspiracy survey at Cooks site where there is no link (climatedaily.wordpress.com)
- Lewandowsky Doubles Down (climateaudit.org)
- John Cook’s new survey – lots of questions, no answers (wattsupwiththat.com)
- Why publishers should explain why papers disappear: The complicated Lewandowsky study saga (retractionwatch.wordpress.com)
- IPCC Lead Author calls Lewandowsky “deluded” (climatedaily.wordpress.com)
- The Moon Hoax has Landed (geoffchambers.wordpress.com)
- Dear John [Cook], you want “deniers” to help you do a fallacious survey eh? (JoNova)
Do you propose that the “truthers” are primarily left-wingers? Neo-Marxists, 99%-ers, “Occupiers” or related far-left malcontents?
I don’t doubt psychology is a science. Unfortunately too few people know enough about it to form valid judgments on whether a report “based on psychology” is legitimate or fabrication. The rise of “pop psychology” has not helped this. Like anyone who thinks they know more about a topic than others, psychologists can descend into “code” speak, where a few important sounding words can confuse a listener and make them think the speaker knows what they’re talking about.
I’ve also occasionally done the same when required to convince a potential client that I know what I’m doing when it comes to I.T. or programming in general. I hate doing it…
Some of Lew’s methodologies and conclusions are way across the line, going from bafflegab to pure fabrication. This is one of those. As others have commented above, including SkS in any form completely invalidates any possible conclusion. I’ve never before seen any forum site on any topic do as much underhanded fiddling of comments left by people. Even in car related forums, which can get VERY confrontational and argumentative, usually the worst thing done to a dissenting poster is to delete them and their posts. Altering posts to change their meaning or make the poster look like an idiot is dishonest at best, and probably should be unlawful.
But I guess if it promotes your fervent belief, almost anything appears justified. This IS how wars start, and IS how people blow others up. The warmist cult might be among the most insidious religions ever formed.
Charlatan is the word that springs to mind!
Sam the First says:
May 5, 2013 at 5:08 pm
“Academics react very negatively to such basic errors.”
The reason why they do that is to avoid the issues and to run the logical fallicy of ad hom as a way to ignore the issue. Also his general points got across just fine and he wasn’t the only ones filing against this paper. Its pretty clear cut they want to bury the isue because they know they will lose.
@Barry please keep us updated on that mess. If possibe can you file with your own ethics department or maybe even have someone file an eithics complaint against you to force your ethics department to look into the issue?
I hope I’m not repeating something that has already been said above….but I am always amazed that Lew can persist with this nonsense about ‘deniers’ who believe that NASA faked the moon landings,when a man who WALKED on the moon, as well as scientists/technicians who helped him get there and back -have declared themselves ‘skeptics’.
Surely this simple fact debunks all Lew’s conspiracy garbage in one fell stroke?
Great jobs, Shub and Barry W. KUTGW.
I guess now all we have to do is direct Loo-and-poo-sky to WUWT, Climate Depot, JoNova, JunkScience.com, CFACT, Bishop Hill, the Global Warming Policy Foundation, Climatism!.com, Jennifer Marohasy and see how few skeptic commenters he still thinks there are.
I used to try to put criticisms on warmist web sites, and I was kicked off and called a troll for not agreeing with them. What could they be sampling besides useful idiots. My views are here: http://www.nov79.com
Tom Curtis (carrying the water for John Cook, anyone know why?), raises questions. As usual, Tom is good for the most base level of factual accuracy, but from there on, all bets are off.
First he says that 80% of commenters are ‘warmists’, which is the same as Cook’s estimate. This misses key points, that commenter JunkPsychology notes above. First, Cook does not state where he got his 1067 commenters from to classify. What is this population? We don’t know. Second, classifying commenters is not an easy task – many offer the bare minimum information to categorically place them in one or the other group. The fact that Cook took a thousand commenters and shoehorned them into neat and easy bins of skeptic and warmist, only shows that he blithely overestimated skeptics in his own audience. A more accurate estimate shows skeptics to be ~10% of the total, a figure which repeats across blogs (Skepticalscience, Deltoid) and between observers. What matters is whether the survey was posted at a venue where skeptics are active, engaged and likely to respond positively to blog owners’ requests. Tom has no answer for this fundamental requirement for survey sampling not being fulfilled.
Then, Tom claims that Junkscience.com posted the survey. The paper itself does not recognize survey participants from Junkscience.com. The paper also claims significant participation from Skepticalscience readers when there are actually none.
The next bit is ironic. Tom writes:
“I should add, Shub also demonstrates the much larger readership at the “skeptical blogs” than at the pro-science blogs. It follows that had the owners of those “skeptical” blogs posted links to the survey, the proportion of “skeptics” to pro-consensus respondents to the survey would have been reversed. The lack of “skeptical” respondents to the survey was not by design of Lewandowsky and his co-authors.”
Everything stated above is correct. Without getting down into the Freudian slips, the lack of skeptical respondents is not from bad study design, though there are enough indications to suggest this, but from a complete failure of translating design into practice. Lewandowsky’s intention in contacting skeptical blogs was to sample skeptical opinion. He and his co-authors never translated this.
The final is funny:
“Rather it is the direct consequence of “skeptics” not wanting their views of their supporters to be known”
Really? Skeptical bloggers conspired to not post Lewandowsky’s survey? Everyone contacted had to go rummaging in their spam folders to pull out emails from Lewandowsky and his associate. This brings things to rock-bottom with Curtis’ arguments.
I’m somewhat surprised that someone hasn’t proceeded with legal action re libel regarding the Recursive Fury paper,or are attempts on defamation allowed now?
Sam the first, when i wrote my complaint, i was on holiday, working on a tablet.
I had previously just asked for some information, given the circumstances from the authors, and had been completely ignored. I am just a member of the public, my expectation from the journal, uwa wascdor them tovreact in horror ar whar lewandowsky et alnwere doing, thank me very much, snd just deal with it and to withdraw both papers.
Because of such blatant activism, and researchers that were so very conflicted, being atagonistic and publically hostile to their research subjects. I don’t have the time nor energy for this, if the field of psychology is not offended by all this, and refuses to take action. What more can I do and why should i care for psychologies reputation.
I yhought psychology was supposed to protect the public from this sort of politicised behaviour!
Still on a tablet and i don’ t care about typos, still on holiday.
When i get back, now that I’ve been given the brush off, am inclined to publish all correspondence and be done with it.
Why are not psychologist offended by The actions of activist hostile to research subjects, psychologists
I think that some of the problems related to this paper is the definition of a Skeptic. In the eyes of most commentators Skeptical Science is not remotely sceptical of any climate issue, but they feel they are and are good example of how a genuine skeptic thinks and feels. Much of their thinking is based on that idea, so in their eyes someone who believes that AGW is only 99% correct is a skeptic, and anyone who questions the fundamentals of the science is beyond the pale.If their classification of attitudes is seen in this light it is more understandable, though it still remains a pretty rotten piece of research.
Yes that is correct. The crackpots are overwhelmingly leftists. And those that appear as “right-wing” are likely to be false flaggers.
This Scientologist, Lewandowsky, merely thought he was being clever pointing the finger elsewhere, rather than right back at himself and his fellow travelers. This thesis, that “truthers” are on the right and are anti-science and climate den!ers is itself a prime example of leftist crackpotism, a so-called conspiracy theory. Connect the dots, secret societies of big oil barons paying anti-science bloggers, a huge coordinated effort to destroy them. It’s all there, only the subject matter is changes from theory to theory. Lewandowsky is exactly what he is writing about. A crackpot.
@barry woods
Sure. Publish it. UWA is a public body. I don’t see any reason why their correspondence with a member of the public should be confidential. Go for it!
PS – sorry to say that I agree with Sam. A proof reader/editor would improve your stuff a lot. On a blog minor errors don’t matter a lot, but in formal correspondence they do. ‘Being on holiday’ doesn’t mean that you shouldn’t present your case as clearly, concisely and crisply as you can.
Remember the aphorism attributed to Churchill
‘I am sorry to send you such a long letter. I do not have time to write a short one’.
Wise words.
Here is what I encountered over at SkS . .
“keitho at 01:13 AM on 6 May, 2013
I do hope that the same chicanery that happened with the last attempt at profiling doesn’t happen again. Shub Niggurath has done some analysis on that paper and it doesn’t look good . .
http://wattsupwiththat.com/2013/05/05/lewandowsky-et-al-2013-surveying-peter-to-report-on-paul/#comment-1297865
Tom Curtis at 07:46 AM on 6 May, 2013
keitho @22, its odd that you should refer to Shub Niggurath’s piece as proof of chicanery in LOG13 (moon landing paper). What he demonstrates after detailed (and likely biased) analysis is that slightly less than 80% of commenters on the pro-science sites that posted LOG13 survey support the consensus on climate change; ie, approximately the same percentage as in the survey itself. Beyond that the analysis seems entirely based on misinterpreting “broad readership” as meaning “readership equally divided between those opposing and those accepting the consensus”.
Curiously, despite his detailed analysis, he fails to mention that a link to the survey was posted on at least one “skeptical” blog, ie, Junk Science. Admitedly Junk Science was not one of the blogs contacted by Lewandowsky, but that is hardly relevant. The fact that the link was posted on a “skeptical blog” undermines the entire basis of Shub’s analysis.”
Not sure about Tom Curtis’ role over there is but he obviously misses Shub’s points completely.
Latimer. I originaly tried:
“The researchers are conflicted, hostile bloggers against the people they research, please ask the authors to remove my name, data and urls from the paper – Recursive Fury”
THAT sentence alone, SHOULD have been enough for both papers to have been pulled, by the journalsand UWA. It is psychology after all. !! Hostile researchers, fighting sceptics on their own blogs!
MUCH better submissions than mine, have recieved the EXACT same response, from UWA
I just thought I just had to put something forward, even if it ruined my holiday, my wife was upset with time wasted on this. I just didn’t want my chidren to come across this in the future, nor their friends.
mine was purely my personal opinion, trying to get my name out of a paper, where I get labelled a conspiracy theorist, where both reseaechers, doing the work, Cook And Marriott are hostilevtowards me, and have been shown to have a stong vested interest, in disparging anything I write at WUWT.
In particular, my WUWT article, What else did 97% of scientists say.
My first sentence to the journals above, shoukd have been enough, (along with links evidence i provided)
If my actions are not good enough for armchair sceptic commentatirs. Why don’t you all take the time to do the work, “properly”, what’s stopping everybody. I have a family, 3 small children, day job, elderly unwell parents, a grandmother who spent Easter holidays, whilst complaints were happening, suffering multiple falls etc. I am done with this.
Let psychology reputation fester, I don’t care, OTHER psychologists should be offended and complaining.
Not me, I really can’t be bothered being a sceptic any more, at the moment, if ‘Science’ doesn’t care about this sort of activism, why should i care about science any more
Curently at a chess tournament, with my son
@barry woods
Whoa…hold on there, tiger!
You’re not the sort to give up just like that.
Sceptics come in all sorts of shapes and sizes…that’s one of the characteristics – uniform they ain’t. Some are great at forensics (McSteve), some at narrative (the Bish), some at ridcule, some at deep diving the ‘science’.
You are especially good at persistence…if we didn’t already have a ‘Tiger’, you’d be him. But ‘Terrier’ Woods will do.
Take a break, go for a walk, have a drink or relax in whatever way floats your boat. Forget about it all for a few days. Then reconsider.
I’m confident that you’ll return even more determined to adhere to your motto
Nil illegitimi carborundum.
which – as all Latin scholars will know – translates as
‘Don’t let the bastards grind you down’
LA
Barry
The psychology game is played differently. If you read the literature, there are several instances of psychology post-graduates and majors researching small communities and vulnerable groups via deception (there was no other way to win their trust). This won the approval of several members of their profession. Psychology researchers anticipate backlash from deceived subjects and take that as a sign that their research conclusions must have hit the mark. I believe Lewandowsky says as much in one of his passages. As a response, the psychology community walled off studying ‘vulnerable groups; (as did all medical research) without strong ethics review procedures and protections in place. Online skeptics are by no stretch members of a vulnerable group, but, if you transport petty insults dressed up as pop psychology into the peer-reviewed domain, you create a situation where your former fellow debater cannot respond.
If your evidence is enough, it has not been presented to those who would take an equanimous look at it. You make the case until someone who’s supposed to get it, gets it. If you don’t have the time, someone else will, and there are other times. Lewandowsky and Cook have behaved in an exploitative manner, utilizing public commentary offered in good faith to a group they belong (at least Cook and Hubble Marriott do) for personal gain.
in response
Wamron says:
May 5, 2013 at 8:33 am
Someone needs to conduct a study….a real one, no con-tricks…investigating the level of correlation between CAGW belief and belief in conspiracy theories that 9/11 was an inside-Bush and Cheney Did It production. i would expect virtually every “Troofer” that draws air believes fervently in CAGW. Its this intensity of correlation on that side of the equation that would be interesting, not how many CAGW believers as a whole are “Troofers”.
It still wont be science…correlation is not evidence of anything by itself in any case.
But thats not the point.
===============
actually most of us “troofers” as you so snarkily call us, gee a denier type slam?
actually ARE informed well read and DO NOT believe in AGW
because?
its a scam like majority govvy led media money and psyops events:-)
GW is about control just as Agenda 21 millenium hype is planning using AGW warmist crud to scare the gullible into compliance
same as the falseflags are to scare the public into handing over right and freedoms.
the Gullible Nonsceptics who believe the govt led agitprop over warming and terrorism are the ones to worry about.
and FYI since Aussies handed legal guns over to our govt..we now have MORE guncrimes and killings as well as glassing stabbing etc than ever we had before
a defenceless public is a criminals delight!
Shub
I have made it very very clear to the editors of Psychological Science and Frontiers and UWA
The editor of Psycholoigical Science has not even acknowledged my existance, the editor of Frontiers treated it as some sort of academic issue. Only by phoning frontiers and spelling it out very robustly, did the Fury paper get, temporarily removed, subject to complaint.
I expectedboth editors to say, omg, we don’t want this activist (conflicted hostile researchers) rubbish in our journal, thank you for bringing it to our attention! and to withdraw it instantly.
Prof Lewandowsky also lied to me personally from my very first enquiry, trying to find the URL for the Skeptical Science survey. It did not exist, yet the paper hangs on it, and he lied to me about having had it, and had lost it!
Yet Lewandowsky, know i kniw, they know, i know (and others) that this is a total lie, yet stillnthey felt able to publish.
I had never heard of Lewandiwsky, i had just read Dr Adam Corners gushing review of it in the Guardian and wrote to the authors asking some questions. And he lied.
In my 20 yr plus career, anybody shown to have lied, once would have been out!
If the editors are happy to publish, there is nothing more i can do, but to hold them in contempt and to get on with my life
Junkpsych, you say Lew didnt have a survey because he failed to get wuwt involved.
It is worse than that. He did not even try to get Wuwt. His 5 so-called skeptic blogs included Pielke jr (not a skeptic) and Morano (not really a blog).
If the editors of psychology journals, nor those responsible for ethics in uwa don’t get it, nobody will.
paulm,
I wouldn’t be surprised to learn that Lewandowsky avoided WUWT on purpose.
“Rather it is the direct consequence of “skeptics” not wanting their views of their supporters to be known”
Conspiracist ideation!
I expected both editors to say, omg, we don’t want this activist (conflicted hostile researchers’) rubbish in our journal, thank you for bringing it to our attention! and to withdraw it instantly.
There are psychology editors who would have reacted in the way Barry Woods expects.
Unfortunately, neither the editor of Psychological Science nor the editor of Frontiers are in this category.
From what I’ve seen of the “conspiracizing” literature, previous authors of studies on “monological” conspiracy theorizing and “conspiracist ideation,” such as Ted Goertzel and Marina Abalakina-Paap and Viren Swami, were never able to get their stuff into Psychological Science.
It was Lewandowsky who broke in, with what looks like the very worst study of them all.