I and (according to Cook) 50 other blogs (with a supposed 50/50 skeptic to advocate split) have received this invitation:
Hi Anthony
As one of the more highly trafficked climate blogs on the web, I’m seeking your assistance in conducting a crowd-sourced online survey of peer-reviewed climate research. I have compiled a database of around 12,000 papers listed in the ‘Web Of Science’ between 1991 to 2011 matching the topic ‘global warming’ or ‘global climate change’. I am now inviting readers from a diverse range of climate blogs to peruse the abstracts of these climate papers with the purpose of estimating the level of consensus in the literature regarding the proposition that humans are causing global warming. If you’re interested in having your readers participate in this survey, please post the following link to the survey:
[redacted for the moment]
The survey involves rating 10 randomly selected abstracts and is expected to take 15 minutes. Participants may sign up to receive the final results of the survey (de-individuated so no individual’s data will be published). No other personal information is required (and email is optional). Participants may elect to discontinue the survey at any point and results are only recorded if the survey is completed. Participant ratings are confidential and all data will be de-individuated in the final results so no individual ratings will be published.
The analysis is being conducted by the University of Queensland in collaboration with contributing authors of the website Skeptical Science. The research project is headed by John Cook, research fellow in climate communication for the Global Change Institute at the University of Queensland.
This study adheres to the Guidelines of the ethical review process of The University of Queensland. Whilst you are free to discuss your participation in this study with project staff (contactable on +61 7 3365 3553 or j.cook3@uq.edu.au), if you would like to speak to an officer of the University not involved in the study, you may contact the Ethics Officer on +61 7 3365 3924.
If you have any questions about the survey or encounter any technical problems, you can contact me at j.cook3@uq.edu.au
Regards,
John Cook
University of Queensland/Skeptical Science
I asked Cook a series of questions about it, because given his behavior with Lewandowsky, I have serious doubts about the veracity of this survey. I asked to see the ethics approval application and approval from the University, and he declined to do so, saying that it it would compromise the survey by revealing the internal workings. I also asked why each of the 50 emails sent out had a different tracking code on it, and he also declined to explain that for the same reason. I asked to see the list of 12,000 papers, so that I could see if the database had a true representation of the peer reviewed landscape, and he also declined, but said the list would be posted “very soon”.
I had concerns about the tracking codes that were on each email sent out, and I ran some tests on it. I also tested to see if they survey could be run without tracking codes, it cannot and I asked him if he would simply provide a single code for all participants so that there can be no chance of any binning data by skeptic/non skeptic blogs or any preselection of the papers presented based on the code. I said this would truly ensure a double blind. He also declined that request.
He stated that he had an expectation (based on past experience) that no skeptic bloggers would post the survey anyway. So why send it then?
Meanwhile many other bloggers shared their concerns with me. Lucia posted a large list of questions about Cook’s survey methodology here:
http://rankexploits.com/musings/2013/dear-john-i-have-questions/
It is a good list, and Lucia’s concerns are valid.
Brandon Schollenberger writes at Lucia’s in comments about some tests he did:
========================================================
Brandon Shollenberger (Comment #112328)
May 3rd, 2013 at 12:48 am
For those following at home, the issue I wanted to talk to Lucia about is the non-randomness of this survey. I was curious when two people at SkS said they got an abstract which said (in part):
Agaves can benefit from the increases in temperature and atmospheric CO2 levels accompanying global climate change
I got the exact same abstract when I clicked on the link at SkS. I wondered if that meant there were only 10 abstracts being used at all. I then had a disturbing thought. The earlier Lewandowsky survey had different versions sent to different people for publishing. What if they had done that here? What if each site was sent a link to 10 different abstracts?
To test this, I contacted lucia to get the link she was sent. I then was able to find a site which had already posted the survey, and I got a different link from it. It turned out all of them resulted in me getting the same survey. I concluded everyone was simply getting the exact same 10 abstracts.
I was going to post a comment to that effect when lucia told me she did not get the Agave abstract I referred to. That made me take a closer look. What I found is by using proxies, I was able to get a number of different surveys. Moreover, some proxies got the same surveys as others. That suggests the randomization is not actual randomization, but instead, different samples are given based on one’s IP address.
Unfortunately, that’s not the end of the story. I’ve followed the links with my original IP address again, and I now get a different sample. However, each time I follow the link with the same IP address now, I get the same sample. That suggests I was right about IP addresses determining which sample you get, but there’s an additional factor. My first guess would be time, but if that’s the case, it’s a strange implementation of it. It would have to be something like an hourly (or even daily) randomization or some sort of caching, neither of which makes any sense to me.
Anyway, my head hurts from trying to figure out what screwy “randomization” John Cook is using. I know it’s nothing normal, and it certainly isn’t appropriate, but trying to figure out what sort of crazy thing he might have done is… difficult. I have no idea why he wouldn’t just use a standard approach like having time in seconds be a seed value for an RNG that picks 10 unique values each time someone requests a survey from the server.
=============================================================
So it appears non random after all and has what I (and others) consider fatal sampling issues.
If you want to look at the survey, you can go to Cook’s website and take it there, because until there are some answers forthcoming, like Lucia, I won’t be posting the coded link for this blog.
See Cook’s survey link: Participate in a survey measuring consensus in climate research
Of course it’s a set up! It’s Cook, how would anyone think it is not a set up?
But I did the test anyway. I want to see where this leads to. Maybe I can be a part of a peer reviewed study about me! :-O
They even asked an email address which I gave. I really, really want to see how they use it. Maybe they even mention me on the paper they write about this study about skeptics?
This is so interesting. It’s like climate skepticism’s history in the making. 😀
Subjective GIGO!
I’m with Admiral Akbar on this one!
John Cook has shown a persistent lack of basic integrity in his intellectual efforts. I decline to participate in his survey.
John
Of course this is merely a political ploy; if you are gracious enough to actually allow the ( slanted) poll, then they’ll refer to the answers- If you fail to allow the poll, you’ll be condemned for not allowing it.
Steven Mosher says:
May 3, 2013 at 11:42 am
Rasey
“To top it off, to believe Cook, you are to link to the survey, read, understand, and question 10 abstracts, give each one some careful consideration to fit into a 7 point rating system, all in about 15 minutes, or approximately one unbiased, carefully considered abstract evaluation every 70 seconds. Maybe a Google webbot might work at that speed. But thousands of human degreed blog visitors? No. Such a pace can be met only if you know the desired answer ahead of time.”
you can take as long as you want. Tested. They estimate about 15 minutes.
The abstracts I read took at best 10 seconds to read and comprehend each.
——————————————————————————-
Mosher, you really need to stop showing off. OK, you are a genius, but very few of the rest of us rabble could make sense of an abstract in 10 seconds.
Perhaps that’s what’s wrong with the current understanding of climate science. Not enough people who can accurately evaluate scientific abstracts in 10 seconds.
I suppose that the slowcoach Willis E.might take 15 seconds, or even longer?
Cynical response:
There’s no need to respond as Cook is quite willing to fabricate data if required, using the Gleick defence of unproven provenance – also if required.
More cynical response:
The creation of the survey is designed to create internet chatter (like this) as “proof” the sceptics are paranoid.
Even more cynical response:
Even discussing this survey is proof that people take “consensus” seriously thus anyone who later says “science is not consensus” can now be labelled a hypocrite or a bad loser. This is just a sophist’s feint.
But may be I am too jaded. Perhaps SkS have learnt humility and so are now genuinely concerned about learning truths they just realised they don’t know.
Perhaps.
According to Mosher: 12:05 pm
From a google search, it appears that what Cook give Mosher as an abstract is really only the first line from an abstract — an abstract fragment. Here is the full abstract from Shaver and Canadell et al, Bioscience, Vol 50, No. 10, Oct 2000, 871-882. (pdf).
Cook doesn’t supply “Abstracts” — only fragments of abstracts. Note, I am taking Mosher at his word. That is, by “real example” Mosher gave us the entire abstract as supplied by Cook.
“As one of the more highly trafficked climate blogs on the web, I’m seeking your assistance . . . ”
Look folks, the prat couldn’t even write a correct sentence inEnglish . . .
End of story.
Just take the survey off of Cook’s own page. It throws a wrench into his goal.
No thanks.
Here are my comments on a survey by James Powell, posted online in various sites last year, of 13,950 papers dealing with climate change. It analyzed their abstracts and “found” that only 24 rejected manmade global warming. I posted the following critical comments on the site below (not the main place it was posted). I suspect it was this survey that inspired what Cook is up to now (5/2013):
http://oilprice.com/The-Environment/Global-Warming/Contrary-to-Popular-Belief-Scientists-are-United-on-Climate-Change.html
The article states:
How many papers that “explicitly state that the theory of global warming is false” would get by peer review with that phrase intact? How many would even be submitted to peer review if they included that phrase? They therefore tend to be more circumspect and merely cite a discrepancy, some flaw (minor perhaps only in the author of this article’s opinion), etc.
Here’s a link to 1100+ peer-reviewed papers supporting skeptical arguments critical of ACC/AGW alarmism:
http://www.populartechnology.net/2009/10/peer-reviewed-papers-supporting.html
========
Strawman. The claim is not that skeptics are 100% “prevented” from being published, but that that it is difficult (and hence rare) to get them published, or to get them published without being watered down, as I hinted above.
==========
IOW, an article will be classified as skeptical only if it presents hard evidence. BUT an article will be counted accepting/endorsing even if it presents no hard evidence, but merely implicit opinion:
Denial must be explicit, but acceptance may be implicit. This double standard biases the results of this article. By how much is unknown. For that, the author should have indicated how many fall into the “implicitly accepting” category.
==========
But the weakness of the warmist case isn’t in the “hard evidence” so much as in the inferences drawn from that evidence, the selectivity applied in deciding which evidence is the most relevant, the inferences drawn from those relevant bits of evidence, the assumptions made, etc. It is at those matters where the main thrust of skepticism has been directed.
But journals want to publish “findings.” This biases them against publishing wide-ranging, argumentative critiques. (To be fair, they rarely publish similar argumentative essays from the warmist side either.) They have a just-the-facts attitude. But the facts don’t speak for themselves. Argumentation has therefore moved to other venues.
What’s needed is an online venue where viewpoints can be argued among credentialed scientists, with the peanut gallery roped off into a separate section where their comments won’t disrupt the discussion, but can be drawn upon by the participants if desired. (Seen but not heard, IOW.) This is what has finally gotten underway with the establishment this month of the Climate Dialogue site, at http://www.climatedialogue.org/
==========
So what? (Irrelevant thesis.) Skeptics don’t deny that. What they deny is that this warming will continue at its current pace; that it would be very harmful if it did so—or even harmful on balance at all; and that there are amplifying factors that will accelerate the current trend. The alarmists’ case rests on the assumptions of strong positive feedbacks and the absence or weakness of negative feedbacks. That’s where their case is weakest.
Hmm . . . There’s nothing in that list by the following skeptical scientists, at least half of whom have presumably published papers properly classified as skeptical:
Syun-Ichi Akasofu, Claude Allègre, John Christy, David Douglass, Don Easterbrook, William M. Gray, Richard Lindzen, Nils-Axel Mörner, Fred Singer, and Roy Spencer.
I took their names from Wikipedia’s “List of [35] scientists opposing the mainstream scientific assessment of global warming” at http://en.wikipedia.org/wiki/List_of_scientists_opposing_the_mainstream_scientific_assessment_of_global_warming
Here are four other names, half of whom I presume wrote articles that were missed: Zbigniew Jaworowski, Augusto Mangini, Nathan Paldor, and Richard Tol
Steven Mosher says:
May 3, 2013 at 12:05 pm
In short, you cant read. You see confirmation of your position where there is none, and you see implicit meanings where no body else does.
———-
I see this survey for what it is — a smear campaign against people who have the audacity to question whether CO2 has any material impact on our climate and our future. That is my position. Judging from the responses in this thread many feel the same. The fact that CO2 has increased dramatically over the last 15 years with no increase warming seems to support this position.
You don’t need a survey to figure that out. In fact, you accuse me of seeing implicit meanings, this is exactly what this survey is trying to do. It’s trying to demonize the heretics that would dare to peak behind the Wizard’s curtain.
If Cook (or anyone else) wants to know what people (deniers) on this blog think and why, he is welcome to come here and participate in an honest and open debate.
Unfortunately, as I and others have experienced this is not the case on his site. That alone is telling.
——————————————-
“Finally, Cook may not be doing this. but If I had this data I would do exactly this.”
Because it’s all about the cause above all else, isn’t it?
Steven Mosher says:
May 3, 2013 at 12:13 pm
blah…blah…blah…blah
=============================================================
Steven, now if you could only ‘splain away 15+ years of material CO2 growth…and no warming.
Would you buy a second hand car from Cook?
Steve Mosher captured most of my thoughts. I think it is a study on confirmation bias.
The issue I see with the survey is we are asked to rate the abstract text. “Your rating should be based on the abstract text.”. Our responses are immediately compared to the authors’ rating of their papers. I rated my abstracts at 3.4 whereas the authors rated their corresponding papers at 2.6.
As SM says – you have to watch the pea. I took my time (much more than 70 seconds per abstract) and carefully parsed the text of each verses the criteria. Now had I rated my abstracts on what I suspected the paper said, I would have gotten closer to the 2.6. If they do a good job on processing the submissions, it will be an interesting look at how readers from skeptic verses alarmist sites analyze the abstracts.
If they try to compare our analysis to the authors’ analysis, then the study is broken because we are comparing abstracts to the full article and what is in the author’s head.
Let’s assume for the sake of argument that our side IS nuts. Cook’s survey will miss it if we’re nuts in contradictory ways, as follows: Half of us rate an abstract as endorsing AGW because it used certain trigger words that we associate with alarmism, and half of us rate that abstract as endorsing skepticism because there was some skeptical assertion in it somewhere.
That sort of split (although not necessarily on a 50/50 basis) is very likely to occur.
would, could, might, may, sorta, maybe, ……….ice free = 1 million sq km (the size of Egypt)
I could publish a million papers if allowed to use those words…..
…of course, they would never let a skeptic paper get away with that
It’s a cook book!
Just looking at the experimental design, my guess would be they intend to test the hypothesis that people on different sides of the debate assess evidence differently. The idea would be to show examples of abstracts that they interpret as AGW-supporting and that sceptics have classified as non-supportive. By presenting it as a survey to assess the consensus, it tempts sceptics into shading their judgements in that direction to try to bias the result towards reporting a lower degree of consensus.
It’s an interesting question, and if you got someone a little more neutral to conduct it, (e.g. someone like Dan Kahan, who researches this stuff) it would be a worthwhile enquiry.
NIV,
This is my best guess as to the point of the study—if it has any at all.
Although when I completed the survey, I was told that the authors rated their own studies as somewhat less supportive of CAGW than I had. And I’m a skeptic.
I started on the survey twice, from different IP addresses, just to see what the samples of articles looked like. Filled it out once. The two lists were completely different, but some items on the first list that I saw have already been mentioned upthread. Not a random selection from Cook’s database of articles…
Two big questions:
(1) What articles are in Cook’s database, and how were they selected? I hope Lucia will enlighten us…
(2) What’s the point of asking people to evaluate the leaning of an article from the abstract? Abstracts often miss the point, are uninformative, sometimes are seriously misleading. I had the feeling that in scoring the abstracts I was relying more on the rhetoric employed than the points stated.
Having already made a burning sacrifice of his credibility, Cook should not be trusted here. Either on the technical side, or on the ethical side.
PS. There is noticeable game-playing in some of Kahan’s writeups, but his collection and analysis methods—unlike those of most contributors to the “conspiracist” literature—do seem to meet professional standards.
To get people to spend their time on something, you need to explain the benefit to them. I have no idea how participating in this exercise would provide anything useful for climate science or policy or for me personally. I would be interested in why Mr. Cook thinks people would or should spend time on this exercise. Just because he asked nicely?
I usually don’t judge people fast, I try to understand and judge them by their acting.
About Cook and Lewandowsky I learned some. It is not without reason that there is a separate category on the blog roll named unreliable.
I read many posts about Lew’s papers on many different blogs that I trust. Trust is something that has to be earned. Cook and Lewandowsky need to learn that.
http://wattsupwiththat.com/2013/03/20/some-thoughts-on-the-recent-lewandowsky-cook-conspiracy-theory/
“Lewandowsky is a charlatan. His latest paper, co-authored by John Cook, is a flight of fantasy that ignores the fact that most of the comments that he labels ‘recursive fury’ were polite mentions of the fact that he Cooked the books in his survey–a survey he claims is published, but is not.
I play a minor role in this. As someone who has worked in the field of online research pretty much from the day online research started, I have participated in literally thousands of online surveys. I commented on Lewandowsky’s weblog posts concerning his survey, pointing out some of the (many) issues with what he had done and asking for a look at the questionnaire.
Lewandowsky deleted all of my comments. And his latest paper, which has a Data Supplement showing the ‘recursive fury’, which apparently means cherry picking a few of the comments he didn’t like, doesn’t mention my deleted comments for some reason.
In addition to biasing the sample, Lewandowsky presented different versions of the survey to respondents coming from different websites. His ‘conspiracists’ from the skeptic world were outnumbered by ‘conspiracists’ from the climate activist community. He has not published the data, despite promising to do so and claiming that he has”
http://wattsupwiththat.com/2013/03/09/stephan-lewandowsky-and-john-cook-making-things-up/
http://wattsupwiththat.com/2013/02/06/lewandowskys-latest-smear-paper-gets-pulled-from-the-journal-website/
…”He, and his coauthors, falsely used my name in order to support some kind of psychology paper on climate skeptic bloggers titled”…
http://wattsupwiththat.com/2013/02/05/lewandowskys-bear-baiting-behavior/
and many, many, many other…
Personally I have no interest whatsoever to invest time and energy in anything that is linked with the 2.
In my view the only way how such cooperation might be accepted by skeptics blogs is if the methodology is clearly explained, agreed and verifiable and if the raw data is made available. I mean only if the complete control is ensured – the best is if the whole survey would be run on a skeptical site and raw data could be made available to all for study, but to contribute to the study of these 2 clowns, to enable them to say they had so many skeptics and to know they will falsify the results to what they desire? No way. Not after what they did, how they “analysed” the data, fabricated their answers to fit their pet theories.
My advice – if anybody asks for it – they had their chance. Look what they did with it.
Under NO circumstances should any realist become involved. A paper bag burning on the front porch? YES, it is! “Please STAY AWAY.”
Let’s understand this…
The frauds get paid a lot of money to ignore the truth about CAGW, publish fabrication after fabrication and make up fake answers and theories about people who doubt the CAGW religion and believe their sorry butts should spend the next hundred years breaking rock for the government.
Now they’re supposedly making goo goo eyes at us and asking us skeptics, after they asked their buds to fill out another flim flam scam.
Don’t dirty your hands! Keep the world sKs free!
Don’t do it.