I and (according to Cook) 50 other blogs (with a supposed 50/50 skeptic to advocate split) have received this invitation:
Hi Anthony
As one of the more highly trafficked climate blogs on the web, I’m seeking your assistance in conducting a crowd-sourced online survey of peer-reviewed climate research. I have compiled a database of around 12,000 papers listed in the ‘Web Of Science’ between 1991 to 2011 matching the topic ‘global warming’ or ‘global climate change’. I am now inviting readers from a diverse range of climate blogs to peruse the abstracts of these climate papers with the purpose of estimating the level of consensus in the literature regarding the proposition that humans are causing global warming. If you’re interested in having your readers participate in this survey, please post the following link to the survey:
[redacted for the moment]
The survey involves rating 10 randomly selected abstracts and is expected to take 15 minutes. Participants may sign up to receive the final results of the survey (de-individuated so no individual’s data will be published). No other personal information is required (and email is optional). Participants may elect to discontinue the survey at any point and results are only recorded if the survey is completed. Participant ratings are confidential and all data will be de-individuated in the final results so no individual ratings will be published.
The analysis is being conducted by the University of Queensland in collaboration with contributing authors of the website Skeptical Science. The research project is headed by John Cook, research fellow in climate communication for the Global Change Institute at the University of Queensland.
This study adheres to the Guidelines of the ethical review process of The University of Queensland. Whilst you are free to discuss your participation in this study with project staff (contactable on +61 7 3365 3553 or j.cook3@uq.edu.au), if you would like to speak to an officer of the University not involved in the study, you may contact the Ethics Officer on +61 7 3365 3924.
If you have any questions about the survey or encounter any technical problems, you can contact me at j.cook3@uq.edu.au
Regards,
John Cook
University of Queensland/Skeptical Science
I asked Cook a series of questions about it, because given his behavior with Lewandowsky, I have serious doubts about the veracity of this survey. I asked to see the ethics approval application and approval from the University, and he declined to do so, saying that it it would compromise the survey by revealing the internal workings. I also asked why each of the 50 emails sent out had a different tracking code on it, and he also declined to explain that for the same reason. I asked to see the list of 12,000 papers, so that I could see if the database had a true representation of the peer reviewed landscape, and he also declined, but said the list would be posted “very soon”.
I had concerns about the tracking codes that were on each email sent out, and I ran some tests on it. I also tested to see if they survey could be run without tracking codes, it cannot and I asked him if he would simply provide a single code for all participants so that there can be no chance of any binning data by skeptic/non skeptic blogs or any preselection of the papers presented based on the code. I said this would truly ensure a double blind. He also declined that request.
He stated that he had an expectation (based on past experience) that no skeptic bloggers would post the survey anyway. So why send it then?
Meanwhile many other bloggers shared their concerns with me. Lucia posted a large list of questions about Cook’s survey methodology here:
http://rankexploits.com/musings/2013/dear-john-i-have-questions/
It is a good list, and Lucia’s concerns are valid.
Brandon Schollenberger writes at Lucia’s in comments about some tests he did:
========================================================
Brandon Shollenberger (Comment #112328)
May 3rd, 2013 at 12:48 am
For those following at home, the issue I wanted to talk to Lucia about is the non-randomness of this survey. I was curious when two people at SkS said they got an abstract which said (in part):
Agaves can benefit from the increases in temperature and atmospheric CO2 levels accompanying global climate change
I got the exact same abstract when I clicked on the link at SkS. I wondered if that meant there were only 10 abstracts being used at all. I then had a disturbing thought. The earlier Lewandowsky survey had different versions sent to different people for publishing. What if they had done that here? What if each site was sent a link to 10 different abstracts?
To test this, I contacted lucia to get the link she was sent. I then was able to find a site which had already posted the survey, and I got a different link from it. It turned out all of them resulted in me getting the same survey. I concluded everyone was simply getting the exact same 10 abstracts.
I was going to post a comment to that effect when lucia told me she did not get the Agave abstract I referred to. That made me take a closer look. What I found is by using proxies, I was able to get a number of different surveys. Moreover, some proxies got the same surveys as others. That suggests the randomization is not actual randomization, but instead, different samples are given based on one’s IP address.
Unfortunately, that’s not the end of the story. I’ve followed the links with my original IP address again, and I now get a different sample. However, each time I follow the link with the same IP address now, I get the same sample. That suggests I was right about IP addresses determining which sample you get, but there’s an additional factor. My first guess would be time, but if that’s the case, it’s a strange implementation of it. It would have to be something like an hourly (or even daily) randomization or some sort of caching, neither of which makes any sense to me.
Anyway, my head hurts from trying to figure out what screwy “randomization” John Cook is using. I know it’s nothing normal, and it certainly isn’t appropriate, but trying to figure out what sort of crazy thing he might have done is… difficult. I have no idea why he wouldn’t just use a standard approach like having time in seconds be a seed value for an RNG that picks 10 unique values each time someone requests a survey from the server.
=============================================================
So it appears non random after all and has what I (and others) consider fatal sampling issues.
If you want to look at the survey, you can go to Cook’s website and take it there, because until there are some answers forthcoming, like Lucia, I won’t be posting the coded link for this blog.
See Cook’s survey link: Participate in a survey measuring consensus in climate research
In 2006 I demolished some economic papers from the University of Queensland, including the head of department, which were used to justify an economically-absurd “state economic development strategy.” I sent a draft to one of the authors, who had moved on to LSE. He fully accepted my critique which demonstrated that there was no basis for certain policies adopted by the state government, and said that “I now laugh at my time in Queensland.” I don’t laugh, I weep at the standards of the Uni just over the river from me.
I am a bit late to this and have just read the comments.
The main thrust of the comments seems to be following Brandon Shollenberger’s issue with the randomization of the sample. But there are broader issues here that can completely undermine the results of the research. This is particularly true if the researcher is trying to infer quality and truth of science is equated with numerical support.
First, is that if funding of research within climatology has been highly skewed towards supporting the AGW hypothesis, and the journals have a strong selection bias, then the sample will be meaningless.
Second, is if not all scientific papers are equal. As any undergraduate course will show, relative importance of papers is extremely highly skewed. If the first point is true, then the average non-consensus paper needs carry more weight than the average consensus paper.
Third, that relative importance changes over time. I hope that no university would say that you should treat the papers of twenty years ago on an equal basis to the latest research.
Fourth, the consensus can shift. For instance, compare the size of the medieval warm period in the original Mann hockey sticks with Esper et al 2012, (or the withdrawn Gergis paper), and you will see a distinct movement.
Most importantly, any significance of the research is totally undermined if science does not progress by consensus, but by new insights that sometimes totally undermine current views.
Second, is if not all scientific papers are equal. As any undergraduate course will show, relative importance of papers is extremely highly skewed. If the first point is true, then the average non-consensus paper needs carry more weight than the average consensus paper.
Third, that relative importance changes over time. I hope that no university would say that you should treat the papers of twenty years ago on an equal basis to the latest research.
Fourth, the consensus can shift. For instance, compare the size of the medieval warm period in the original Mann hockey sticks with Esper et al 2012, (or the withdrawn Gergis paper), and you will see a distinct movement.
Most importantly, any significance of the research is totally undermined if science does not progress by consensus, but by new insights that sometimes totally undermine current views.
Steven Mosher says:
May 3, 2013 at 9:56 am
Come on guys have some fun.
take the survey at skeptical science as if you read it.
Whatever the abstract says answer that the abstract endorses AGW in some way.
hehe. confirmation bias
—————————————————————————-
I disagree. Why lie? Do you lie often?
If you take the survey at least work to see if his results are reasonable.
Who knows? Maybe skeptics are lying, conniving, conspirators, getting rich off government and oil money hell bent on control of mankind and the earth.
otoh….
I believe the statistics that Cook will create will skew skeptic’s beliefs.
The survey will be used as propaganda. They still use the 97% as real numbers.
Stupid answers will demonstrate skeptics are nutty paranoids and the news of the unbelievable wacko denier’s answers will be broadcast worldwide. Even if they can only say “2% of the total respondents are as crazy as Steven Mosher.”
Cook has shown no reason to make me believe otherwise and legitimizing his bias is unsat!
If he wants to make things up I can’t stop him.
Sending an invitation (to Anthony et al) doesn’t mean skeptic participation.
If his survey is only on his site few skeptics will see it without a direct link.
I don’t like the way the Team does business.
I have chosen to notsupport John Cook in this effort.
Thanks for the invitation but I’ll pass.
cn
Regarding this “survey”: Fools rush in where wise men fear to tread…..
I just emailed young Cook with the following:
John
I read with interest your plan for a survey of the published climate literature abstracts with a view to finding a consensus on the level of AGW.
Unfortunately, your project is pursuing a false premise – that the abstracts are a true reflection of the actual research data in the studies.
One of the things I quickly became aware of while researching the book Air Con was how nearly every study carried an obligatory tip of the hat to AGW, yet often the data did not necessarily support that conclusion. I realised that a number of good researchers were banging the required phraseology into their papers to keep their masters and grants controllers happy, but they were also letting the data speak for itself in the study proper.
A survey of abstracts will be meaningless.
Regards
Ian Wishart
Next week we’ll be asking the infidels to rate suras to see why they don’t accept thetruth of Islam.
Why does one need participants to reach a foregone conclusion? Can’t Mr. Cook just cut to the chase?
For a psychology class, I had to write a proposal for a study — not necessarily a study that would actually be conducted — but a proposal that could be used for a real study. This nonsense would have been thrown out immediately.
My how times have changed.
dave38 says: “Timeo danaos dona ferentes!”
How does one say Geek in Latin?
A colossal waste of time, which I think is the intent.
Focus on disproving our own assumptions, as that is true science. Hang all else.
Justin Templer (@justintempler) says:
May 4, 2013 at 12:52 am
Cook’s “scientific” survey looks to be nothing more than a propaganda piece as a way to label skeptics as deniers. They have already released a paper
“http://www.earth-syst-dynam-discuss.net/4/451/2013/esdd-4-451-2013.pdf”
“Cook et al. (2013) reviewed nearly 12000 climate abstracts and received 1200 self-ratings from the authors of climate science publications.Using both methodologies, they found a 97% consensus in the peer-reviewed climate science literature that humans are causing global warming. There appears to be a gap in the understanding of the climate between experts and the lay public, and a common denominator between all the examples reported here and in the supporting material is that they all represent a contribution towards the agnotology associated with the climate change issue.”
I expect Cook to draw a comparison between the survey from scientists with the survey contrasting AGW believers with skeptics to be able to apply the denier label to skeptics.
I think you pretty summed it up Justin. The conclusion is already there, they need now the “data”.
I wonder if the set-up was to “Prove” that the skeptic blogs when invited WILL NOT PARTICIPATE. Remember that this was one of Lewandowsky’s defenses of his
Bafflegabpaper. Therefore Cook is setting up an ‘Experiment” that ‘proves’ when skepitcs are invited they won’t play.I think that and not the survey itself is the purpose.
Any response we give is going to be a lose-lose since the purpose is to ‘Prove’ skepitcs are crazy and therefore can be sidelined.
On a slightly different note:
Someone needs to alert Lewandowsky that a leftist magazine, Rolling Stone is conceding Conspiracy Theorists were RIGHT!
A Matt Taibbi article in Rolling Stone: Everything Is Rigged: The Biggest Price-Fixing Scandal Ever: The Illuminati were amateurs. The second huge financial scandal of the year reveals the real international conspiracy: There’s no price the big banks can’t fix – the markets are completely rigged by the big boys. SURPRISE!
Now where is _Jim to try and refute this….
WUWT could start its own survey and set the rules. Let it be a shining example of how to do this properly. Anyone interested in testing the putative consensus with a truly objective assessment?
REPLY: I’m not a ward of the state with unlimited time and grants to do this work as Mr. Cook is. While I could certainly take on the task here, I have a business to run which is already neglected due to spending too much time here. Plus, AGW advocates, people just as yourself, would simply pooh pooh the results.
That said, why don’t you step up? I’ll be happy to host such a survey setup if somebody else such as yourself does the design and that design isn’t flawed like Cooks. Of course for it to be credible, you’ll have to stop hiding behind a pseudonym and disposable email account. – Anthony
Gail Combs says:
May 4, 2013 at 9:37 am
I wonder if the set-up was to “Prove” that the skeptic blogs when invited WILL NOT PARTICIPATE. Remember that this was one of Lewandowsky’s defenses of his Bafflegab paper. Therefore Cook is setting up an ‘Experiment” that ‘proves’ when skepitcs are invited they won’t play.
I think that and not the survey itself is the purpose.
Any response we give is going to be a lose-lose since the purpose is to ‘Prove’ skepitcs are crazy and therefore can be sidelined.
Gail you are right, however they are proven to manufacture data. They are proven to be unreliable.
With this proven facts, I see no reason whatsoever to cooperate with these repetitive science distorters, in the idea to try to show cooperation, in the hope that sometime they will learn the scientific method.
They will not apply the scientific method as it does not lead to their pre-conceived “facts”.
If the data does not fit they will make it fit, as we have seen again and again (I must think again at Josh’s cartoon – make data fit anyhow :))
http://bishophill.squarespace.com/blog/2011/7/7/ipcc-science-fit-for-purpose-josh-111.html
But as many have posted we may put some of our own surveys. However I have the impression that many skeptics are working, have also families and manage barely to save some free time to read and post on blogs. In contrast the CAGW blogs are run by people payed heavy amounts of money to do that (and have the pressure to show results).
As posted above by Justin they have the results already.
http://bishophill.squarespace.com/blog/2012/9/8/conspiracy-of-one-josh-183.html
Peter Miller (May 3, 2013 at 11:55 am) “I think this is yet another attempt by the alarmist community to muddy the waters between AGW and CAGW.”
I know this is what they are doing because that is all they ever do at SkepticalScience. How much scientific support is there behind further albedo based feedback from CO2 warming? Certainly not 97% probably more like 10% How much scientific support for giant AlGoresque hurricanes? Basically science fantasy. How about scientific support for the nonexistence of naturally caused megadroughts? What about blocking patterns that can’t be natural, must be caused by “low sea ice”? Neither of those are based on consensus science, but mostly on ignorance of historical weather and current weather processes.
They will simply use the 97% scientific support for CO2 absorption of LW radiation as a proxy for all of those things and many more. That is all they ever do.
Waaay back up the postings, Willis had it right. Do not have one iota of trust in this fake survey.
There can only be the truth, the whole truth and nothing but the truth. It matters not how many people ‘believe’ otherwise. The numbers of otherwise ‘believers’ does not, somehow, change falsehood into truth.
Consensus? Here’s quote from the late Baroness Thatcher about consensus: “Consensus. The process of abandoning all beliefs, principles, values, and policies in search of something in which no one believes, but to which no one objects; the process of avoiding the very issues that have to be solved, merely because you cannot get agreement on the way ahead. What great cause would have been fought and won under the banner: ‘I stand for consensus?”
Dear Jonathan Frodsham,
THANK YOU! #{:)]
You brightened my day.
i haven’t read all the comments and it is likely this is repetitive, but why does anyone care whether there is a consensus on any of this? As long as real science is a possibility, a consensus lacks convincing power.
If I didn’t have 3 jobs seven days a week at the moment, I might undertake to learn how to format such a survey. I would have time to participate in the survey over a few days, though, which would be a much less time-consuming task.
The email address I post under is my primary email address. I’ve had it since 1998 and never use another except for internal work emails.
Out of curiosity, is it a new rule that article authors must post under their real name? There have been numerous posts here that do not. I have considered asking to post here, when time permits, hopefully under conditions equal to those authors (justthefactswuwt, charles the moderator, for example).
If anyone undertook to do a literature survey under the auspices of WUWT, I’d recommend they keep participants anonymous to give primacy to the information rather than to personality.
There’s an interesting parallel here with how people perceive terrorists. (Nailing my colours to the mast, I’m firmly in the sceptic camp.) A paper I read years ago (and can’t put my finger on – dammit) argued that terrorism is generally likely to fail. The reason was that people simply didn’t trust the stated aims of the terrorists. Apparently, most Russians supported complete or partial autonomy for Chechnya, before the Moscow apartment blocks bombings. Most/many Israelis supported Palestinian independence before the second Intifada. Osama Bin Laden had very specific demands on the US before 9/11 (US out of the Middle East, no support for illegitimate – i.e. not sufficiently Muslim – Middle East governments, etc). However, after the events, no-one believed the stated aims of the perpetrators. It was like, “This is what they say they want, but what do they REALLY want?” Cook MAY (!?!??) actually want to do a genuine crowd-sourced survey, but who is going to believe him after the last debacle with Lewandosky? Fascinating.
Use that to your advantage since a number
of players won’t be as alert as you. Although many prefer pit games like Black Jack and Roulette, I have always preferred opportunities that offer a reasonable rate of return. This is due to the reason, that letting you win will make you come back for more than just cheating you for once and losing the business.
Since a very large majority of the papers on global warming support it being AGW, what Cook is trying to show is that skeptics blantantly lie or deny what the vast majority of authors (climate scientists) say about global warming.
Lets use Mann’s Hockey stick paper as an example… I assume it maintains in the abstract that the warming in the mid to late 20 century is unprecedented in the last 1000+ years. Now if you get that paper as an abstract… do you
a) Agree with the author that indeed the data does show that?
b) Disagree with the author and say no it doesn’t… knowing full well that the data presented has been tortured and processed in such a way that the conclusion reached in the paper is bs.
If you select A, then Cook can claim “even skeptics, after reading scientific papers, come to the same conclusions as the vast majority of scientists that global warming is happening and it is primarily the result of human activities… primarily CO2.”
If you select B (more likely the case with true skeptics who have studied up on the Mann Hockey stick) then Cook can claim “Skeptics have a propensity to lie or deny what an author says in a scientific paper. The published paper clearly says one thing, as even the authors (insert world reknown scientists name here for appeal from authority) attest, but skeptics say the paper says just the opposite. Obviously skeptics are completely bonkers and nothing but irrational fruitcakes.”
WUWT and Climate Audit are very good at taking major landmark papers published by the “Team” and ripping them to shreds… exposing them to be quite flawed, if not fraudulent. Most of the readers of WUWT are much more knowledgeable about what is really going on in climate science so are much less inclined to agree with the authors of global warming papers about exactly what the data supports.
Contrary to what Mosher maintains, this survey is a total setup. It is more akin to answering the guestion “Have you stopped beating your wife yet?”
Best to just totally ignore Cook in my opinion.
Alcheson has it dead right. Cook wants to show that skeptics won’t believe the evidence when it’s put under their noses. That’s why the links sent to the various blogs are uniquely identified. It’s advocacy pure and simple.
@barry says:
May 4, 2013 at 4:50 pm
That said, why don’t you step up? I’ll be happy to host such a survey setup if somebody else such as yourself does the design and that design isn’t flawed like Cooks. Of course for it to be credible, you’ll have to stop hiding behind a pseudonym and disposable email account. – Anthony
“If I didn’t have 3 jobs seven days a week at the moment, I might undertake to learn how to format such a survey. I would have time to participate in the survey over a few days, though, which would be a much less time-consuming task. […]”
barry, barry, barry… just do like the rest of us and declare your CAGW skepticism loud and long. Pretty soon you’ll be rolling in checks from Big Oil. Heck, all you have to do is – just once – whisper, “Maybe it isn’t CO2 after all,” and that’s generally good for $50k from Shell ;o)
It’s a conspiracy.