Brandon Shollenberger writes: I’ve been mulling over an idea I had, and I wanted to get some public feedback. What would you think of a public re-analysis of the Cook et al data set?
A key criticism of the Cook et al paper is they didn’t define the “consensus” they were looking for. There’s a lot of confusion as to whether that “consensus” position is weak (e.g. the greenhouse effect is real) or strong (e.g. humans are the primary culprits). The reason for that is Cook et al tried to combine both definitions into one rating, meaning they had no real definition.
You can see a discussion of that here.
I think it’d be interesting to examine the same data with sensible definitions. Instead of saying there’s a “97% consensus,” we could say “X% believe in global warming, Y% say humans are responsible for Z% of it.” That’d be far more informative. It’d also let us see if rating abstracts is even a plausibly useful approach for measuring a consensus.
My current thinking is to create a web site where people will be able to create accounts, log in and rate a particular subsample of the Cook et al data. I’m thinking 100 “Endorse AGW” abstracts to start with should be enough. After enough ratings have been submitted (or enough time has passed), I’ll break off the ratings, post results and start ratings on another set of abstracts.
The results would allow us to see tallies of how each abstract was rated (contrasted with the Cook et al ratings). I’m thinking I’d also allow raters to leave comments on abstracts to explain themselves, and these would be displayed as well. Finally, individual raters’ ratings could be viewed on a page to look for systematic differences in views.
What do you guys think? Would you be interested in something like this? Do you have things you’d like added or removed from it? Most importantly, do you think it’d be worth the effort? I’d be happy to create it, but it would take a fair amount of time and effort. It’d also take some money for hosting costs. I’d like to have an idea of if it’d be worth it.
An added bonus to doing it would be I could move my blog to that site as well. Self-hosting WordPress takes more effort than using WordPress.com, but it allows for far more customization. I’d love that.
So, thoughts? Questions? Concerns?
By the way, don’t hesitate to tell me I’m a fool if you think I’m spending too much time on the Cook et al issue. I’ve been telling myself that for the last two weeks.
Source: http://hiizuru.wordpress.com/2014/05/20/a-re-analysis-of-the-consensus/
===============================================================
My opinion is that given the vast number of people interested in this at WUWT, we could likely crowd-source this work much more accurately and quickly than Cook did, without having to fall back on a small cadre of “like minded friends”. Both sides of the aisle can participate.
I don’t know what the result will be of such an analysis proposed by Brandon, but I do know that we can get far more participants from a much broader venue (since WUWT has almost an order of magnitude more reach than “Skeptical Science”) and that Brandon’s attention to detail will be an asset.
We already know many of the mistakes made in Cook’s work, so a re-do has the advantage out of the gate. This disadvantage may be that the gatekeepers at IOP may refuse to publish it, and University of Queensland may publish yet another bogus legal threat, since they seem tickled that Cooks 97% is the subject of worldwide gossip – Anthony
For folks who claim there is no consensus I would say this:
A) what evidence do you have
B) why resist an attempt to properly measure the consensus present in the literature.
For folks who claim that consensus doesnt matter, well that’s a separate question.
Doing a proper content analysis of the published science is good.
There are many ways to assess the state of the science and level of agreement.
1. Do a review of the science. The IPCC approach.
2. Interview scientists ( see the Vision prize)
3. Content Analysis.
All three are interesting and necessary.
For #3 All the studies Ive seen are flawed by some easy to remedy problems.
The only reasons not to do it.
A) you think its a waste of time.. well DONT HELP THEN
B) you are afraid of the answer.
“What would you think of a public re-analysis of the Cook et al data set?”
I’d would think it had little more value than the Cook et al analysis, except maybe as a PR exercise (and only if it showed that the conclusion of the original was contrary to the evidence).
97% of witch doctors agree that toads cause warts – so what?
Just my 2p (or 2c depending on your currency)
gareth
” It’d also let us see if rating abstracts is even a plausibly useful approach for measuring a consensus.”
What are you aim for? A socio-political analysis of what gets published?
Consensus has nothing to do with the factual science and all to do with peer group pressure, orthodoxy and manipulation.
We already know the whole field is corrupted by those problems, we don’t to count the beans.
By all means prove they’re study was a fraud if you can but don’ t legitimise the mythi of consensus by trying to do it “properly”.
@ur momisugly tteclod says: May 21, 2014 at 9:38 am
“Go to the source. Ask the authors of each paper for an opinion regarding climate science.”
Better, but risky. As we observe, “abstracts” are frequently at variance to actual content, eg the ACC/AGW “sell” is in the abstract but less apparent or entirely absent from the paper. IMO abstracts are sometimes as bad as press releases.
It’s a terrible idea on several levels. By carrying out your own ratings you legitimise the method, which is a nonsense. You also invite derisory coverage, because of course ‘den!ers’ will disagree with Cook et als findings, you will be no more neutral than them. You will also provide the excuse for more supportive coverage of the original paper. It had already been torn to shreds, don’t reanimate it.
Bad idea (in ‘as they came to mind’ order:
– many times we have seen reported that papers do not get published unless they include proof that ‘global warming’, ‘climate change’ or numerous variations of apocalyptic disaster are included. Thus, the papers being reviewed may not provide a balanced view.
– abstracts can be at odds with the finer detail and nuances within the paper
– opinions (of reviewers) are like fundamental orifices; we all have one but some smell worse than others. You could possibly have many arguments and rationalisations over 1 paper as to it’s authors’ beliefs, never mind 100’s or 1,000’s.
– it is too open to trolls
– it would be better, as others have suggested, to survey the authors rather than guess at their motivations and beliefs
– counting the number of papers produced over a period that DO NOT include CAGW would not be proof that it does not exist? The opposite applies (see Piltdown Man, stomach ulcers, etc)
I’m inclined to say “No” because of one of my dad’s old witticisms: “Never give a prick the satisfaction.”
“There’s a lot of confusion as to whether that “consensus” position is weak (e.g. the greenhouse effect is real) or strong (e.g. humans are the primary culprits). The reason for that is Cook et al tried to combine both definitions into one rating, meaning they had no real definition.”
If you a looking for the genuine consensus, you will also need to look at scientist’s qualifications and fields of expertise to determine whether or not they are appropriately qualified to make comment on the physical basis of AGW.
If you do this, Brandon, note that raters should get a minimum amount of training in judging aspects of summaries. You should use at least three raters for each summary, which allows you to detect outliers among the raters. Compute rater reliabilities in a pilot phase. If these are too low, you should better stop.
I’m with David in Cal, Col Mosby et al: Short answer “Yes”, but approach scientists not abstracts.
The questions and participants are critical. But first, clearly identify the objective. I suggest:
“To find out what scientists think about certain aspects of climate science.“.
Participants : Obviously an unbiased selection process is needed, such as “All authors of climate-related papers published between dates x&y” or crowd-sourced. The former could be biased towards the mainstream because of funding, gatekeeping, etc, but in this context that’s not necessarily a bad thing (it establishes an ‘upper limit’). The latter is fraught with difficulty and open to corruption by non-scientists.
Questions: Unslanted, of course, and unambiguous. Do make sure that the questions make it very straightforward for anyone with mainstream views to make them clear. Do make sure that important sceptical ideas are tested, and that it is easy to express disagreement with them. Do make sure that ‘shades of grey’ can be expressed. Do make sure that they test knowledge not just belief. Suggestions:
1- Have human activities, such as burning fossil fuels, increased atmospheric CO2 concentration (ie, above what it otherwise would have been)?
2- Is CO2 a ‘greenhouse gas’ (ie, will increasing atmospheric concentrations tend to raise global temperature)?
3- About the history of global temperature-
-a- Was the Medieval Warming Period a global event, either concurrent or phased?
-b- Was the the Little Ice Age a global event, either concurrent or phased?
-c- How did temperatures in the early part [define] of the Holocene compare with today’s?
-d- How did sea levels in the early part of the Holocene compare with today’s?
-e- How did temperatures in the Eemian compare with today’s?
-f- How did sea levels in the Eemian compare with today’s?
4- At what date do you think that man-made CO2 began to have a measurable effect, say 0.2 deg C, on global temperature? [I suggest 0.2 because it is the currently expected decadal increase]
5a- Excluding feedbacks, what is the climate sensitivity (“CS”) to CO2? [define CS carefully].
5b – Including feedbacks, what is CS?
6- What proportions of observed global warming since say 1970 were -i- caused by man-made CO2, -ii- caused by other human activity such as land-clearing, -iii- natural, -iv- unreliable (eg, measurement or calculation error)?
7- How far into the future are climate model temperature forecasts reliable (multiple choice answers range from ‘not reliable’ and ‘a few days’ up to ‘1-2 centuries’ and ‘more than 2 centuries’)?
8- Does the current halt in global temperature increase (aka the ‘hiatus’ or ‘pause’) invalidate or cast doubt on the climate models?
9- Over the next 20 years, and over the next 100 years, with ‘business as usual’, by how much do you expect the global temperature to change? (multiple choice answers include ‘up an unspecified amount’ and ‘down an unspecified amount’)
10- With ‘business as usual’, how long will it be before fossil fuel usage falls below today’s usage?
Two reasons not to do it.
Never wrestle with pigs. You both get dirty and the pig likes it.”― George Bernard Shaw
and
“Never argue with a fool, onlookers may not be able to tell the difference.”― Mark Twain
I think it’s a good idea but just how statistically significant such a body of work would be is another question.
The criticism of sampling bias could so easily be leveled at the authours surely?
Just a thought that’s all.
The magnitude or rate of the warming needs to be a factor. For instance, the longer the pause goes on without some cooling, the more I find plausible the hypothesis that most of the warming that we are left with is anthropogenic. The direct effects of CO2 could never explain more than about 30% of the rapid rate of warming of the 80s and 90s without significant net positive feedback. With this extended pause, it seems likely that most of that unexplained rate of warming was natural or due to multi-decadal ocean modes. But the residual much lower rate of warming we are left with, might well be mostly anthropogenic. So while the extended pause destroys the credibility of the high sensitivity climate models and their extreme projections, it increases the plausibility of a largely anthropogenic 1C to 2C of warming by 2100 under business as usual emission scenarios.
I agree with you. I have been of the opinion for several days now, that this should be done. Just like Steve and Ross destroyed MBH98/99 with a re-analysis, this should have only an upside, as far as I can see. Like Donna L says, step by step things such as this matter. It adds up. Count me in if I can be helpful.
That’s not the intent of this re-analysis. The intent is to discredit Cook et al., those who pumped it, and SkS. That’ll do!
I think you should survey those attending the Church of England General Synod and try to establish how many of them believe in God. I think you’ll find it’s somewhere in the region of 97%.
Yesterday on Brandon Shollenberger’s blog I did not have enthusiasm for his idea primarily because the ‘Consensus’ paper is based on flawed premises and anything that uses the basic concept cannot avoid being tainted. And I argued this project seems low on a list of research projects.
Also, unless the idea proposed by Brandon Shollenberger is targeted and put on a track for publication in a major scientific journal, why spend the effort? Authors and co-author selection is crucial to put it on a journal publication track as well as affiliation to a organization that has a good track record of assisting in getting a paper published. In addition, the project needs a board of experienced and published mentors to guide it.
Though I am not enthusiastic about the idea by Brandon Shollenberger , I think if an objective group carries the idea forward then it will be with science in mind, In that case I will be very interested in following its development, its progress and in seeing the results.
John
Reading through the comments here, It’s clear that there are a lot of major faults with the Cook 97% Consensus. Any one of the criticisms, throws the survey result into a downward spiral.
Perhaps, putting all of these points, into a single paper, and written with the humble layperson,(like me) in mind, could dismantle the 97% myth. Getting it to appear in the public domain seems to be difficult. Maybe the Times would oblige again.
Ignore Cook et al, they will go the way of gas lighting.
What a waste of time – a warmest group analyzed the data and (SURPRISE!) found 97% agreement. Non-warmests are now proposing to analyze the exact same data with the expectation of finding (SURPRISE!) less than 97% agreement.
Who is the target audience for this colossal waste of time & why would ANYBODY find this to be credible?
Can any reasonable person answer “yes” to either of the following questions:
1) Would anything of substance actually change if non-warmists perfectly re-did this meaningless & silly survey?
2) Given reputational damage already heaped on the survey & its authors, would re-doing the survey have a positive impact on the scientific or public discussion (if, so, how?)?
Steven Mosher says:
May 21, 2014 at 2:00 pm
For folks who claim there is no consensus I would say this:
A) what evidence do you have
B) why resist an attempt to properly measure the consensus present in the literature.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
News flash for Steven Mosher: the scientific consensus that Earth was the center of creation was overturned in about 1675; consensus against the photon was overturned about 1920; consensus against plate tectonics overturned about 1970.
Frankly, where in the scientific method is weight given to consensus?
Great idea. Maybe have the good folks at scienceofdoom participate.
Brandon, here’s one suggestion for handling the stats (I would not be surprised if you have already planned this):
a) Given the full set of abstracts {W}
b) Select (in secret) a subset {Q} < {W} of maybe 20 abstracts that is representative of the range of possible assessments
c) Have a balanced group of experts (some warmers, some skeptics) assess and discuss the set {Q}, to determine the "canonical" classification of each abstract in {Q}. Keep this information confidential (it might be best to do it at the end of the data collection phase).
d) For each volunteer assessor k, assign a subset {Wk} of abstracts to evaluate. In every case, {Wk} should include all elements of {Q} though in unmarked random positions in the list.
e) Ensure that there are enough volunteers to assess all abstracts multiple times.
f) Run a special statistical analysis to detect patterns in the assessments, such as: are some assessors biased; do the average assessments change as the assessors gain experience, etc.
g) Remove data contributed by assessors who performed poorly on the canonical set.
When I first heard of the 97% consensus it reminded me of the election results of an election in North Korea or Russia. The kind of an election where 3% of the electorate might eventually disappear. As for your idea, please don’t. I believe it would lead to an inaccurate public poll of their opinion.
Perhaps a scientific poll with a control for responses, double blind so no one knows who responded with what answer would be the best methodology. But it is unnecessary. We know the flaws of the Cook survey mentioned above.
Mr. Watts, keep up the good work and the good fight.
I think I would volunteer. I would however suggest to have a pre-question about ECS to allow the reviewers to be classified. This would think provide a view about bias.
Do you know what ECS is?
Yes: What do you think ECS will be : 0-1, 1-2 etc
No: Do you consider yourself to be a Skeptic, Alarmist, (provide examples to be sure the definition is understood)