Brandon Shollenberger writes: I’ve been mulling over an idea I had, and I wanted to get some public feedback. What would you think of a public re-analysis of the Cook et al data set?
A key criticism of the Cook et al paper is they didn’t define the “consensus” they were looking for. There’s a lot of confusion as to whether that “consensus” position is weak (e.g. the greenhouse effect is real) or strong (e.g. humans are the primary culprits). The reason for that is Cook et al tried to combine both definitions into one rating, meaning they had no real definition.
You can see a discussion of that here.
I think it’d be interesting to examine the same data with sensible definitions. Instead of saying there’s a “97% consensus,” we could say “X% believe in global warming, Y% say humans are responsible for Z% of it.” That’d be far more informative. It’d also let us see if rating abstracts is even a plausibly useful approach for measuring a consensus.
My current thinking is to create a web site where people will be able to create accounts, log in and rate a particular subsample of the Cook et al data. I’m thinking 100 “Endorse AGW” abstracts to start with should be enough. After enough ratings have been submitted (or enough time has passed), I’ll break off the ratings, post results and start ratings on another set of abstracts.
The results would allow us to see tallies of how each abstract was rated (contrasted with the Cook et al ratings). I’m thinking I’d also allow raters to leave comments on abstracts to explain themselves, and these would be displayed as well. Finally, individual raters’ ratings could be viewed on a page to look for systematic differences in views.
What do you guys think? Would you be interested in something like this? Do you have things you’d like added or removed from it? Most importantly, do you think it’d be worth the effort? I’d be happy to create it, but it would take a fair amount of time and effort. It’d also take some money for hosting costs. I’d like to have an idea of if it’d be worth it.
An added bonus to doing it would be I could move my blog to that site as well. Self-hosting WordPress takes more effort than using WordPress.com, but it allows for far more customization. I’d love that.
So, thoughts? Questions? Concerns?
By the way, don’t hesitate to tell me I’m a fool if you think I’m spending too much time on the Cook et al issue. I’ve been telling myself that for the last two weeks.
Source: http://hiizuru.wordpress.com/2014/05/20/a-re-analysis-of-the-consensus/
===============================================================
My opinion is that given the vast number of people interested in this at WUWT, we could likely crowd-source this work much more accurately and quickly than Cook did, without having to fall back on a small cadre of “like minded friends”. Both sides of the aisle can participate.
I don’t know what the result will be of such an analysis proposed by Brandon, but I do know that we can get far more participants from a much broader venue (since WUWT has almost an order of magnitude more reach than “Skeptical Science”) and that Brandon’s attention to detail will be an asset.
We already know many of the mistakes made in Cook’s work, so a re-do has the advantage out of the gate. This disadvantage may be that the gatekeepers at IOP may refuse to publish it, and University of Queensland may publish yet another bogus legal threat, since they seem tickled that Cooks 97% is the subject of worldwide gossip – Anthony
I regard this as a very BAD IDEA. Scientific results are not verified by a popularity poll. They are verified by ongoing research which confirms their utility.
If the rating operation is to be open to all comers, it would need to be possible for readers to slice and dice the results in accordance with who the (anonymized?) raters were: which ones rated most or least consistently with the others, for instance, and what are examples of a given rater’s more-controversial ratings.. This is because there will no doubt be saboteurs, so attempts will have to be made to cull the data–which will no doubt be called cherry picking.
Incidentally, it would be informative to perform a parallel categorization of the papers in accordance with whether they actually attempt to establish values for climate sensitivity or man’s percentage responsibility for warming. If a mycologist, for example, starts his paper out with”100% of global warming is caused by humans’ CO2 emissions, which are certain to raise mean temperatures at least 10 Celsius degrees by Friday, so this paper reports our study of how installing incandescent lamps in cellars affects mushroom growth,” that may actually reflect his belief (as opposed to his desire for further funding), but the paper gives no reason for joining that belief.
Anthony don’t waste your time no one will believe your results anyway at least none of the warmists.
What I believe we need is advertising in large papers telling people the truth about temps telling people about sea levels, if they knew the truth I’m sure more people would come over to our side. It would certainly make the Establishment have to think differently if they were constantly being asked pertinent questions about the climate.
I think we need a site where we could raise money to pay for this, I’m sure even business’s would contribute if they knew there electricity bills would come down if this BULL was shot down.
Imagine the frustration of all the doom mongers, they wouldn’t be able to run and hide like they do, they would also have to answer difficult questions.
Anyway i’ll keep dreaming.
I think the whole idea of trying to use abstracts as evidence of anything is futile. It’s just one step improved over science by press release. Trying to prove any position requires facts which, by definition, are missing from an abstract. The facts are what are, or should be, contained in the papers themselves.
As anybody who actually goes to the effort of actually reading scientific papers has experienced, there can often be a large discrepancy between the paper’s factual contents and what a summary of that same paper purports it’s contents to be. I’ve often seen an interesting press release or paper abstract and gone on to read the full paper with all it’s caveats and conditions. Even when the science itself may be above my understanding, you can still see how it was conducted and to what degree the authors are confident of their conclusions. Sometimes the paper itself is a fairly straight forward analysis of data to test a theory, and the conclusions have minimal errors and assumptions. Other times, the paper spends more time trying to justify their assumptions and exclusions of potentially contradictory data than they spend on the data and methodology that was used to produce the conclusions mentioned in the abstract or press release.
It’s this “the devil is in the details” issue that makes it impossible to use just an abstract to determine what the real “scientific” consensus is. An abstract unsupported by data in the actual paper would have the same “weight” as an abstract for a well documented paper. Without reading and rating each individual paper, for not only end opinion but robustness of data, any conclusion derived from abstracts alone would just tell you what the writers want the readers to believe and not what the actual science may show.
Geology Joe says: (May 21, 2014 at 10:51 am) “Consensus is irrelevant to science.”
Really? 😉
By now you’ve realized this whole thing is not, and never was, about science. It is about 100 billion dollars a year to send to Africa along with the usual global governance wish list items by people that are not being careful about what they wish for.
You need a clever slogan — 97 percent of scientists…
You need a couple of scary movies: “Day After Tomorrow” had me worried for about an hour. Then I remembered that adiabatic compression will warm stratospheric air as it comes down to sea level. It just doesn’t stay frozen and freeze everything it touches, removing megajoules of energy from the sea without itself warming up in the process.
You need people to worry about cute furry polar bears.
That is “magical thinking” at its finest and has already moved tens of billions of dollars, per year, just in the United States.
Don’t do it. Go for a retraction. At least 97% of these proverbs concur:
http://www.giga-usa.com/quotes/topics/proverbs_t318.htm
The more you stir a turd, the more it stinks.
– Proverb, (Dutch)
The more you stir filth the worse it stinks.
– Proverb, (Danish)
The more you stir it the more it stinks.
– Proverb, (French)
The more you stir it the worse it stinks.
– Proverb
The more you stir the mire, the more it stinks.
– Proverb, (German)
The more you stir, the worse it will stink.
– Proverb
This will be a second mention of this report but it is worth mentioning AGAIN (just as the 97 percent gets mentioned in everything)
http://wattsupwiththat.com/2013/09/03/cooks-97-consensus-disproven-by-a-new-paper-showing-major-math-errors/
“Only 41 out of the 11,944 published climate papers Cook examined explicitly stated that Man caused most of the warming since 1950. Cook himself had flagged just 64 papers as explicitly supporting that consensus, but 23 of the 64 had not in fact supported it.”
Tim Ball says:
“The sad truth is the claim of 97% achieved its objective, which was political from the start. When it is quoted by the US President it comes with all the unearned authority people ascribe to that position.”
So, depending on how you select your papers, you get anywhere from 0.3 percent consensus to 100 percent consensus.
I am starting to see why this whole exercise is really, really pointless — it has been done, the “debunk” published. I will try out this new statement on my TBW (True Believing Warmist) family members and see if it makes the slightest difference. The TBW’s themselves readily concede lack of consensus — but to them it is irrelevant. They understand that the REASON for the 97 percent is “social justice” and that’s a Good Thing (TM).
I didn’t read the Legates, Monckton et al. paper, so I repeat my question:
Is this not redundant rework?
How is this proposal different from Legate’s paper?
Harry Passfield says:
If I get this right, you seem to think that Cook’s original idea was good to start with: to demonstrate there is a ‘consensus’…
Exactly. Whether the fact that Cook’s survey was done correctly or not isn’t the issue here. The issue is that consensus is politics, not science, and we skeptics should have not part of it.
My opinion may be a bit strong, but I think the idea of doing it again will bring the same shame to skeptics. As Michael Crichton put it:
Slightly OT but hopefully entertaining and illuminates the problem:
The religious equivalent is saying that 97 percent of all theologians believe in the Book of Mormon. You’d simply filter the list of theologians to include only Mormons. That way you’d get a small percentage of Mormons that don’t believe in the book of Mormon and thus seem very objective and scientific. (*)
When challenged, you would say, “Since the study is about the Book of Mormon, it seems proper to include only those people whose opinions are valid on the subject, and that necessarily means including only Mormons in the survey.”
It ought to be 100 percent but that doesn’t seem very “scientific”. 97 percent is a nice prime lucky number that’s nearly 100 percent.
* I actually have no idea what is the percentage of Mormons that believe in the B of M.
If the goal is to see how many people agree with the proclaimed “consensus” position, then the criterion for agreeing with the “consensus” should to be the position that the IPCC authoritatively states as the consensus position (from a near final draft of AR5, would have to look up what the final says):
I suspect that the actual degree of agreement with that claim is quite low amongst climate scientists. The previous IPCC “consensus” had been that the signal of human caused warming was now detectable. To leap from “detectable” to a high degree of certainty that humans had caused MOST recent warming, when the only new data was a cessation in warming, is bizarre. A person would have to pretty much be a lunatic (as some climate scientists certainly are) to think the evidence supports this leap.
Just for fun, I would also ask how many think warming is still happening.
John Barrett suggests advertising. That’s not a bad idea. One graph with a short explanation would go a long way in educating the undecided public.
Re: peer review. Remember this? Peer review in the climate field is corrupt. It has little credibility. Really, you have to go back about 15 – 20 years to get some balance.
José Tomás says: (May 21, 2014 at 12:03 pm)
“How is this proposal different from Legate’s paper?”
It has the limited utility of quantifying various options other than “bad AGW”. It might also illuminate that a different pool of reviewers would rate differently. But I suspect everyone here already is pretty sure of both outcomes.
The Cook study reveals a huge burden on a small number of volunteers with over a thousand abstracts reviewed by some volunteers in fairly short span of time. It is no wonder that some were considered “consensus” papers when they weren’t about climate whatsoever — just throw a sentence in the abstract to get some grant money or get published.
The “Sokal Affair” relates to this as does the SciGen scandal. Put in a few leftwing words (climate change or social justice) and it will get published — even if it is otherwise about improving catalytic efficiency.
If a climate paper is about climate science but does not specifically mention AGW — well, actually it DID convey an opinion — there’s nothing there to write ABOUT.
It is the same as counting only “Yes” votes and then coming up with a conclusion that all papers say “yes” — but only 41 out of 12,000.
large scale attribution of motive to thousands of ‘scientists’ by a bunch of blog readers – what’s not to like?
get this:
“Although the art of trolling is measured by ROI in word count, sometimes the only object of an exercise is to turn a thread into a quagmire of nonsensical fits and blurts.
Some of the tried and true techniques are listed below:”
“:GO META!
Question or attribute motive for the statement rather than addressing the content of it. If the motives are of 3rd parties who are not present to confirm or deny any assertion, this will invite endless, pointless speculative debate.”
so sure- do it – add another skidmark to the cook paper.
there is plenty of room for more arrows in that straw man- knock yourself out; tell yourself you wont something.
but you’ll have been neutralized by idiots smarter than you!
fighting and winning are 2 very different things.
stop paying them- that’s the achilles heel, pokerguy.
keep paying them and you bleed yourself to feed your enemy.
but i suppose that’s unthinkable to eloi.
I can see the headlines now….
“Racists attack scientific papers…that 97% of scientists agree on”
….film at 11
I say do it … with real definitions about what the “consensus” is … the 97% meme needs to be shutdown … it has become a trump card for AGW proponents (or Real World Deniers as I call them) to try and end all debate …
would be happy to volunteer and would gladly hit a tip jar to help defer expenses …
I suggest you do this with a twist.
Evaluate each paper as to whether or not it gives EVIDENCE
1) to support CAGW,
2) to support AGW,
3) to support GW
4) researches the impact of GW if it occurs
5) is unrelated at all to climate
or
6) it is a modeling exercise.
Don’t ask for their opinions. Ask what the papers show.
Nearly a decade-and-a-half after the contested presidential election in Florida, the die-hards still believe Gore won, no matter what an independent study by the AP concluded.
Can’t win this one, cat’s already of the bag. I
I like the idea, but I think you’ll be attacked for bias. Critics will say that, by using Watts Up With That to recruit reviewers, you effectively solicited people who would lowball the consensus numbers. If you could find ways to control for this bias, then I think it’s an interesting idea.
Boromir was wrong. You dont use your enemies strongest weapon against them
Chuck it into the fire,where it belongs, along with cookie-gollum
original survey smacks of “4 out of 5 doctors smoke lucky strikes”. Cook’s paper smacks of a backhanded attempt to prop up the original (what a coinky dink both came up with the exact same number). Do a new survey and make it clear that if > 60% believe in CAGW, then all funding of climate science will be diverted to wind farms.
There’s no great likelihood that a properly-conducted survey will improve on the results in the original paper. The definitions were in fact clear enough: only abstracts marked “level 1” explicitly stated that most of the recent global warming was manmade – and Cook et al. themselves marked only 0.5% of the sample as assenting to that definition. See Legates et al., 2013.
A far more useful approach would be to arrange for the investigating authorities’ attention to be drawn to the flagrant misreporting of the authors’ results not only in the paper itself but also in Bedford & Cook (2013).
To my thinking, the best way to counter the 97% study would be to point out that several other studies have shown other results. Finding all of them would be a good start? The Bray/von Storch surveys have been mentioned. Here are two others, admittedly a few years old.
http://pielkeclimatesci.wordpress.com/2008/02/22/is-there-agreement-amongst-climate-scientists-on-the-ipcc-ar4-wg1/
http://www.forbes.com/2009/12/19/climategate-copenhagen-science-opinions-contributors-s-robert-lichter.html
97% of climate scientists believe in global warming.
The key word is “believe”.
This consensus is right up there with the consensus that 100% of creation scientists believe that all geologic strata were deposited in a Noachian flood.
Did they get a number of scientists to self rate their papers IAW their scale?