All you can do is laugh.
Brandon Shollenberger writes at Lucia’s about the new Fuzzy Math consensus “proof” paper from the ever entertaining John Cook at Skeptical Science, rated with the help of 27 of the SkS kidz club. The method is simple:
“Each abstract was categorized by two independent, anonymized raters.”
With a simple premise like that, what could go wrong? Well for starters, they don’t seem to understand what the word “independent” means. Shollenberger continues:
That makes sense. What doesn’t make sense is that people would make topics in the SKS forum like:
Does this mean what it seems to mean?
how to rate: Cool Dudes: The Denial Of Climate Change…
That’s right. The “independent” raters talked to each other about how to rate the papers. This must be some new form of independence I’ve never heard of. I’m not the only one thrown off by this. Sarah Green, one of the most active raters, observed the non-independence:
But, this is clearly not an independent poll, nor really a statistical exercise. We are just assisting in the effort to apply defined criteria to the abstracts with the goal of classifying them as objectively as possible.
Disagreements arise because neither the criteria nor the abstracts can be 100% precise. We have already gone down the path of trying to reach a consensus through the discussions of particular cases. From the start we would never be able to claim that ratings were done by independent, unbiased, or random people anyhow.
One must wonder at the fact an author of the paper calls the work independent despite having said just a year earlier, “we would never be able to claim” it is independent. Perhaps there is some new definition for “never” I’m unaware of.
And it gets even more hilarious. Read it all at Lucia’s.
For those who don’t know Inigo Montoya: