While Oxburgh writes a 5 page book report that most college professors would likely reject due to incompleteness, we have this report from Donna Laframboise of Toronto and a team of citizen auditors. The mission? Determine how much of AR4 met IPCC’s own standards for peer review by reviewing every reference in the report to determine if it comes from peer reviewed literature, grey literature, or if they “simply made stuff up”, like glacier melt dates.
21 of 44 chapters in the United Nations’ Nobel-winning climate bible earned an F on a report card we are releasing today. Forty citizen auditors from 12 countries examined 18,531 sources cited in the report – finding 5,587 to be not peer-reviewed.
Contrary to statements by the chairman of the Intergovernmental Panel on Climate Change (IPCC), the celebrated 2007 report does not rely solely on research published in reputable scientific journals. It also cites press releases, newspaper and magazine clippings, working papers, student theses, discussion papers, and literature published by green advocacy groups. Such material is often called “grey literature.”
We’ve been told this report is the gold standard. We’ve been told it’s 100 percent peer-reviewed science. But thousands of sources cited by this report have not come within a mile of a scientific journal.
Based on the grading system used in US schools, 21 chapters in the IPCC report receive an F (they cite peer-reviewed sources less than 60% of the time), 4 chapters get a D, and 6 get a C. There are also 5 Bs and 8 As.
In November, IPCC chairman Rajendra Pachauri disparaged non-peer-reviewed research in an interview with the Times of India (see the end of the article):
IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.
Between Oxburgh’s failure to write a credible report and this obvious failure of IPCC to follow their own rules, is it any wonder why people are beginning to laugh at the “robustness” oft touted in climate science?
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Kate
“That the UK is a world leader in a field of climate research is not necessarily a bad thing. But it does speak about the priorities and prejudices that dominate the political sphere, which, in turn, is reflected in the constitution of academic climate science, represented in Ch.10. ”
The numbers (UK and US dominance) just reflect general excellence in
Science and possibly an edge for English speaking nations: if an
international review on totally neutral disciplines, say fundamental
chemistry or neuroscience would be commissioned you would find about
the same distribution of nationalities.
When you’re in a hole, stop digging!
James F. Evans (12:38:54) :
Does this confirm that a political agenda was more important than quality science?
And, those that are pro-AGW, what say you?
You mean pro-AGW theory; pro-AGW or AGW enthousiast would make
me a criminal wouldn’t it (language matters). Since sceptics maintain
that the peer reviewed bulk of climate science material is corrupted
and IPCC in your view undoubtedly would cherry-pick the grey material
also. What would be the point in their manipulating through this
mechanism?
Maybe it was just pure common sense: to cover relevant material in
an optimal way. A ratio of 70/30 or 65/35 sounds like a report firmly
based in peer-reviewed science, but also reaching out for e.g. extensive
synthesis reports and other relevant material? Doesn’t have to be more
sinister than that.
Very interesting article and certainly destroys any remaining credibility that Pachuri might have had left.
Of the grey literature, does anyone know how much was published by advocacy groups or similar organisations who might have had a very specific agenda? Given the concentration of “F”s in WG2 and 3 I’ve got a hunch that it might be quite high.
Some of the criticism of this article also seems a bit misplaced. If one were to write a paper for an exam with a similar level of unreviewed references would it get an “F”? To my mind the question isn’t really whether the IPCC used unreviewed citations, simply that by using so many in what really are the critical parts politically, do they in fact deserve to get an “F”?
If we agree that they would, would the fix simply be to go back and rework it so that they don’t use so many?
Then, by leaving out much of the unreviewed material does it materially affect either the impacts of AGW or its mitigation?
@Sphaerica (10:26:05) :
You miss the whole point. It does not matter what the IPCC’s guidelines to their tame scientists are. What matters is the perception among policy makers and the public that the IPCC reports are infallible because they are *entirely* based on peer-reviewed studies. This perception is one which has been actively promoted by the IPCC (a list of quotes is here: http://www.noconsensus.org/ipcc-audit/not-as-advertised.php ).
For example:
“As IPCC Chairman Rajendra K. Pachauri recently stated: ‘IPCC relies entirely on peer reviewed literature in carrying out its assessment…'” – US Environmental Protection Agency, December 2009 (bottom of PDF’s page 7)
Also, having read and reread Laframboise’s claims, she doesn’t say that the report claims only to use peer reviewed material. She does, however, say that it is claimed over and over again, not least by its Chairman and not even all that long ago, that it only uses peer reviewed material, and that such a claim is incorrect.
I think that Max and Todd both need to apologise for the pointless introduction of their straw man. Sorry guys, you can’t criticise her for an incorrect claim that she doesn’t make.
Nice try at hiding the pea though.
At some point, the legal issues will have to be addressed….were the IPCC authors committing outright fraud? Seems like it to me….
For all those doubters who think that ‘grey-papers’ are allowed in IPCC reports, here’s a quote from the head of the IPCC.
“IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.”…..Rajendra Pachauri.
So if he is correct, a mistake has been made regarding IPCC Policy – if he is wrong, the head of the IPCC is a fool or a liar. The whole organisation is a joke, much like the rest of the UN.
Regarding the peer reviewed papers used in the report, it would be useful to know how many have been refuted by subsequent papers? This would be another massive job, but I suspect it would be revealing!
The whole IPCC climate scam really built on the shallowest of foundations, just like the conjecture of CAGW.
The chart at http://www.ipcc.ch/publications_and_data/publications_and_data.htm indicates that “Expert and Government Review” occurs in the 2nd and 3rd draft and “Expert Review” occurs in the 1st Draft.
If questionable material made it into the drafts, its reasonable to assume it was sanctioned by all the “Experts and Government Reviewers”?
Allegedly, some material was inserted after the 3 Review stages. If true, I’m surprised Governments aren’t more outspoken about this apparent breach of faith.
Do all the government reviewers support the final version?
So we should dismiss what Hansen says based on the facts that he sells books based on it? Good point!
We should also dismiss everything the Club Of Rome says. They sell their reports in book form.
Oh, and a lot of peer reviewed pro AGW papers are behind pay walls.
You’re a concern troll.
By DirkH on April 14, 2010 at 1:00 pm
I am saying exactly what I said. And the point was quite obvious. We should be in this to make a point, not money out of it. This lady is.
And don’t you dare accuse me of being a troll.
@ur momisugly Johnny D (13:00:54) :
“Funny that all the chapters in Working Group 1, where the actual climate science is laid out, seemed to get A’s. One could argue that Working Group 1 is the only one that really matters, considering that that’s where the climate science is. Of course Working Groups 2 and 3, on fuzzier stuff like adaptation and energy, are going to be less rigorous scientifically.”
Johnny, I’m not clear here. Peer-review literature only matters where you believe science matters or is it that peer review literature doesn’t matter on “fuzzy stuff” such as energy? Or that peer-review literature only matters when using it as claims for veracity? Or it doesn’t matter towards claims of veracity?
To be perfectly honest, I really don’t care about whether the science backs up climate change or not. I’ve known our climate was in a constant state of change for some time now. It is the “fuzzy stuff” that bothers me. You know, mitigation for a non-existent problem, the starving large groups of people of energy, the laughable attempts to force adaptation to perceived problem,…ect. I really could care less if a group of people want to waste time and money on trying to prove an unprovable, as long as it’s not on my dime and the reaction to their ludicrous conclusions is thwarted by a sense of reality. Sadly, it is my dime, and there is no sense of reality to thwart these idiots.
When do they release the Fourth Assessment Report-Corrected Edition (FARCE)? When it’s complete?
mikael pihlström (12:22:59) :
Your phrase 2 – maybe a problem here: with rejection rates 90%, you will
have a lot of people in the world disagreeing with Nature or Science editors
——————————
You are new here. There have been observations on threads in the distant past (pre-Climategate) about how both these journals refuse to publish articles which do not support the prevailing narrative of environmental catastrophe due to human influences. While I have grave concerns about the state of many of the world’s fisheries, when Boris Worm made his pronouncements in 2006 they were greeted by myself and others who know something about fisheries science with derision. He has since recanted and now produces more reasonable projections, although his focus remains the same. Indeed, at a conference last year there was some progress in dialogue between fisheries biologists and Worm and a contingent of other ecologists. While Science would not publish the critiques of his inflammatory projections, other journals specializing in fisheries science, where the scientific authors and readers are more familiar with fish population dynamics, fisheries management and fish biology, did publish them. Does the fact that Science refused to publish scientific criticisms by fisheries biologists reduce the value of these papers? Hardly. The point is that Nature and Science selectively publish ‘science’ relating to environmental issues only if it supports a given, ideological slant.
I know this is really OT and moderator I fully understand if you cut this but is anybody doing this on the ObamaCare bill? It needs 10,000 auditors on it like Mr Watts has on climate. Two channels: 1) take the bill apart line by line, do the sums, check the facts; 2) work together to compile reform that would really keep the US healthy at a better price. We people in Europe need a functional US ‘main street’ to keep us out of the bloodbath that has been much of our history.
Reply: Seriously off topic. You can go make your own blog? ~ ctm
Fair point ctm
What proportion are peer review but not scientific, ie, based on the guesswork of computer models?
In DirkH (13:00:47), DirkH calls climategatestuff a concern troll.
In blackswhitewash.com (14:17:30), blackwhitewash says in direct reply to DirkH (13:00:47) “I am saying exactly what I said. (…) And don’t you dare accuse me of being a troll.”
Did someone get their two different aliases mixed up? 🙂
“blackswhitewash.com (14:17:30) :
[…]
I am saying exactly what I said. And the point was quite obvious. We should be in this to make a point, not money out of it. This lady is.
And don’t you dare accuse me of being a troll.”
What about Steve Mosher? Do you extend your accusations on him? He makes money out of climategate, doesn’t he? He’s written a book.
climategatestuff,
“My concern with this is related to the fact that Donna Laframboise just happens to be launching a book about this.”
Awesome. Can’t wait to see it. I have been buying and reading every book on AGW that comes out. Highly entertaining. Well worth every penny.
I suppose I am atoning for all those years of paid subscriptions to Scientific American and Newsweek.
It’s a lot of work writing a book – hardly qualifies as “feeding at the trough” in my opinion. I mean, what public funds is Donna getting to write her book? If she is getting no such funds, then whence cometh said trough feeding. Right on Donna. Bring it on.
blackswhitewash, are you identical to climategatestuff or what’s going on here? Please don’t do that.
vigilantfish (14:22:52) :
The point is that Nature and Science selectively publish ’science’ relating to environmental issues only if it supports a given, ideological slant.
Well, that is well reasoned and it worries me (such bias is obviously destructive). But, is your claim that Science magazine has a slant
solid? It can be somewhat random: one rejects, you go to the next,
etc. In the end you don’t always understand the logic.
More likely in this case; being generalists the editors of Science did
not understand the importance as well as the fisheries journals and
they could argue that they have to reject a lot of good papers on
account of the huge supply of manuscripts. Were the critical responses
of yours, an article or in the letters compartment?
have so many
Max Westin (09:33:29) :
I was going to make the same point Max. I couldn’t find any claim by the IPCC that supported the statement “We’ve been told it’s 100 percent peer-reviewed science.”
Perhaps it’s just our favorite railway engineer shouting his mouth off.
Interesting work all the same.
So many people have tried to shut down any debate with the “Is it peer reviewed?” line that it was well worth the time spent. There were many other issues spotted as well — self citing, (“academic check kiting” is a term I have heard), circular citing, etc. There are lots of issues, but the first one to resolve is the “throw it in the dustbin” line. Now that’s done. Now we can move on to something important!
For those who want more — I say “Go for it!” start up a new project. You could also audit horrible things like the Ontario (Canada) “Climate Change” literature which is based on IPCC literature — and uses many of the same (non)mathematical techniques to arrive at suspect conclusions. The point is that it can be done.
Now we don’t have to accept “It isn’t peer reviewed.” Why? We now have agreement here at least that peer review is not necessary. So grab your crayons, sharpen them up, and draw a graph — then “Get Published” (non peer-reviewed of course!).
Kate.
Excellent article. I read it twice but it was worth it.
Mikael P.
Do you really think that one’s mother tongue determines success in publishing in journals? Actually, I probably agree which says everything about so called peer review.
As someone said earlier (metallurgist) it is essential for all young researchers to be published and most papers are fairly ordinary but they keep their fingers crossed. Certainly, in medicine, one might be advised as to which journal might be appropriate i.e. different standards of ‘validity’, whether in design or statistical interpretation etc. But, once published they often seem to have the same ‘truthfulness’ at least as far as the media is concerned. Peer review only means that the paper is not complete rubbish, on a first cursory read, but says nothing about the true validity of the paper.
I’m don’t think it is reasonable that the IPCC report should be expected to only reference peer-reviewed work. This excludes a huge body of information including reports from scientific institutions, government bodies, NGO’s, the Media and even skeptical sources.
If you have a look at the report you will get an idea of the sort of references it involves.
Having said that, this is an interesting piece of analysis, and I’m sure that Laframboise’s book will be a great success.