IPCC AR4 also gets a failing grade on 21 chapters

While Oxburgh writes a 5 page book report that most college professors would likely reject due to incompleteness, we have this report from Donna Laframboise of Toronto and a team of citizen auditors. The mission? Determine how much of AR4 met IPCC’s own standards for peer review by reviewing every reference in the report to determine if it comes from peer reviewed literature, grey literature, or if they “simply made stuff up”, like glacier melt dates.

She writes:

21 of 44 chapters in the United Nations’ Nobel-winning climate bible earned an F on a report card we are releasing today. Forty citizen auditors from 12 countries examined 18,531 sources cited in the report – finding 5,587 to be not peer-reviewed.

Contrary to statements by the chairman of the Intergovernmental Panel on Climate Change (IPCC), the celebrated 2007 report does not rely solely on research published in reputable scientific journals. It also cites press releases, newspaper and magazine clippings, working papers, student theses, discussion papers, and literature published by green advocacy groups. Such material is often called “grey literature.”

We’ve been told this report is the gold standard. We’ve been told it’s 100 percent peer-reviewed science. But thousands of sources cited by this report have not come within a mile of a scientific journal.

Based on the grading system used in US schools, 21 chapters in the IPCC report receive an F (they cite peer-reviewed sources less than 60% of the time), 4 chapters get a D, and 6 get a C. There are also 5 Bs and 8 As.

In November, IPCC chairman Rajendra Pachauri disparaged non-peer-reviewed research in an interview with the Times of India (see the end of the article):

IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.

Between Oxburgh’s failure to write a credible report and this obvious failure of IPCC to follow their own rules, is it any wonder why people are beginning to laugh at the “robustness” oft touted in climate science?

Advertisements

  Subscribe  
newest oldest most voted
Notify of
Calum

Well done to everyone for revealing the flimsy nature of the IPCC AR4.
You could send a link to the secretary of the Inter-Academy Council who are reviewing (supposedly) IPCC reports.

Brilliant research – well found! We need to keep raising awareness of the pseudo-mystical guff that the IPCC continues to recucle and pass of as science. How about the formation of a Campaign for Real Science?

Robert of Ottawa

I couldn’t find a location on Donna Laframboise’s web page to congratulate her on a job well done. May I do it here? Another Great Canadian 🙂

Leaks just keep popping every where.
The IPCC AR4 is an advocacy work that uses cherry-picked science to drive an agenda, while completely ignoring a huge scientific body that is not in agreement. They failed, or even refused, to discern scientific literature from fantasy.

oakgeo

So a total of 30% of the citations were of non-peer reviewed writings. That’s actually far worse than I expected. It also would be interesting to know how many of the 70% peer reviewed citations were actually referenced correctly (i.e. supported the relevent IPCC text), but that would be a monumental undertaking.

MattE

Would be nice if the listing mentioned, generally, what was in each of the failing chapters…

Jeff C.

The IPCC really are a bunch of shysters. And the response will be….
“Nothing to see here. Move along now”
I’ll echo Calum – well done to everyone involved. Copy of this going to my MP (once the election is over and to Cameron).

Loco

Perhaps we should report on all the major players and give them a grading…?
Al Gore – for inaccurate, misleading and totally unscientific pronouncements, oh, and for making a truck load of money by scaring people… F
John Kerry – for being in a position of power and influence, but for being so DUMB (I’ve never seen a peer reviewed paper that refutes global warming)… F
George Monbiot – For years of rampant alarmism, but then finally having the balls to admit that Climategate was both damming and a blow to the AGW cause… C+
James Hansen – for.. um, where do you start? For some of the most outrageous pronouncements on global warming every to dribble from a person’s lips… G (only because it’s below an F!)
Steve McIntyre – For vigilance and determination in the face of almost overwhelming odds… A++
There’s plenty more to choose from!

vigilantfish

Well done Donna LaFramboise and the citizen volunteer auditors! I agree with MattE. It would be interesting to see a semi-detailed analysis of the chapters that are the worst offenders.

The breakdown of the grading by chapters is interesting – Group one, describing ‘Climate Science’ appears to have been put together somewhat professionally. The part where they start ‘making things up’ comes in with the work from Groups two and three, Impacts and Mitigations, respectively.
So, basically, they may be able to claim to talk about ‘something’ happening, but as far as what that means, and what should be done about it, they don’t seem to have a the first frakkin clue other than their hairshirt agenda.

To be fair, AR4 (70%) is an improvement over TAR (62%).
http://pgosselin.wordpress.com/2010/04/11/ipccs-scientific-assessment-reports/
At this rate the IPCC will be using only 100% peer-reviewed literature by 2030. But by then we might be in a little ice age (but at least AGW will be proven!).

OUTSTANDING !!!!
Will anyone pay attention ???

Dr T G Watkins

Great job guys. Eventually the masses will catch on to the entirely political nature of the IPCC and many of its lead authors.
A tipping point will be reached, hopefully soon.(note the number of hits on this blog alone, steadily rising).

Bob Laban

An excellent idea, but the auditors have not followed it up with the necessary analysis. Were the references to news articles, etc. *appropriate*? If the cited publication was accurate authority for the proposition footnoted, then it is correct, regardless of whether it is peer-reviewed.
For example, if you are referring to the Kyoto Protocol, you may want to cite to the text of the treaty (not peer reviewed) rather than to a peer-reviewed article about the treaty.

RockyRoad

Any bets whether Al Gore will now follow through on his promise to get with Fox News for an interview?
What would be the odds?

Max Westin

Unfortunately for Framboise she bases her report card (much like many of her past criticisms) on the wrong information. Regardless of what Pachauri has said publicly, the IPCC guidelines do not state that it will only use peer reviewed literature. See for yourself: http://www.ipcc.ch/publications_and_data/publications_and_data.htm
“[The IPCC will reference] Peer reviewed and internationally available scientific technical and socio-economic literature, manuscripts made available for IPCC review and selected non-peer reviewed literature produced by other relevant institutions including industry.”
[snip]
REPLY: That’s a point of consideration, but consider this: warmists will never again be able to say “but it’s not peer reviewed, and therefore irrelevant without looking like complete fools in the context of citing the IPCC -A

@vigilantfish and all
If you click through you’ll find all the raw data on each chapter

Riku Mellin

Hi, Riku Mellin, one of the auditors from the project.
I noticed that few people requested more details.
http://www.noconsensus.org/ipcc-audit/findings-detailed.php
You can find the list of the individual workgroup and under the chapter number you can find “A”, “B” and “C” options, wich all of them have the individual audit.

GregF

MattE and vigilantfish, click on the link to the report on Donna’s page above, then click on detailed findings. Under each chapter listing is an a, b and c. Those link to the individual audits and also link to the text of the chapter and the reference list. That should be enough to get you started. Cheers,
GregF

fredb

How many of these grey literature citations stood in isolation *without* a partner peer reviewed citation. My (incomplete) reading of the IPCC is that most often the grey literature citation was a complement to a peer reviewed citation. Unless that distinction is made in this sort of analysis, this posting has little meaning.

Robert M

edit… yo???
professors would likely reject due yo incompleteness, we have

Frank Lansner

This is extremely useful. Its becoming harder and harder for the global warming movement. Now they cant even get away with “It must be peer reviewed” anymore. Poor AGW crowd.

Todd Brunner

Regardless of what Pachauri is quoted as saying, the IPCC guidelines clearly state that they use non-peer reviewed literature at times. Looks like everyone gets an A after all….except Laframboise and her in-depth research team.

Layne Blanchard

Someone posted here recently that there was a clause somewhere in AR4 allowing grey literature. Don’t know myself. But is a fluff piece by WWF even considered “grey”?

Todd Brunner

Forgot a link to the IPCC publication guidelines: http://www.ipcc-wg1.unibe.ch/publications/publications.html

RockyRoad

fredb (09:45:45) :
How many of these grey literature citations stood in isolation *without* a partner peer reviewed citation. My (incomplete) reading of the IPCC is that most often the grey literature citation was a complement to a peer reviewed citation. Unless that distinction is made in this sort of analysis, this posting has little meaning
———————–
Reply:
Of what use and of what authority are companion citations that come from non-peer-reviewed sources (regardless of whether or not they ahppen to be paired with peer-reviewed sources)?
Answer? NADA!
The only reason they’re thrown in is to increase “thud factor” — toss the document on the
floor and because it’s thicker, it makes a bigger thud.
Is that what science has come to? (It does’t work for a mid-term English paper, either!)

higley7

The models must be right because they are so expensive and complicated. How could they be wrong?.
The IPCC AR4 must be right because so many “scientists” and real scientists worked on it. Ah, but who edited it?
Of course, I would really like to see which chapters earned A’s or B’s, as they were probably topics which were not crucial to the AGW fabrication of assumptions. The editors could let these few goods chapters slide.
A friend used to say “We should eat cow dung. How can 14 trillion flies be wrong!” (His version of Truth by Democracy.)

F. Ross

Well done auditors.
As far as the results go, who would expect anything else from any United Nations organization?

Ron

In section 6 of Appendix A to the Principles Governing IPCC Work it does say that not all material must be peer reviewed but it must be clearly labled as such.
6. IPCC SUPPORTING MATERIAL
All supporting material should be formally and prominently described on the front and other introductory covers as:
“Supporting material prepared for consideration by the Intergovernmental Panel on Climate Change. This supporting material has not been subject to formal IPCC review processes.”

higley7

RockyRoad (09:59:51) said:
“Of what use and of what authority are companion citations that come from non-peer-reviewed sources (regardless of whether or not they ahppen to be paired with peer-reviewed sources)?”
Ah, but the grey companion citations often use much less objective (more alarmist) language, which is what they really want. Exaggerated conclusions and wantom, scary warmings, abound.

Peer review is no guarantee of quality or protection against advocacy it is important in the scientific process. This is good work, thank you volunteers. We must remember the IPCC process is more political then anything else. This is not an excuse for not even following its own rules, it just is the way it is.

After Climategate last year it was a relief to find that AGW was shall we say, exaggerated. And it’s good news too – man-made CO2 is nothing more than much needed plant food. Public opinion polls show concern for the AGW conjecture is low and we can hope it falls even lower as there is nothing to be concerned about.
The bad news is that the exaggerations and flat out lies continue to inform our leaders and the electorate and MSM couldn’t seem to care less. Donna LaFramoise and her citizen volunteers have shown that so-called gold-standard peer-reviewed science informing the UN and world governments is more exaggeration and in some cases pure hearsay. There is an old saying – nothing ruins the truth like stretching it.
So this what the face of “settled climate science” looks like. Get past the letters after the names; get past the august institutions of academia and government; that’s the mask: rip it off and you get the real face of this so-called settled science. It looks a lot to me like a lynch mob of politically posturing petty intellectuals hungry for totally undeserved public recognition. Rip off the mask; what do you see? I see evil.

Don Shaw

Max Westin (09:33:29) :
Max, Did you know that the IPCC recently changed the wording and removed the claim re: only using peer reviewed material?

Henry chance

The anti science mis informers are which side?

bubbagyro

This audit is a great start, but let’s not fall into the “if it’s peer reviewed, it’s OK”. How many of the “peer reviewed” papers are from controversial journals like Science or Nature? How many papers are mutually reviewed? (One lies, the other swears to it, or you scratch my back, I’ll scratch yours!).
I think the whole field of climate science has to move from opinion or speculation to hard science, with more input from physicists, chemists, geologists, statisticians, and biologists. I think meteorology has to be stepped up a notch so that divinity students like Al Gore or Railroad Engineers like Pachauri don’t get first dibs in the debate. Although if their arguments were sound, I don’t care who their mothers are.

Agassiz

Part of the standard of classifying a paper as “non-peer reviewed” by this report is that the paper in question was published before the modern, scientific peer review process existed. Here is some of scientific literature cited in Working Group 1, Chapter One, that falls in that category:
1. Agassiz, L., 1837: Discours d’ouverture sur l’ancienne extension des glaciers. Société Helvétique des Sciences Naturelles, Neufchâtel
2. Buys Ballot, C.H.D., 1872: Suggestions on a Uniform System of Meteorological Observations. Publication No. 37, Royal Netherlands Meteorological Institute, Utrecht, 56 pp.
3. Croll, J., 1890: Climate and Time in Their Geological Relations: A Theory of Secular Changes of the Earth’s Climate, 2nd ed. Appleton, New York, 577 pp.
4. Langley, S.P., 1884: Researches on the Solar Heat and its Absorption by the Earth’s Atmosphere. A Report of the Mount Whitney Expedition. Signal Service Professional Paper 15, Washington, DC.
In the case of the first one listed, the IPCC report reads “Louis Agassiz (1837) developed the hypothesis that Europe had experienced past glacial ages … .” That’s the sum total of the Agassiz reference.
My question is: do the authors of this report card — and everyone here writing in support of this document — believe that it is appropriate to treat a reference of scientific literature like this one (Agassiz) on the exact same basis that contemporary peer-reviewed scientific literature (e.g., Mann) is referenced?

Sphaerica

This would be great if it weren’t complete garbage, because there is no such rule that only peer-reviewed studies may be used. Quite to the contrary, they have explicit rules on how and when non-peer reviewed publications may be used.
Read the PROCEDURES FOR THE PREPARATION, REVIEW, ACCEPTANCE, ADOPTION, APPROVAL AND PUBLICATION OF IPCC REPORTS Adopted at the Fifteenth Session (San Jose, 15- 18 April 1999) amended at the Twentieth Session (Paris, 19-21 February 2003), Twenty-first Session (Vienna, 3 and 6-7 November 2003), and Twenty-Ninth Session (Geneva, 31 August – 4 September 2008) to see the “rubric” which should have been used for “grading”.
And in the future, please do your own research before you publish nonsense, spread disinformation, and waste everybody’s time.
From the rules…

PROCEDURE FOR USING NON-PUBLISHED/NON-PEER-REVIEWED SOURCES IN IPCC REPORTS
Because it is increasingly apparent that materials relevant to IPCC Reports, in particular, information about the experience and practice of the private sector in mitigation and adaptation activities, are found in sources that have not been published or peer-reviewed (e.g., industry journals, internal organisational publications, non-peer reviewed reports or working papers of research institutions, proceedings of workshops etc) the following additional procedures are provided. These have been designed to make all references used in IPCC Reports easily accessible and to ensure that the IPCC process remains open and transparent.

Note that working groups 2 and 3, the ones with all of the “failing” grades because the (gasp) used non-peer reviewed studies, are concerned with mitigation and adaptation activities.

Fred

I still think they’d be more accurate and honest if the used a Beer Review Process.
Couldn’t be any worse than their current process.

Todd Brunner

The IPCC PROCEDURES FOR THE PREPARATION, REVIEW, ACCEPTANCE, ADOPTION, APPROVAL AND PUBLICATION OF IPCC REPORTS has the official work on non-peer reviewed literature in its section entitled PROCEDURE FOR USING NON-PUBLISHED/NON-PEER-REVIEWED SOURCES IN IPCC REPORTS:
“Because it is increasingly apparent that materials relevant to IPCC Reports, in particular, information about the experience and practice of the private sector in mitigation and adaptation activities, are found in sources that have not been published or peer-reviewed (e.g., industry journals, internal organisational publications, non-peer reviewed reports or working papers of research institutions, proceedings of workshops etc) the following additional procedures are provided…”
Read more here http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles-appendix-a.pdf

bubbagyro

I agree with Sphaerica.
Almost all of the articles in the IPCC, peer reviewed or not, are cherry-picked to support their hare-brained hypotheses, or else their membership in the Club Gore and access to lucrative grants are jeopardized. This hypothesis, the earth is warming unprecedentedly, it is caused by humans, and CO2 is the culprit, has been falsified on all three issues in the last 10 or more years in both peer-reviewed and “rogue” journals (today’s rogue journals, with more honorable peers doing the reviewing, I predict will be the bellwether of science in the not-too-distant future) since it has been shown that a conspiracy has kept “skeptical” papers out of the “reputable” journals.

Michael

“When the Sun’s magnetic output is low, winters in Europe tend to be cooler than average – whereas higher output corresponds to warmer winters. That is the conclusion of a new study by physicists in the UK and Germany that looked at the relationship between winter temperatures in England and the strength of the Sun’s magnetic emissions over the last 350 years. The group predicts that, global warming notwithstanding, Europe is likely to continue to experience cold winters for many years to come.”
Sun Blamed for Europe’s Colder Winters
http://physicsworld.com/cws/article/news/42298

James Sexton

Sphaerica (10:26:05) :
“IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.”…..Rajendra Pachauri.
So, Sphaerica, I ask you, is the head of this group that stupid that he doesn’t know the rules governing his work product, or is he so dishonest that he’d wish to mislead the world regarding his work product?
Further, you know, like I know, the words “peer reviewed” has been part and parcel of the alarmist mantra. So much so, the sentiments have echoed the above quote.
In the end, it doesn’t really matter if the rules allowed “grey” material or not. The head of the group (Pachy) either doesn’t know what the rules are or is completely dishonest. This alone and in itself is enough to invalidate the work product. The reasons for the invalidation is exemplified by the many statements that came out of the IPCCs work product that were shown to be so blatantly wrong. (Rapidly melting glaciers in the Himalayan, for example .) They were either too stupid for such an undertaking or grossly dishonest. I suspect both, but will be difficult to prove.

Neil

Al Gore et al will dismiss this research as ‘typical of the sceptics and ‘deniars’. Sadly, this will not get the airtime it deserves, particularly on the ‘impartial’ BBC

From Don Shaw above:
“Max, Did you know that the IPCC recently changed the wording and removed the claim re: only using peer reviewed material?”
I wonder if Sphaerica has the language that was changed in 2008? I’d like to see all the changed language posted for the record.

Summer is coming, the arctic sea ice starts its seasonal decline and the cryosat ready to start beaming down info.
It’s going to be an interesting year.

Sphaerica

James Sexton,
Funny you should mention glaciers. WUWT loves to post articles on weather and anecdotal events (look! it snowed this winter! global warming can’t exist!), but there’s no mention of the glacier that collapsed in peru, destroying fifty homes in the process.
http://www.foxnews.com/scitech/2010/04/13/massive-glacier-triggers-tsunami-lake/
REPLY: I’m so sorry, my all seeing eye is defective today. Please be sure to note weather events on other countries, continents, moons, and planets that WUWT didn’t write about so that you can build up and knock down those ridiculous straw man arguments as well.
-Anthony “one eye” Watts

One of my favorite pages from our report is titled How the IPCC report has been Advertised. It’s a list of quotes – from Dr. Pachauri, from the US Enivironmental Protection Agency, and media outlets ranging from the The Economist to the Associated Press.
All these quotes declare that the IPCC report is based only and solely on peer-reviewed literature. This is the grand myth of the IPCC. And Pachauri is not the only one who has peddled it for years.
It is all well and good to point out that the IPCC has a policy regarding non-peer-reviewed references. But that policy says non-peer-reviewed references are supposed to be “will be followed by a statement that they are not published”. I recall seeing no such statement. Not once in the thousand of IPCC references I personally examined.
That such a policy exists in no way changes the fact that the 100% peer-review myth has been widespread and long-lived – and that it has been used to shut down debate and to squelch dissenting voices.
If the IPCC’s own chairman has spent years mis-characterizing this report what possible explanation could there be? Has he deliberately misled the citizenry of the world – or is he shockingly ignorant of the true nature of his own organization’s primary product? Whatever one’s views on climate change, surely reasonable people see the problem here.
If you can’t even describe your own report accurately, don’t expect folks like me to believe anything else that comes out of your mouth. Nothing Pachauri now says about his organization or his report has any credibility.

vigilantfish

bubbagyro (10:24:34) :
This audit is a great start, but let’s not fall into the “if it’s peer reviewed, it’s OK”. How many of the “peer reviewed” papers are from controversial journals like Science or Nature?
————-
Good point. It’s not just climate science that’s a problem at these journals. Anything with an alarmist spin will be defended and the opposing science is suppressed. A case in point:
Fisheries Research 86 (2007) 1–5
Alan Longhurst
“Doubt and certainty in fishery science: Are we really headed for a global collapse of stocks?”
“…a recent article in ‘Science’ by Worm et al. (2006) was more alarmist: their analysis of a spatially interpolated version of the FAO archives led these authors to conclude that the rate of stock collapses had accelerated throughout the period 1950–2004. They extrapolate this trend in such a way that many readers understood that all exploited fish and invertebrate stocks
could collapse before mid-21st century if remedial action was not taken: even more extreme interpretations of the text were widely reported in the news media….”
” …critical comments concerning the article have been offered to the Science editors by several fishery scientists: one of these suggested that the ‘collapse’ data had been obtained by Worm et al. from FAO archives using unreliable assumptions. But, at the time of writing, in January 2007, no critical comment has yet been published by Science; the only response that I have seen from the editorial staff concern my own (rejected) comments on this apparent failure of peer review, and are meaningless: “We are aware that fisheries science and ocean ecology are contentious subjects, and that areas of legitimate disagreement exist” (Sugden, in litt.).”
After presenting evidence for the flawed use of statistical tools and other problems with Worm’s research, Longhurst comments:
“The results reported by Worm et al., and the very public interpretation of their text, must raise in our minds the issue of doubt and certainty in science (Young, 1951), and the increasing difficulty of certifying that important scientific results have a reasonable probability of being correct. The managing editor of Science, whose response I quoted above, had it completely wrong: if fisheries science and ocean ecology are contentious subjects, it is perhaps only because the process by which contributions are assessed by disinterested reviewers is starting to fail us.
Inevitably we are drawn to consider the apparent failure of peer reviewing in cases like this, and to the essential question: how should we resolve doubt and certainty in science now that individual scientists appear more likely than in the past to espouse social issues that are related to their research? If this is really the case, how may we now ‘certify’ that a scientific conclusion is the result of unbiased investigations and has received a disinterested review? More specifically, what are the consequences of part of the scientific community losing confidence in the review processes in what the press refers to as ‘influential scientific periodicals’?”
I wanted to draw WUWT readers’ attention to the fact that climate science is not an outlier in this problem. I had seen other people here comment that Science and Nature were tainted journals, and thought this might be a good opportunity to give evidence that this is a general problem. Longhurst also critiques ‘faith-based fisheries science’ – another familiar theme here.
(For those who might want to link to this article, it is available through a paywall at Scholar’s Portal.)
bubbagyro’s argument that articles published in these two journals should not be counted as being properly peer-reviewed should be taken seriously.

observa

[snip]