While Oxburgh writes a 5 page book report that most college professors would likely reject due to incompleteness, we have this report from Donna Laframboise of Toronto and a team of citizen auditors. The mission? Determine how much of AR4 met IPCC’s own standards for peer review by reviewing every reference in the report to determine if it comes from peer reviewed literature, grey literature, or if they “simply made stuff up”, like glacier melt dates.
21 of 44 chapters in the United Nations’ Nobel-winning climate bible earned an F on a report card we are releasing today. Forty citizen auditors from 12 countries examined 18,531 sources cited in the report – finding 5,587 to be not peer-reviewed.
Contrary to statements by the chairman of the Intergovernmental Panel on Climate Change (IPCC), the celebrated 2007 report does not rely solely on research published in reputable scientific journals. It also cites press releases, newspaper and magazine clippings, working papers, student theses, discussion papers, and literature published by green advocacy groups. Such material is often called “grey literature.”
We’ve been told this report is the gold standard. We’ve been told it’s 100 percent peer-reviewed science. But thousands of sources cited by this report have not come within a mile of a scientific journal.
Based on the grading system used in US schools, 21 chapters in the IPCC report receive an F (they cite peer-reviewed sources less than 60% of the time), 4 chapters get a D, and 6 get a C. There are also 5 Bs and 8 As.
In November, IPCC chairman Rajendra Pachauri disparaged non-peer-reviewed research in an interview with the Times of India (see the end of the article):
IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.
Between Oxburgh’s failure to write a credible report and this obvious failure of IPCC to follow their own rules, is it any wonder why people are beginning to laugh at the “robustness” oft touted in climate science?
RobJM (14:48:18) :
What proportion are peer review but not scientific, ie, based on the guesswork of computer models?
Really, you cannot reason like that. Computer models are part of
science. Period.
You could possibly maintain that IPCC use of models is guesswork,
meaning that they apply the method in a bad way .
I don’t agree with that conclusion.
mikael pihlström (15:12:27) :
The critical responses were not mine, but belonged to a scientist (Alan Longhurst). They were in the form of an article. Please note that Longhurst echoes the fears of other scientists that the peer-review system is failing, first in the failure to get articles properly reviewed in the first place, and then second, in the refusal of scientific journals such as Science and Nature to publish the criticisms of published work by highly qualified scientists. Ironically, Longhurst is very concerned about overfishing. He just does not want public confidence in science to be eroded by distortions and exaggerations. Science requires a dialogue, not a diktat, if it is to prosper, have relevance, and more importantly, not become a tool for political abuse.
If there were any clearer evidence that this whole thing is one great “groupthink” mistake, it is the way all these reports are full of bogus science. In proper science, there is proper scrutiny and stupid stuff like this would never have found its way into the reports.
Instead, not only is their dodgy dossier full of bogus science, they even are stupid enough to try to defend their own failings. And by doing so, not only do we know that their reports are dodgy, we also know that the whole system of review and oversight is also highly dodgy: the whole system is corrupt, and so it is inevitable that anything produced by this corrupt system is also corrupt.
@ur momisugly Sphaerica (11:19:01) :
James Sexton,
Funny you should mention glaciers. WUWT loves to post articles on weather and anecdotal events (look! it snowed this winter! global warming can’t exist!), but there’s no mention of the glacier that collapsed in peru, destroying fifty homes in the process.
Yes, I’ve read the story, calling it even anecdotal is a stretch. Unlike alarmists, I know of no instance where someone here claimed glaciers are or should be static. You simply read a story about glacial movement, and a bunch of simpletons that grouped by it expecting the ice never to move or break off.
BTW, is “look! it snowed this winter! global warming can’t exist!” anything akin to saying “Look! We’ve had a very warm March!!! This is proof of the existence of global warming!!!” Well, that and glacial movement.
Didn’t see this in any previous comments. My apologies if I’m simply repeating known information.
IPCC chapters receiving a grade of A:
Chapter 2: Changes in Atmospheric Constituents and in Radiative Forcing
Chapter 3: Observations: Surface and Atmospheric Climate Change
Chapter 5: Observations: Oceanic Climate Change and Sea Level
Chapter 6: Palaeoclimate
Chapter 8: Climate Models and their Evaluation
Chapter 7: Couplings Between Changes in the Climate System and Biogeochemistry
Chapter 9: Understanding and Attributing Climate Change
Chapter 10: Global Climate Projections
“Sphaerica (11:19:01) :
James Sexton,
Funny you should mention glaciers. WUWT loves to post articles on weather and anecdotal events (look! it snowed this winter! global warming can’t exist!), but there’s no mention of the glacier that collapsed in peru, destroying fifty homes in the process.
http://www.foxnews.com/scitech/2010/04/13/massive-glacier-triggers-tsunami-lake/”
When I read this story yesterday the following sentence stood out.
“Patricio Vaderrama, a Peruvian glacier expert, said: “The tsunami wave breached the lake’s levees, which are 23m high — meaning the wave was 23m high,” or 75 feet. ”
I guess this guy must have spent his life taking showers and never made waves in the bath.
It is getting to the surreal stage. Is it just me, or does every report in the MSM that has anything to do with global warming include stupid exaggerations from so called experts? BTW, this report is linked to FOX, but actually is a Reuters story.
Donna Laframboise (11:30:28) :
Thank you very much, both for the report, and this sensible and reasoned riposte.
But who shall audit the auditors?
I picked one of the low-scoring chapters [24% or 12/50] and double checked.
http://www.noconsensus.org/ipcc-audit/2007/WG3chapter1-A.html
Now 12 of the references classified as ‘not reviewed’ are to other IPCC reports or chapters. These may not be academic journals, but they are the most reviewed scientific documents on the planet – going through 2 public drafts and a final review by politicians. To place them in an ‘unreviewed’ category is laughable.
I then took a reference classified as ‘not peer-reviewed’ and looked it up:
Gritsevsky, A., and N. Nakicenovic, 2002: Modelling uncertainty of induced technological change. In: Technological change and the environment, A. Grubler, N. Nakicenovic, W.D. Nordhaus, (eds.). Resources for the Future, pp. 251-279.
Certainly, the cited article is a book chapter, and books are not peer-reviewed, but somehow our ‘citizen auditors’ managed to miss the fact that the it is a reprint from the very much peer-reviewed Energy policy, and it has been cited an impressive 131 times in the literature. It took me about 2 minutes with Google Scholar to discover this
http://scholar.google.co.uk/scholar?q=Modelling+uncertainty+of+induced+technological+change&hl=en&btnG=Search&as_sdt=2001&as_sdtp=on
Amateur auditing? So the figure derived by this ‘report’ for this Chapter is at best, less than 50% of the actual number. Is there any reason to suppose the rest is any more accurate?
Sphaerica (11:19:01) :
Very little gets past WUWT, but not everything gets turned into a page. If you write something, perhaps Anthony will make you a guest poster.
Over in Tips & Notes I see references on April 13th at 06:18 and on the 14th at 00:45. Try http://wattsupwiththat.com/tips-notes-to-wuwt-5/#comment-367599
There are likely other references in in other pages, but those all belong in Tips & Notes too.
And Pachauri won’t leave, even if he is fired!
http://motls.blogspot.com/2010/04/pachauri-wont-leave-even-if-he-is-fired.html
@ur momisugly magicjava (16:13:58) :
“Didn’t see this in any previous comments. My apologies if I’m simply repeating known information.
IPCC chapters receiving a grade of A:………………”
Yes, the grading was based upon how many peer-reviewed references were used vs. “gray” material. So, I’m a bit confused about your post. Is it that peer-reviewed material only matters in your mentioned chapters? Or that it doesn’t matter at all? Or perhaps only those that are important to you? I’m not sure.
What I took away from the article……. A quote from our favorite train engineer, Pachy(man in charge of the IPCC report) “IPCC studies only peer-review science. Let someone publish the data in a decent credible publication. I am sure IPCC would then accept it, otherwise we can just throw it into the dustbin.”
Pachy was being either blatantly stupid or blatantly dishonest. Either way, the man in charge of the report was/is obviously unfit. His entire work product is invalidated, regardless of the few “A”s they got based on the few chapters that included some peer-reviewed studies.
Moreover, if he truly believed that only peer-reviewed material was being used, then his subordinates were picking and choosing the material they wished in the report and ignoring other material without his knowledge. If he did have knowledge then he has shown a propensity to mislead and misinform the public. However you wish to perceive Pachy, his body of work is invalidated by his own statement. (Or one could look at the fantastic claims made by the report that has already been discredited.)
Congratulations to Ms. Laframboise and her team. This is a very significant bit of work.
vigilantfish (15:55:25) :
Please note that Longhurst echoes the fears of other scientists that the peer-review system is failing, first in the failure to get articles properly reviewed in the first place, and then second, in the refusal of scientific journals such as Science and Nature to publish the criticisms of published work by highly qualified scientists.
Dr T G Watkins (15:28:50) :
Do you really think that one’s mother tongue determines success in publishing in journals? Actually, I probably agree which says everything about so called peer review.
Peer-review system is indeed failing; for sure it is overburdened by the
shee volume of articles submitted. Still doubtful on Science & Nature bias though. There is of course nothing magic with S & N, only we have made
them so important for impact points etc. Peer-review was a bad system all along, but the best we have. As TCW says, it is a minimum standard. The
slight(?) edge of being a native English speaker cannot perhaps be blamed on peer-review, since a native speaker objectively tends to express himself
better. The effect of US/UK being dominant in excellence is hard to separate
from any language effect.
Phil Clarke (16:30:20) :
Amateur auditing? So the figure derived by this ‘report’ for this Chapter is at best, less than 50% of the actual number. Is there any reason to suppose the rest is any more accurate?
The value of this panel is more participation? and raising interest for the
issues? If the whole venture has been started with an aggressive agenda
in the shadows, the volunteers are put into an unfair spotlight. But, you
are right it would be logical to audit the auditors.
If tomorrow morning i presented the IPCC with hard evidence that falsified AGW, would they disband?
Anser: NO
Why? Answer: A tangled web of vested interests (‘research’ funding, salaries, Pachauri, Gore, Oxburgh – insert others – …….)
If and when AGW collapses it is going to be very embarrasing indeed.
For those who wish to audit the auditors and re-do the work… Please! Feel free to do so. The checks would be most welcome.
For those who noted that some articles were re-printed as chapters of a book — Indeed it was noted by some auditors. The issue then was to read the book, read the paper and see if any conclusions or data had been changed. This would be an onerous task. Would it not be simpler for the author(s) or editors(s) to state that the article was included in an unchanged fashion? Indeed in some cases an article re-published in book form was included, however the task of verification proved quite onerous. Again, feel free to re-audit and question the work. Is this not the very basis of the scientific method? Why would an auditor complain that you rechecked the work? Simply do as Donna did and state your methods, assumptions and resolution processes. We look forward to your articles and notes. You will get your work peer-reviewed — right?
I will say that to me it proved the point that not even the IPCC and its spokespeople truly understood the nature of their own work. Certainly consumer laws protect us when companies misrepresent and over-sell their work. Perhaps the same mechanism can be used here. Should Pachouri be prosecuted for overselling his “product”? You tell us!
Donna Laframboise
I, also, thank you , for an excellent, well written, report.
It would be such a positive step forward if, somehow, someone, would initiate a class-action lawsuit against the IPCC and the lead scientists, for fraud.
[ie accepting government/taxpayers funds for fraudulent scientific reports.]
Steven K (12:07:29) said it beautifully. Thanks Steven.
For those who wanted to know the chapter headings, I hope this table formats readably:
Working Group 3, Chapter 12 F 37 414 63 Sustainable Development and mitigation
Working Group 2, Chapter 13 F 40 354 60 Latin America
Working Group 2, Chapter 11 F 42 372 58 Australia and New Zealand
Working Group 3, Chapter 2 F 46 302 54 Framing Issues
Working Group 2, Chapter 7 F 46 244 54 Industry, Settlement and Society
Working Group 3, Chapter 10 F 47 191 53 Waste management
Working Group 3, Chapter 13 F 48 491 52 Policies, instruments, and co-operative arrangements
Working Group 2, Chapter 9 F 53 361 47 Africa
Working Group 2, Chapter 17 F 53 275 47 Assessment of Adaptation Practices, Options, Constraints and Capacity
Working Group 2, Chapter 20 F 53 220 47 Perspectives on Climate Change and Sustainability
Working Group 2, Chapter 18 F 56 270 44 Inter-Relationships Between Adaptation and Mitigation
Working Group 3, Chapter 9 F 56 229 44 Forestry
Working Group 2, Chapter 14 F 58 562 42 North America
Working Group 2, Chapter 16 F 58 194 42 Small Islands
Working Group 3, Chapter 3 F 58 358 42 Issues related to mitigation in the long-term context
Working Group 2, Chapter 10 F 59 391 41 Asia
Working Group 3, Chapter 11 D 62 330 38 Mitigation from a cross-sectoral perspective
Working Group 2, Chapter 6 D 65 443 35 Coastal Systems and Low-Lying Areas
Working Group 2, Chapter 2 D 67 374 33 New Assesment Methods and the Characterisation of Future Conditions
Working Group 2, Chapter 15 D 68 361 32 Polar Regions (Arctic and Antarctic)
Working Group 2, Chapter 5 C 70 444 30 Food, Fibre, and Forest Products
Working Group 2, Chapter 3 C 71 377 29 Fresh Water Resources and their Management
Working Group 2, Chapter 12 C 71 633 29 Europe
Working Group 2, Chapter 19 C 71 273 29 Assessing Key Vulnerabilities and the Risk from Climate Change
Working Group 2, Chapter 8 C 75 557 25 Human Health
Working Group 3, Chapter 8 C 77 317 23 Agriculture
Working Group 1, Chapter 1 B 80 264 20 Historical Overview of Climate Change Science
Working Group 2, Chapter 4 B 82 917 18 Ecosystems, their Properties, Goods and Services
Working Group 1, Chapter 4 B 85 257 15 Observations: Changes in Snow, Ice and Frozen Ground
Working Group 2, Chapter 1 B 86 650 14 Assessment of Observed Changes and Responses in Natural and Managed Systems
Working Group 1, Chapter 11 B 89 609 11 Regional Climate Projections
Working Group 1, Chapter 6 A 93 609 7 Palaeoclimate
Working Group 1, Chapter 8 A 94 686 6 Climate Models and their Evaluation
Working Group 1, Chapter 9 A 94 535 6 Understanding and Attributing Climate Change
Working Group 1, Chapter 2 A 95 759 5 Changes in Atmospheric Constituents and in Radiative Forcing
Working Group 1, Chapter 10 A 95 545 5 Global Climate Projections
Working Group 1, Chapter 3 A 96 804 4 Observations: Surface and Atmospheric Climate Change
Working Group 1, Chapter 5 A 96 289 4 Observations: Oceanic Climate Change and Sea Level
Working Group 1, Chapter 7 A 96 869 4 Couplings Between Changes in the Climate System and Biogeochemistry
mikael pihlström (15:42:38) :
Really, you cannot reason like that. Computer models are part of
science. Period.
Terrific. Another diktat from the hive-mind to go along with “The Science is Settled.” The models are nothing more than the articulation of a theory – a tool to generate hypotheses that can be tested through empirical observation. In the case of the climate models, the predictions are mostly so far in the future that many of us won’t live to see them validated or falsified. The few predictions that have been generated for the near term have failed badly. Sure, computer models are part of science, in the same way that a theory is part of science, but the existence of the theory is not evidence or proof of anything. Neither are the outputs of the models.
Even if the predictions of the models were validated, that still would not mean the science is settled. Incorrect theory can and has generated lots of empirically verifiable (and verified!) predictions. Ptolemaic astronomy generated predictions that were verified by events for over a thousand years. The problem is that the sun really and truly does not orbit around the earth on crystalline spheres.
The price of investing in “post normal” Science…..
http://thehill.com/blogs/hillicon-valley/technology/92117-neil-armstrong-criticizes-obama-space-plan
Armstrong is a humble man who rarely makes public statements.
He is to be applauded.
Oops – sorry Mod – I missed pasting these 5 lines to the top of my previously posted table
Working Group 3, Chapter 4 F 15 360 85 Energy supply
Working Group 3, Chapter 7 F 23 352 77 Manufacturing and process industries
Working Group 3, Chapter 1 F 24 50 76 Introduction
Working Group 3, Chapter 5 F 27 260 73 Transport and its infrastructure
Working Group 3, Chapter 6 F 29 379 71 Residential and commercial buildings
Todd Brunner (12:29:39) :
[regarding IPCC rule that non-peer-reviewed material is to be clearly identified as such in the references by use of the word “Unpublished”]
“Just do a quick search for the term “unpublished” in IPCC AR4 and you will find several references labeled as such…unpublished references. Here’s one example page […]”
I happen to have all 44 chapter references as Word docs sittng on my h/d. So, I’ve just completed such a search. Here are the results:
WG2 CH5 = 1
WG2 CH6 = 1
WG2 CH9 = 1
WG2 CH10 = 1
WG2 CH11 = 1
WG3 CH5 = 1
WOW! 6 instances of “Unpublished”. In the context of 18,631, I would not consider a grand total of 6 to be “several”. YMMV.
Apart from the fact that “Unpublished” – perhaps not unlike “entirely“solely“, “only” and “all” (even “trick”, come to think of it) – seems to have taken on a whole new meaning in IPCC-speak, I don’t find 6 out of 5,587 to be particularly indicative of, well, anything. Except perhaps that the IPCC doesn’t follow its own rules.
Is it your contention that the other 5,581 references that were not sourced to a publication which indicates that material contained therein has been peer-reviewed must be reclassified as “peer-reviewed” because they do not contain the word “Unpublished”?! Or do you contend that the IPCC has .. uh … redefined “peer-reviewed” journal, without letting the rest of the world know?
As one of the auditors involved in Donna’s project, what I found quite interesting about the results is that numbers in our raw data are quite close to the preliminary numbers resulting from work undertaken by Peter Bobroff in Australia. He’s developing a program that will make such research much faster and accessible to all in the not too distant future. My understanding is that this program will enable the user to very quickly get answers to some of the questions posed elsewhere in the comments. Stay tuned!
Considering the lack of consistency in the IPCC’s citation of source material – which presents even more of a challenge to a computer program than to three pairs of eyeballs – I’d say that Bobroff’s results were remarkably close to ours, as you will see for yourself when you view our raw data – because I’ve included his, for comparison purposes. All declines are in full view!
and no FOI request required 😉
http://tinyurl.com/citizenauditdata
But the bottom line is that we now have an unequivocal NO in response to the question:
James Sexton:
“To be perfectly honest, I really don’t care about whether the science backs up climate change or not.”
That pretty much says it all right there.
there will be auditing of the auditors by the ipcc report originators and by every warmer in world as well as every media hack on the CAGW bandwagon.
@WillR (17:31:21) :
For those who wish to audit the auditors and re-do the work… Please! Feel free to do so. The checks would be most welcome
”
Thank you WillR
The audit was undertaken with good intentions and was for the most part a laborious and thankless task, though it did on occasions give us some insight into the science. In many cases this was a positive experience.
Sure, there were cases when we missed out “reprints” or had to make decisions about historical papers that pre-dated the modern peer-review process, but we believe we did this in as fair and transparent manner as possible.
If anyone wants to review or do further work on this then I am sure Donna would be happy to engage.