It has been an interesting couple of days. Today yet another scientist has come forward with a press release saying that not only did their audit of IPCC forecasting procedures and found that they “violated 72 scientific principles of forecasting”, but that “The models were not intended as forecasting models and they have not been validated for that purpose.” This organization should know, they certify forecasters for many disciplines and in conjunction with John Hopkins University if Washington, DC, offer a Certificate of Forecasting Practice. The story below originally appeared in the blog of Australian Dr. Jennifer Marohasy. It is reprinted below, with with some pictures and links added for WUWT readers. – Anthony
J. Scott Armstrong, founder of the International Journal of Forecasting
Guest post by Jennifer Marohasy
YESTERDAY, a former chief at NASA, Dr John S. Theon, slammed the computer models used to determine future climate claiming they are not scientific in part because the modellers have “resisted making their work transparent so that it can be replicated independently by other scientists”. [1]
Today, a founder of the International Journal of Forecasting, Journal of Forecasting, International Institute of Forecasters, and International Symposium on Forecasting, and the author of Long-range Forecasting (1978, 1985), the Principles of Forecasting Handbook, and over 70 papers on forecasting, Dr J. Scott Armstrong, tabled a statement declaring that the forecasting process used by the Intergovernmental Panel on Climate Change (IPCC) lacks a scientific basis. [2]
What these two authorities, Drs Theon and Armstrong, are independently and explicitly stating is that the computer models underpinning the work of many scientific institutions concerned with global warming, including Australia’s CSIRO, are fundamentally flawed.
In today’s statement, made with economist Kesten Green, Dr Armstrong provides the following eight reasons as to why the current IPCC computer models lack a scientific basis:
1. No scientific forecasts of the changes in the Earth’s climate.
Currently, the only forecasts are those based on the opinions of some scientists. Computer modeling was used to create scenarios (i.e., stories) to represent the scientists’ opinions about what might happen. The models were not intended as forecasting models (Trenberth 2007) and they have not been validated for that purpose. Since the publication of our paper, no one has provided evidence to refute our claim that there are no scientific forecasts to support global warming.
We conducted an audit of the procedures described in the IPCC report and found that they clearly violated 72 scientific principles of forecasting (Green and Armstrong 2008). (No justification was provided for any of these violations.) For important forecasts, we can see no reason why any principle should be violated. We draw analogies to flying an aircraft or building a bridge or performing heart surgery—given the potential cost of errors, it is not permissible to violate principles.
2. Improper peer review process.
To our knowledge, papers claiming to forecast global warming have not been subject to peer review by experts in scientific forecasting.
3. Complexity and uncertainty of climate render expert opinions invalid for forecasting.
Expert opinions are an inappropriate forecasting method in situations that involve high complexity and high uncertainty. This conclusion is based on over eight decades of research. Armstrong (1978) provided a review of the evidence and this was supported by Tetlock’s (2005) study that involved 82,361 forecasts by 284 experts over two decades.
Long-term climate changes are highly complex due to the many factors that affect climate and to their interactions. Uncertainty about long-term climate changes is high due to a lack of good knowledge about such things as:
a) causes of climate change,
b) direction, lag time, and effect size of causal factors related to climate change,
c) effects of changing temperatures, and
d) costs and benefits of alternative actions to deal with climate changes (e.g., CO2 markets).
Given these conditions, expert opinions are not appropriate for long-term climate predictions.
4. Forecasts are needed for the effects of climate change.
Even if it were possible to forecast climate changes, it would still be necessary to forecast the effects of climate changes. In other words, in what ways might the effects be beneficial or harmful? Here again, we have been unable to find any scientific forecasts—as opposed to speculation—despite our appeals for such studies.
We addressed this issue with respect to studies involving the possible classification of polar bears as threatened or endangered (Armstrong, Green, and Soon 2008). In our audits of two key papers to support the polar bear listing, 41 principles were clearly violated by the authors of one paper and 61 by the authors of the other. It is not proper from a scientific or from a practical viewpoint to violate any principles. Again, there was no sign that the forecasters realized that they were making mistakes.
5. Forecasts are needed of the costs and benefits of alternative actions that might be taken to combat climate change.
Assuming that climate change could be accurately forecast, it would be necessary to forecast the costs and benefits of actions taken to reduce harmful effects, and to compare the net benefit with other feasible policies including taking no action. Here again we have been unable to find any scientific forecasts despite our appeals for such studies.
6. To justify using a climate forecasting model, one would need to test it against a relevant naïve model.
We used the Forecasting Method Selection Tree to help determine which method is most appropriate for forecasting long-term climate change. A copy of the Tree is attached as Appendix 1. It is drawn from comparative empirical studies from all areas of forecasting. It suggests that extrapolation is appropriate, and we chose a naïve (no change) model as an appropriate benchmark. A forecasting model should not be used unless it can be shown to provide forecasts that are more accurate than those from this naïve model, as it would otherwise increase error. In Green, Armstrong and Soon (2008), we show that the mean absolute error of 108 naïve forecasts for 50 years in the future was 0.24°C.
7. The climate system is stable.
To assess stability, we examined the errors from naïve forecasts for up to 100 years into the future. Using the U.K. Met Office Hadley Centre’s data, we started with 1850 and used that year’s average temperature as our forecast for the next 100 years. We then calculated the errors for each forecast horizon from 1 to 100. We repeated the process using the average temperature in 1851 as our naïve forecast for the next 100 years, and so on. This “successive updating” continued until year 2006, when we forecasted a single year ahead. This provided 157 one-year-ahead forecasts, 156 two-year-ahead and so on to 58 100-year-ahead forecasts.
We then examined how many forecasts were further than 0.5°C from the observed value. Fewer than 13% of forecasts of up to 65-years-ahead had absolute errors larger than 0.5°C. For longer horizons, fewer than 33% had absolute errors larger than 0.5°C. Given the remarkable stability of global mean temperature, it is unlikely that there would be any practical benefits from a forecasting method that provided more accurate forecasts.
8. Be conservative and avoid the precautionary principle.
One of the primary scientific principles in forecasting is to be conservative in the darkness of uncertainty. This principle also argues for the use of the naive no-change extrapolation. Some have argued for the precautionary principle as a way to be conservative. It is a political, not a scientific principle. As we explain in our essay in Appendix 2, it is actually an anti-scientific principle in that it attempts to make decisions without using rational analyses. Instead, cost/benefit analyses are appropriate given the available evidence which suggests that temperature is just as likely to go up as down. However, these analyses should be supported by scientific forecasts.
The reach of these models is extraordinary, for example, the CSIRO models are currently being used in Australia to determine water allocations for farmers and to justify the need for an Emissions Trading Scheme (ETS) – the most far-reaching of possible economic interventions. Yet, according to Dr Armstrong, these same models violate 72 scientific principles.
********************
1. Marc Morano, James Hansen’s Former NASA Supervisor Declares Himself a Skeptic, January 27,2009. http://epw.senate.gov/public/index.cfm?FuseAction=Minority.Blogs&ContentRecord_id=1a5e6e32-802a-23ad-40ed-ecd53cd3d320
2. “Analysis of the U.S. Environmental Protection Agency’s Advanced Notice of Proposed Rulemaking for Greenhouse Gases”, Drs. J. Scott Armstrong and Kesten C. Green a statement prepared for US Senator Inhofe for an analysis of the US EPA’s proposed policies for greenhouse gases. http://theclimatebet.com
Sponsored IT training links:
Get guaranteed success in 312-50 exam in first try using incredible 642-374 dumps and other 310-200 training resources prepared by experts.
foinavon (08:02:51) :
First, of course, because a climate model, and it’s output, is not a “forecast”!
Call it what you will, when the story line is that the world will continue to warm and the basis is a climate model, then that sounds like a forecast to me.
Additionally we’d have to know what these “principles of forecasting” are. Can you provide a link/citation?
No, but then I’ve not looked yet.
I’ve had a look at Armstrong’s published papers, and these principles seem not to be in the scientific arena.
I’m interested, what would be classed as IN the scientific arena? This argument is a non starter as we can pick rules that suit our cause all day long.
Perhaps he’s put them on a website somewhere? Or written a book/pamphlet?
Perhaps, but as I say, I’ve not looked yet
It seems rather extraordinary to be discussing this issue without anyone seeming to know what the “principles of forecasting” actually are,
I agree, but here we are. I doubt a lot of people on here from either side of the camp know what the “principles of forecasting” are including you and I. I guess that is the reason why we are debating it.
or whether they have any relevance to climate-related science, let alone climate-related policy…
Again, I’m interested, what is “relevent” and would be classed as IN the scientific arena? This argument is a non starter as we can pick rules that suit our cause all day long.
G Alston (08:48:09)
First, the climate models and their output are not “forecasts”.
Second, while climate models are certainly part of a “mathematical discipline”, the “forecast principles” are not mathematical in any sense. I’ve finally tracked some of these “forecast principles” down (in a magazine called Energy and Environment vol 18 no 7-8..they don’t seem to be described at all in the peer-reviewed scientific literature) and they’re completely non-mathematical. They are of the form (here’s some of them):
“do not address uncertainty in a traditional (unstructured” group meeting”
“obtain forecasts from heterogeneous experts”
“describe conditions associated with the forecasting problem”
“identify possible outcomes prior to making forecast”
and so on. How this sort of thing is relevant to a scientific paper describing implementation and analysis of some runs of a climate model is rather difficult to see! Why not download a scientific paper on climate modelling and consider yourself how these “principles” might help…
I would also say “not really”! It’s nothing like a “spreadsheet”. It’s a computer program which analysis the evolution of time series of many elements of the climate system and their interactions, each of which is incorporated in the model by mathematical formulae based on our physical understanding, and with the physical elements parameterized using empirical analysis of real world measurements.
So it is scientific analysis. The extent to which a model is able to reproduce observations either in a hindcast (of the sort that Simon Evans linked to [see Simon Evans (05:57:41)], or predictively, as in the sort described rather successfully by Hansen ((sorry about that, but it’s one that I could find quickly that’s directly linkable!) is an indication of our ability to correctly encapsulate the contributions to the evolution of the climate system wthin a computational model and to paramaterize this successfully. So climate models are a very useful contribution to the development of our understanding.
http://pubs.giss.nasa.gov/docs/2006/2006_Hansen_etal_1.pdf
That’s a reasonable thought. A model should be thought of as an encapsulation of our current knowledge of the subject being modelled, as well as our ability to parameterize this (or its components) adequately. I think that most modelers would agree with that. On the other hand, while a model is not “data” it is comprised of data (that informs the modeller of the parameterizations), and provides a means of assessing how the climate will evolve given a particular emission scenario (for example), and within our current understanding of the climate system and its responses.
Smokey (09:15:15) :
TonyB:
‘Ilk’ means merely ‘of the same’ it is in no way derogatary.
Not so in the U.S., where ‘ilk’ is deemed to be mildly derogatory [eg, “I want you and your ilk off my property.”]
Another problem word is to “table.” IIRC, in GB tabled means to put something on the table for discussion. In the U.S. it means to postpone. I’m not sure which way Canada leans.
This is primarily related to usage. For instance, someone can put an issue on the table or take it off the table. You can belong to an ilk that you may believe is something positive or something negative.
foinavon:
First, of course, because a climate model, and it’s output, is not a “forecast”!
Ok, the “output” is only made to appear to be a forecast. Just as the U.N.’s Intergovernmental Panel on Climate Change does not really study “climate change”.
Agreed, all of this is really a massive propaganda operation.
John W. (09:15:07) :
http://www.forecastingprinciples.com/handbook.html
It’s the link to a book titled, oddly enough, “Principles of Forecasting: A Handbook for Researchers and Practitioners.” Equally odd, I found the link on a website named “forecastingprinciple.com.”
Many thanks for that, saved me some work!! 🙂
“‘Ilk’ means merely ‘of the same’ it is in no way derogatary. ”
Gonna have to recuse myself on the derogatory call but I hear this word nearly every drive I take. With ‘road-rage’ issues can’t listen to NPR or National News so its ‘Sports Radio’. ‘Nuff said.
G Alston (08:48:09) :
This is the one thing that bugs me about climate science, this perception by AGW proponents that model outputs are data. They’re not.
Indeed. Those of us who have to use modeling and simulation in the course of work that is measured against real world outcomes make a clear distinction:
“Data” consists of measurements taken in the course of observation, test, and/or experimentation.
“Information” consists of outputs from computer programs.
The distinction is one of the principle reasons behind the concept of “Independent Validadtion and Verification.”
John W. (09:15:07)
I agree with the top article and with Dr Trenberth referred to there that a climate model and its output are not “forecasts” in the common sense of the word. After all we don’t really know what the emission scenarios are going to be. So climate models are explorations of the evolution of the climate system under a range of different emission scenarios parameterized according to our current understanding of the physical elements of the climate system and its interactions.
There’s a lot of hoo-haw about climate models on all sides. In reality, the concerns of scientists and policymakers are not derived from the results of models. The concerns come from our basic understanding of the climate system and the consequences of enhancing the greenhouse effect. This understanding goes into the development and implementation of climate models by way of parameterization, but nothing really comes out that we didn’t know already, except that we might be able to assign a more generalized regional scale distribution of warming-related changes…and perhaps find the evolution of non-predicted phenomena that we could explore in relation to likely veracity…that sort of thing.
So for example that fact that scientists might predict that with an emission scenario leading to a CO2 concentration of 600 ppm by 2100, the earth will likely be 2 oC warmer than now, with a sea level rise of such-and-such, is a conclusion that is essentially independent of models.
In scientific fields, one expects the important information to be sourced in the scientific literature. So (as I would expect form a student, although it’s difficult to keep them away from Wikipedia and such like!) I looked in the scientific data base. One would presume that Armstrong would have published this stuff in the scientific literature. But no.
As you point out it is in a handbook. I also found some information in a magazine. If one is attempting to develop “principles” of any note, one might expect this to be addressed in the peer-review scientific literature!
So the logical extension of this is that if weather persists so does climate, and “persistence” will predict that next year will be much like the last.
Until it isn’t.
The 20th century climate is a series of 20-30 cycles.
Cooling then warming then cooling again. Dustbowl and drought in the 30’s, cold enough to make people fear an ice age in the 70’s. , warm in the 90’s and cooling the 2007/2008 years.
If you use the previous year to predict the next you are right until the cycle switches, then you are wrong for a year or so and then you are right again for 10 or 20 years.
Climate models that don’t predict the ups and downs are useless.
The IPCC models are useless. They didn’t predict 2007/2008.
Luis Dias (06:40:00) :
“What astounds me is the sheer gullibility of almost every single one reader of this blog.”
I went over and read the article at RC you referenced. Did you somehow miss all the Ad Hominem attacks and strawman arguments? While I still have no overall feelings on this report I can only wonder how “gullible” you are.
Chris:
By the way, Al Gore is:
A “C” level undergrad
A divinity school flunkout
A claimant to the invention of the internet
A failed politician
A global warming billionaire.
Is America a great place, or what? All you gotta do is find a scam.
I’m none of these things, and I don’t think we need to worry about CO2.
It kind of tortures my mind to hear that a report that clearly predicts temperature changes over the next century is not a “forecast”. Sort of reminds me of trying to define “is”. And this criticism that the IPCC report is not a forecast but only a scenario was, in fact, forecast by Armstong and addressed directly in his paper. Just for easy reference, here is the what is said about it:
Does the IPCC report provide climate forecasts?
Trenberth (2007) and others have claimed that the IPCC does not provide forecasts but rather presents “scenarios” or “projections.” As best as we can tell, these terms are used by the IPCC authors to indicate that they provide “conditional forecasts.”
Presumably the IPCC authors hope that readers, especially policy makers, will find at
least one of their conditional forecast series plausible and will act as if it will come
true if no action is taken. As it happens, the word “forecast” and its derivatives
occurred 37 times, and “predict” and its derivatives occurred 90 times in the body of 1006 Energy & Environment · Vol. 18, No. 7+8, 2007
Chapter 8. Recall also that most of our respondents (29 of whom were IPCC authors
or reviewers) nominated the IPCC report as the most credible source of forecasts (not “scenarios” or “projections”) of global average temperature. We conclude that the IPCC does provide forecasts.
This is the key. If you haven’t waited to see if the results are correct then you can’t make any pronouncements about the accuracy of your prediction/forecast/scenario. So, you have to find another way.
Paul Shanahan (09:37:52)
The question I was addressing relates to why climate modellers don’t use these “principles of forcasting” when doing their modelling. The answer is because the model is not a forecast. It’s an exploration of the evolution of a climate system to explore a particular chosen phenomenon (e.g. a possible greenhouse gas emission scenario or range of these).
Now that we’ve seen what the “principles of forecasting” actually amount to (see examples in my post [foinavon (09:40:25)] and Armstrong’s magazine article cited there), it’s easy to see why these aren’t used by climate modellers in their work.
If someone devised some “principles” of any note, that might be designed to address scientific methodologies, for example, one would expect these to be published in the peer-reviewed scientific literature, where their relevance, applicability and usefulness can be assessed.
One can’t just devises a set of “principles”, put these in a handbook, and then consider that these can be used to judge the methodologies of researchers! As you say, such a strategy allows the proposer to “pick rules that suit our cause all day long”. And I agree that this approach is likely to be considered a “non starter” in serious scientific circles…
I’ve been snipped for name calling, but the paragraph was more than that. I’ll repeat myself, now more mr. Nicey style:
What I find astounding in this blog is the ability to call one self “Skeptic” while parading everything one can against GW, without any *ahem* SKEPTIC *ahem* filter against what is pure BULL. I linked to RC’s post and it kills this nonsensical article which is already almost 2 years old!!
It’s a rehash of a non-story which is being bought by gullible people as a “sign” that “change is blowing in the wind” or something to that effect.
And yes, to hilarity:
Did you somehow miss all the Ad Hominem attacks and strawman arguments?
Just like the one you just did? Yes I did. All I watched was RC destroying the entire argument and showing how ridiculous the IIF’s study really is. If they compound it by calling “Stupid” what is definitely “Stupid”, I can hardly see that as “Ad Hominem”.
A duck is a duck is a duck.
And this is a non-story.
REPLY: Opinion noted. But I’m not going back and forth on your opinion any further. You don’t like it, we get it. -Anthony
This is probably OT but getting on for 40 years ago, as a young man, I knew and worked with an old Ghillie in highland Perthshire. He had a reputation for being a bit of a weather forecasting guru. Apart from a couple grandad rules such as “if the mist is rising like smoke from the hills in the morning then it will rain by the afternoon”; He had a very simple method of telling what the weather would be like tomorrow. He just described today’s weather. If the wind was moving from S&W to N&E then it would be colder and drier and equally the reverse was true. Even in Scotland renown for four seasons in one day this was a pretty reliable forecasting method.
I suppose it could be described as naive as well
Here’s another useless forecasting example.
I’m flying out to Washington DC next week (Sunday through Friday), so yesterday I took a snapshot of the 10-day “forecast” from weather.com.
Today I took the same snapshot. Every single day is now different, some drastically so. I understand that GCMs are not the same as weather models, but, how confident am I supposed to be in these things with something so unpredictable?
Here are the two snapshots:
http://jalberts.net/images/DC_01-28-2009.jpg
http://jalberts.net/images/DC_01-29-2009.jpg
I’ll take another one tomorrow and the next day…
“Is America a great place, or what? All you gotta do is find a scam.”
Word. Kinda makes it hard for us nominally decent folk, avarice-wise.
The IPCC makes up their own rules and “home-made guidelines” too, your point? – Anthony
My point is that the IPCC doesn’t have to agree with “rules” that are made up by third parties. Until those guidelines are accepted in mainstream, the sheer gall of these guys of judging models by their own made-up guidelines and just declaring the models unscientific is completely mind-boggling and a total red-herring.
Had they judged models according to their own set of principles and then just concluded that they violated them, it would be fair and balanced, because alas! they do not “own” copyright terms on how is science “defined”, bokay?
RC’s post is more substantive on the method itself, which is horrible. If this is science, then I’m Santa. Wanna’ present this fall, Anthony? Then don’t snip me again ;).
REPLY: Thanks Luis, following your own advice on rules, I’ll snip as needed. Personally I think you are too closed minded on the issue and give RC too much credit. When have they ever agreed with anyone who has suggested a change in their methods? Here we have an agency that certifies forecast methods, they are accepted for coursework by a major university. Businesses use them for risk assessment and economic forecasts where if the methods they use don’t work, they get fired or not paid.
RC climate modelers say “pooh on that, they don’t need such things”. And they may be right, because there is no accountability for them. If the climate model fails 50 years into the future. They aren’t around to be held accountable. Thus it appears they aren’t even interested in looking outside their own realm to embrace something that might improve the model forecasts. The arrogance on display there at RC is stunning.- Anthony
“What astounds me is the sheer gullibility of almost every single one reader of this blog.”
That was my nickname in HS, ‘Gullible’.
Y ahorita Usted tambien? Cuidado.
foinavon (11:09:51) :
Every engineer reading what you’ve written today has had numerous laughs. Those principles (which I took a moment to read), are very much like every other principles in any certification type organization in engineering (See the IEEE).
That you fail to understand the obvious need for these types of checks on GCM’s says to me that you are either being deliberately obtuse, or are wholly ignorant on the importance of certification organizations in a modern world.
Smokey
I said
‘Ilk’ means merely ‘of the same’ it is in no way derogatary.
You said
“Not so in the U.S., where ‘ilk’ is deemed to be mildly derogatory [eg, “I want you and your ilk off my property.”]”
Smokey I think you must have misheard, he wasn’t talking to you he was talking to the elk:)
TonyB
foinavon:
Why suffer the abuse here? Come back to the Stratospheric Heating thread.
Tom M (10:50:10) :
For my part I have been addressing the “Improper peer review process” point (point #2 in the top article) and the question of whether “forecasters” should review papers on climate models as part of the peer-review process as described in my post [foinavon (09:40:25) ] in response to [G Alston (08:48:09)], and [Paul Shanahan (07:16:32)] who asked “On the flip side, why should “climate modelers” build their “forecasts” without the understanding of the principles of forcasting.”, and so on…
If we agree that it doesn’t make sense for climate modellers to incorporate these “”principles of forecasting” into their work, since a cliimate model isn’t a “forecast”, nor for forecasters to review their publications as part of the peer-review process, then we could move on to a more sensible point which is something like:
should scientific advisors to policymakers or policymakers themselves incorporate a set of defined “principles of forecasting” when preparing/assessing forecasts relevant to policymaking?
Then we would have to ask whether they might not be doing this already, and if not, whether they should, and if they should, what these “principles” might be…