It has been an interesting couple of days. Today yet another scientist has come forward with a press release saying that not only did their audit of IPCC forecasting procedures and found that they “violated 72 scientific principles of forecasting”, but that “The models were not intended as forecasting models and they have not been validated for that purpose.” This organization should know, they certify forecasters for many disciplines and in conjunction with John Hopkins University if Washington, DC, offer a Certificate of Forecasting Practice. The story below originally appeared in the blog of Australian Dr. Jennifer Marohasy. It is reprinted below, with with some pictures and links added for WUWT readers. – Anthony
J. Scott Armstrong, founder of the International Journal of Forecasting
Guest post by Jennifer Marohasy
YESTERDAY, a former chief at NASA, Dr John S. Theon, slammed the computer models used to determine future climate claiming they are not scientific in part because the modellers have “resisted making their work transparent so that it can be replicated independently by other scientists”. [1]
Today, a founder of the International Journal of Forecasting, Journal of Forecasting, International Institute of Forecasters, and International Symposium on Forecasting, and the author of Long-range Forecasting (1978, 1985), the Principles of Forecasting Handbook, and over 70 papers on forecasting, Dr J. Scott Armstrong, tabled a statement declaring that the forecasting process used by the Intergovernmental Panel on Climate Change (IPCC) lacks a scientific basis. [2]
What these two authorities, Drs Theon and Armstrong, are independently and explicitly stating is that the computer models underpinning the work of many scientific institutions concerned with global warming, including Australia’s CSIRO, are fundamentally flawed.
In today’s statement, made with economist Kesten Green, Dr Armstrong provides the following eight reasons as to why the current IPCC computer models lack a scientific basis:
1. No scientific forecasts of the changes in the Earth’s climate.
Currently, the only forecasts are those based on the opinions of some scientists. Computer modeling was used to create scenarios (i.e., stories) to represent the scientists’ opinions about what might happen. The models were not intended as forecasting models (Trenberth 2007) and they have not been validated for that purpose. Since the publication of our paper, no one has provided evidence to refute our claim that there are no scientific forecasts to support global warming.
We conducted an audit of the procedures described in the IPCC report and found that they clearly violated 72 scientific principles of forecasting (Green and Armstrong 2008). (No justification was provided for any of these violations.) For important forecasts, we can see no reason why any principle should be violated. We draw analogies to flying an aircraft or building a bridge or performing heart surgery—given the potential cost of errors, it is not permissible to violate principles.
2. Improper peer review process.
To our knowledge, papers claiming to forecast global warming have not been subject to peer review by experts in scientific forecasting.
3. Complexity and uncertainty of climate render expert opinions invalid for forecasting.
Expert opinions are an inappropriate forecasting method in situations that involve high complexity and high uncertainty. This conclusion is based on over eight decades of research. Armstrong (1978) provided a review of the evidence and this was supported by Tetlock’s (2005) study that involved 82,361 forecasts by 284 experts over two decades.
Long-term climate changes are highly complex due to the many factors that affect climate and to their interactions. Uncertainty about long-term climate changes is high due to a lack of good knowledge about such things as:
a) causes of climate change,
b) direction, lag time, and effect size of causal factors related to climate change,
c) effects of changing temperatures, and
d) costs and benefits of alternative actions to deal with climate changes (e.g., CO2 markets).
Given these conditions, expert opinions are not appropriate for long-term climate predictions.
4. Forecasts are needed for the effects of climate change.
Even if it were possible to forecast climate changes, it would still be necessary to forecast the effects of climate changes. In other words, in what ways might the effects be beneficial or harmful? Here again, we have been unable to find any scientific forecasts—as opposed to speculation—despite our appeals for such studies.
We addressed this issue with respect to studies involving the possible classification of polar bears as threatened or endangered (Armstrong, Green, and Soon 2008). In our audits of two key papers to support the polar bear listing, 41 principles were clearly violated by the authors of one paper and 61 by the authors of the other. It is not proper from a scientific or from a practical viewpoint to violate any principles. Again, there was no sign that the forecasters realized that they were making mistakes.
5. Forecasts are needed of the costs and benefits of alternative actions that might be taken to combat climate change.
Assuming that climate change could be accurately forecast, it would be necessary to forecast the costs and benefits of actions taken to reduce harmful effects, and to compare the net benefit with other feasible policies including taking no action. Here again we have been unable to find any scientific forecasts despite our appeals for such studies.
6. To justify using a climate forecasting model, one would need to test it against a relevant naïve model.
We used the Forecasting Method Selection Tree to help determine which method is most appropriate for forecasting long-term climate change. A copy of the Tree is attached as Appendix 1. It is drawn from comparative empirical studies from all areas of forecasting. It suggests that extrapolation is appropriate, and we chose a naïve (no change) model as an appropriate benchmark. A forecasting model should not be used unless it can be shown to provide forecasts that are more accurate than those from this naïve model, as it would otherwise increase error. In Green, Armstrong and Soon (2008), we show that the mean absolute error of 108 naïve forecasts for 50 years in the future was 0.24°C.
7. The climate system is stable.
To assess stability, we examined the errors from naïve forecasts for up to 100 years into the future. Using the U.K. Met Office Hadley Centre’s data, we started with 1850 and used that year’s average temperature as our forecast for the next 100 years. We then calculated the errors for each forecast horizon from 1 to 100. We repeated the process using the average temperature in 1851 as our naïve forecast for the next 100 years, and so on. This “successive updating” continued until year 2006, when we forecasted a single year ahead. This provided 157 one-year-ahead forecasts, 156 two-year-ahead and so on to 58 100-year-ahead forecasts.
We then examined how many forecasts were further than 0.5°C from the observed value. Fewer than 13% of forecasts of up to 65-years-ahead had absolute errors larger than 0.5°C. For longer horizons, fewer than 33% had absolute errors larger than 0.5°C. Given the remarkable stability of global mean temperature, it is unlikely that there would be any practical benefits from a forecasting method that provided more accurate forecasts.
8. Be conservative and avoid the precautionary principle.
One of the primary scientific principles in forecasting is to be conservative in the darkness of uncertainty. This principle also argues for the use of the naive no-change extrapolation. Some have argued for the precautionary principle as a way to be conservative. It is a political, not a scientific principle. As we explain in our essay in Appendix 2, it is actually an anti-scientific principle in that it attempts to make decisions without using rational analyses. Instead, cost/benefit analyses are appropriate given the available evidence which suggests that temperature is just as likely to go up as down. However, these analyses should be supported by scientific forecasts.
The reach of these models is extraordinary, for example, the CSIRO models are currently being used in Australia to determine water allocations for farmers and to justify the need for an Emissions Trading Scheme (ETS) – the most far-reaching of possible economic interventions. Yet, according to Dr Armstrong, these same models violate 72 scientific principles.
********************
1. Marc Morano, James Hansen’s Former NASA Supervisor Declares Himself a Skeptic, January 27,2009. http://epw.senate.gov/public/index.cfm?FuseAction=Minority.Blogs&ContentRecord_id=1a5e6e32-802a-23ad-40ed-ecd53cd3d320
2. “Analysis of the U.S. Environmental Protection Agency’s Advanced Notice of Proposed Rulemaking for Greenhouse Gases”, Drs. J. Scott Armstrong and Kesten C. Green a statement prepared for US Senator Inhofe for an analysis of the US EPA’s proposed policies for greenhouse gases. http://theclimatebet.com
Sponsored IT training links:
Get guaranteed success in 312-50 exam in first try using incredible 642-374 dumps and other 310-200 training resources prepared by experts.
foinavon (06:57:21) :
TWO. All papers published in the scientific literature are subject to peer review. Papers are submitted to appropriate journals, editors select appropriate reviewers. Why should “forecasters” be used as reviewers of scientific papers that they’re unlikely to understand? If a forecaster (of the type suggested by Armstrong) is involved, it might more usefully be in the arena of policymaking.
On the flip side, why should “climate modelers” build their “forecasts” without the understanding of the principles of forcasting? This is a two way road…
Dr. Pielke, Sr. has a clarifying column on weather and climate models. See: http://www.climatesci.org/2009/01/29/real-climate-suffers-from-foggy-perception-by-henk-tennekes/
I recommend everyone to follow the link Luis Dias placed at (06:25:45). What strikes me is that Armstrong already came out a year and a half ago with his critique of IPCC forecasting, but it is presented here at WUWT as though it’s coming completely out of the blue. It’s put on the level of the John S. Theon story, though it’s completely different.
Some remarks following the article on RC are quite interesting:
-The authors also seem to put a large weight on “forecasting principles” developed in different fields. While there may be some valuable advice, and cross-field cooperation is to be encouraged, one should not assume that techniques developed in say, econometrics, port directly into climate science.
-When I see statements like: “Based on our Google searches, those forecasting long-term climate change have no apparent knowledge of evidence-based forecasting methods, so we expect that the same conclusions would apply to the other three necessary parts of the forecasting problem” and I just cringe.
A “Google search” sounds like something I’d see on a second-rate undergraduate research paper; how about using the Science Citation Index for example and show me you’ve really done a thorough lit. review. This is just lazy.
-It seems to me that much of the failure of the G&A article comes from the fact that they are economists. Economics doesn’t have anything resembling physics or thermodynamics, it only has models. For a long time, they thought the velocity of money was stable. Then it changed. For a long time, the P/E ratio of most stocks stayed in the range of 10 to 20. Then the range changed.
-A look at the list of authors and reviewers is enlightening. While a smattering are from information sciences or mathematics the majority are from business schools or business related institutions. Psychology is also represented. Unless I missed someone there is no one from the physical sciences as either an author or a reviewer.
And here’s another interesting link explaining exactly how the IPCC violates Armstrong’s scientific principles of forecasting: http://www.dailykos.com/story/2007/6/27/0249/19852
I have to say that after doing some extensive reading today about the subject Mr Armstrong’s method is in my view far from objective and hardly scientific.
Can we not use the word “ilk”. It is a disgusting word in the context of a science debate that should be focused on data and analysis. I have done research and I can tell you that statistical analysis is one way to get the data to say what you want it to say. So data and its analysis is open for serious consideration and critique. That does not make either side “ilk’s”.
Perhaps a little OT, but here it is anyway – another blow for the AGW folks from John Coleman (via Drudge): “The Amazing Story Behind The Global Warming Scam.”
It just gets better and better…
http://www.kusi.com/weather/colemanscorner/38574742.html
Anthony – your blog is HUGE. Thanks for your hard work and good science!
Scientists Rank Global Cooling Hacks
And by “Hacks” they’re not referring to Algore.
Pamela
‘Ilk’ means merely ‘of the same’ it is in no way derogatary. It often refers to members of the same class or a family, and was formerly used to denote descendants of someone-often high born
People use far worse words than that on this blog-although thankfully it is normally well mannered and swearing is minimal.
TonyB
JP (07:03:41) :
The IPCC issued scenario projections. After looking at the scenarios it became apparent that the IPCC hasn’t a clue what our climate will be like in 100 years. Call it what you want, but this isn’t science. If the the science is settled why the lack of confidence? Why all the scenarios? This why the Alarmists are all over the board on everything from sea levels, to rainfall projections, to TC predictions.
Output can only be predicted from known inputs. Projections are made in respect of different sets of presumed inputs. They can be assessed once the actual inputs (aka forcings) become known for the period. The IPCC can’t be expected to predict what humans will choose to do!
@Sekerob (22:02:29) :
By all means, let’s follow the money:
http://www.kusi.com/weather/colemanscorner/38574742.html
Now my first thought was “the tide is changing against AGW”, and indeed it is. But my second though was, how “coincidental” it is that this is happening now that there is a Democratic administration. I mean, think about it. These are HUGE bombshells being dropped, both within the last two days, and both not 10 DAYS after Obama was inaugurated. To me, this screams manipulation. And it makes perfect sense in this economic climate. Here we are, about to spend billions on on stimulus, and you’ve still got the Greenies screaming that we need a carbon tax and other craziness. And the Dems CANNOT afford to do this. They know it. Not during a recession. So along comes information that basically says this AGW thing is all completely overblown, just in time for the Dems to have political cover to NOT do any green-type taxes that would kill the economy. This is just more proof that Obama just might be a masterful politician. He gets to look all concerned about the environment, he gets elected in large part because of Greenies who want drastic “change”, and now he is off the hook. So he can / will still pursue renewable energy as a policy (and he should), but he does it with a different goal – energy independence – and he doesn’t need to levy any huge carbon taxes that will cripple the recovery.
Now the real question might be HOW is he doing this? How is this anti-AGW campaign being orchestrated? I’m not sure, it might just be calling in some favors, getting some people who were told to keep quiet before to speak out. But it really does seem AWFULLY convenient. Combine this with the recent Pew poll that says that AGW is WAY down the list of priorities for the American people (something like 30% still think it is a big deal), and you see how the math adds up. This may have been planned all along once these guys got elected, because they cannot give in to their extreme base, and they know it. I think the really AGW-crazed people are about to get marginalized within the party. They were used to get the Dems elected. The Dems never really believed in this AGW stuff, and now that they are on the verge of permanent power (I mean, after thiey spend all of this money, will they really lose again any time soon?), they get to jettison some of the stuff that would threaten their control. I don’t like ’em, but you gotta admire the gamesmanship.
Paul Shanahan (07:16:32) :
First, of course, because a climate model, and it’s output, is not a “forecast”!
Additionally we’d have to know what these “principles of forecasting” are. Can you provide a link/citation? I’ve had a look at Armstrong’s published papers, and these principles seem not to be in the scientific arena. Perhaps he’s put them on a website somewhere? Or written a book/pamphlet? It seems rather extraordinary to be discussing this issue without anyone seeming to know what the “principles of forecasting” actually are, or whether they have any relevance to climate-related science, let alone climate-related policy…
I realize the above is all supposition, and just as likely the Dems / Obama will be forced to react to events (such as sudden cold and scientific dissent) that are happening. And I understand that there is still funding for the climate models and C02 capture technology in the spending bill. But again, the big issue is carbon-taxing, and I haven’t heard one peep about that. Doesn’t that seem odd, given the “cataclysmic” impact of not removing CO2? It just seems to me like the groundwork is being laid for a tactical retreat, all the while trying to keep the base happy and keeping the scientists happy with funding. I sometimes see more than is what really there, but politics is all strategy. What you see in the press is rarely what is really going on.
For example, think about Pelosi’s recent “backdown” on funding of family planning services in the stimulus bill. She goes on TV, basically says how less poor people IS a stimulus, and then “backs down” because Obama asks her too? This seems all to convenient for me, especially given Pelosi’s reputation as a tiger not to be messed with. She even told Obama to stay off her turf recently. IMO, this looks like a setup, pure and simple. I mean, think about it. They KNEW this would provoke a Rep backlash. There was no doubt about it. If Obama wanted to show real bipartisanship, they would have done this behind closed doors before it ever hit the bill. And Pelosi NEVER likes to look bad. You think she would let this purposely leak out and then “back down” to the Reps? Even after she went on This Week with ‘lil shorty to defend how birth control is STIMULUS because poor people drain services? Why would she make such a big issue out of a small part of the bill? And then back down for Obama’s sake? She would never do this unless it was part of a bigger gaming strategy. Later on, when they want something from the Reps (perhaps voting for this bill when it’s done), they will surely point back to this and say “look what we did for you” to put pressure on the Reps to vote for it.
BTW, I am an independent. I see politics for what it is. There are no saints in politics.
I suspect that Dr. Armstrong’s opinion of and involvment in the AGW debate went through an evolution similar to mine (and probably many reading this). Starting out as sympathetic to the notion of AGW (it is intuituve that we have some warming impact – look at the stuff we burn). Slowly gathering more information over time. Getting drawn into the discussion in some substantive way. Growing incredulity at the tenuous substance of Al Gore’s postion and the credulous MSM that supports it. Growing disgust at the politization of the science.
And then to go over to a site like Real Climate and to compare the discourse there to that here and at Climate Audit. I’ve spent a lot of time sorting through the links at Real Climate that do not establish or prove what is claimed.
The irony is that a few years ago, the AGW position had essentially achieved consensus. Rather than taking yes for an answer, and keeping their heads down during the current cooling, the alarmist kept pushing and making more extreme claims and policy demands. One of those (polar bear extiction) has apparently drawn Dr. Armstrong into the discourse.
Anthony,
I agree with this post (below) and I hope you did have a good evening. As I watch the news this am I see where President Obama is about to make some major decisions regarding “Global warming/change/cooling/…”
In the next moment we see Mr. Gore proclaiming that we can’t wait another second – or else the sky will fall.
IMHO we need a visible personality on TV – interviewed by the media (read Fox) to champion this cause. Again, IMHO, that might be you.
So, what do you think?
You certainly have done some impressive fact gathering and excellent presentation.
Thanks for all your work.
Lou
“MC (19:21:08) :
Anthony,
You remind me of Colombo. Persistent, methodical, constant, questioning, and always after the truth.
I know you would rather we talk about the post but I wanted to give my first thought after reading it.
Thank you for being like Colombo and have a great evening.”
http://www.forecastingprinciples.com/Public_Policy/WarmAudit31.pdf for the paper.
DaveE.
According to the Nasa.gov website, Dr. James Hansen has no degrees in climatology, climate science, geology, earth sciences, chemistry or meteorology. Many other climate change experts are biologists or environmental scientists? What makes them experts on the climate?
Thanks for that tip. I haven’t used FORTRAN since 1981. Is there a FORTRAN compiler I can get for Windows? I’m pretty sure they don’t teach that programming language anymore. That’s a government agency for you.
Check your premises. Look and see who is really going to profit if we mandate more green energy, force people to buy more wind power and force a cap-and-trade scheme upon everybody.
As to why these people are starting to speak out now? They have probably been scared out of their silence by the prospect that some foolish save-the-planet, stop AGW legislation will actually pass this Congress and leave us economically crippled while doing nothing to save the global climate.
foinavon — Why should “forecasters” be used as reviewers of scientific papers that they’re unlikely to understand?
Non sequitor. Forecasting is a mathematical discipline. You don’t need to be an expert in climate physics to have a valid opinion on the forecasting. I don’t think it would hurt if papers that showing a forecast were forwarded to those who are experts in the field. It could even help. That said, I’m not convinced that the result presented would be any different.
But we’re not talking about “expert opinion”. We’re talking about scientific analysis.
Not really. We’re talking about the output of what amounts to a massive spreadsheet, albeit bigger, faster, and using iterative algorithms rather than formulae, but still, a spreadsheet in concept and execution. (Doubtful that the formula algorithms are self-modifying, in which case an iterative algorithm is different than a formula only in scope — it’s still repeatable.)
This is the one thing that bugs me about climate science, this perception by AGW proponents that model outputs are data. They’re not.
I think I owe Mr. D. Porter, on a prior thread, an apology. Right street, right building, just around the back and down the stairs.
Neven (07:18:49) : “-It seems to me that much of the failure of the G&A article comes from the fact that they are economists. Economics doesn’t have anything resembling physics or thermodynamics, it only has models.
-A look at the list of authors and reviewers is enlightening. While a smattering are from information sciences or mathematics the majority are from business schools or business related institutions. Psychology is also represented.”
Excellent observation, Neven. You were taking notice of the IPCC correct? Head of the IPCC is Dr. R K Pachauri — an economist. One of the greatest criticisms of the “2500 top scientists” signing on to the 1992 IPCC Working Assessment was that they were mostly social scientists, political scientists, psychologists, and economists with a smattering from atmospheric sciences, climatology, meterology, and math. It is, after all, all about the money.
[Reply: It should also be kept in mind that all IPCC members are political appointees from their various countries. ~ dbstealey, mod.]
It is hard to fathom why everyone wants to make simple, well known principles more complex.
The naive model is just an extension of what every weatherman learns, which is that weather persists. A good rule of thumb is that tomorrow will be pretty much like today.
So the logical extension of this is that if weather persists so does climate, and “persistence” will predict that next year will be much like the last.
I just don’t understand why this principal is suddenly controversial, nor why demonstrated high success rates with such an algorithm are startling.
I think there is a big disconnect between what the IPCC claims and what the AGW alarmists and profiteers claim. The IPCC says they do “scenarios”. You are correct that the IPCC doesn’t claim to do forecasts.
But of what use are the scenarios? First thing we have to ask, did they get the science right? This has not been independently verified. A scientific theory must be falsifiable and it must correctly predict future events to be valid. The IPCC climate models fall short of this.
I’m sure the climate models have use in helping to understand the climate. Right now, they are helping us to learn how the climate doesn’t work. But at a practical level, these models tell us nothing about future climate.
Climate models can’t predict human behavior and can’t predict nature. Will humans continue to use fossil fuels at the assumed rate for the assumed number of decades, or will hydrogen power become so economical that the world switches to it in a few years? Will the sun cooperate and continue to output energy at the rate assumed by the models? Will there be massive volcanic eruptions in the decades ahead that lead to severe global cooling (by blocking the sun). These things are completely unpredictable.
Lastly, it’s generally accepted that increasing atmospheric CO2 from 280 PPM to 560 PPM will be itself lead to only 1.6 degree C of warming. It’s well known the climate system is not linear, so we can’t even take that degree of warming as a given (this 1.6 degree C of warming will only occur if everything else remains the same and it won’t). The IPCC climate models and others have to add in various positive feedbacks and forcings to get their projected warming. None of those feedbacks and forcings have been shown to actually exist. In other words, nobody has shown the climate actually works the way the IPPC models assume it does. Most of the evidence shows the opposite.
foinavon (08:02:51) :
Paul Shanahan (07:16:32) :
On the flip side, why should “climate modelers” build their “forecasts” without the understanding of the principles of forcasting.
First, of course, because a climate model, and it’s output, is not a “forecast”!
Additionally we’d have to know what these “principles of forecasting” are. Can you provide a link/citation? I’ve had a look at Armstrong’s published papers, and these principles seem not to be in the scientific arena. Perhaps he’s put them on a website somewhere? Or written a book/pamphlet? It seems rather extraordinary to be discussing this issue without anyone seeming to know what the “principles of forecasting” actually are, or whether they have any relevance to climate-related science, let alone climate-related policy…
It’s not forecasting to make statements about what the climate will be in 100 or 1000 years? Then, please, tell us what that sort of activity is called?
Regarding your request, see http://www.forecastingprinciples.com/handbook.html
It’s the link to a book titled, oddly enough, “Principles of Forecasting: A Handbook for Researchers and Practitioners.” Equally odd, I found the link on a website named “forecastingprinciple.com.”
While I’d expect a bit more from a student in the way of sourcing than a Google search, I’d be more than a bit peeved if one of my engineers did not perform one as a first, rough step to finding out what information might be out there.
TonyB:
Not so in the U.S., where ‘ilk’ is deemed to be mildly derogatory [eg, “I want you and your ilk off my property.”]
Another problem word is to “table.” IIRC, in GB tabled means to put something on the table for discussion. In the U.S. it means to postpone. I’m not sure which way Canada leans.
A great many “sceptics” of scientific methods and principles have made disparaging comments regarding Dr. Armstrong’s qualifications to speak to the subject of climate change.
While I disagree with them, I understand their frustration. Only yesterday, an individual with a B.A. in journalism from an obscure college spoke in front of congress, peresenting himself as a world class expert on climate change.
It certainly frustrates me to see people with no apparent relevant training, experience, knowledge or skills presented as authorities.
Simon Evans (07:49:29) :
Output can only be predicted from known inputs. Projections are made in respect of different sets of presumed inputs. They can be assessed once the actual inputs (aka forcings) become known for the period. The IPCC can’t be expected to predict what humans will choose to do!
I don’t see how this can be true. Our climate is chaotic at best (don’t confuse this with statement 7 above where the result is that the anomily is stable) so how is it possible to project fron input, through random scenrio to an output? Thats like saying that 50 numbered balls (1 – 50) go into a lotto draw and that the expected result is that every 6 number draw results in the same numbers being pulled out every time. (apologies for the poor analogy, it was a spur of the moment thing) The fact is there are millions of potential outcomes for this draw.
I remember seeing some research where a number of robots were programmed with the same logic (to interact with each other and learn). The expected result was that they would all eventually react the same, but the result was very different. If I remember correctly, one became aggresive, one became submissive and one had a string of bad luck interacting and decided to shut down completely.
My point is (after rambling on) that the climate is extremely complex, with a high element of randomness and choas, where micro climates, gasses, sun, cosmic radiation, land use change effects, etc etc etc all react with each other and never in the same way, so how can we really be able to build a climate model that can give us a limited number of scenario’s when by nature, it should produce millions, if not billions of scenario’s? In my (humble) opinion, It’s just not possible. It’s essentially a holy grail.