Forecasting accuracy of Global Climate Models is something that has been at the very heart of the global warming debate for some time. Leif Svalgaard turned me on to this paper in GRL today:
Reifen, C., and R. Toumi (2009), Climate projections: Past performance no guarantee of future skill?, Geophys. Res. Lett., 36, L13704, doi:10.1029/2009GL038082.
PDF available here
It makes a very interesting point about the “stationarity” of climate feedback strengths. In a nutshell, it says that climate models break down after a time because both forcings and feedbacks don’t remain static, and the program can’t predict such changes.
Gavin Schmidt of NASA GISS says something similar in a recent interview:
The problem with climate prediction and projections going out to 2030 and 2050 is that we don’t anticipate that they can be tested in the way you can test a weather forecast. It takes about 20 years to evaluate because there is so much unforced variability in the system which we can’t predict — the chaotic component of the climate system — which is not predictable beyond two weeks, even theoretically. That is something that we can’t really get a handle on.
From Edge: THE PHYSICS THAT WE KNOW: A Conversation With Gavin Schmidt [with video]
Some excerpts from the paper:
The principle of selecting climate models based on their agreement with observations has been tested for surface temperature using 17 of the IPCC AR4 models.
…
There is no evidence that any subset of models delivers significant improvement in prediction accuracy compared to the total ensemble.
With the ever increasing number of models, the question arises of how to make a best estimate prediction of future temperature change. The Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) combines the results of the available models to form an ensemble average, giving all models equal weight. Other studies argue in favor of treating some models as more reliable than others [Shukla et al., 2006; Giorgi and Mearns,
2002]. However, determining which models, if any, are superior is not straightforward. The IPCC comments:
‘‘What does the accuracy of a climate model’s simulation of past or contemporary climate say about the accuracy of its projections of climate change? This question is just beginning to be addressed. . .’’[Intergovernmental Panel on Climate Change, 2007, p. 594].
One key assumption, on which the principle of performance-based selection rests, is that a model which performs better in one time period will continue to perform better in the future. This has been studied in terms of pattern-scaling using the ‘‘perfect model assumption’’ [Whetton et al., 2007]. We examine the question in an observational context for temperature here
for the first time. We will also quantify the effect of ensemble size on the global mean, Siberian and European temperature error. [3] The principle of averaging results from different
models to form a multi-model ensemble prediction also has potential problems, since models share biases and there is no guarantee that their errors will neatly cancel out. For this reason groups of models thus combined have been termed ‘‘ensembles of opportunity’’ [Piani et al., 2005]. Various studies have showed that multi-model ensembles produce more accurate results than single models [Kiktev et al., 2007; Mullen and Buizza, 2002]. Our examination of ensemble performance aims to address the question in the context of the current generation of climate models.
…
In our analysis there is no evidence of future prediction skill delivered by past performance-based model selection. There seems to be little persistence in relative model skill, as illustrated by the percentage turnover in Figure 3. We speculate that the cause of this behavior is the non-stationarity of climate feedback strengths. Models that respond accurately in one period are likely to have the correct feedback strength at that time. However, the feedback strength and forcing is not stationary, favoring no particular model or groups of models consistently. For example, one could imagine that in certain time periods the sea-ice albedo feedback is more important favoring those models that simulate sea-ice well. In another period,
El Nino may be the dominant mode, favoring those models that capture tropical climate better. On average all models have a significant signal to contribute.
…
While the authors of this paper still profess faith in model ensembles, the issues they point out with non-staionarity call into question the ability for any model to remain on-track for an extended forecast period.
“Testing a model against past climate (hindcasting) is an advanced exercise in curve fitting, nothing more and proves absolutely nothing.”
I totally agree. This is especially so if failed models can be quietly discarded or tweaked until they match history.
“With a computer model there are an infinite number of ways to match the temperature curve but only one way that represents the real world. ”
One way at the very most. It’s possible, even likely, that we simply cannot predict the climate 50 or 100 years from now any more accurately than simply guessing that the climate will be roughly the same as it is now.
“At best one is right and the rest are wrong, though it’s certainly possible for them all to be wrong”
I agree. And yet, all of those models match history. The reasonable inference is that “matching history” does not mean a model is a good one.
Any second year IT tech school student could construct a model to predict whatever you want. That’s exactly what the “climate researchers” have done. They have delivered us into the hands of mendacious politicians who will seize absolute power. When that happened in late 19th century Montana the citizenry solved the problem with a rope.
Off topic, but I had an idea for a new blog post, which is to ask the following question:
Let’s assume for the sake of argument that the warmists are correct that most of the observed warming in the second half of the 20th century was due to human CO2 emissions.
In that case, one can ask: What would global surface temperatures be like now but for such warming? I did a calculation (I’m happy to share it!). It turns out that current global surface temperatures would be shockingly low right now without such warming.
The conclusion is that if warmists are right, then we should be happy so much CO2 has been emitted.
A bit OT, but NASA kicked out another news blurb stating the arctic ice is melting faster now because of the reduction in multi-year ice. Global Warming to blame!
http://www.kcautv.com/Global/story.asp?S=10658610&nav=1kgl
After watching Gavin’s video, a few things that he says stand out in my mind. 1) He claims that they have nearly perfect knowledge of cloud formations and behavior, precipitation production and behavior, yet, if this were true, why is there not a single climate model that is capable of representing such skillful knowledge? His model’s do not. and 2) He describes increasing CO2 in the atmosphere will warm the ground. Well, any self respecting physicist will tell you, that is impossible. Even if increased CO2 is capable of increasing atmospheric temperature, it is NOT capable of increasing surface temperature, as that would violate some very fundamental physical laws.
Although I can relate to, and I agree with, some of what he says, several of his assertions are simply false, and false in areas he is well versed.
I have often thought about this same thing. If we are warming at an unprecedented rate, even faster than we thought, while setting cold temperature records all around the globe, one has to wonder just how cold we would really be with out this “unprecedented global warming”
I shiver to think…
Keep in mind that Dr. Spencer has said that the 30 years of Satellite data are not sufficient to disprove the models at this point. Spencer has stated that 50 years might be a minimum but that 100 years of data might be required to sort out natural fluctuations vs human induced temperature changes.
Thanks
Ed
I read Gavin’s quote as meaning something different from the effect this paper is discussing. I think Gavin is saying that it will take 30 years before we know if the current divergence between models and measurements is the effect of an oscillation or measurement noise, or if it reflects a collection of models which are inaccurate. To some extent his point has merit. The 1970-2000 period is far too short to provide robust validation of the models (and we have poor understanding of a good period). It seems the logic is that ‘we only know of one effect which could have caused the sudden rise…’ yet the current pause is just a pipeline stall…
What i believe the paper is suggesting is that if we assume that the climate system has (coincidentally) a well balanced set of feedback mechanisms then it is possible to achieve a good fit whilst neglecting any feedback term so long as the influence of that term is small in the testing period. If clouds can provide a regulating effect, we would only expect to see that regulation on action once other conditions are met – and we are in the realms of non-linear (total) feedback (which is required in order to achieve stable oscillation – ask any EE. Amplifiers are easier to build than oscillators).
Question is, how do we determine if the calibration period covers sufficient of the forcings space to be valid? Have the models been tested to demonstrate stability in different geological time periods? Would they be expected to be stable?
“I have often thought about this same thing. If we are warming at an unprecedented rate, even faster than we thought, while setting cold temperature records all around the globe, one has to wonder just how cold we would really be with out this ‘unprecedented global warming”'”
My guess is that some of the months in 2009 would be the coldest months in the last 100 years.
Bill Illis (05:56:38) :
“The models seem reasonably accurate when they are hindcasting – running the models against the known temperature record.”
Absolutely false Bill.
I have posted here recently that, in order to hindcast, the models use fabricated (false) aerosol data to reproduce the cooling from ~1945-1975. Actual measurements as described by Doug Hoyt et al show no such aerosol trends.
Edward (11:10:10) :
“Keep in mind that Dr. Spencer has said that the 30 years of Satellite data are not sufficient to disprove the models at this point. Spencer has stated that 50 years might be a minimum but that 100 years of data might be required to sort out natural fluctuations vs human induced temperature changes.”
______________
I disagree.
We can say from satellite data that there has been no global warming since 1979, and we can also say from pre-satellite data that there has been no global warming since 1940, and perhaps even ~0.3C of cooling. And now we have experienced further global cooling for the past decade or so.
Yet the models continue to predict catastrophic warming.
I can confidently conclude that there is adequate data to demonstrate that the models are invalid.
In today’s ‘Daily Mail’ (popular UK newspaper) a sensible article!
http://www.dailymail.co.uk/debate/article-1198188/Hysteria-real-threat-global-warming.html
This is a fair point. I have a random number generator which will generate an integer from 1-6. I roll 18 dice to see if any of them can predict the number before it is generated. For the next run I only select the dice which predicted the right answer as of course they will be more likely to predict it the next time.
Okay, maybe this is a dumb question, but with regard to hindcasting, has anyone ever run the GCM models against a given period of ice-core data? I mean, initializing from ice-core values for CO2 and temp, and making no attempt at curve fitting, what would result? Or do these models simply not function over so wide a range of CO2 and temp?
My favorite model is Kathy Ireland.
I work with people who do plasma physics models. (Polywell Fusion Reactor) All the equations are known to a very high degree of precision. (8 or 10 significant figures) And yet due to the fact that EVERY particle affects ever other particle simulations are not very good. They may give you the general trend (increase the density and the value of x rises), but exact predictions are out of the question. The joke we often use is that we need a real time computer that can run the equations to perfect precision (experiment).
Now compare this to climate where all the equations are NOT known to a high degree of precision and in fact ALL the equations are not even known.
To predict the future with such (#$@*&!!) is not possible. In fact with so many still unknown significant factors (Svensmark) it is useless. And we are just now getting a handle on cosmic rays and clouds. And there are likely still unknown unknowns.
My favorite model is Louisa Lockhart. I have yet to see a good simulation and she is a very good model.
My guess is that some of the months in 2009 would be the coldest months in the last 100 years.
I live in the northern Illinois area and today we had a day where the temperature did not get above 63F. This seems rather unusual for July.
A perfect model of our climate would, like our climate, behave chaotically. Give it identical starting conditions ten times and run for a 100 virtual years then I’m sure 10 very different endpoint climates would be produced.
It seems to me that the climate system is not calculable because any given set of starting parameters will not produce the same result if the model is run twice.
If clouds can provide a regulating effect, we would only expect to see that regulation on action once other conditions are met – and we are in the realms of non-linear (total) feedback (which is required in order to achieve stable oscillation – ask any EE. Amplifiers are easier to build than oscillators.
Yes. Say you want to make a low distortion audio sine wave directly from an oscillator. We have ways of doing that (a Wein bridge with a light bulb in the feed back circuit) that is not too bad. After the oscillator settles we can without to much difficulty get a wave with distortion 60db (.1%) down. Going to 80 db without other tricks (filters) is tough. And even filters are rough because they can introduce distortion. And even then there are limits due to the intrinsic noise of resistors and amplifiers. A PERFECT sine wave is impossible to generate.
I suspect Gavin will be slowly distancing himself from the politicans who are driving this thing. It is, after all government science, and thus politically directed.
Allan M. R. MacRae (13:42),
I agree with you. I’ve posted a lot about the made-up Aerosols (and volcano) plugs that are used to make the hindcasts work.
This is GISS Aerosols forcing from 1880 to 2003. Take these numbers and multiply by 0.32 to change the forcing to temperature impact. Total direct and indirect temp impact from Aerosols in GISS models is -0.6C. Obviously, these forcings are manufactured in Hansen’s laboratory.
http://img58.imageshack.us/img58/855/modelaerosolsforcingp.png
The latest study on sulfate aerosols is that they combine with black carbon and soot to produce warming in the atmosphere rather than cooling. This matches better with the temperature experience of China, south Asia, southern California and the northern hemisphere for example.
Friends:
There is only one fact that needs to be known about ensemble climate models, and it needs no discussion: i.e.
Average wrong is still wrong.
Richard
henrychance (07:09:00) :
We could include some periods that were totally made up readings but readings within a sensible range.
Haven’t they already done that one?
DaveE
This thread has been coming to an end, and I am tempted to tell one of my stories, relevant to my reaction to the “skill” of models.
A man goes to the next village to get himself a wife. He meets many young girls but one of them, who smiles very sweetly and only says “yes” and “no” appeals to him, and the marriage is arranged.
He takes the sweet girl to his home and they have a lovely honeymoon, the wife is a good cook too, the only thing is, she keeps saying only “yes”, “no”, and smiling sweetly.
After a while this gets on his nerves, and he tries to get some other reactions from her, tries to make her angry by doing irrational and sometimes cruel things.
Once he brought back a piece of marble pretending it was cheese, asked her to bring it to the table, and made a big show of anger when she did not. Still, she trembled sweetly and did not get angry or say more than “yes”.
Once he bought her a tight pair of shoes and forced her to wear them, still no reaction from her.
Once he hid behind the door and jumped at her scaring her out of her wits, but not out of the “yes” or “no”.
He decided on drastic measures. He pretended he dropped dead, not responding to anything she tried to do with him, just lay there dead. After a while she was convinced he was dead. She started a dirge crying and crying:
Oh, deal huthband, what thall I lemember filst?
The malble cheethe, the tight thoes, or the BAH behind the dool?
She had a speech impediment and had been told not to speak because she would lose the bridegroom.
The dirge is what comes to my mind when I think of the skill of GCM models”
What shall I remember first?
The lack of error propagation? the spaghetti graphs? the insolent use of linearity in a chaotic system?The manipulated data?
So as not to leave the story hanging, the man resurrected himself, hugged his wife, and asked her to please speak up, and he did not care about the lisp! A happy ending, which I do not foresee for the GCMs.