Guest Post by Willis Eschenbach
There’s a lovely 2005 paper I hadn’t seen, put out by the Los Alamos National Laboratory entitled “Our Calibrated Model has No Predictive Value” (PDF).
Figure 1. The Tinkertoy Computer. It also has no predictive value.
The paper’s abstract says it much better than I could:
Abstract: It is often assumed that once a model has been calibrated to measurements then it will have some level of predictive capability, although this may be limited. If the model does not have predictive capability then the assumption is that the model needs to be improved in some way.
Using an example from the petroleum industry, we show that cases can exist where calibrated models have no predictive capability. This occurs even when there is no modelling error present. It is also shown that the introduction of a small modelling error can make it impossible to obtain any models with useful predictive capability.
We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.
There are three results in there, one expected and two unexpected.
The expected result is that models that are “tuned” or “calibrated” to an existing dataset may very well have no predictive capability. On the face of it this is obvious—if we could tune a model that simply then someone would be predicting the stock market or next month’s weather with good accuracy.
The next result was totally unexpected. The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.
The third unexpected result was the effect of error. The authors found that if there are even small modeling errors, it may not be possible to find any model with useful predictive capability.
To paraphrase, even if a tuned (“calibrated”) model is perfect about the physics, it may not have predictive capabilities. And if there is even a little error in the model, good luck finding anything useful.
This was a very clean experiment. There were only three tunable parameters. So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.
I leave it to the reader to consider what this means about the various climate models’ ability to simulate the future evolution of the climate, as they definitely are tuned or as the study authors call them “calibrated” models, and they definitely have more than three tunable parameters.
In this regard, a modest proposal. Could climate scientists please just stop predicting stuff for maybe say one year? In no other field of scientific endeavor is every finding surrounded by predictions that this “could” or “might” or “possibly” or “perhaps” will lead to something catastrophic in ten or thirty or a hundred years. Could I ask that for one short year, that climate scientists actually study the various climate phenomena, rather than try to forecast their future changes? We still are a long ways from understanding the climate, so could we just study the present and past climate, and leave the future alone for one year?
We have no practical reason to believe that the current crop of climate models have predictive capability. For example, none of them predicted the current 15-year or so hiatus in the warming. And as this paper shows, there is certainly no theoretical reason to think they have predictive capability.
The models, including climate models, can sometimes illustrate or provide useful information about climate. Could we use them for that for a while? Could we use them to try to understand the climate, rather than to predict the climate?
And 100 and 500 year forecasts? I don’t care if you do call them “scenarios” or whatever the current politically correct term is. Predicting anything 500 years out is a joke. Those, you could stop forever with no loss at all
I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe. I mean, it must get tiring for them, seeing their predictions of Thermageddon™ blown out of the water by ugly reality, time after time, without interruption. I think they’d welcome a year where they could forget about tomorrow.
Regards to all,
w.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Lief:
I see you have still not addressed the fundamental point made to you in my post at October 31, 2011 at 3:43 pm. I would be grateful if you were to answer it.
Richard
Richard S Courtney says:
November 1, 2011 at 11:04 am
I see you have still not addressed the fundamental point made to you in my post at October 31, 2011 at 3:43 pm. I would be grateful if you were to answer it.
I didn’t see an explicit question to answer, just your [somewhat muddled] opinion.
I must remark that the discussion of physics or curve fitting is interesting as is the discussion of chicken entrails. The sniping back and forth has some levity but gets tiresome after a while.
The thing that everyone must realize is that despite your considerable skill and knowlege, there are days that you just ain’t going to hit that curve ball.
Leif Svalgaard,
The time step depends on the numerical method, implicit methods that allow longer time steps are being sought, and different physics is stepped at different intervals. Even at the AR4, parameterizations, averaging and smoothing must be tuned not just for physics but for numberical requirements over different time scales. However, great our faith is in physics, simulating physics on a global scale is far from that ideal:
“Coupling frequency is an important issue, because fluxes are
averaged during a coupling interval. Typically, most AOGCMs
evaluated here pass fluxes and other variables between the
component parts once per day. The K-Profi le Parametrization
ocean vertical scheme (Large et al., 1994), used in several
models, is very sensitive to the wind energy available for
mixing. If the models are coupled at a frequency lower than
once per ocean time step, nonlinear quantities such as wind
mixing power (which depends on the cube of the wind speed)
must be accumulated over every time step before passing to
the ocean. Improper averaging therefore could lead to too
little mixing energy and hence shallower mixed-layer depths,
assuming the parametrization is not re-tuned. However, high
coupling frequency can bring new technical issues. In the
MIROC model, the coupling interval is three hours, and in this
case, a poorly resolved internal gravity wave is excited in the
ocean so some smoothing is necessary to damp this numerical
problem. It should also be noted that the AOGCMs used here
have relatively thick top oceanic grid boxes (typically 10 m or
more), limiting the sea surface temperature (SST) response to
frequent coupling (Bernie et al., 2005).”
http://www.ipcc.ch/pdf/assessment-report/ar4/wg1/ar4-wg1-chapter8.pdf
commieBob says:
November 1, 2011 at 10:44 am
Surely you would agree that cosmic rays influence the climate.
Actually not. There is, at best, conflicting evidence, and overall, no compelling evidence for any such observable influence.
Martin Lewitt says:
November 1, 2011 at 11:12 am
The time step depends on the numerical method, implicit methods that allow longer time steps are being sought, and different physics is stepped at different intervals.
True, but irrelevant, as the time step is very small compared to the length of the simulation. The smallest time step [5 minutes] are used for the most important dynamical processes.
Steve In S.C. says:
November 1, 2011 at 11:12 am
I must remark that the discussion of physics or curve fitting is interesting as is the discussion of chicken entrails.
It illustrates the level of ignorance by participants.
Martin Lewitt says:
November 1, 2011 at 11:12 am
The time step depends on the numerical method, implicit methods that allow longer time steps are being sought, and different physics is stepped at different intervals. Even at the AR4, parameterizations, averaging and smoothing must be tuned not just for physics but for numerical requirements over different time scales. However, great our faith is in physics, simulating physics on a global scale is far from that ideal:”
Your comment goes to stability of the numerical algorithm used in the climate model. Unfortunately, it is difficult or impossible to prove anything about the stability of systems of partial differential equations, unless they are linearized model equations. The “optimal” time step is therefore usually found by trial and error (i.e. the largest value that doesn’t cause the solution to “blow up”).
This also gets us into fuzzy area of why the numerical algorithms for climate model atmospheric dynamics are any different from numerical weather prediction models…
“For time step check slide 7 of http://uscid.us/08gcc/Brekke%201.PDF”
You’ve got to be kidding?! Where is the stability analysis to prove the values asserted? There aren’t even any equations in the entire presentation!
Frank K. says:
November 1, 2011 at 11:43 am
“For time step check slide 7 of http://uscid.us/08gcc/Brekke%201.PDF”
You’ve got to be kidding?! Where is the stability analysis to prove the values asserted? There aren’t even any equations in the entire presentation!
It was not the purpose to prove anything, just to substantiate wit a quote what is actually used.
I especially like the title to this piece. As, indeed, this is much like Jonathan Swift’s original proposal. So seemingly simple in concept but impossible to execute.
Leif Svalgaard says:
November 1, 2011 at 11:51 am
“It was not the purpose to prove anything, just to substantiate wit a quote what is actually used.”
OK. Sounds like a trial and error time step to me… ;^)
RACookPE1978 says:
October 31, 2011 at 10:54 pm
That’s a good contribution. You provide lots of examples of how climate models are, in Leif Svalgaard’s words, “too simple”.
Frank K. says:
November 1, 2011 at 12:09 pm
OK. Sounds like a trial and error time step to me… ;^)
A more important issue is the amount of computer time spent and also the relation between time step and spacial resolution. Last time I asked Gavin Schmidt about this, he told me that the 5 minutes is a compromise between all these factors [including numerical stability].
“We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.”
As noted here: GLOBAL WARMING: FORECASTS BY SCIENTISTS
VERSUS SCIENTIFIC FORECASTS
by
Kesten C. Green and J. Scott Armstrong
Isn’t that the problem here? No knowledge of the subject.
OK so the computer has merely to be the size of the solar system.
commieBob says:
November 1, 2011 at 12:17 pm
OK so the computer has merely to be the size of the solar system.
We actually have such a computer, it is called ‘our solar system’ and shows us what is happening.
Leif Svalgaard wrote: And if the model predictions match reality we have not learned anything new. Only when there is discrepancies can we learn something.
That is not absolutely or universally true. We will have learned that the model makes accurate predictions. Enough of that and we might be able to base policy (including the design of the next experiment, or the design and construction of a device) on the model predictions. If the model contains a novel conjecture, then the accurate prediction can increase confidence that the conjecture is not false. Learning need not be all-or-none, but can be an increase or decrease in confidence of reliability. “We” denotes a diverse group: “we” learned a great deal when Eddington’s expedition confirmed the predictions of Einstein’s General Theory of Relativity, and when Meitner wrote out the model that was accurate for the latest in the series of experiments by Meitner, Hahn and Strassmann. That last ignited, so to speak, a rush of confirmatory experiments and a tidal wave of effort to construct some new devices. Perhaps “we” could say that Einstein learned nothing new from those model confirmations, bet everyone else learned a great deal.
Leif Svalgaard says:
November 1, 2011 at 10:01 am
“Willis Eschenbach says:
November 1, 2011 at 9:43 am
Then why did they start by mentioning “climate models” in the very first paragraph of their work?
They and you and most others go wrong by assuming that climate models are curve fitting as in the paper. They are not, they are solving differential equations forwards with a small time step [every ~5 minutes]. That is why the models do not compare. This is a qualitative difference.”
Leif, don’t confuse the genetic algorithm that does the curve fitting by selecting a model variation with the individual model runs. They say their genertic algorithm runs a total of 7,000 individual model runs. It is these model runs that they compare to GCM runs.
There is no qualitative difference to a Gavin Schmidt running a GCM, and tunes its parameters and runs it again to see whether it matches historical data better now.
And we KNOW that they do this – they constantly publish papers in which they argue that they now have a better estimate for climate sensitivity to a doubling of CO2 because they tried out some values…
So the human climate modelers in their entirety correspond to a huge curve-fitting algorithm that runs GCMs over and over again and modifies them.
Leif Svalgaard says:
November 1, 2011 at 10:01 am
Leif, surely you must know that the models contain a variety of parameters, and that these are “tuned” to make sure the models don’t go off the rails. I can provide you with citations if you wish …
So no, climate models not just physics based, that’s a huge exaggeration. They have a number of parameters which are tuned by comparison of the model outputs to the historical data. This is not a qualitative difference at all.
w.
Steve In S.C. says:
November 1, 2011 at 11:12 am
I love people who post simply to tell us it’s all inutterably boring … hey, if we’re boring you, go away, there’s a good fellow.
No need to insult the locals before you leave with your superior knowledge of what’s boring and what’s not, and please, don’t go away mad … but if you’re bored, STFU and leave.
w.
Willis Eschenbach says:
November 1, 2011 at 12:39 pm
They have a number of parameters which are tuned by comparison of the model outputs to the historical data. This is not a qualitative difference at all.
I don’t think so. Perhaps you have some references.
DirkH says:
November 1, 2011 at 12:34 pm
Leif, don’t confuse the genetic algorithm that does the curve fitting by selecting a model variation with the individual model runs.
The climate models are numerical solutions to differential equations. I see no such in the paper under discussion. That is the qualitative difference.
re: Eric Anderson says: November 1, 2011 at 9:00 am
Exactly.
Add :
8) A basic understanding of the massive amount of uncertainty with regard to how much various factors influence climate (e.g., soot, underwater volcanoes, natural & man made aerosols, atmospheric residence time of CO2, clouds, cosmic rays, etc).
9) A good concept of the degree of unknowns that still exists – e.g., how many “new” but highly relevant factors have been and are being recently discovered, how little understanding there is of naturally existing climate cycles, etc.
10) A basic understanding that the model results are being used, by their producers and operators, to claim severe damage will occur to the Earth, when current conditions and at least some of the model results don’t deviate from natural cycles seen in the past, in terms of rate of temp increase, amount of temp increase, and maximum temps – and in those past cycles, the warmest times were the best times for man and other species as best we can tell.
w.-
Asking professional crystal ball readers to cease prognisticating is akin to asking a politician to stop talking. Whether they admit it or not, both classes of men get a thrill out seeing their name in print or hearing their voice broadcast. It’s an addictive drug and, as is the case for most addicts, asking them to go “cold turkey” is asking the impossible.
Rational Debate says:
November 1, 2011 at 12:53 pm
10) A basic understanding that the model results are being used,
That last one is completely irrelevant as to the veracity or lack thereof of the models. Many of the other factors cited sound more like ignorance on part of the doubter or [worse] are agenda-driven.
Leif Svalgaard says:
November 1, 2011 at 12:15 pm
“A more important issue is the amount of computer time spent and also the relation between time step and spacial resolution. Last time I asked Gavin Schmidt about this, he told me that the 5 minutes is a compromise between all these factors [including numerical stability].”
Actually, numerical stability is the ONLY thing that matters. The basic relationship between spatial resolution and time step is given by the CFL (or Courant) numnber: C = u*dt/dx. For most explicit schemes, C is generally less than about 1. You can push this higher for implicit schemes (and hence the time step), but temporal accuracy will begin to suffer if the time step is too large. Again, stability proofs can only be made for simple model equations. Once you begin to add equation coupling, non-linearity, spatially-varying source terms and properties, Lagrangian advection of scalars, etc., proofs go out the window and you rely on trial and error (which I suppose is the “compromise” Gavin was referring to).