Guest Post by Willis Eschenbach
There’s a lovely 2005 paper I hadn’t seen, put out by the Los Alamos National Laboratory entitled “Our Calibrated Model has No Predictive Value” (PDF).
Figure 1. The Tinkertoy Computer. It also has no predictive value.
The paper’s abstract says it much better than I could:
Abstract: It is often assumed that once a model has been calibrated to measurements then it will have some level of predictive capability, although this may be limited. If the model does not have predictive capability then the assumption is that the model needs to be improved in some way.
Using an example from the petroleum industry, we show that cases can exist where calibrated models have no predictive capability. This occurs even when there is no modelling error present. It is also shown that the introduction of a small modelling error can make it impossible to obtain any models with useful predictive capability.
We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.
There are three results in there, one expected and two unexpected.
The expected result is that models that are “tuned” or “calibrated” to an existing dataset may very well have no predictive capability. On the face of it this is obvious—if we could tune a model that simply then someone would be predicting the stock market or next month’s weather with good accuracy.
The next result was totally unexpected. The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.
The third unexpected result was the effect of error. The authors found that if there are even small modeling errors, it may not be possible to find any model with useful predictive capability.
To paraphrase, even if a tuned (“calibrated”) model is perfect about the physics, it may not have predictive capabilities. And if there is even a little error in the model, good luck finding anything useful.
This was a very clean experiment. There were only three tunable parameters. So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.
I leave it to the reader to consider what this means about the various climate models’ ability to simulate the future evolution of the climate, as they definitely are tuned or as the study authors call them “calibrated” models, and they definitely have more than three tunable parameters.
In this regard, a modest proposal. Could climate scientists please just stop predicting stuff for maybe say one year? In no other field of scientific endeavor is every finding surrounded by predictions that this “could” or “might” or “possibly” or “perhaps” will lead to something catastrophic in ten or thirty or a hundred years. Could I ask that for one short year, that climate scientists actually study the various climate phenomena, rather than try to forecast their future changes? We still are a long ways from understanding the climate, so could we just study the present and past climate, and leave the future alone for one year?
We have no practical reason to believe that the current crop of climate models have predictive capability. For example, none of them predicted the current 15-year or so hiatus in the warming. And as this paper shows, there is certainly no theoretical reason to think they have predictive capability.
The models, including climate models, can sometimes illustrate or provide useful information about climate. Could we use them for that for a while? Could we use them to try to understand the climate, rather than to predict the climate?
And 100 and 500 year forecasts? I don’t care if you do call them “scenarios” or whatever the current politically correct term is. Predicting anything 500 years out is a joke. Those, you could stop forever with no loss at all
I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe. I mean, it must get tiring for them, seeing their predictions of Thermageddon™ blown out of the water by ugly reality, time after time, without interruption. I think they’d welcome a year where they could forget about tomorrow.
Regards to all,
w.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Paddy says:
October 31, 2011 at 4:32 pm
Perhaps someday … my goal actually is to be the Mark Twain of climate science.
w.
What a great picture choice.
People have to keep in mind that the outcome of a computer program is in fact a mechanical outcome. In other words, the outcome is ALWAYS going to be the same with the same given set of inputs. This is no different than math and how the math outcome is fixed for a given set of numbers.
The fixed outcome includes the “random” number generator code that runs when you hit the random song button on your mpeg player or CD player. So the computer code that runs on that music device uses the seed of time. If you hit the random button at the exact same time again for the internal clock value used for seeding the code then the random sequence will ALWAYS be the same EVERY time. In other words, the outcome is fixed based on math.
In the physical world (nature) there is no such thing as a random event, but only events in which we don’t have quality instruments to know the inputs. A particle moving through space does not have a staff meeting and decide to change its direction without a CAUSE.
And quantum mechanics does not change this issue one bit. Simply put objects cannot move themselves. In other words if we know the weight, velocity (vectors) and all inputs of a baseball when hit means we can determine the outcome using math. The SAME math will ALWAYS give the same answer for a given set of input numbers. The base ball will do the same thing for the same given set of inputs.
However because we don’t at the quantum level or even in the simple case of a baseball have the input numbers then we simply revert to a statistical math answer. We thus state that we have “X” amount of certainly that the baseball will land inside of the ball stadium 90% of the time. Such estimates are fine and legitimate math for these cases as that all we have to go on.
However this does not mean the particle or baseball is not subject to a given set of laws and a pre-determined math outcome.
There is no demonstrable experiment or observation that shows or proves ANY kind of random event in which objects change their course without a cause.
So it is fitting to keep in mind that nature, electronic computers or that mechanical tinker toy computer are subject to a fixed set of laws and math and the outcome is pre-determined in a mechanical way for a given set of math inputs.
Albert D. Kallal
Edmonton, Alberta Canada
You say projections,
I say predictions,
Let’s call the whole thing off..
Willis, given the astonishing number of things you have done in life, it wouldn’t surprise me one bit to hear that you’d been a Steamboat navigator. 🙂
Willis, in case you haven’t noticed, most model studies are aimed at gaining understanding. But part of that understanding comes from comparing model predictions of what we should be seeing with data collected in the real world.
See http://www.oldweather.org for an on-going project using volunteer efforts to compile a computerized database of weather data found in old ship logs (UK).
Rattus Norvegicus says:
October 31, 2011 at 6:39 pm
Willis, in case you haven’t noticed, most model studies are aimed at gaining understanding. But part of that understanding comes from comparing model predictions of what we should be seeing with data collected in the real world.
And if the model predictions match reality we have not learned anything new. Only when there is discrepancies can we learn something.
@Rattus Norvegicus on October 31, 2011 at 6:39 pm:
Speak up, you appear to be mumbling.
If models were so very good there would be no role for test pilots
If I build a model of the Titanic I would have to make it sink (tuned to known events). If the Titanic had not hit the iceberg and not sunk I would have had to build my model not to sink (and would I model it to survived WW1 ?).
Is my ‘Titanic model’ curve fitting or is my the model is based on sound physics and would my Titanic models really have had any predictive value ?
Leif Svalgaard says:
October 31, 2011 at 1:46 pm
If the model is based on sound physics it usually will have predictive capability, unless it is too simple.
That is the illusion of Victorian era physics. Physics has limited predictive skills for the future even when dealing with very simple problems. For example, Gravity can’t predict the orbits of 3 or more objects, except in very special cases.
There is a fundamental reason that Willis outlined the unexpected results. Most of us assume the word follows the Victorian era view of physics. That tomorrow is just like today, but one day removed, and thus our laws of physics that work today should apply to tomorrow as well.
However, quantum mechanics tells us that this view of the world is fatally flawed. Tomorrow does not exist as a “place”. Tomorrow exists as a probability only, which introduces uncertainty into all attempts to predict the future.
That above should read.
Is my ‘Titanic model’ curve fitting or is my model based on sound physics and would my Titanic model(s) really have had any predictive value ?
People don’t expect that : “The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.”
Let’s have an example: the lottery machine. The physics and data are known but it is very difficult to forecast the lottery numbers. The runs are not repeatable – you get different results. That is what chaos theory is talking about.
Is this what the paper is preferring to?
http://www.scientificamerican.com/article.cfm?id=finance-why-economic-models-are-always-wrong
At a minimum, climate models should be subject to sensitivity analysis as outlined in this article in Scientific American. If the models truly are predictive, they should show very low sensitivity to small input errors. However, if they show high sensitivity to small input errors (chaotic) then they can never hope to be predictive, because there is always small errors in the source data (eg: temperature records) that drive the models. (Willis’ third unexpected result).
Leif do you believe the various climate models to be reasonably physics based mathematical simulations or are they highly trained fits that contain little physical information?
Pick one or the other Leif.
Lief, you are correct in this, to an extent. Negative results sometimes result in improvement to the measurement systems, sometimes the result in improvements to the models.
The most interesting negative result (which we might be seeing at the LHC) is the lack of a Higgs boson. That would be really, really interesting.
“For example, none of them predicted the current 15-year or so hiatus in the warming.”
Actually several did predict this and some predicted less warming.
the MEAN of all the models is another matter.
A model is a set of equations used to describe reality. F=MA is a model.
all physical laws are ‘models’ of reality and not reality themselves.
Leif Svalgaard says:
October 31, 2011 at 1:46 pm
“If the ‘model’ is plain curve fitting it may indeed have no predictive capability. If the model is based on sound physics it usually will have predictive capability, unless it is too simple.”
Leif, you cannot get away with this hand waving forever. What do you mean when you say that the model is based on sound physics? My guess is that you mean that the people who constructed the model understand the relevant physics. But that claim only brings up the additional question: How do they get the physics into the model? That question does not have a hand -waving answer. One answer might be that they have actually programmed the statements that make up a physical theory into a deductive engine. That would put a physical theory into the model. But you cannot have done that because if you had then you could exhibit the set of statements that make up the physical theory; after all, you just programmed them into the model. But you cannot exhibit those statements Leif because you do not have them. No climate scientist has them. In fact, the reason that you are using a model is that you have not been able to formulate your piece of climate science as a physical theory. Having no physical theory by which to judge your model, you have no idea whether your model amounts to a physical theory at all and no clue whether it has predictive ability.
Now, tell me where I am wrong.
ferd berple says:
October 31, 2011 at 7:15 pm
For example, Gravity can’t predict the orbits of 3 or more objects, except in very special cases.
Of course we can, see: http://ssd.jpl.nasa.gov/?horizons which predicts with high accuracy the orbits of a system consisting of 568670 asteroids, 3113 comets, 171 planetary satellites, 8 planets, and the Sun.
DocMartyn says:
October 31, 2011 at 7:38 pm
Leif do you believe the various climate models to be reasonably physics based mathematical simulations or are they highly trained fits that contain little physical information?
Pick one or the other Leif.
Since I know what goes on, I must pick the first choice. They are certainly not highly trained fits. That the models don’t perform well is another matter that people are very hard working on.
Whether or not, the models will ever work well enough to be useful is not known at this time.
steven mosher says:
October 31, 2011 at 7:42 pm
On this topic, you are hopeless. “F=MA” is an abbreviation for a law of physics. The statement is either true or false. In the form presented here, it is not rigorously stated and so is just an abbreviation.
You can create a model of our solar system. The model can be created out of physical objects, created on a computer, or created in some other fashion. If it is a good model it will have exactly the same behavior as our solar system. Note that the model is not true or false and, in fact, says nothing about our solar system. It need not be a set of statements at all.
How do we know that a model is a good model? We have a physical theory that enables us to predict the behavior of our solar system. If that physical theory predicts the behavior of our model with no conflicts then we know that our model of our solar system is as good as a model can get, given the existing physical theory. What is the moral of this story? The only way to judge the quality of a model is by reference to a physical theory that can predict all the behavior of the model without conflicts.
The very idea of a model in the physical sciences makes no sense whatsoever except by reference to some physical theory. No judgments about the quality of a model can be made except by reference to some physical theory. There is another moral.
If our model of our solar system turns out to be a perfect model by reference to our existing physical theory, roughly the Big Bang Theory, the model still cannot be used for prediction. The model is not a set of statements and declares nothing about our solar system. A prediction of a future event is one or more statements that will prove to be true or false at some future time. A statement that formulates a prediction can be obtained only from other statements that imply it, namely, our physical theory plus some statements of initial conditions that we have specified. Models are not statements and imply no statements and, for that reason, cannot be used for prediction.
Could I make it any clearer, any starker, any more black and white? To claim that a model can be used for prediction is what upperclass Englishmen call a “category mistake.” Such a mistake is like reporting that your tour of the campus at Berkeley revealed some beautiful buildings but you saw nothing that could be called a university. (If you don’t get the punch line, the category mistake is thinking that a university is something like a building. Can’t see the forest for the trees.)
Theo Goodwin says:
October 31, 2011 at 7:57 pm
How do they get the physics into the model? That question does not have a hand -waving answer. One answer might be that they have actually programmed the statements that make up a physical theory into a deductive engine. […] Now, tell me where I am wrong.
Physical systems are controlled by a [sometimes large] set of coupled differential equations. Given the equations and an initial set of values, the equations can be integrated forward in time. A good example is [the simple] physical system consisting of 568670 asteroids, 3113 comets, 171 planetary satellites, 8 planets and a star that make up our solar system. This is where you are wrong.
If not all of the boundary conditions are known or the processes are poorly understood, the model will not perform as well as JPL’s calculation of orbits in the solar system. This is a condition for the climate system that might [and probably will] change as time goes on. The principle is clear, though, and must be understood.
>quantum mechanics tells us that this view of the world is fatally flawed.
No that is not what it tells you. Because you don’t know where the baseball is going to land is NOT an excuse to tell me the laws of physics don’t apply to that baseball.
As I pointed out there is nothing wrong with using statistics to tell me that the baseball going to land 80% of the time in the baseball park. Howver we are doing this because we DO NOT have the ability to gather the needed information to make a precise math outcome. So this is not a proof that such a perfect math and set of rule based outcome does not exist, it simply means we don’t have that information (very big differnce here).
In other words because you don’t know where the ball is going to land is not a logical argument that the ball is not going to land in a particular spot based on a set of rules.
There is NO experiment of ANY kind that shows or proves that such outcomes are not based on a cause and outcome. Given a correct set of inputs the outcome WILL ALWAYS be the same. This is no different than telling me that one day you add up some math, and another day the numbers will now be different? They are always going to be the same.
Objects do not move or change their behavior without a cause or force being acted upon them. Objects do not move themselves.
So simply not having the ability to measure the outcome is not an excuse or logical reason to toss out causality and quantum mechanics does not change or alter this fact one bit.
>The physics and data are known but it is very difficult to forecast the lottery numbers. The runs are not repeatable – you get different results. That is what chaos theory is talking about.
If you repeat the experiment with the same input values the outcome will be the same EVERY TIME. There is not some chaos here there is simply not the means to gather the input values to plugs into the laws of physics. If you have all of the correct input values when the “lottery button” is pressed, then you can most certainly determine the outcome and it will be the SAME every single time base on the same input values.
The idea that we don’t know where the baseball is going to land is NOT an excuse or proof that we throw out the laws of physics and that the lottery balls don’t follow a set of math and rules here.
The baseball or lottery balls don’t think as they fly through the air, but are subject to a set of rules based on math. The outcome of math and numbers are the same every time for a given set of input numbers.
The balls or particles in quantum mechanics ALSO adhere to this rule and there no demonstrable experiment that shows otherwise. Not being able to gather the correct inputs is not an excuse or proof that these things don’t follow a given set of rules. Using “stats” as we do in qanta is simple done because we don’t have the abiliity to get the input values we need.
Albert D. Kallal
Edmonton, Alberta Canada
The second definition is pretty much the definition of chaos : you can know everything about a system and still can not predict the future. Chaos theory was developed to decribe the impossibility of accurate long term weather forecasting [longer than 3 days!]
With the climate change models it is equivalent to measuring a sine wave from the trough to the steepest slope and then projecting that as a trend! As the sine wave flattens out you just ignore it, or claim it is just temporary, or claim there is a counter effect, etc. What you don’t do is admit you are wrong, ever!
Normally when people favour fantasy over reality they are regarded as insane, but when you get huge amounts of money for this behaviour accusations of insanity moves to the funders, governments. But then you realise rich friends of the governmnets can make huge amounts of money ‘fighting climate change’. So there is no insanity – just a gullible public.
One way to come to some conclusion about the accuracy of GCMs is to write down all of the differential equations, and their boundary/initial conditions used in these models so we can make an assessment. I’ll wait for someone to do this for the purposes of our discussion in this thread. …Take your time…
(PS – It would also be useful to write down all of the numerical algorithms used to solve the coupled, non-linear, partial differential equations …again, take as much time as you need … I’ll wait).
steven mosher says:
October 31, 2011 at 7:42 pm
Cite? I’ve never seen one, doesn’t mean there isn’t one.
w.