Guest Post by Willis Eschenbach
There’s a lovely 2005 paper I hadn’t seen, put out by the Los Alamos National Laboratory entitled “Our Calibrated Model has No Predictive Value” (PDF).
Figure 1. The Tinkertoy Computer. It also has no predictive value.
The paper’s abstract says it much better than I could:
Abstract: It is often assumed that once a model has been calibrated to measurements then it will have some level of predictive capability, although this may be limited. If the model does not have predictive capability then the assumption is that the model needs to be improved in some way.
Using an example from the petroleum industry, we show that cases can exist where calibrated models have no predictive capability. This occurs even when there is no modelling error present. It is also shown that the introduction of a small modelling error can make it impossible to obtain any models with useful predictive capability.
We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.
There are three results in there, one expected and two unexpected.
The expected result is that models that are “tuned” or “calibrated” to an existing dataset may very well have no predictive capability. On the face of it this is obvious—if we could tune a model that simply then someone would be predicting the stock market or next month’s weather with good accuracy.
The next result was totally unexpected. The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.
The third unexpected result was the effect of error. The authors found that if there are even small modeling errors, it may not be possible to find any model with useful predictive capability.
To paraphrase, even if a tuned (“calibrated”) model is perfect about the physics, it may not have predictive capabilities. And if there is even a little error in the model, good luck finding anything useful.
This was a very clean experiment. There were only three tunable parameters. So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.
I leave it to the reader to consider what this means about the various climate models’ ability to simulate the future evolution of the climate, as they definitely are tuned or as the study authors call them “calibrated” models, and they definitely have more than three tunable parameters.
In this regard, a modest proposal. Could climate scientists please just stop predicting stuff for maybe say one year? In no other field of scientific endeavor is every finding surrounded by predictions that this “could” or “might” or “possibly” or “perhaps” will lead to something catastrophic in ten or thirty or a hundred years. Could I ask that for one short year, that climate scientists actually study the various climate phenomena, rather than try to forecast their future changes? We still are a long ways from understanding the climate, so could we just study the present and past climate, and leave the future alone for one year?
We have no practical reason to believe that the current crop of climate models have predictive capability. For example, none of them predicted the current 15-year or so hiatus in the warming. And as this paper shows, there is certainly no theoretical reason to think they have predictive capability.
The models, including climate models, can sometimes illustrate or provide useful information about climate. Could we use them for that for a while? Could we use them to try to understand the climate, rather than to predict the climate?
And 100 and 500 year forecasts? I don’t care if you do call them “scenarios” or whatever the current politically correct term is. Predicting anything 500 years out is a joke. Those, you could stop forever with no loss at all
I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe. I mean, it must get tiring for them, seeing their predictions of Thermageddon™ blown out of the water by ugly reality, time after time, without interruption. I think they’d welcome a year where they could forget about tomorrow.
Regards to all,
w.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Actually Leif, according to NASA, at least last I can find and recall reading, while they don’t discount the possibility of some emission from Pioneer causing the slow down, they suspect it’s more likely that Voyager is experiencing the same effect as Pioneer – only the many atitude adjusment thrusts are likely covering it up.
http://www.space.com/448-problem-gravity-mission-probe-strange-puzzle.html
http://science.nasa.gov/science-news/science-at-nasa/2006/21sep_voyager/
http://www.guardian.co.uk/science/2002/feb/28/physicalsciences.research
steven mosher says:
October 31, 2011 at 7:42 pm
Indeed it is. However, it is not an iterative model, as climate models are. More to the point, it is also not a “tuned” model.
As a result, the situations are far from the same, far enough that an iterative tuned model has very different limitations from a “model” of the “F=MA” type.
And it is those limitations which are the subject of the cited paper.
w.
re: Leif Svalgaard & Kev-in-Uk says: October 31, 2011 at 2:45 pm
I can’t help but think of the Heisenburg Uncertainty Principal.
Besides, what does a sound physics based model have to do with AGW, Climate Models, etc? What are the sound physics that allow modelling of biota’s interaction with climate, especially on a global scale? Same re: clouds. Cosmic and solar radiation? ENSO, AMO, PDO, NAO, etc? Or any number of other major variables involved in forming climate trends?
While I’m sure that there are sound physics involved in all of these things, I’m also quite certain that humans don’t even begin to know enough to put it all together such that a “sound physics model” can be generated to model climate with any meaninful accurracy. Which is what this article is about, after all.
Fron p. 197 of the article: “The very large spike, with h roughly 10, corresponds to the truth case. We can also see notable local optima with 0 < h < 8, 30 < h < 38 and 40 < h < 45. The global optimum has a small basin of attraction around it and has proved diffcult to identify in previous
work[2], the easiest optimum to find has been the one with 30 < h < 38. The rather noisy structure of the objective surface is largely an artifact of the of the way that kg is sampled."
Now I'm not a physicist. But I am an experimenter and statistician. When I read this, it sounds to me like the problem here is an artifact of either: (1) the estimation method (a genetic algorithm); (2) the data sampling interval; and/or (c) the nature of the model itself, which gives rise to an irregular objective function in the parameters.
If I were leading a group of grad students, I would be asking them questions like these. Suppose we sample this at a much higher rate; would that help? Suppose we use a different estimation algorithm: Would that help? Suppose we don't worry about estimating one of these parameters, but instead use some outside estimate (or prior information)…Would that help?
Or, put differently. I frequently tell students that 90% of the statistical thought should have happened before you ever started collecting data (or running an experiment if you are lucky enough to be able to). You should already have simulated your model and asked: What is a good sampling plan? How many observations do I need, and at what interval? If I have a lot of confidence in some prior information, should I incorporate that, and how?
I don't see much of this kind of thinking in the article Willis cites. Perhaps that is irrelevant to the evaluation of climate science business-as-usual. But I would hate for people to take away the wrong message from an example like this one. In this example, the experimental design is given, and the poor message is perhaps as much a result of that as the underlying physics (or whatever). This is, to my mind, a good example of what can happen when you don't think carefully about the design of experiments… at least as much as an example of fundamental uselessness of models.
Just sayin.
re: Leif Svalgaard says: October 31, 2011 at 3:05 pm
I sure agree there – they ought to keep trying to improve them. But in the meantime, it seems pretty clear that the models aren’t anywhere close to accurate, and that there are reams of interactions occuring that they don’t understand yet or know well enough to model. So long as the uncertainties are so massive, it sure seems to me that repeated proclamations of how the world will be in dire straights in 100 years if we don’t massive change our behavior, standard of living, etc., all based on those highly questionable and clearly flawed model predictions is beyond absurd and far into the realm of doing serious harm to many people.
RE: Leif Svalgaard
Define “sound physics” in terms of a complete, validated and verified, system model for the climate. Get back to me when you are finished.
steven mosher says:
“… F=MA is a model”.
More to the point, it is a Law.
Climate models can’t accurately predict. They are BEST at generating taxpayer loot. But as for accurate predictions… nope. Sorry. But that’s the truth.
I’m betting that F=MA is not strictly speaking perfectly correct.
Rational Debate says:
October 31, 2011 at 8:52 pm
Actually Leif, according to NASA, at least last I can find and recall reading
Those were old news. Here is more up-to-date stuff http://planetary.org/programs/projects/pioneer_anomaly/
http://arxiv.org/PS_cache/arxiv/pdf/1107/1107.2886v1.pdf
Rational Debate says:
October 31, 2011 at 8:58 pm
I can’t help but think of the Heisenburg Uncertainty Principle.
Does not apply to macroscopic systems.
While I’m sure that there are sound physics involved in all of these things, I’m also quite certain that humans don’t even begin to know enough to put it all together such that a “sound physics model” can be generated to model climate with any meaningful accuracy
The people who do this for a living think otherwise and I will agree with them in principle. That the models may not perform well yet, should not stop us from trying very hard to improve them.
Why is there never a mention of Edward Lorenz’s seminal Chaos Theory (1960) allied to Benoit Mandelbrot’s Fractal Geometry (c. 1974)? Good lord, from Newton’s “three-body problem” on down, physical science has known that any and all “complex dynamic systems” –those with three or more mutually interacting variables– are in principle non-random but indeterminate, self-similar on every scale.
Despite cycling in context of over-broad parameters, non-linear processes are by nature effectively random-recursive/stochastic, meaning that chance-and-necessity in combination are forever beyond forecasting ken. “No-one is expert on the future,” nor indeed could so-called experts ever agree on their prognostications if they were, for nothing is ever fixed or given: “No world is beyond surprise.”
In AGW contexts, no credentialed practitioner of integrity could possibly deny these elemental constraints on even short-term, nevermind centuries-long projections. To knowingly pretend otherwise means the Green Gang comprises not fools but rather charlatans or knaves; in all too many cases, both.
Jaye Bass says:
October 31, 2011 at 9:21 pm
Define “sound physics” in terms of a complete, validated and verified, system model for the climate. Get back to me when you are finished.
You wouldn’t understand the physics anyway, nor the computer code, so what would be the point?
Jaye Bass says:
October 31, 2011 at 9:21 pm
Define “sound physics” in terms of a complete, validated and verified, system model for the climate. Get back to me when you are finished.
You can prove me wrong by reading and understanding Jacobsen’s text book describing the physics: http://www.stanford.edu/group/efmh/FAMbook/FAMbook.html
Jaye Bass says:
October 31, 2011 at 9:21 pm
Define “sound physics” in terms of a complete, validated and verified, system model for the climate. Get back to me when you are finished.
Updated presentation at: http://www.stanford.edu/group/efmh/FAMbook2dEd/index.html
and ‘come back to me when you are finished’.
re: steven mosher says: October 31, 2011 at 7:42 pm
Would you please show me where that is the case, because I don’t see it in the IPCC 2001 synthesis report figures: http://www.ipcc.ch/ipccreports/tar/wg1/figspm-5.htm
Thank you for the link.
OK. That book has been ordered from Amazon. (Stanford.edu apparently doesn’t let you order dead-tree versions of the whole volume?)
I need a reference for the climate circulation models – Who has written the best? (Yes, I saw Trenberth’s edition in the Amazon page – but I don’t trust his judgement nor his accuracy.)
Albert D. Kallal says:
October 31, 2011 at 8:26 pm
“The idea that we don’t know where the baseball is going to land is NOT an excuse or proof that we throw out the laws of physics and that the lottery balls don’t follow a set of math and rules here. ”
No is not, but that this not the issue.
Both physics and computer programs are deterministic. Same inputs give same outputs. But models are simplifications of reality. The output of reality and output of the model do not match. Real computer programs have limited precision in data storage and calculations. Real life physics can’t be expressed by simple equations. You can’t have indefinitely precise measurements of the input parameters either. When run your model thousands of steps the errors aggregate.
Cracks me up. We cant even measure temperature properly. Why would we think that a model could even come close for long term predictions. Is it worth the effort to try? Yes, but don’t confuse this with reality and try and convince me the end of the world is nigh. Just last Tuesday the models said that it was going to be clear and sunny sky’s for the next 5 days. It was raining on Thursday. Most of the time they get pretty close but even short term it is so chaotic as to almost be useless.
Speaking of proper measurements. How accurate are the CO2 readings say for the last 20 years?
Willis Eschenbach says:
October 31, 2011 at 8:56 pm
(Replying to)
steven mosher says:
October 31, 2011 at 7:42 pm
… F=MA is a model.
Indeed it is. However, it is not an iterative model, as climate models are. More to the point, it is also not a “tuned” model.
As a result, the situations are far from the same, far enough that an iterative tuned model has very different limitations from a “model” of the “F=MA” type.
I ‘m going to disagree with both of you in this way: F=MA is a valid model. FEA models of stresses and strains and engineering “models” of movement and distortion are used thousands of times ad ay – and they are accurate representations (simplifications) of the items they are trying to model.
BUT It is a correct application only with the approximations and simplifications that we (everybody!) needs to use to make the finite element analysis (FEA) “work” in the simulated worlds of perfect solids and exact-to-the-last-degree-computer models used in engineering FEA work.
You can’t make a exact FEA model of the “real world” flaws and stress risers and lattice holes and stresses and crystal structure inside an actual casting. You HAVE to make a simplified “perfect” material with a “perfect” geometry as a FEA starting point. THEN you apply the stresses and strains as the metal (or plastic or composite or liquid) moves as reacts to the stresses being studied.
But you have to begin with the assumptions of the geometry. To be accurate, your F=MA model must be simulated with near-identical, symmetric small equal-sided lattices and nodes.
Your boundary conditions need to be accurate and match the real-world you are trying to simulate .
Your transfer equations from node to node (across the “planes” of each cube to the next cube) MUST be accurate in every detail. Assumptions and simplifications must be understood and “tested” (or validated) back against the real world.
The model “edges” (sharp points, corners, radii, chamfers, fillets, holes, dividing lines or planes) must be as accurate as possible: The model will respond at these sharp edges (changes) and will exaggerate the problems actually found there.
Every FEA model running every different FEA software running on every different computer worldwide is REQUIRED to come out with the same results from the same input conditions.
The F=MA model is applied worldwide, and we get back the same results every time it is run worldwide.
THEN we get the same (within experimental accuracy!) results when every F=MA “test” is made using real materials and real crystals and real mechanical and thermal and fluid-flow tests.
But none of these are applied to the CAGW favored “models” of the climate.
They start from 0,0,0,0,0,0,0 conditions (x,y,z,Temp,time,pressure,humidity,etc) then let them run for years to see if the result approximates the world’s climate and temperatures.
They don’t “rotate” the sun to simulate night and day and the changing distance to the sun.
They don’t “rotate” the simulated earth to generate Coriolis effects on winds and ocean currents and jet streams.
They don’t model coasts and islands and mountains and local surface conditions.
Their “cells” are huge – but don’t change to match the poles as the curvature closes in.
Their “cells” are huge – but they are very, very thin with respect to height against width and depth.
Their “cells” are huge – but even these slices of atmosphere are too coarse in height (altitude) to approximate the actual changes in pressure, temperature, air flow and clouds w/r to altitude.
Their “cells” are huge – but still too large to even simulate or approximate one hurricane per cell.
There are some 23 different models, and every model is “averaged” after thousands of runs – and “bad” results get thrown out based on the prejudices of the climate “team” – .Even so, every result is known to be different, with each model starting from different assumptions and slightly different parameters and using slightly different .. So, 22 of the 23 models are wrong, even if they come out with the same result. (But they don’t come out with the same result even when run two times in a row with the same input.)
But we don’t know which of the 23 is closest to being “not too wrong.”
re: Leif Svalgaard says: October 31, 2011 at 9:35 pm
Thanks for the updated info Leif, interesting reading.
Of couse not – but your statement was: “If the model is based on sound physics it usually will have predictive capability, unless it is too simple.” In other words, you didn’t specify macro v. micro, you simply said models based on sound physics. Who couldn’t help but think of Heisenburg UP with a statement like that?
That they don’t perform well simply proves my statement. We’re not there yet. The multiple papers that are published frequently finding new unexpected major effects is further evidence. What is the accurate physics on soot? On natural variability from ENSO, AMO, PDO, NAO, etc.? On comsic ray effects? On clouds wrt positive, negative or neutral forcing? Aerosols? It wasn’t that many years ago when they discovered that biota produce aerosols under certain conditions, aerosols that affect cloud cover… the list goes on and on. All of these things show that currently we don’t begin to know enough to create a sound physics model of climate. That said, again I totally agree with you that scientists ought to keep trying to improve what we do know with regard to these issues.
Frankly, I believe that the funds expended would serve far more purpose (including resulting in usefully accurate climate models sooner) if put towards understanding these underlying variables, especially naturally occuring variables and the basic physics behind known issues such as soot, rather than towards trying to generate climate models that are based on so many different gross assumptions and which are intended primarily only to predict, er, project future climate. Particularly considering that the temperature changes we’ve seen over the last 60 years or so don’t even begin to break out of the null hypothesis, e.g., what we know to be natural variability that has occurred during this interglacial.
Leif writes : “If the model is based on sound physics it usually will have predictive capability”
Parametisation is fitting not “sound physics”. You might argue that the frequency at which the parametisation changes adequately represents sound physics but where is your evidence?
timg56 (October 31, 2011 at 2:15 pm), thanks for the Scientific American reference. I found that easier to understand than the original article.
This post reminds me of high-school math (or was it early college) about interpolation, extrapolation and polynomials. Through any set of n points one can draw a n-1-order polynomial and there’s even an explicit expression for it, the Lagrange interpolation formula (if I remember correctly). It will have no error in this context but NO predictive value, i.e. it will be totally useless for extrapolation. However, if one knows something about the physics behind the data, one can allow interpolation errors, lower the interpolation polynomial and retain some predictive value of the model. Better yet, one can fit the data to a function that has something to do with the physics and have even more predictive power (but that’s another story).
Point as nicely made as ever, Willis – and thanks indeed for introducing me to the Tinkertoy computer, which looks beautifully eccentric. May I also recommend, for those who appreciate such baroque machinery, Tim Robinson’s Meccano Computing Machinery website at http://www.meccano.us/ … the videos of his things working are among the few things which are, beyond all argument, awesome Dunno that he’s done a lot of useful computing on them, though!
[Meccano is/was a constructional toy, rather more popular in the 50s and 60s than in an age when kids grow up knowing they won;’t be doing any real engineering, ever.]
Willis,
I’d say this only shows that their objective functions are not suitable.
I find it rather weird that a calibration on a seven month period (Fig. 2b, Equation 2) gives a better-defined optimum calibration result than a calibration on a 36-month period (Fig. 2a, Equation 1). You’d expect 5 times more data to result in a less noisy objective function landscape, not the other way around.
This point is only made more clear when they add modeling error and the best tuning of their parameters (Fig. 4a) actually gives a completely wrong throw (h).
I don’t think this article says anything about predictive value of models in general.
What the “Climate Scientists” are trying to do is to piggyback on the successes of physics where first principles are well understood. The difference is huge.
Several years ago LLNL had developed a computer model of the impact of multiple laser beams on a small, aluminum covered, spherical volume of deuterium. The model showed a shockwave racing around the ball. Did it really exist? Searching, they found that a group of physicists in Italy had used an ultra high-speed camera to record laser beams striking an aluminum ball. The shock wave was there. The model can be said to be capable of doing physics from first principles and correctly predicted a physical result.
The difference between the understanding of physics that predicted the shockwave on the surface of the aluminum sphere and the models predicting the climate should be obvious to a school child. Scientists working to predict the climate are not remotely able to use physics at a fine enough level to predict lightning in their models, let alone the weather.