Guest Post by Willis Eschenbach
There’s a lovely 2005 paper I hadn’t seen, put out by the Los Alamos National Laboratory entitled “Our Calibrated Model has No Predictive Value” (PDF).
Figure 1. The Tinkertoy Computer. It also has no predictive value.
The paper’s abstract says it much better than I could:
Abstract: It is often assumed that once a model has been calibrated to measurements then it will have some level of predictive capability, although this may be limited. If the model does not have predictive capability then the assumption is that the model needs to be improved in some way.
Using an example from the petroleum industry, we show that cases can exist where calibrated models have no predictive capability. This occurs even when there is no modelling error present. It is also shown that the introduction of a small modelling error can make it impossible to obtain any models with useful predictive capability.
We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.
There are three results in there, one expected and two unexpected.
The expected result is that models that are “tuned” or “calibrated” to an existing dataset may very well have no predictive capability. On the face of it this is obvious—if we could tune a model that simply then someone would be predicting the stock market or next month’s weather with good accuracy.
The next result was totally unexpected. The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.
The third unexpected result was the effect of error. The authors found that if there are even small modeling errors, it may not be possible to find any model with useful predictive capability.
To paraphrase, even if a tuned (“calibrated”) model is perfect about the physics, it may not have predictive capabilities. And if there is even a little error in the model, good luck finding anything useful.
This was a very clean experiment. There were only three tunable parameters. So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.
I leave it to the reader to consider what this means about the various climate models’ ability to simulate the future evolution of the climate, as they definitely are tuned or as the study authors call them “calibrated” models, and they definitely have more than three tunable parameters.
In this regard, a modest proposal. Could climate scientists please just stop predicting stuff for maybe say one year? In no other field of scientific endeavor is every finding surrounded by predictions that this “could” or “might” or “possibly” or “perhaps” will lead to something catastrophic in ten or thirty or a hundred years. Could I ask that for one short year, that climate scientists actually study the various climate phenomena, rather than try to forecast their future changes? We still are a long ways from understanding the climate, so could we just study the present and past climate, and leave the future alone for one year?
We have no practical reason to believe that the current crop of climate models have predictive capability. For example, none of them predicted the current 15-year or so hiatus in the warming. And as this paper shows, there is certainly no theoretical reason to think they have predictive capability.
The models, including climate models, can sometimes illustrate or provide useful information about climate. Could we use them for that for a while? Could we use them to try to understand the climate, rather than to predict the climate?
And 100 and 500 year forecasts? I don’t care if you do call them “scenarios” or whatever the current politically correct term is. Predicting anything 500 years out is a joke. Those, you could stop forever with no loss at all
I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe. I mean, it must get tiring for them, seeing their predictions of Thermageddon™ blown out of the water by ugly reality, time after time, without interruption. I think they’d welcome a year where they could forget about tomorrow.
Regards to all,
w.
re: Leif Svalgaard says: November 1, 2011 at 1:22 pm
Oh really Leif? The null hypothesis no longer has meaning in the scientific method. When did that get overturned and tossed? Researcher bias and agenda also no longer merits concern? Both were included in 10), factors you amazingly claim are irrelevant.
As to the other factors sounding more like ignorance or agenda driven (your claim ironically coming immediately after you claim that researcher bias is completely irrelevant wrt models) – try reading the research from the past decade, Leif. Then tell me, just how much does “black carbon”/soot affect climate? How much agreement is there over when clouds act positive feedback vs. negative in the equation – and just how much they affect overall albedo. What is the residence time of CO2 in the atmosphere? Research tells us anywhere from 5 years to hundreds of years. How well dispersed is CO2 in the atmosphere? To what degree and in which direction under what conditions do biota produce aerosols affect the climate? Have plankton levels radically decreased in the past 40 or so years, or increased?
Then tell me how these and so many other clearly uncertain factors have the underlying physics involved accurately captured in the climate models.
Willis Eschenbach says:
November 1, 2011 at 12:39 pm
“They have a number of parameters which are tuned by comparison of the model outputs to the historical data. This is not a qualitative difference at all.”
Leif says:
“I don’t think so. Perhaps you have some references.”
Wiki says:
“Climate models are systems of differential equations based on the basic laws of physics, fluid motion, and chemistry.”
BUT. . . it then adds the following:
“Parametrizations are used to include the effects of various processes. All modern AGCMs include parameterizations for:
* convection
* land surface processes, albedo and hydrology
* cloud cover”
So, it appears that although based on differential equations, there are parameterizations. The paper is therefore correct in referring to the GCM’s as tuned models, unless you assume that changing the parameters won’t alter their predictive power.
Frank K. says:
November 1, 2011 at 1:39 pm
you rely on trial and error (which I suppose is the “compromise” Gavin was referring to).
I’m sure that that was part of the equation as well. Although it is hard to know when the models ‘blow up’ because of numerical instability.
Rational Debate says:
November 1, 2011 at 1:40 pm
Oh really Leif? The null hypothesis no longer has meaning in the scientific method. When did that get overturned and tossed? Researcher bias and agenda also no longer merits concern?
The null hypothesis says that researcher bias and agenda are not relevant.
Then tell me how these and so many other clearly uncertain factors have the underlying physics involved accurately captured in the climate models.
I didn’t say that they were accurately captured. Each is an item of ongoing research and the results will be incorporated at their appropriate time. My view is that this will make the models better with time. What I complained about is the notion that this cannot be done, in principle.
Vince Causey says:
November 1, 2011 at 1:53 pm
So, it appears that although based on differential equations, there are parameterizations. The paper is therefore correct in referring to the GCM’s as tuned models, unless you assume that changing the parameters won’t alter their predictive power.
As far as I know, the parameterizations are not driven by comparison with observed data and hence are not ‘tuned’. Other [physical] concerns and empirical data determine the parameters.
Leif Svalgaard says:
November 1, 2011 at 2:08 pm
As far as I know, the parameterizations are not driven by comparison with observed data and hence are not ‘tuned’.
To make that clear: not driven by comparing model output with observed behavior to change the parameters. You might prove me wrong if you can show some examples of the opposite.
Regarding Parameterisation, Leif writes : “It is if the parameter is based on the physics involved. ”
At the end of the day, they’re approximations for large areas over large timesteps and two items that are parameterised cant properly interact within a timestep. Its not physics Leif. Whether its useful is another matter but the point is you cant wave away the uncertainty that arises from the computational efficiencies made through parameterisation.
Leif Svalgaard says:
November 1, 2011 at 2:02 pm
“I didn’t say that they were accurately captured. Each is an item of ongoing research and the results will be incorporated at their appropriate time. My view is that this will make the models better with time. What I complained about is the notion that this cannot be done, in principle.”
Their conclusion is that you cannot determine IN ADVANCE whether your model has predictive skill. You can, of course, wait and see and improve your model when you see it failed.
That’s exactly the situation the climate modelers are in. Skepticalscience has defended Hansens wrong forecasts, saying “if he had assumed a climate sensitivity of 3.4 deg C per doubling of CO2 instead of 4.5 he would have been right” – and that’s what they say NOW; they might have to revise it AGAIN in a few years (and again… and again… and again. And during all those years climate models will slowly get better.)
re: Leif Svalgaard says: November 1, 2011 at 2:02 pm
Leif, clearly they are two separate issues. The null hypothesis says nothing about researcher bias – researcher bias, however, can lead to claiming the null hypothesis has been overturned when it hasn’t. The scientific method only works when researcher bias has been eliminated or at least mostly eliminated – this aspect is quite justifiably suspect with many heavily involved in AGW research, the IPCC, and climate models based on their own statements, discounting of the null hypothesis, discounting natural variability, etc.
And I never said this cannot be done, in principle, and in time. I simply said it’s impossible to actually do NOW, in reality, and isn’t being done in the current models primarily because of insufficient knowledge of multiple factors. Yet you keep arguing against this, as if it can and is currently done in existing climate models, going so far as to imply anyone who doesn’t agree is simply ignorant or agenda driven. Perhaps the question of agenda is better asked of yourself.
I went further to note that those creating and running the climate models’ claims of reasonably accurate long term projections from these models (accurate enough to be used to affect the lives of billions of people), and considering that modern climate hasn’t breached the null hypothesis of natural variability and the state of uncertainty wrt major variables involved, leads to quite reasonable questioning of the agenda and bias of the scientists involved. You claim that any bias or agenda on their part is meaningless. WUWT??
Leif Svalgaard says:
November 1, 2011 at 12:46 pm
Of course I have references, Leif, I don’t emphatically state things without references. Here’s Gavin Schmidt on one of the many ways the GISS-E model is tuned:
Unfortunately, that tuning throws off other things (GISS-E model says ~59% cloud cover, globe actually has ~69% cloud cover), but that’s a separate question. Here’s another example from the same paper:
Now that you know that you were completely wrong, and that the GCMs are indeed “tuned” by comparison to the historical data, can we agree that the paper in question applies to GCMs?
w.
Willis Eschenbach says:
November 1, 2011 at 3:04 pm
The model is tuned (using the threshold relative humidity U00 for the initiation of ice and water clouds) to be in global radiative balance (i.e., net radiation at …
The model is tuned to be in global radiative balance which is a physical constraint, not driven by comparing model output with observations, so, no, you did not respond with a relevant reference.
Leif Svalgaard says:
October 31, 2011 at 8:22 pm
“Physical systems are controlled by a [sometimes large] set of coupled differential equations. Given the equations and an initial set of values, the equations can be integrated forward in time. A good example is [the simple] physical system consisting of 568670 asteroids, 3113 comets, 171 planetary satellites, 8 planets and a star that make up our solar system. This is where you are wrong.”
You are referencing our existing physical theory, are you not? In that case, I agree with you completely but your example is no counter example to my argument. I heartily embrace your differential equations. You are allowed to calculate. But if those differential equations do not introduce additonal primitive predicates then they cannot be treated as statements that are true or false. Do you really want to dispense with truth? Many do.
“If not all of the boundary conditions are known or the processes are poorly understood, the model will not perform as well as JPL’s calculation of orbits in the solar system. This is a condition for the climate system that might [and probably will] change as time goes on. The principle is clear, though, and must be understood.”
The only principle that is clear in your statement is that climate science, so-called, could someday become a genuine science. I certainly agree with that principle. My criticism is that climate science is not a science at this time and no one should be asserting that it is. Paraphrasing your words, “because not all of the boundary conditions are known and the processes are poorly understood, the model will not perform as well as JPL’s calculations of orbits in the solar system.” Well, right, that is exactly what I am saying. At this time you cannot claim to have a decent model of the solar system. So, why are you, in the analogous case of so-called climate science, claiming that you have a decent model there? You do not and you should lead the way among your colleagues in explaining that you do not. I wish you the greatest and quickest success but at this time you have neither model nor science.
I take my hat off to several who have replied to Leif. (Of course, I take my hat off to Leif also for being not only an eminent scientist but a good egg who engages in rational argument.) I do not have time to mention all but the following deserves special mention:
[Leif says:]
“I didn’t say that they were accurately captured. Each is an item of ongoing research and the results will be incorporated at their appropriate time. My view is that this will make the models better with time. What I complained about is the notion that this cannot be done, in principle.”
[Rational Debate says:]
“And I never said this cannot be done, in principle, and in time. I simply said it’s impossible to actually do NOW, in reality, and isn’t being done in the current models primarily because of insufficient knowledge of multiple factors. Yet you keep arguing against this, as if it can and is currently done in existing climate models, going so far as to imply anyone who doesn’t agree is simply ignorant or agenda driven. Perhaps the question of agenda is better asked of yourself.”
Yes. No one is arguing that climate science is doomed to incompleteness in principle or something similar. What we argue is that at this time, NOW, there is nothing in climate science that can qualify as either a genuine physical theory that goes beyond Arrhenius or as a validated model that captures temperature changes since 1850. Someday there will be. But all the claims of doom following from people such as Hansen and Trenberth should be withdrawn because the science is simply not there NOW.
Leif Svalgaard:
I admit that I had anticipated much better from you than your post at November 1, 2011 at 11:12 am which says;
“Richard S Courtney says:
November 1, 2011 at 11:04 am
“I see you have still not addressed the fundamental point made to you in my post at October 31, 2011 at 3:43 pm. I would be grateful if you were to answer it.”
I didn’t see an explicit question to answer, just your [somewhat muddled] opinion.”
To save others finding it, I copy my point here (below) and perhaps you can
(a) try to address it
(b) explain what you think is “muddled” in it
and
(c) explain what you think is not factual but is merely “opinion” in it .
Or is your response that I copy here (above) your admission that you know my post (copied below) shows you are wrong?
Richard
Leif Svalgaard:
Several people have rebutted your assertion at October 31, 2011 at 1:46 pm which said;
“If the ‘model’ is plain curve fitting it may indeed have no predictive capability. If the model is based on sound physics it usually will have predictive capability, unless it is too simple.”
And you have attempted to refute the rebuttals. However, you miss the important point; viz.
No model (of any kind) should be assumed to have more predictive skill than it has been demonstrated to possess.
Assuming undemonstrated model predictive skill is pseudoscience of precisely the same type as astrology. But no climate model has existed for 30, 50 or 100 years and, therefore, it is not possible for any of them to have demonstrated predictive skill over such times.
In other words, the predictions of future climate by
(a) the climate models
and
(b) examination of chicken entrails
have equal demonstrated forecast skill (i.e. none) and, therefore, they are deserving of equal respect (i.e. none).
Leif Svalgaard says:
November 1, 2011 at 3:19 pm
I see … and the fact that the albedo is tuned, not to a “physical constraint” but to be about what the historical data says? How did they do that, without reference to historical data?
Or consider Kiehl, Twentieth century climate model response and climate sensitivity:
Now, if the models were never tuned to the historical record as you claim, how do you explain that paradox? Is it just good physics? Can’t be, they give wildly different numbers for the climate sensitivity.
Kiehl points out that the models that assume large anthropogenic forcings have small climate sensitivities, and vice versa. Again, this shows that the models are not physics based. Instead, they are tuned by comparison to the historical record. How else could the forcings vary inversely with the climate sensitivity? Please don’t say that’s a result of the physics, it can’t be.
The good news? There’s fewer degrees of freedom in the models than in the observational datasets they are compared to. Or at least according to the IPCC, Chapter 8, page 596:
I like the claim, but I’m not sure I believe it.
In any case, the models have tunable parameters. To evaluate the model they compare results to observations like, as Gavin says, the albedo. Then they reset the parameters and try again, until the albedo is tuned in.
w.
Leif Svalgaard says:
October 31, 2011 at 9:47 pm
Jaye Bass says:
October 31, 2011 at 9:21 pm
Define “sound physics” in terms of a complete, validated and verified, system model for the climate. Get back to me when you are finished.
You can prove me wrong by reading and understanding Jacobsen’s text book describing the physics: http://www.stanford.edu/group/efmh/FAMbook/FAMbook.html”
A textbook does not a physical theory make. Back around 1940, Hans Reichenbach published a really wonderful book with the title “Axiomatization of the Theory of Relativity.” I have one here somewhere and you can buy it on Amazon. It is several hundred pages long. It sets out Einstein’s physical theory and does all kinds of wonderful things such as identifying Einstein’s primitive predicates. Those predicates tell you what the theory is about.
As soon as climate science has one of these babies ready to go, I will be the first to buy it and review it. However, no such book is on the horizon for climate science because they have no rigorous formulation of their (part of) physical theory, axiomatized or not. They cannot so much as guess what their primitive predicates will be in ten years. Climate scientists should accept this reality and address the world accordingly.
Theo Goodwin says:
November 1, 2011 at 3:36 pm
because the science is simply not there NOW.
But it will be eventually, and that is my point. On parameters: if the models were constantly tuned to match observations they would always be correct ‘up to yesterday’. The parameters are constrained by other considerations and the models are thus not tuned to the data, but to reasonable physics, such as radiative balance [what goes in must come out].
Leif Svalgaard says:
November 1, 2011 at 2:02 pm
“I’m sure that that was part of the equation as well. Although it is hard to know when the models ‘blow up’ because of numerical instability.”
Well, unless there is a bug somewhere (always a possibility) a “blow up” (which refers to solutions oscillating wildly until a divide by zero or NAN occurs) is usually due to a numerical instability being amplified, hence the importance of stability analysis of the numerical algorithms. This is not hard to know…
By the way, let me know what other considerations are required for determining the appropriate solution time step outside of numerical stability and accuracy…
Willis Eschenbach says:
November 1, 2011 at 4:10 pm
I see … and the fact that the albedo is tuned, not to a “physical constraint” but to be about what the historical data says? How did they do that, without reference to historical data?
I don’t think you see. Some parameters depend on measured quantities and those change with time as instruments and data get better. It makes sense to use the best data available, no?
Leif Svalgaard says:
November 1, 2011 at 12:22 pm
commieBob says:
November 1, 2011 at 12:17 pm
OK so the computer has merely to be the size of the solar system.
“We actually have such a computer, it is called ‘our solar system’ and shows us what is happening.”
Our solar system is a model of the physical theory that describes it. This is a clear and definite use of the word “model” in this scientific context. You are getting clear on the relationship between model and theory.
Frank K. says:
November 1, 2011 at 4:29 pm
hence the importance of stability analysis of the numerical algorithms. This is not hard to know…
There are more subtle ‘blowups’ where the variations just get too large without catastrophic failures.
By the way, let me know what other considerations are required for determining the appropriate solution time step outside of numerical stability and accuracy…
Computing time. If we had 1000 layers and a spatial resolution of a few meters and 4096-bit floating point numbers, there is little doubt that we can go to a much smaller time step as we can go to, say, a 500th-order Runge-Kutta method, instead of the 4th order used now. There is nothing magical about the 5 minutes.
Theo Goodwin says:
November 1, 2011 at 4:42 pm
You are getting clear on the relationship between model and theory.
Good to know that you agree.
Leif, I still see no explanation from you for the puzzle posed by Kiehl … how can the models converge on the historical temperature record, when they are using different forcings and output different climate sensitivities?
w.
PS—I said the models were tuned, and gave an example. You said no, that was tuning to a physical constraint (radiation balance). I said, OK, they’re also tuning to the albedo, that’s not a physical constraint. That is a historical measure.
Oh, no, you say … that’s a “measured quantity”.
I begin to despair, Leif. Every serious scientist I know will tell you that the climate models are tuned. The modelers say so themselves. Keihl wrote a whole paper on it.
So I invite the interested reader to peruse this piece about Kiehl’s paper, and then y’all can tell me if you think Leif is right that the models are not tuned to the historical record.
From Environmental Research Letters:
Yep. Sure ’nuff …
w.
Willis Eschenbach says:
November 1, 2011 at 5:03 pm
So I invite the interested reader to peruse this piece about Kiehl’s paper, and then y’all can tell me if you think Leif is right that the models are not tuned to the historical record.
Kiehl said: “In many models aerosol forcing is not applied as an external forcing, but is calculated as an integral component of the system. Many current models predict aerosol concentrations interactively within the climate model and this concentration is then used to predict the direct and indirect forcing effects on the climate system.”
Pielke Sr. on how models are tuned …
w.