Guest Post by Willis Eschenbach
There’s a lovely 2005 paper I hadn’t seen, put out by the Los Alamos National Laboratory entitled “Our Calibrated Model has No Predictive Value” (PDF).
Figure 1. The Tinkertoy Computer. It also has no predictive value.
The paper’s abstract says it much better than I could:
Abstract: It is often assumed that once a model has been calibrated to measurements then it will have some level of predictive capability, although this may be limited. If the model does not have predictive capability then the assumption is that the model needs to be improved in some way.
Using an example from the petroleum industry, we show that cases can exist where calibrated models have no predictive capability. This occurs even when there is no modelling error present. It is also shown that the introduction of a small modelling error can make it impossible to obtain any models with useful predictive capability.
We have been unable to find ways of identifying which calibrated models will have some predictive capacity and those which will not.
There are three results in there, one expected and two unexpected.
The expected result is that models that are “tuned” or “calibrated” to an existing dataset may very well have no predictive capability. On the face of it this is obvious—if we could tune a model that simply then someone would be predicting the stock market or next month’s weather with good accuracy.
The next result was totally unexpected. The model may have no predictive capability despite being a perfect model. The model may represent the physics of the situation perfectly and exactly in each and every relevant detail. But if that perfect model is tuned to a dataset, even a perfect dataset, it may have no predictive capability at all.
The third unexpected result was the effect of error. The authors found that if there are even small modeling errors, it may not be possible to find any model with useful predictive capability.
To paraphrase, even if a tuned (“calibrated”) model is perfect about the physics, it may not have predictive capabilities. And if there is even a little error in the model, good luck finding anything useful.
This was a very clean experiment. There were only three tunable parameters. So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.
I leave it to the reader to consider what this means about the various climate models’ ability to simulate the future evolution of the climate, as they definitely are tuned or as the study authors call them “calibrated” models, and they definitely have more than three tunable parameters.
In this regard, a modest proposal. Could climate scientists please just stop predicting stuff for maybe say one year? In no other field of scientific endeavor is every finding surrounded by predictions that this “could” or “might” or “possibly” or “perhaps” will lead to something catastrophic in ten or thirty or a hundred years. Could I ask that for one short year, that climate scientists actually study the various climate phenomena, rather than try to forecast their future changes? We still are a long ways from understanding the climate, so could we just study the present and past climate, and leave the future alone for one year?
We have no practical reason to believe that the current crop of climate models have predictive capability. For example, none of them predicted the current 15-year or so hiatus in the warming. And as this paper shows, there is certainly no theoretical reason to think they have predictive capability.
The models, including climate models, can sometimes illustrate or provide useful information about climate. Could we use them for that for a while? Could we use them to try to understand the climate, rather than to predict the climate?
And 100 and 500 year forecasts? I don’t care if you do call them “scenarios” or whatever the current politically correct term is. Predicting anything 500 years out is a joke. Those, you could stop forever with no loss at all
I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe. I mean, it must get tiring for them, seeing their predictions of Thermageddon™ blown out of the water by ugly reality, time after time, without interruption. I think they’d welcome a year where they could forget about tomorrow.
Regards to all,
w.
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
The AGW modelers are just a money sink. Money sinks are crucial for Keynesian economics to work. Without money sinks, the money wouldn’t know where to go.
Wonderfully done. Concise and clean. Bravo.
BTW, love the tinker toy computer. Folks forget that if you can make an AND and OR gate (or even just a NAND gate) you can make a computer out of anything. Even ropes and pulleys. There are ‘hydraulic computers’ for various uses (even in old non-electronic automatic transmissions – of a sort) and you could make one out of jelly beans and Rube Goldberg buckets et. al. if you liked. Jevon’s made a “logic piano” back in the 1800s.
http://www.rutherfordjournal.org/article010103.html
Nice to see folks keeping up the ‘vision’ in a modern form 😉
Dear Mr. Eschenbach
A slight typo:
“So it looks like John Von Neumann was right, you can fit an elephant with three parameters, and with four parameters, make him wiggle his trunk.”
however the PDF states:
“you can fit an elephant with four parameters and with five, make him wiggle his trunk.”.
Appreciate the article and the PDF!
John :-#)#
Willis,
Great article. As far as the your proposal, rather than no predictions for just ONE year, I would like to see none for at least THREE years . . . with the latest in the solar cycle and ocean cycles, one year to me isn’t long enough to hold off on the predictions . . .
“I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe.”
Nope Mann is hard at it but this time Manns appearance in Minneapolis gets occupied by protesters!
http://www.msnbc.msn.com/id/45107032/ns/us_news-environment/#.Tq8DE_SP5LI
Good luck with that Willis, it’s already known there’s no clothes if there’s no models either this fashion show of showing the evil ways of man would be for nothing.
Although you would hope that at some point these model experts would realise that there lifes work really doesn’t cut the mustard and simply put there feet up and milk the cow.
Oh wait…..never mind.
That is a great find! many thanks.
“Could climate scientists please just stop predicting stuff for maybe say one year?”
My finely tuned skeptic model predicts that won’t happen.
If the ‘model’ is plain curve fitting it may indeed have no predictive capability. If the model is based on sound physics it usually will have predictive capability, unless it is too simple.
Willis: I would think that after the unbroken string of totally incorrect prognostications from Paul Ehrlich and John Holdren and James Hansen and other failed serial doomcasters, the alarmists would welcome such a hiatus from having to dream up the newer, better future catastrophe.
That has been said many times, many ways (That is not a criticism of you or of your post.) After having been wrong now for 4+ decades, shouldn’t Holdren and Ehrlich at least be modest? Except for the possibility that they exemplify the effects of the same neurological mechanism by which other people think they understand The Revelation of Saint Paul, it’s mystifying.
Even the inventor of the computer ‘had problems’….
“On two occasions, I have been asked [by members of Parliament],
‘Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’
I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a question.”
– Charles Babbage (1791-1871)
Nice post, Willis. Unfortunately all you’ll get from the peanut gallery is lots of fingers in ears and off key renditions of “La, la, la, I can’t hear you!”
When their sound physics models start doing something useful, like winning the lotto…
…I’ll start paying attention
..or even predicting tomorrow’s weather….or hurricanes
From what I read it looks like their model has a grid for the simulation, each cell in the grid represents a block of “good sand” or “poor sand”, the permeability of each cell is randomly perturbed. So one simulation run corresponds to a run of a nontrivial 2 D cellular automate. (At least I think they do only 2 dimensions, somebody correct me if I’m wrong)
Wolfram has in “A New Kind Of Science” demonstrated that even one-dimensional cellular automata with only one bit of state information per cell can show irreducibly complex behaviour; i.e. behaviour that cannot be predicted without performing the entire computation of the cellular automaton; there is no possible shortcut. He actually has a classification of his automata, a lot show trivial behaviour but then there are some that fall into this complex class. (One could say chaotic if the automaton also had an infinite number of cells; but that’s not necessary for the complex behaviour)
I won’t even go into this calibration of parameters… I think it’s safe to say that their model is, compared to what Wolfram did, already hopelessly complex; climate models even more so.
Maybe the people who still believe in predictive power of climate models would be well advised to get themselves a copy of Wolfram’s ANKOS.
Here’s a lecture by Wolfram about it.
Willis:
The models are tuned to match past climate, without a strong understanding of the factors involved and the complex relationships between those factors. There is quite a bit of fudge factoring and outright guessing in the models. For example, Hansen recently talked about the failures of the models and their overestimation of ocean heating. The problem was, they just made up that ocean heating based upon their theories, not any facts. It was the only way they could think of to make the models work.
It’s sort of like how GISSTemp works — the models assume the poles will warm faster, and since there aren’t any surface stations at the poles, just project the temps based upon the models. Viola! Global warming!
Climate models should best be considered as hypothesis. As we’ve seen time and time again, they fail to properly predict the real climate. This should take the modelers back to the drawing board, but their is too much time, money, prestige and ego invested, and too much at stake, to admit they are wrong and start over.
Leif Svalgaard says:
October 31, 2011 at 1:46 pm
Leif, the paper is about “tuned”, also called “calibrated” models like the current GCMs.
w.
Pete in Cumbria
I can see the church where Babbage was married from my living room window
http://en.wikipedia.org/wiki/Charles_Babbage
tonyb
Willis,
This looks like it may be the paper referenced in the recent Scientific American article.
http://www.scientificamerican.com/article.cfm?id=finance-why-economic-models-are-always-wrong
And then there’s Pioneer 10… off course with no explanation. Sound physics, prediction wrong.
As Stephen Wolfram has shown in Chapter Two of A New Kind of Science, some simple systems can generate complexity as complex as any complex system can… and this is the result of the simple system generating internal randomness. What this means is that you can not predict the next state of the system let alone states equivalent to a year or fifty years or five hundred years into the future. The only way to know the future states of these systems is to observe them as they unfold one state change at a time.
Many if not all of Natural systems such as we see in climate systems are this kind of simple system that generate internal randomness and maximal complexity, which means that the only way to know the future states of the climate systems is to watch them unfold in real time. This renders predicting the future of climate impossible due to first principles.
Note that this internal randomness is not the same as randomness from outside of the system ( which can still be occurring in climate systems for sure) and doesn’t derive from initial conditions.
http://pathstoknowledge.net/2009/05/01/a-new-kind-of-science-by-stephen-wolfram
I’d like to add another Larry’s Law. (I’ve lost count of what the number should be.)
No clothes does not necessarily imply no emperors.
Two points from someone who has worked inside “Big Science”:
1) Asking climate scientists to stop making people worried by predicting possibly bad outcomes is equivalent to asking them to stop looking for new grants to replace the ones that are running out.
2) When a group of scientists and engineers are sitting around a table trying to reach a decision, those with a model (however bad) to back up their suggestions will always have a tremendous advantage over those who have nothing but skepticism and general principles. The whole point of a decision by committee is to lessen the blame for bad decisions, and when committee members can in turn shift blame to a model — by saying they were just following its suggestions — well, let’s just say that this is almost always too tempting to resist. A corollary to this point is that models with more bells and whistles are more influential than those with fewer.
Willis, all these doom casters are good for media sales of newspapers and TV shows so they get plenty of encouragement for what they do. If you don’t know how deeply corrupt the media is then you’re not paying attention.
Interesting. Only this evening did a chap on BBC radio 4’s material world suggest that climate models should be used to predict the past and that way they can be checked for accuracy.
He also went on to say what I posted only a few days ago, that every RN ship has recorded the weather (temp, pressure, wind speed and direction, cloud type etc ), seawater temp, sea state, ship’s postion, speed and course on every watch without fail for a very long time and no doubt other navies have too.
Jeremy says:
October 31, 2011 at 2:17 pm
And then there’s Pioneer 10… off course with no explanation. Sound physics, prediction wrong.
All physics must be included. For the spin-stabilized Pioneers thermal emission from the spacecraft are likely the cause. For the attitude stabilized Voyagers there is no anomaly.