This article was sent to me by reader Peter Yodis. I found it interesting and germane to current events, so I’m sharing it here. Just a note for clarification, the very last sentence is in his original article, it is not commentary from me. – Anthony
Reposted from Machine Design.com from editor Leland E. Teschler Feb 17th, 2009
Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.
Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.
What went wrong with the modeling? That’s a subject of keen interest to engineers who must model the behavior and risks of their own complicated systems. Insights about problems with the mathematics behind financial systems come from Huybert Groenendaal, whose Ph.D. is in modeling the spread of diseases. Groenendaal is a partner and senior risk analyst with Vose Consulting LLC in Boulder, a firm that works with a wide variety of banks and other companies trying to mitigate risks.
“In risk modeling, you use a lot of statistics because you want to learn from the past,” says Groenendaal. “That’s good if the past is like the future, but in that sense you could be getting a false sense of security.”
That sense of security plays directly into what happened with banks and financial instruments based on mortgages. “It gets back to the use of historical data,” says Groenendaal. “One critical assumption people had to make was that the past could predict the future. I believe in the case of mortgage products, there was too much faith in the idea that past trends would hold.”
Therein lies a lesson. “In our experience, people have excessive confidence in their historical data. That problem isn’t unique to the financial area,” says Groenendaal. “You must be cynical and open to the idea that this time, the world could change. When we work with people on models, we warn them that models are just tools. You have to think about the assumptions you make. Models can help you make better decisions, but you must remain skeptical.”
Did the quantitative analysts who came up with ineffective financial models lose their jobs in the aftermath? Groenendaal just laughs at this idea. “I have a feeling they will do fine. If you are a bank and you fire your whole risk-analysis department, I don’t think that would be viewed positively,” he says.
Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.
“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.
We might add there is one other similarity he didn’t mention: It is doubtful anyone was ever fired for screwing up a climate model.
If you read the story of the destruction of AIG, you will see a high stakes model error that destroyed one of the largest comanies in the world. There was a great deal of skepticism at AIG at the models that said the investments that were proposed were extremely low risk. The financial modellers carried the day and AIG invested heavily into these “low risk” assets. The company went from a $70/share value to a $0.70/share because the models were wrong.
Climate models extending a couple of years may be of some value – barring any natural surprises. But climate models predicting decades or even a century into the future is utter lunacy. Who can predict what volcanos, the sun or space will deliver?
It’s the greatest swindle ever pushed onto humanity. Just goes to show there are enough gullible blokes out there waiting to be taken in hook, line and sinker.
Excellent piece! It seems to me there is one marked difference between models that quantify risk and climate models. Risk models can run for years without actually being tested against reality. Predictive climate models are tested against reality on a continuing basis and have regularly been found wanting. It seems to me the entire history of climate modeling has boiled down to a series of “backfilling” exercises every time either the model’s predictions failed to materialize or additional “past history” was unearthed that could not be duplicated within the envelope of the modeler’s assumptions.
The entire “Manmade Clobal Warming” position is based on a modeled hypothesis that has regularly failed many tests to which it has been subjected.
Or perhaps finance is far more complex than climate. 😉
Don’t necessarily blame the models here. I think the bankers just ignored the ones that gave them answers they didn’t like. For example, in the UK, the senior risk manager was fired for saying to the boss that the bank strategy was headed for disaster. See here:
http://business.timesonline.co.uk/tol/business/industry_sectors/banking_and_finance/article5701380.ece
Didn’t Gore say that the climate models could be trusted, because the economic risk models were similar?
(if anyone has a reference to this, I would appreciate it. I can’t find a reference, so I will have to put this in my Urban Myths section.)
I once saw a warning on a web page about control systems modeling. It went as follows:
Models have limits, stupidity has no limit.
I would love to provide attribution on this, but it was years ago and I don’t know where it came from.
The financial risk models had a another serious problem. In addition to the assumption regarding the past data (and note, they only used the last few years!), a more serious assumption comes from using volatility as a proxy for risk. Volatility is useful in that it can be measured. All kinds of calculations of volatility underlie the models. But in the end, volatility is a reflection of normal market conditions. Serious risk is, almost by definition, not part of the normal market.
The models should never have been used the way they were. As used, they gave a false sense of security.
Hubris.
Wouldn’t the idea of sensitive dependence on initial conditions in dynamic complex systems rule out, in principle, the possibility of accurately modeling the climate? Without perfect knowledge of initial conditions to feed into the models, how can climate modelers expect the models to be accurate for any extended time period? Could it be that meteorologists are more familiar with the work of Edward Lorenz (the father of chaos theory) and the limitations of computer models, and that’s why they are more skeptical as a group than other scientists? I haven’t heard this addressed anywhere, and I wonder what you guys think about it.
I sometimes wonder how nimble with fortran Gavin really is.
Pierre Gosselin (10:15:14) :
Or perhaps finance is far more complex than climate. 😉
For finance Kodratieff’s cycles. For climate Jose’s sun cycles around the barycenter?
Here’s a good article on why the financial models failed, by placing all the risk on a very very low probability event (called a martingale) … Which won’t ever happen, right?
http://www.slate.com/id/2201428
Has anyone ever had a go at developing a model which inserts certain disrupting events at a given frequency (e.g. volcanic eruption etc)?
Is the result that you have no ability to predict very far ahead?
Or would it say that although you can’t predict exact troughs and peaks, the long-term system performance is quite predictable?
Would such modelling give insights into the sorts of stresses needed to change the steady state e.g. instigate an ice age or a period of prolonged warmth?
Questions from a scientifically trained non-climatologist…….
Here is another example of modeling based on prior history which I believe illustrates the points made above — Dikpati el in Geophysical Research Letters 2006 published their solar cycle predictions based on their model which showed a correlation coefficient of 0.958 for simulated and observed cycle peaks for cycles 12 to 23 and 0.987 for cycles 16 through 23. — Their prediction for cycle 24: peak of 120 – 140 (Zurich sunspot number) — cycle starting 2007 – 2008.
PDF paper is available at 192.211.16.13/z/zita/articles/Dik06GRLMar.pdf
Time will tell how close their prediction came, but it doesn’t appear to me to be very accurate at present.
“In our experience, people have excessive confidence in their historical data.”
This might be a great motto for WUWT.
Computer modeling has worked so well for financial markets and the Global Warming industry. I can’t wait for it to be used with the Census.
Douglas Cootey
☆ @SplinteredMind on Twitter
✍ http://TheSplinteredMind.blogspot.com
Chicago Tea Party?!
http://www.cnbc.com/id/15840232?video=1039849853
h/t Drudge
I don’t need a complex model to tell me that making sub-prime loans to people who can’t afford them and/or have sketchy credit histories is a recipe for disaster.
And that’s what happened.
The current financial crisis is entirely anthropogenic, and involves factors completely within our power to measure and control. Given that the risk models involved did not perform correctly, and in fact led to disastrous results, why would anyone assume that the current climate models, which are inherently much more complex and extremely difficult to test, could yield a description of reality even close to being accurate
The thing is, finance is a purely human thing, weather and climater are mostly natural without much human interference.
What can happen with financial modelling is that when someone finds some sort of corelation between things, and acts upon it, that act changes the rules of finance and money so that the rule ceases to work. I think that Goodhart came up with what is now called Goodhart’s law explaining how difficult it was to control inflation by the money supply.
And these chaps did the same with all of these mortgage rates. The worked out that a certain percentage of them would be paid back based on historical figures. They overlooked the fact that if billions and billions of dollars were lent on sub-prime loans, that would change the existing rules, leading to the bust.
I am sure that even a butterfly can flap its wings and change the weather. But I dont think that man is yet capable of bringing about an ice age or melting our ice caps.
The question “Who killed Wall street” is answered here:
“We see the same problem with “climate science.” Our politicians are in the same position as bank CEO’s, looking at flawed projections, and planning policy from wrong premises”.
“Perilous Models”
Tuesday, 17 Feb 09, business
“Who killed Wall Street? Harvard MBA’s. This analysis makes more sense to me than any other explanation of the crisis. Consider also the way in which financial models were confused with reality. We see the same problem with “climate science.” Our politicians are in the same position as bank CEO’s, looking at flawed projections, and planning policy from wrong premises. With the economy already slumping, we need to expedite energy production, not tax and distort it”.
http://www.seablogger.com/?p=12875
and http://www.bloomberg.com/apps/news?pid=20601039&refer=columnist_hassett&sid=a_ac69DqFutQ
The current administration is ramping up NOAA’s climate modeling program in a big way (via the stimulus):
NOAA – NOAA will receive $230 million for operations, research and facilities, $600 million for procurement, acquisition and construction. The conference report states that $600 million is for “construction and repair of NOAA facilities, ships and equipment, to improve weather forecasting and to support satellite development. Of the amounts provided, $170,000,000 shall address critical gaps in climate modeling and establish climate data records for continuing research into the cause, effects and ways to mitigate climate change.”
Source: American Geological Institute
URL: http://www.agiweb.org/gap/legis111/update_stim0209.html
I think that the original stimulus called for a minimum of $140M to be spent on climate modeling.
I guarantee you that the administration is going to use the results from the new models to drive policy changes.
tallbloke (10:31:54)
“I sometimes wonder how nimble with fortran Gavin really is.”
Here you go…
http://www.giss.nasa.gov/tools/modelE/modelEsrc/
I encourage everyone with some knowledge of numerical modeling and basic Fortran to examine the Model E source code, and decide for yourself whether you could place any trust in the output…
Modeling finance and climate have some similarities and some dissimilarities. Here are a few, with my take on whether there MAY be similarities.
1. Basic Assumptions:
A. In business, most modelers assume Keynesian economics is correct: Austrians (myself included) would say GIGO. (Similar to AGW modeling)
B. You may assume government policies will be constant (eg. interest rates), but political winds can change. (Dissimilar – while Mother Nature can change, it’s not a political decision)
2. Feedback: If “everybody” acts the same way to business models, the effects can be large enough to affect the conditions of the markets. (Dissimilar)
3. Bias: There are so many financial indicators that is possible to set up models which correspond with preconceived ideas. Economists start with blank-slate minds, so it is easy for bias to be there without their even realizing it. (If you laid all the economists in the world end to end, they wouldn’t reach a conclusion.) (Similar)
4. Government Influence: Agencies routinely report doctored economic reports (see. shadowstats.com) and encourage rosy forecasts of the economic future. (Similar)
5. Corruption: Personal agendas for personal gains led to ratings shopping by investment banks with S&P and other ratings businesses. (Similar)
6. Sloppiness: Data may be accepted for model use which should be verified (CPI vs Surface Station data). (Similar)
7. Randomness: Trends may be “identified” which really are random fluctuations. Tossing a coin 10 times may show HHHHHHHHHH or HHTHTTTHTH; while the latter seems more reasonable, both outcomes are equally as likely, and both are random events. (Similar)
I appreciate the Due Diligence that Anthony and the readers of this site provide.
A huge difference between financial modelling (fm) and climate modelling (cm) is that the system that fm models has a potential for learning built into it whereas cm doesn’t.
Gregory Bateson (I think in his book Steps to an Ecology of Mind) uses the example of kicking a dog and a football. One can model the outcome of kicking a football pretty accurately using Newtonian physics because there is no potential learning capacity in a football. However modelling the dog’s behaviour is another matter because the system that is the dog does have an inherent learning capacity. So one might kick a dog say twenty times and model the results – say it runs away and hides – but on the twenty first kick it might bite your bum. Because a football has no capacity for learning you can forecast the result of kicking it very accurately.
A financial system has huge potential learning built into it because it contains human beings. Confidence is a function of learning. Say no more.
Physically the world has pretty well as much wealth today as it had a year ago so why has the world’s GDP/rate of growth dropped off? Answer – because we’re human and we know things today that we didn’t last year.
In my opinion it wasn’t greed that caused the credit crunch but rather a belief that fm has the same characteristics of cm (or other purely Newtonian extrapolated models).