This article was sent to me by reader Peter Yodis. I found it interesting and germane to current events, so I’m sharing it here. Just a note for clarification, the very last sentence is in his original article, it is not commentary from me. – Anthony
Reposted from Machine Design.com from editor Leland E. Teschler Feb 17th, 2009
Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.
Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.
What went wrong with the modeling? That’s a subject of keen interest to engineers who must model the behavior and risks of their own complicated systems. Insights about problems with the mathematics behind financial systems come from Huybert Groenendaal, whose Ph.D. is in modeling the spread of diseases. Groenendaal is a partner and senior risk analyst with Vose Consulting LLC in Boulder, a firm that works with a wide variety of banks and other companies trying to mitigate risks.
“In risk modeling, you use a lot of statistics because you want to learn from the past,” says Groenendaal. “That’s good if the past is like the future, but in that sense you could be getting a false sense of security.”
That sense of security plays directly into what happened with banks and financial instruments based on mortgages. “It gets back to the use of historical data,” says Groenendaal. “One critical assumption people had to make was that the past could predict the future. I believe in the case of mortgage products, there was too much faith in the idea that past trends would hold.”
Therein lies a lesson. “In our experience, people have excessive confidence in their historical data. That problem isn’t unique to the financial area,” says Groenendaal. “You must be cynical and open to the idea that this time, the world could change. When we work with people on models, we warn them that models are just tools. You have to think about the assumptions you make. Models can help you make better decisions, but you must remain skeptical.”
Did the quantitative analysts who came up with ineffective financial models lose their jobs in the aftermath? Groenendaal just laughs at this idea. “I have a feeling they will do fine. If you are a bank and you fire your whole risk-analysis department, I don’t think that would be viewed positively,” he says.
Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.
“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.
We might add there is one other similarity he didn’t mention: It is doubtful anyone was ever fired for screwing up a climate model.
During my study Political Science (1970-1976, VU Amsterdam) I made in 1975 a simulation model of the growth of the city of Amsterdam (people, houses, companies, etc) for the period 1820-1970. For calculating my model I used the programming language DYNAMO from Forrester and Meadows with mathematics was also used for the original Club of Rome report “The Limits to growth: a global challenge” (Dennis Meadows , 1972).
My model contains around 50 variables and 120 parameters/initial values of the variables); most of the relationships between the variable were third order feedbacks.
After a intensive period of I came to the surprising conclusion that any model with that amount of degrees of freedom, you must be able to construct any desirable time-curve of all the relevant variable you want.
After that experience I distrust the outcome of many complex simulation models, and I think that also the IPCC-models can “prove” any desirable outcome.
PaulHClark (12:27:33) :
“By way of a little background – I have been predicting a massive downturn in the economy here in the UK since 2003 with the comment to my friends, “that one day a major bank, a household name major bank, somewhere in the world will declare it has run out of capital.”
Sorry but you were pre-empted by John E. Brignall at numberwatch in Number of the month. I believe it was August 2001.
DaveE.
Alan the Brit,
“This point I would like to direct to Roger Sowell, yes computers are wonderful things, but they are after all just a tool to do a job;-) I spend many hours recommending to graduate engineers they sit down with a pad & a pencil & sketch things out by hand before they ever get near a computer programme. As a 51 yo luddite I mistrust computers, & with the current political administration losing personal data left, right, & centre I feel vindicated.”
I also am/was an engineer, dating from the slide rule days. I completely agree that it is usually best to think it through first with a pad and pencil, perhaps even research a bit to see what others have published. There are, no doubt, many thousands of good software routines in regular use that are just doing what humans can do, only faster and error-free. I have written and implemented my share of those.
I think this all comes down to semantics, just what is artificial intelligence. To me, if a human cannot do it (whatever “it” is), but the computer can, that is a form of AI. The examples I gave earlier are on point.
We as humans give a label to people with great memories, or abilities to solve problems that no one else can. That label is usually “intelligent.” There are even standardized tests (albeit controversial) that purport to give a score that measures IQ. As an attorney, I had to take quite a few rather difficult tests to prove a certain level of ability before I was awarded my license to practice law. Other professions do too, and I have no intention to place attorneys in a spotlight. Professional engineers, PhDs, MDs, CPAs, CFAs, the list is long. I have a lot of respect for others without fancy degrees, too, especially my auto mechanic. Even he uses a computerized diagnostic tester from time to time; I think it has a rules-based expert system in it.
Hence, when a computer can solve a problem no human could or ever will, is it also “intelligent?”
Anthony, if this is too far off-topic, I can take this over to my energyguy’s musings blog so as not to waste your time. — Roger
“They have, at least most of them, compensation that is hugely tied to the performance of their company …”
In the short run.
In the March issue of “Wired” there’s a good cover-story article by Felix Salmon titled, “Formula for Disaster,” about David Li’s formula upon which the financial house of cards was built.
Did the models predict Congress changing the rules to require risky mortgage loans which Fannie Mae or Freddie Mac was given a quota to buy? How did the models predict fraud in those agencies so the executives could get bonuses? Why are the models being blamed for the mess?
But therein lies the problem with statistical models. It may even only ever happen, once, ever; but that doesn’t mean it will not happen NOW.
And it doesn’t tell you when it will happen again, but you know darn well it can happen. Risk Management is a convenient sedative to lull the daredevil into thier impending demise. It uses the bellcurve to deadly effect, the user unaware of the poison dulling thier senses.
I read the Black Swan.
Most of us never saw the sinister effects of Y2K because an army of programmers was called up to fix the bugs.
On jan 1, 2000, as I was filling up at the gas station, the numerical display on the pump was reading the underlying code.
I told the station attendant, who seemed to not care. Cheapest gas I have ever had the pleasure to pump.
They only fixed the code for the expected years of usage remaining.
It’s still there, waiting for the next round.
The people who knew the bugs were there, sounded the alarm, then got called up to implement the fix. They won’t be around for the next time,
and the bugs are there waiting to strike.
Who wants to bet all the outsourced programmers will come to the rescue next time?
Black Swan.
Interesting but somewhat off topic article at:
http://bighollywood.breitbart.com/dschultz/2009/02/20/ikes-not-so-famous-second-warning/
There’s a lot of good comments above, which regrettably I haven’t read, as I am impatient.
I can sensibly claim to be an expert on Y2K. I have been a computer programmer for nearly 50 years now, and when I first heard of the “Y2K bug”, I thought is was a put-on. Unfortunately, it appears that you can clean up by promoting the most utter nonsense in the world, because the vast majority of the people (these days) don’t really understand computers, and are fully prepared to believe that someone or something is out to get them. I got a call from a friend in 1997 saying that he had a very rich friend who was being cajoled into funding a Y2K company, and he asked me to advise him. I told him that there was no Y2K problem, because everybody who might care about Y2K had already fixed their software decades ago. For example (and I gave several well-known examples), the Social Security System had no Y2K problem, because at the time that they started, most retirees had been born in the 19th century, and over the next 30 years, as people born in the 20th century became dependent on SS, the software was corrected to handle it. I’m pretty sure that in the 90s, most of the retirees or disabled on their books had been born in the 20th century, but a small number had been born in the 19th century. For another example (perhaps more to the point), it was frequently claimed that the banking system would collapse in the year 2000, but in reality, almost all banks had to deal with maturities in the 21st century starting around 1970. Many mortgages at the time (and even now) had a 30 year term, so a mortgage obtained in 1970 would have matured in the year 2000. By 1997, even the most routine money market instrument, e.g. a 3 year certificate of deposit, matured in the next century. Perversely, the one thing you could be completely sure of is that all century-related issues would disappear as soon as we reached the year 2000. The notion that airplanes would fall out of the sky was even more ludicrous, because the airlines don’t keep track of the current year. (Try making a reservation more than a year in advance sometime.) The silliest one was that elevators would plunge to the ground. Can anyone imagine a reason why an elevator would care what year it is? Like, “Ohmigod, it’s the year 2000, we’d better fall to the ground.” Anyway, all I got for my trouble was a threat from one of the lawyers trying to separate this guy from his money.
Oh well, enough nattering. The same friend who asked me to provide advice about Y2K has been beating on me for years to get me interested in modeling. He’s a very smart guy, and he knows perfectly well that none of the existing models is worth the paper it’s written on. For my own part, I won’t do that, because there’s no way I could imagine every relevant detail. He was very hot on a spot Forex model that some guy named Mario from Cuba devised. I tried working with Mario, but I ended up getting blamed for everything that didn’t work out the way he wanted. Mario never told us what his great idea was.
Here’s a small detail that I don’t understand very well. In around 1963, I applied for an advertised programming job at the Harvard Meteorology Department. They turned me down, on the basis that they were looking for a Meteorology student. As it happens, they then went on to produce some of the more outrageous models that we have ever seen. I’m guessing that I wouldn’t have fit in.
One last point on financial vs climate models.
The economy is 90+% controlled by man (regulation & implementation), while the climate is 99+% controlled by God (the laws of physics, thermodynamics, etc.).
To model the economy is to model man. To model the climate is to model God. Neither one is easy, but at least God is rational and repeatable.
They say that capitalism only works with perfect knowledge and rational actors. Personally I believe it is socialism that has these requirements. Capitalism works because we as humans understand our limitations & irrationality, and if unfettered are able to quickly react to the ever-changing conditions.
Finally, the current mortgage based banking crisis is a valuation problem. Mark-to-market accounting regulations put in place in the wake of Enron are responsible for making the banks insolvent (on paper). Enron, which “bubbled up” during the California energy crisis of 2000 benefitted from: Greenies who did not want power plants in California, a hot summer, a drought, and an ill conceived deregulation scheme. Just another one of those perfect storms we seem to be getting more of lately.
Climate model says CO2 will kill planet
Governments start banning CO2
Business (capital) investment plummets
Bubbles grow and burst as dollars flow into commodities instead of capital.
Governments react by further redirecting dollars into political programs
Last man standing please turn off the light (or blow out the candle..)
“Hence, when a computer can solve a problem no human could or ever will, is it also “intelligent?””
No.
“To me, if a human cannot do it (whatever “it” is), but the computer can, that is a form of AI.”
IBM’s Deep Blue Supercomputer can compete with world class chess players. Does that make it “intelligent”?
In some respects you can say it looks intelligent. But fundamentally what Deep Blue can do is evaluate more moves further ahead in a shorter period of time than a human. Its program was intelligently written to use the brute force compute power of the supercomputer to give it an advantage over a human. That’s not intelligence.
In order to program something a programmer has to think through every step of the process and write instructions to achieve the results desired. In essence, computers are “repeaters” of the instruction sequences humans write. Intelligence is not contained in a preprogrammed sequence of binary operations no matter how “intelligent” the results appear.
Go back to that professor and ask him how that program worked. I’m certain that he’ll explain that the program was intelligently written to give whatever it controlled the ability to be “smart” with respect to how it handled various situations. That doesn’t make the program smart. It means the programmers were.
“But I will not get further into a Did so! Did Not! contest, as it is fruitless and a waste of Anthony’s and moderators’ time.”
I wasn’t going to go here but I felt obliged to respond after your latest comments.
There is no “Did so! Did not!” component to my earlier responses. I wasn’t voicing an opinion. I was explaining how computers work.
This leads me to my original point. Computer models are simply reflections of the knowledge and understanding of the people who program them.
Mr Lynn (20:41:34) :
——
‘Skeptic’ is the wrong name. It should be called the REALIST Party. It’s high time that those who oppose the politicization of science in the service of collectivist politics stopped letting the Acolytes of the Goracle define them with names like ‘denier’ and ’skeptic’ and (yes, even ‘heretic’).
Mr Lynn,
I agree with your qualification.
“Realist Party” would have been a better name.
I reject any form of stigma but let’s leave this discussion for another time because it’s OT.
DaveE (16:39:31) :
“PaulHClark (12:27:33) :
“By way of a little background – I have been predicting a massive downturn in the economy here in the UK since 2003 with the comment to my friends, “that one day a major bank, a household name major bank, somewhere in the world will declare it has run out of capital.”
Sorry but you were pre-empted by John E. Brignall at numberwatch in Number of the month. I believe it was August 2001.
DaveE”.
Gentlemen,
As we know (every body knows) our economy is a cyclic system and therefore it is not difficult to make “any” prediction. There will always be a moment where a prediction comes true. It’s only a matter of time.
The same goes for our climate.
Only the IPCC does not know that.