This article was sent to me by reader Peter Yodis. I found it interesting and germane to current events, so I’m sharing it here. Just a note for clarification, the very last sentence is in his original article, it is not commentary from me. – Anthony
Reposted from Machine Design.com from editor Leland E. Teschler Feb 17th, 2009
Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.
Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.
What went wrong with the modeling? That’s a subject of keen interest to engineers who must model the behavior and risks of their own complicated systems. Insights about problems with the mathematics behind financial systems come from Huybert Groenendaal, whose Ph.D. is in modeling the spread of diseases. Groenendaal is a partner and senior risk analyst with Vose Consulting LLC in Boulder, a firm that works with a wide variety of banks and other companies trying to mitigate risks.
“In risk modeling, you use a lot of statistics because you want to learn from the past,” says Groenendaal. “That’s good if the past is like the future, but in that sense you could be getting a false sense of security.”
That sense of security plays directly into what happened with banks and financial instruments based on mortgages. “It gets back to the use of historical data,” says Groenendaal. “One critical assumption people had to make was that the past could predict the future. I believe in the case of mortgage products, there was too much faith in the idea that past trends would hold.”
Therein lies a lesson. “In our experience, people have excessive confidence in their historical data. That problem isn’t unique to the financial area,” says Groenendaal. “You must be cynical and open to the idea that this time, the world could change. When we work with people on models, we warn them that models are just tools. You have to think about the assumptions you make. Models can help you make better decisions, but you must remain skeptical.”
Did the quantitative analysts who came up with ineffective financial models lose their jobs in the aftermath? Groenendaal just laughs at this idea. “I have a feeling they will do fine. If you are a bank and you fire your whole risk-analysis department, I don’t think that would be viewed positively,” he says.
Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.
“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.
We might add there is one other similarity he didn’t mention: It is doubtful anyone was ever fired for screwing up a climate model.
Dave (11:12:57)
Thanks for Goodhart’s Law. I love collecting all these laws of behavior. Was home ownership the statistic that the government tried to manipulate?
My financial law is: Any model of the economy will fail as soon as it is accepted as correct.
We pay an awful lot of money to people to maximize returns. As soon as a correct model is created and accepted as correct people will act in such a manner as to make the model fail. In the short run, the economy is close to a zero sum game. The losers in in the financial markets will rapidly start using a correct model that made the winners successful. All the assumptions in any model will immediately fail.
For the climate, we have no effect, so a model could be successful because we cannot, by human behavior, change the assumptions of a correct model.
Dave Wendt
Dave, here is a study, Radiative Forcing of Climate Change:
Expanding the Concept and Addressing Uncertainties, http://www.nap.edu/catalog.php?record_id=11175.
Changes in climate are driven by natural and human-induced perturbations of the Earth s energy balance. These climate drivers or “forcings” include variations in greenhouse gases, aerosols, land use, and the amount of energy Earth receives from the Sun. Although climate throughout Earth s history has varied from “snowball” conditions with global ice cover to “hothouse” conditions when glaciers all but disappeared, the climate over the past 10,000 years has been remarkably stable and favorable to human civilization. Increasing evidence points to a large human impact on global climate over the past century. The report reviews current knowledge of climate forcings and recommends critical research needed to improve understanding. Whereas emphasis to date has been on how these climate forcings affect global mean temperature, the report finds that regional variation and climate impacts other than temperature deserve increased attention.
Radiative forcing is reported in the climate change scientific literature as a change in energy flux at the tropopause, calculated in units of watts per square meter (W m−2); model calculations typically report values in which the stratosphere was allowed to adjust thermally to the forcing under an assumption of fixed stratospheric dynamics.
Efficacy: The ratio of the climate sensitivity parameter λ for a given forcing agent to λ for a doubling of CO2. The efficacy E is then used to define an effective forcing Fe = f E.
Projections of climate change: An estimate of future climate, typically produced by a climate model, in response to estimates of future natural and anthropogenic forcings. Note that most projections consider only a subset of possible forcings.
————————————————————-
A radiative forcing seems to be a variable whose value is climate model dependent.
I have harbored the thought that radiative forcing, as presented in the study, is the D’Artagnan of the Three Musketeers, Finagle, Bougerre and Diddle.
First off – been reading the site for a couple months – Outstanding job!!
Funny thing about models. Guys that are geniuses program them, and are often wrong. A land surveying buddy of mine knew 2 years ago that the housing market was going to tank because a) he could see the activity decline and b) he has lived through a couple of housing corrections. The geniuses could learn something from the “dumb” surveyor.
It is the same way with climate change. The talking heads speak calamity when the summer is hot and get some geek to talk about the end of the planet. A “dumb” farmer realizes that many events like floods, droughts, etc. are on 40-50 year cycles.
If some of the stimulus money goes into a well designed automated climate network, that has many stations far from civilization, the whole thing might be worth it in savings on AGW in the long run.
What is really needed is good data, even if that data starts this year.
Roberto @ur momisugly 15:45:24):
…calculated that the kinds of losses that eventually drove LTCM under were a ten sigma event which, in their calculation, could only happen every 5 or so billion years.
But therein lies the problem with statistical models. It may even only ever happen, once, ever; but that doesn’t mean it will not happen NOW.
Goodhart’s Law: Whatever you adopt as a target ceases to be a relevant target once you have adopted it.
e.g., once a bill passes giving $$$$$ X 10^9 to those who pretend they can change the climate, their ostensible target evaporates; a new target is put in place, and the ratchet makes another click.
Ed Scott @ur momisugly 15:31:04
If the EPA does go for this, I trust that at the public hearings (there will be public hearings, won’t there?) someone mentions that this is a tax on breathing!!!
And joggers should pay more as they generate more CO2 than lazy bastards like me.
Tell me why can’t I set my climate models to the year 1 AD and ask it to predict the last 2,000 years of temperature and see how the predicted compares against the measured?
It’s called validation. It is how financial institutions do it with their market prediction models as well. Both suffer the same problem, unknown events.
Robert Wood
Public hearings will probably be handled as the public review period of five days for the stimulus pork and public debt bill.
They will have a review, but I wonder how effective public opinion will be on a preconceived outcome.
Holdren, Chu, Browner, Jackson and Salazar are hard-core greenies.
Robert (09:54:10) : (first post)
“The company went from a $70/share value to a $0.70/share because the models were wrong.”
WRONG I think. The companies managers didn’t care whether the models were right or wrong. They were making $10sM/yr in bonuses. Their personal agenda’s trumped any fiduciary responsibility they might have felt.
Sound familiar? Wall street. Climate science. Two peas in a pod.
“Statistical sampling of large populations has a good theoretical base, and the methods of assessing error are quite robust.
Your comment smacks of a political commentary clothed as technical critique. Is there a reason to only do things the way they did in the 18th century?”
Sorry, but it just scares the heck out of me when somebody says the word “robust”. It sounds too much like climate “science”.
I am surprised no one has mentioned how the weather naturally drives the economy, and how the unnatural effects of climate change hysteria has slowly strangled the economy over the past 2 decades.
A glut of sub-prime mortgages is nothing compared to world-wide cap and trade policies. Monies that would otherwise have flowed into industrial growth have been diverted into commodities, currencies, and credit markets. One bubble after another…
The elephant in the room is AGW; the economy is the pile of crap at its feet.
I’ve found the answer to both the financial problem and AGW.
It’s 48
With apologies to the late Douglas Adams – who with typical British humor is laughing at us all.
Mike
It’s 48
I thought that it was 42, unless these problems are 6 more, kinda like Spinal Tap’s amps being one louder.
The financial meltdown is not an issue of modeling, and was not caused by reliance on models. The meltdown was caused by lack of transparency in the markets (absence of proper exchanges), and by ignorance on the part of a very large firm whose area of expertise is in insuring non-financial risks, not in derivative contracts.
It is a mistake to draw parallels between the collapse of the financial system, presumably under the noses of expert modelers, and the GW controversy. Physics modeling, which pertains to GW, and financial risk control are different fields with very different paradigms.
Physical systems respond to rigid conservation laws – all the modeler needs to do is capture such laws in their analysis and computational implementation. Financial models, on the other hand, rely on assumptions about human responses and market efficiencies, issues that cannot be cast in the same mathematically rigid manner as the physical laws.
One point about GCMs vs financial models – the financial models are generally pure statistical models, while the GCMs are ‘physical’ models ie. they’re based on physics and chemistry etc.
Doesn’t make either of them good, bad, or indifferent, but comparing the two is not really apples to apples.
Way To Go, W. James!
W. James (17:57:20) :
“I am surprised no one has mentioned how the weather naturally drives the economy, and how the unnatural effects of climate change hysteria has slowly strangled the economy over the past 2 decades.
A glut of sub-prime mortgages is nothing compared to world-wide cap and trade policies. Monies that would otherwise have flowed into industrial growth have been diverted into commodities, currencies, and credit markets. One bubble after another…
The elephant in the room is AGW; the economy is the pile of crap at its feet.”
The love of truth, transparency, accountability — and science — has been trampled beneath “its” feet.
I’m going with agesilaus (11:37:40) and the Austrian School of Economics on this one.
I believe the financial models get trumped when a government implements programs that effectively privatize profits while socializing the risks. When financial institutions no longer need to factor in real risk (will I pay back my loan) because another entity (government) subsidizes that risk (don’t worry, Freddie and Fannie will cover it) they can focus totally on the reward side of the risk/reward equation. Add a little greed and a few decades of Government prodding (home ownership stimulus programs and Federal Reserve money pumping), and… Well, here we are.
RE: P. Hager (10:26:40)
“If Stupidity got us into this mess, then why can’t it get us out?” Will Rogers 😉
Roberto,
“I thought that it was 42, unless these problems are 6 more, kinda like Spinal Tap’s amps being one louder.”
Um,
Well, now you know why the financial markets failed, and AGW models are, well, less than useful…
Mike
Excellent W.James! Both AGW and Real Estate problems in one statement.I’m
a Realtor now.I saw this bubble coming and warned people-but what do I know.
AGW and its adherents are sitting on top of the AGW/Carbon credits charade,
and we had better not go there.At least selling apples on streetcorners may have a future…
John Galt (12:37:19) :
While you weren’t watching, Fortran has evolved. Fortran 95 has lots of OO features and does array operations in a single command. Sometimes something is just plain good. Minor tweaks is all it needs because it is intuitive and thus a great tool. I use it every day — including writing new code.
Who would have thought we’d still be making auto engines with round pistons and a crankshaft. When you understand how piston rings seat, you realize why the Wankel couldn’t last. The right design has a long lifespan.
The business of banks is to lend money for profit. Simple as that. Nothing else. To lend money for profit.
It wasn’t very long ago that each application for a loan was looked at by a human banker with a few years experience and judged on its merits to the best of that banker’s ability. If he wasn’t sure whether to lend, he would refer it to someone of more experience. Whether the decision was about a personal loan to buy a car, a mortgage loan to buy a house or a business loan, the same procedure applied. A significant aspect of the decision was the banker’s assessment of the likely ability of the applicant to repay the loan.
In the mid 1980s in the UK a new procedure was adopted by aggressive entrants into the mortgage loan business. They relied on the perceived value of the property first, the borrower’s ability to repay was relegated to secondary importance to such an extent that in many instances they sought no proof of employment or income other than the borrower’s assertion on the application form. They wrote a lot of business and posted massive profits, then the housing market fell and repossessed properties were sold at a loss. Vast sums were lost by the companies that took these risks. The old crusty banks and building societies suffered very few losses because they based their lending decisions on the apparent ability of the borrowers to repay, the apparent value of the property against which the loan was to be secured was also relevant but only once the borrowers were seen to be a good bet.
This type of “equity lending” went out of fashion as a result, but resurfaced in the early 2000s when exactly the same products which had proved such a disaster more than a decade before were offered once again. And now exactly the same losses are being witnessed again. Well who’d-a-thunk-it? I know one group who thunk it, crusty old lawyers like me who were involved in the litigation resulting from the last property crash. The aggressive lenders wanted to pass some of their losses onto others, so they sued valuers who they claimed had overvalued houses and conveyancing solicitors who missed defects in title. There was thousands of these cases. Even where the valuers or solicitors were found to be negligent the damages awarded were reduced to reflect the unreasonable risks the lenders took by not investigating the borrowers’ means. The most critical judgments were copied and circulated around banks to warn them of the dangers of lending to the impecunious.
Models cannot replace experienced human assessment of the risks involved in individual transactions. Indeed, it is absurd to think of models having any part to play in the process. It is only at more remote stages that models come into the picture. A bank wants to sell-on a bundle of mortgage loans (the bank gets a wad of cash now, the purchaser buys the right to receive repayments from the borrowers), how much should be paid? A bundle of mortgage loans is put up as security for a commercial loan, what is it worth? How will that bundle of loans perform over the next 2, 5, 10 years? Pump some assumptions into a computer and you’ll get an answer, but it will be unreliable unless the assumptions are sound and the assumptions cannot be sound unless average rates of default can be predicted. They can be predicted (within a reasonable margin) under old-fashioned banking procedures, but not when the borrowers’ ability to pay has never been factored into the equation.
Many of the models were concerned with the anticipated performance of financial packages several stages removed from the underlying transactions. At each stage of remove more assumptions have to be made and the input error is increased. To some extent, the assumptions are inferences from known facts, but only to some extent. Inferences can only be drawn if you have an established set of values against which to make them. For example, it is reasonable to infer that lending to someone with a good work history and restricting the loan to no more than 3 times his annual income and no more than 75% of the current market value of a house, will result in a loan that performs well. That is an inference because it can be measured against historic default levels for loans applying those criteria. Lending 5 times income and 125% of the value of the house cannot be measured against established data because there aren’t any, so you have to make assumption which are little more than semi-educated guesswork. Even solid inferences are not guaranteed to prove accurate.
I suppose it must be the same with climate models. Input data based on recent measures of temperature will be subject to error (as Mr Watts’ project is demonstrating) but it is nonetheless likely to be more accurate than data which itself is the result of assumptions about what I believe are known as “proxies” (I’ve always thought that sounds like a 1930s chocolate bar, but that’s an aside). You then have to add more and more layers of inference and assumption. The result is inevitably something so full of ifs and buts that it is matter of chance whether it produces anything even vaguely approaching reality.
Tallbloke, you take Obama’s 170 million for “modeling” and research to be a sign of further research; humm? the skeptic in me says it is funding for propoganda, as the administration is already preparing to declare CO2 to be a pollutant
I believe I saw the same program not long ago. If I remember correctly, he also stated that this (or these) models were not intended to forecast and/or predict anything, and warned that they should not be used as such.
Similar has been said about many of the GCM’s as well. I believe GCM’s are tools being used not for climate forecasting, but as a supporting actor to perpetuate a socialistic agenda, as their outcome is pre-determined!
This being my humble opinion, as a ~28 year computer science veteran who has been creating computer models of various types and derivatives the vast majority of my life.
“In my experience, computers can do many things humans cannot do. As just one example, when I studied artificial intelligence theory, algorithms, and systems, it was eye-opening to discover that a properly programmed computer can do “things” that humans just cannot do.”
My original statement is correct. There is nothing a computer can do that a human can’t do. The computer can just do it faster.
Computers are machines. Programs are instructions to the machine to do things. Humans design the programs. Humans write the programs. Humans test the programs. And humans run the programs. Therefore, humans can do the same thing the programs do but just slower.
Computers aren’t creative. They have no independence of thought. They don’t think at all. They have not independence of action. They have no cognitive understanding. They simply execute the programs. One of the biggest misnomers in Computer Science is Artificial Intelligence. There is no intelligence in a computer. And we’ve never been able to put it in there.
I first studied Artificial Intelligence in the early 80’s. Neural nets, which are often purported to be advanced, self-learning computers, are fundamentally self-weighting algorithms that can varying their behaviour based on feedback mechanisms. Expert systems are simply rule-based approaches to decision systems. Humans build the neural nets and humans write the rules. There is nothing about how these programs work that we don’t understand. The HAL 9000 of 2001: A Space Odyssey doesn’t exist today or maybe ever.
As someone who has worked in computers for over 26 years from programmer to Chief Technology Officer, I can tell you with a high-degree of confidence I understand how computers work and what they can do. They don’t do anything we don’t tell them to do. And since everything they do is something we tell them to do we can do it.
Don’t confuse that computers can do things much faster than humans with what humans can do. The point is they are just doing what we program them to do. Of course they do it orders and orders of magnitude faster than we can. Hence, the old joke, “To err is human, but to really f#*k up takes a computer.”