This article was sent to me by reader Peter Yodis. I found it interesting and germane to current events, so I’m sharing it here. Just a note for clarification, the very last sentence is in his original article, it is not commentary from me. – Anthony
Reposted from Machine Design.com from editor Leland E. Teschler Feb 17th, 2009
Amid all the hand-wringing about financial systems in meltdown mode, the subject of modeling hasn’t gotten a lot of notice. Banks and other financial institutions employed legions of Ph.D. mathematicians and statistics specialists to model the risks those firms were assuming under a variety of scenarios. The point was to avoid taking on obligations that could put the company under.
Judging by the calamity we are now living through, one would have to say those models failed miserably. They did so despite the best efforts of numerous professionals, all highly paid and with a lot of intellectual horsepower, employed specifically to head off such catastrophes.
What went wrong with the modeling? That’s a subject of keen interest to engineers who must model the behavior and risks of their own complicated systems. Insights about problems with the mathematics behind financial systems come from Huybert Groenendaal, whose Ph.D. is in modeling the spread of diseases. Groenendaal is a partner and senior risk analyst with Vose Consulting LLC in Boulder, a firm that works with a wide variety of banks and other companies trying to mitigate risks.
“In risk modeling, you use a lot of statistics because you want to learn from the past,” says Groenendaal. “That’s good if the past is like the future, but in that sense you could be getting a false sense of security.”
That sense of security plays directly into what happened with banks and financial instruments based on mortgages. “It gets back to the use of historical data,” says Groenendaal. “One critical assumption people had to make was that the past could predict the future. I believe in the case of mortgage products, there was too much faith in the idea that past trends would hold.”
Therein lies a lesson. “In our experience, people have excessive confidence in their historical data. That problem isn’t unique to the financial area,” says Groenendaal. “You must be cynical and open to the idea that this time, the world could change. When we work with people on models, we warn them that models are just tools. You have to think about the assumptions you make. Models can help you make better decisions, but you must remain skeptical.”
Did the quantitative analysts who came up with ineffective financial models lose their jobs in the aftermath? Groenendaal just laughs at this idea. “I have a feeling they will do fine. If you are a bank and you fire your whole risk-analysis department, I don’t think that would be viewed positively,” he says.
Interestingly enough, Groenendaal suggests skepticism is also in order for an equally controversial area of modeling: climate change.
“Climate change is similar to financial markets in that you can’t run experiments with it as you might when you are formulating theories in physics. That means your skepticism should go up,” he says.
We might add there is one other similarity he didn’t mention: It is doubtful anyone was ever fired for screwing up a climate model.
It makes one wonder who the individuals were who developed the financial modeling software. Were they connected to Maurice Strong, Al gore, The Club of Rome, anyone in the IPCC, or any extreme political organization.
I also have no doubt that the press has no interest in looking into it further.
I’m not going to defend modeling but at least in the mortgage market the government stuck its oar in with predictably disasterous results. The Congress passed laws with the intent of forcing banks to make loans to low income buyers that they never would have made under normal circumstances. The risks were too high.
But the government creatures Freddie and Fannie ordered the banks to proceed with the promise that they would guarentee those loans. And the entire failure can be traced to that socialistic plan.
Of course government is likewise interfering with science in the climate argument.
“Don’t necessarily blame the models here. I think the bankers just ignored the ones that gave them answers they didn’t like.”
Blame the models! Models are written by people. They reflect what people know, think and believe. At best, they are no better than the understanding of the people who write them. They are often much worse.
And then there is the data that goes into the models. It is often inaccurate, incomplete, biased, or otherwise flawed. There is a basic rule in computer science that students are taught the first day they begin classes. Garbage In, Garbage Out. Bad data in, bad data out.
Computers can’t do anything humans can’t do. Computers can’t think. Computers can’t create. What computers can do is some of what humans can do only faster.
They can compute. They can and, or, xor, add, subtract, multiply, shift, and otherwise manipulate bits of data that we humans can apply meaning too. Computers have no understanding of what they do. They are simply machines.
When humans can predict the future, computers will be able to be programmed to predict the future too. Until then, all models that project forward are guesses at best. They are subject to a plethora of possible failings that ultimately leave them utterly unreliable.
Google “black swans” (Nassim Taleb)
Pierre Gosselin (10:50:59) :
I have always been a Rick Santelli fan from across the pond – and trust me – he is spot on, on this one.
George A. Reilly (11:01:20) wrote: “The current financial crisis is entirely anthropogenic, and involves factors completely within our power to measure and control. Given that the risk models involved did not perform correctly, and in fact led to disastrous results, why would anyone assume that the current climate models, which are inherently much more complex and extremely difficult to test, could yield a description of reality even close to being accurate?”
Answer: Because he/she WANTS to believe–especially anything that supports his/her philosophy and view of the world.
I think William makes the key comment here:
“For example, in the UK, the senior risk manager was fired for saying to the boss that the bank strategy was headed for disaster.”
If you are a grad student and you rock the Climate Change gravy train, you are toast. If you are a grade school teacher it is your duty to traumatize the children with drowning polar bears. If you are an auto executive begging for money you must shout ‘I BELIEVE!’ If you are an average Joe with a brain, you must keep it to yourself.
Not being an advocate is equivalent to having a swastika on your forehead and spouting the N-word.
Economic and financial models are not reality. Human nature, especially of those in government, is reality.
@David Holliday
Actually you really can’t blame the models. Why? Because there were models that showed that the disaster was looming. Thing is those were traditional (meaning old) models.
The business managers *chose* to believe the newer (and therefore better right?) models despite their obvious flaws.
William (10:24:18)
I wish I could name the program, but I did see the guy in charge of the AIG model in an interview on TV. If I remember correctly, he said that they delivered a “first pass” model to AIG, and were starting to work on the improved product when they were defunded. AIG then went out and sold the first pass model to the market.
William (10:24:18) :said
“Don’t necessarily blame the models here. I think the bankers just ignored the ones that gave them answers they didn’t like. For example, in the UK, the senior risk manager was fired for saying to the boss that the bank strategy was headed for disaster.”
William, your example highlights how “group think” evolves and reinforces acceptance of a faulty model-and thus inhibits healthy skepticism.
Likewise in academia! How many phD’s get hired into their department if they say they don’t believe in “string theory” or AGW? An academic department hires people who will work together with the other faculty. The other faculty often have a large say in who gets hired.
This group think can be so powerful that it takes a major failure of the model before the established institution accepts skepticism of primary assumptions.
For climate models, the actual historical data to work with is very thin. Accurate records go back barely 100 years and those are only for cities primarily in North America and Western Europe. We have very spotty data for the rest of the globe and that is true for much of the 20th century.
Climate models have to make assumptions for most of their ‘historical’ data. This makes estimates of warming even trickier to substantiate and is yet another reason why climate models are unable to accurately predict future climate at any useful level.
By way of a little background – I have been predicting a massive downturn in the economy here in the UK since 2003 with the comment to my friends, “that one day a major bank, a household name major bank, somewhere in the world will declare it has run out of capital.
I did not put any timescale on it but I was sure that it would resemble the South Sea Bubble. (It is worth googling if you do not know about it because the relevance to today is real).
In brief – models have been created for valuing derivative financial instruments based on a set of assumptions that typically work – very much like climate models. When things started to go wrong, adjustments were made – much as we see with the current scenario in climate models today.
When things became really bad – banks valued these esoteric securities not on a ‘mark to market’ basis but on a ‘mark to model’ basis. When they became implausible they managed to value them on a ‘mark to aspiration’ basis (pun intended).
The reality is we still do not know (and neither do any governments funding the banks) just how bad this position really is – because no-one does.
The analogy is that surely we should be challenging the climate modellers to justify their models against real world values. Had the Boards of the Banks been diligent in their questioning maybe the severity of this downturn could have been limited.
We should not let a similar lack of diligence on climate models and AGW theory drive us into making immensely costly and unwise policies on carbon taxation.
Interesting article on financial modeling. Anyone looking at models has got to look at the underlying assumptions. In the financial world however, you have a human created system. When it looks at history, particularly with the rates of subprime defaults prior to this century, subprime lending was such a small part of the mix that it did not really influence prices. Between 2003 and 2006, the extensive use of subprime lending made it possible for almost anyone to qualify for a mortgage and average home prices were driven well beyond their historical range of ~3x income and could not be sustained. The outlet for subprime borrowers who got too stretched financially, sell at a profit and move on, shut down when prices fell so the house of cards collapsed. The model made sense when subprime was a bit player but not when it was a major player. In essence, the assurances from history that it was safe to expand it was its undoing.
Climate is a natural system but there are human influences (hence all the hullabalu about CO2). Renewable systems however draw power from the wind, sun, tides, currents or whatever from very low energy density sources so are widely dispersed. In addition they tend to deliver power at less than 1/5 their rated capacity. To obtain a really significant portion of our electrical power renewlable systems, an extremely large footprint will be required and the energy drawn from the environment will change that environment in some small way. When you start adding all these small things up however, you might suddenly find air and water circulation patterns changing. We’ve already seen increases in the dead zone in the Gulf of Mexico since Biofuels boomed. Then there is loss of rain forest in Indonesia to grow palm oils, again for fuels. What I am saying in a very long winded way is has anyone looked at the changes to climate system when renewable systems are extensively deployed? Would they have more or less of an impact than fossile fuel use? I suspect that if you asked this question to someone like Roger Pielke Sr. he’d have some very interesting insights.
I remember when personal computers were in their infancy. A friend of mine wrote a program that would take a list of names and alphabetize them. He was very excited about the result of his considerable effort.
It seems to me that the Global Climate Models are pretty much the same thing. They have been written to produce the desired result, however the tremendous, and always increasing, complexity of these programs somehow clouds our minds to this simple and undeniable fact.
Pierre Gosselin (10:50:59) :
Excellent Link that you provided (below) – Loved it.
http://www.cnbc.com/id/15840232?video=1039849853
G
I read the other day that IBM has contracted to build a new supercomputer for the government that when completed will supposedly have more computational power than the top 500 supercomputers now operating combined. One might think that such a machine might approach the level of computational power needed to model something even as complex the climate system, but the problem of establishing initial conditions still applies. As far as I can tell there is no data set out there right now that would qualify as an undisputed gold standard for providing precise initial conditions for a single one of the myriad of parameters which would need to be established to have any hope of creating a reliably predictive climate model. The recent dustups about NSIDC and Stieg et al and Antarctic temps are just the most recent examples of how far we are, even with our supposedly highly advanced state of technology, from being able to derive an accurate representation of what’s happening in the climate right now, let alone hundreds, or thousands, or millions of years in the past. Despite these seemingly insurmountable logical barriers, we are told we must immediately embrace programs and policies that have immense and highly detrimental economic and social costs based on the catastrophic predictions of climate models that have never demonstrated even the flimsiest gift of predictive ability and which logic dictates they never will.
About a millennium ago the Mayans supposedly predicted the end of the world as we know it for the end of 2012. As I watch the continued daily bludgeoning of liberty and reason by the forces of authoritarianism and emotional ignorance I find myself overcome by a wistful hope that maybe they were right.
I frequently wonder how anybody can be using Fortran for anything in this day and age. It’s obsolete, but once was the primary computer language used in science and engineering.
When was Fortran last part of the curriculum for data processing or computer science? Has it been 20 years or more? Do any schools still teach it? Do engineering firms still have a large Fortran code base in use, or is it only found in government these days?
I’m not saying the models would produce better output if they were written in C++ or Java instead (GIGO: Garbage In, Garbage Out always applies), but I would like to see the models written in a contemporary language that can be run on most current operating systems.
Some good Object Oriented Programming techniques (available with a modern programming language) could also help sort out the spaghetti code and reduce possible coding errors.
A while ago I read an extensive explanation about how and why AIG’s risk assessment model failed. I believe it was in the Wall Street Journal. The model designers had difficulty simulating and projecting some of the “known unknowns” and missed altogether on the “unknown unknowns.”
Climate science is loaded with both types of unknowns. You do not have to be a scientist to understand that we do not have the skill or knowledge to design multi-decadel GCMs that simulate, much less project, future global climate.
It is refreshing to follow this blog and realize that there are many honest scientists without political agendas. But, your voices are drowned out by religious zealots, snake oil salesmen and crooks and their enablers in the media.
In Australia, the model based Global Warming Scare has resulted in the inevitable.
A SKEPTIC POLITICAL PARTY to confront the political establishment from within.
http://heliogenic.blogspot.com/2009/02/new-political-party-in-australia.html
With 90% of the population denying a human link to Global Warming this party has good perspectives to become a succes.
This is a great idea for the USA to. Anthony for President?
Frank K. (11:27:09) :
tallbloke (10:31:54)
“I sometimes wonder how nimble with fortran Gavin really is.”
Here you go…
http://www.giss.nasa.gov/tools/modelE/modelEsrc/
I encourage everyone with some knowledge of numerical modeling and basic Fortran to examine the Model E source code, and decide for yourself whether you could place any trust in the output…
I used to program large Finite Element Systems in Fortran77. It is possible to write such systems reasonably structured with a lot of experience and effort, but if you don’t have the experience, the result is a mess. I have seen many such examples, and the one you refer to is one. It is also a mixture of styles and Fortran dialects (77 vs. 90 for example), so it appears like old code that has grown and grown.
I still do programming for a living, but left Fortran years ago since it did not offer the essential and far more powerful structuring features that languages like e.g. C++ and other modern languages have.
I would say the above code does not appear to be up to current professional standards by any measure. Unless unit and integration tests exists to verify the various parts, it is very hard to tell whether it does anything useful. As it is so large, it is likely to contain many serious bugs.
With precious few exceptions, the financial risk models of which I am aware implicitly or explicitly employ Gaussian Normal Distributions. Why? Because they are simple to model and because they are simple to teach/learn.
However, there is little evidence that return distributions of any asset class–stocks, bonds, real estate, currency–are distributed normally. The Gaussian Normal Distribution is a special case. In my published research, we have found that real estate returns are distinctly non-normal with an alpha of about 1.5, which contrasts with an alpha of 2.0 for the normal distribution and an alpha of 1.0 for the Cauchy distribution typical of bonds.
The normal distribution has a real second moment, a variance, which is considered the simple measure of risk. A distribution with alpha less than 2.0 does not have a second moment. Oops!! Heck, the Cauchy distribution doesn’t even have a first moment, a mean. This is anathema to risk modelers who opt for the easy normal distribution.
We’ve seen disasters with the normal distribution before. Long Term Capital was, in large measure, an over reliance on the normal distribution in the Black-Scholes Option Model, for example.
Dave Wendt (12:35:10) :
No the Mayan’s did no such thing. Their calendar simply ended then, they made no prediction about what that meant. Having said that, it understand you sentiment. I just dislike the propagation of myths.
http://skepdic.com/maya.html
Dave Wendt says:
Using a climate model to predict general climate trends in the presence of forcings is not an initial condition problem. That is to say, the results that one gets for the general climate 100 years from now under a scenario of increasing CO2 emissions is independent of the initial conditions one chooses. (The initial conditions are important if you want to get every little up-and-down jog in the climate right, e.g., El Nino – La Nina oscillations and the like, which is why these are so difficult to forecast.)
I remember someone making the point in relation to harmonics that the further along the line you go you come to a point which is impossible to predict. You may start with two or three notes but then you’ll get harmonics and interactions and then more harmonics heading towards something of too great a complexity.
This ultimately makes a fool out of any decision however well meant the further away you get from the day of decision. If you take something to extremes it becomes its opposite someone else said. Then again the earth system is bogglingly complex so what do we expect. Maybe this is why politicians are all enigma and not too many facts. They know there will be a hatful of contradictions guaranteed.
The hope lies in our ability to see in the relatively short term (and other nonsense statements)