Guest Post by Willis Eschenbach
Following up on his brilliant earlier work “The Black Swan”, Taleb has written a paper called Error, Dimensionality, and Predictability (draft version). I could not even begin to do justice to this tour-de-force, so let me just quote the abstract and encourage you to read the paper.
Abstract—Common intuitions are that adding thin-tailed variables with finite variance has a linear, sublinear, or asymptotically linear effect on the total combination, from the additivity of the variance, leading to convergence of averages. However it does not take into account the most minute model error or imprecision in the measurement of probability. We show how adding random variables from any distribution makes the total error (from initial measurement of probability) diverge; it grows in a convex manner. There is a point in which adding a single variable doubles the total error. We show the effect in probability (via copulas) and payoff space (via sums of r.v.).
Higher dimensional systems – if unconstrained – become eventually totally unpredictable in the presence of the slightest error in measurement regardless of the probability distribution of the individual components.
The results presented are distribution free and hold for any continuous probability distribution with support in R.
Finally we offer a framework to gauge the tradeoff between added dimension and error (or which reduction in the error at the level of the probability is necessary for added dimension).
Dang … talk about alarmism, that’s scary stuff. Here’s one quote:
In fact errors are so convex that the contribution of a single additional variable could increase the total error more than the previous one. The nth variable brings more errors than the combined previous n-1 variables!
The point has some importance for “prediction” in complex domains, such as ecology or in any higher dimensional problem (economics). But it also thwarts predictability in domains deemed “classical” and not complex, under enlargement of the space of variables.
Read the paper. Even without an understanding of the math involved, the conclusions are disturbing, and I trust Taleb on the math … not that I have much option.
H/T to Dr. Judith Curry for highlighting the paper on her excellent blog.