Inaccurate Estimation in a Gaussian World

How accurate is estimation in a Gaussian world?

Critics of finance lavish lots of attention on "fat-tail distributions" and how they call into question the way finance deals with low probability events. Nicholas Nassim Taleb is particularly angered by the way financiers use the Gaussian bell curve, affectionately known as the Great Intellectual Fraud, to "predict" and optimally hedge bets. While sympathetic to this argument, I still find the Gaussian bell curve to be a useful tool to help demonstrate how fragile and unpredictable low probability events are even in a normal world. Working on the problem also proves to be a convenient time to orient myself in R, a statistical package that I will likely be using in my undergraduate research at the University of Michigan this fall.

So here's the problem I want to look at:

Given a sample from a normally distributed population
1) How accurate of an estimate of the population standard deviation is the sample standard deviation?
2) As a result of (1), by how much do you over or under-estimate the probability of tail events?
3) How does the over or underestimation change as the event you're trying to estimate becomes more extreme (ie higher sigma event?)

I've previously written about (2), but the point of this post is to run the full simulation and what are the results.

To start, I generate a 500 element population, which has both a standard deviation and mean of 1.

From here, I take 1000 samples of 50 each, and from each sample I calculate the standard deviation. Below is the distribution of the percent errors of those standard deviations estimates.

As you can see, in spite of the fact that the standard deviation was measured 1000 times, there's still a substantial amount of spread in the distribution of standard deviations. While they average a 3% underestimate, they range from an underestimation of around 40% to an overestimation of over 20%.

This spread in the standard deviation estimate is particularly worrisome when one starts estimating the probability of low probability tail events. Below is the ratio of the actual left tail probability of a 3 sigma event divided by the estimated probability based on the standard deviation estimates. A big number implies an underestimation of the tail risk, while a small number implies an overestimation of the tail risk. Note the skew.

So when we're talking about a 3 sigma event, there's a very sizable risk of a dramatic underestimate. While most of the underestimation is concentrated around 1.14, which corresponds to a 13% underestimation, the ratio can go up to around 15, suggesting that risks can be massively underestimated even in a Gaussian world. Astute readers may note that the estimation error distribution looks to be log-normal, which at least has finite variance. But if a normal population can lead to a log-normal distribution of estimation error, it means that they end up with even more potential for underestimation. As pictured below, a log-normal population means your average underestimate is even higher at around 5 times, while the tail is even fatter.

Now that we've explored part 2, we can look at part 3; how does this change at different levels of standard deviation? I generate a population with a standard deviation , and collect data on it as per the sampling procedure above. I do this over and over again at different populations at different standard deviation levels. Surprisingly, I don't get a robust result for the estimation of error, as it's highly dependent on the population generated. However, most of the time, I do get the general trend for the mean of the log of the error ratio, pictured below. Note that it becomes more and more negative, suggesting that the mean underestimation goes down as the severeity of the tail event increases. This is likely because of the way the skewness works out with the over and under estimation of the standard deviation.

But is this necessarily good news? Not really. While the mean might be tending towards less underestimation, the proportion of underestimated risk stays relatively constant.

While the mean value of the underestimation given the risk is underestimated shoots up.

From these graphs, we can see that the average magnitude of the underestimation is increasing very quickly, while the proportion of underestimation stays relatively constant. It is actually increasing faster than it looks because the y-axis is the log of the ratio. Combined with the fact that the proportion of underestimation is staying constant, this implies the distribution is getting flatter and more dispersed, opening the possibility of catastrophic loss. The mean underestimation likely understates the actual damage that would result, as only one extremely bad underestimation can send the firm bankrupt, along with possibly the rest of the industry.

These graphs show with great detail at how insufficient risk measurement techniques like VaR or even ES are. Small probabilities are hard to estimate, even in a normal, Black-Swan free world. But add some grey swans, fragile balance sheets, and large banks, it's a time bomb that's just waiting to explode.

Take-the-Best Statistical Model

Why do we do multiple regression?

Multiple regression is the workhorse of econometrics.  Almost every empirical paper in economics relies on it, and it also forms the basis for a large majority of political science research.  But is this a valid model for prediction?  How sensitive are the results?

As it turns out, the answer is "very".  Multiple regression is sensitive to a host of issues, including normality, linearity, and low error data.  But if the real world doesn't always fit these assumptions, why do we try to use the model to predict the real world?  One thing to notice when reading the empirical papers is that they often tell you that a certain coefficient is statistically significant, but rarely does one see a given confidence interval for that coefficient.  Of course, listing confidence intervals for every coefficient, especially when there are so many, is quite cumbersome.  Yet this convenient omission often leads us to be too confident about our estimates.  How much do we really know?

Behavioral economists have long criticized this model of human decision making because there's no feasible way that we can run a regression in our head and then make a decision.  At least, I know I don't.   Although many of my friends may have used excel spreadsheets to decide where to go to college, they did not end up basing their decision on some kind of complex regression model.  It's just too computationally intractable for everyday use.

Furthermore, one key problem of multiple regression is ecological validity.  We know that the regression model predicts the sample pretty well, but does it predict the future with any accuracy?  Especially if the future is highly variable and uncertain, why should we trust our Gaussian methods that are highly sensitive to outliers? According to Gerd Gigerenzer, the most accurate rules are often not the high powered intensive statistics methods.  Rather, fast and frugal algorithms that actually limit the information they evaluate can create more accurate results.

One of the prototypical fast and frugal algorithms Gigerenzer describes is Take the Best.  While multiple regression would look at all the data and perform various tests on individual data's contribution to the dependent variable, Take the Best does a sequential evaluation of a list of key determinants.  For example, multiple regression would decide between two restaurants by looking at all the data: food quality, wait time, location, parking spots.  It would then weight each input carefully according to an equation, and then look at the results of the equations for the two restaurants.  Whichever restaurant returns the higher value is the restaurant that's chosen.  On the other hand, Take the Best would look at whether the food quality gap is high.  If so, then pick the restaurant with better food.  If the gap is not high enough, move on to the next rule and repeat this simple process.  Computationally, this would require M+1 evaluations, which is in linear time and computationally quite tractable.

Gigerenzer applied this heuristic to predicting Chicago high school dropout rates.  Given two high schools and all the associated statistics: attendance rate, proportion of low income students, social science test scores, and more, what's the most accurate way to predict which high school had a higher dropout rate?   Gigerenzer and his fellow researchers took half the population of schools and built a Take the Best model and a multiple regression model to explain the data.  No surprise, multiple regression did better, predicting over 70% of the pairs correctly while Take the Best only managed around 65%.  Yet when the two models were tested on the other half of the population, Take the Best had about a 60% accuracy rate and multiple regression barely manged around the low 50%'s.

Surprising?  Gigerenzer in Gut Feelings explains:

But why did ignoring information pay in this case? High school dropout rates are highly unpredictable-in only 60 percent of the cases could the better strategy correctly predict which school had the higher rate, (Note that 50 percent would be chance.) Just as a financial adviser can produce a respectable explanation for yesterday's stock results, the complex strategy can weigh its many reasons so that the resulting equation fits well with what we already know. Yet, as Figure 5-2 clearly shows, in an uncertain world, a complex strategy can fail exactly because it explains too much in hindsight. Only part of the information is valuable for the future, and the art of intuition is to focus on that part and ignore the rest. A simple rule that relies only on the best clue has a good chance of hitting on that useful piece of information.

Looking backwards can hurt; you might end up blindsided by what the future can hold. The data might show trends that are only valid for the sample, and not the population as a whole.  Your results won't be ecologically valid if all the data is taken into consideration.  Especially since correlations change substantially over time, Gaussian methods are more likely to offer the pretense of knowledge than knowledge itself.

This has massive policy implications.  From Gigerenzer:

According to the complex strategy, the best predictors for a high dropout rate were the school`s percentage of Hispanic students, students with limited English, and black students-in that order. In contrast, Take the Best ranked attendance rate first, then writing score, then social science test score. On the basis of the complex analysis, a policy maker might recommend helping minorities to assimilate and supporting the English as a second language program. The simpler and better approach instead suggests that a policy maker should focus on getting students to attend class and teaching them the basics more thoroughly. Policy, not just accuracy, is at stake.

Yet with these policy issues at stake, it's surprising that fast and frugal algorithms aren't used more in economics research.  One disadvantage of take the best is that it doesn't give much quantitative accuracy.  It only tells which value is higher, but not by how much.  But how much does that matter?  While multiple regression may give more statistically significant coefficients, do we really have the power to tune the economy that much?  Even the best of natural experiments don't result in parameters that don't change through time.  Romer and Romer beautifully estimate tax elasticity, but how arrogant would a person need to be to build our entire tax policy based on an estimation from an almost 80 year old data set?  DSGE quantitative accuracy is such a joke that peripheral ad-hoc models are needed to make them even somewhat useful.

These alternative statistical tools are likely to add much to our insight of models.  The development of robust heuristics will be critical in a complex world, in which calculation becomes increasingly difficult and Gaussian methods increasingly fragile.