Inaccurate Estimation in a Gaussian World

How accurate is estimation in a Gaussian world?

Critics of finance lavish lots of attention on "fat-tail distributions" and how they call into question the way finance deals with low probability events. Nicholas Nassim Taleb is particularly angered by the way financiers use the Gaussian bell curve, affectionately known as the Great Intellectual Fraud, to "predict" and optimally hedge bets. While sympathetic to this argument, I still find the Gaussian bell curve to be a useful tool to help demonstrate how fragile and unpredictable low probability events are even in a normal world. Working on the problem also proves to be a convenient time to orient myself in R, a statistical package that I will likely be using in my undergraduate research at the University of Michigan this fall.

So here's the problem I want to look at:

Given a sample from a normally distributed population
1) How accurate of an estimate of the population standard deviation is the sample standard deviation?
2) As a result of (1), by how much do you over or under-estimate the probability of tail events?
3) How does the over or underestimation change as the event you're trying to estimate becomes more extreme (ie higher sigma event?)

I've previously written about (2), but the point of this post is to run the full simulation and what are the results.

To start, I generate a 500 element population, which has both a standard deviation and mean of 1.

From here, I take 1000 samples of 50 each, and from each sample I calculate the standard deviation. Below is the distribution of the percent errors of those standard deviations estimates.

As you can see, in spite of the fact that the standard deviation was measured 1000 times, there's still a substantial amount of spread in the distribution of standard deviations. While they average a 3% underestimate, they range from an underestimation of around 40% to an overestimation of over 20%.

This spread in the standard deviation estimate is particularly worrisome when one starts estimating the probability of low probability tail events. Below is the ratio of the actual left tail probability of a 3 sigma event divided by the estimated probability based on the standard deviation estimates. A big number implies an underestimation of the tail risk, while a small number implies an overestimation of the tail risk. Note the skew.

So when we're talking about a 3 sigma event, there's a very sizable risk of a dramatic underestimate. While most of the underestimation is concentrated around 1.14, which corresponds to a 13% underestimation, the ratio can go up to around 15, suggesting that risks can be massively underestimated even in a Gaussian world. Astute readers may note that the estimation error distribution looks to be log-normal, which at least has finite variance. But if a normal population can lead to a log-normal distribution of estimation error, it means that they end up with even more potential for underestimation. As pictured below, a log-normal population means your average underestimate is even higher at around 5 times, while the tail is even fatter.

Now that we've explored part 2, we can look at part 3; how does this change at different levels of standard deviation? I generate a population with a standard deviation , and collect data on it as per the sampling procedure above. I do this over and over again at different populations at different standard deviation levels. Surprisingly, I don't get a robust result for the estimation of error, as it's highly dependent on the population generated. However, most of the time, I do get the general trend for the mean of the log of the error ratio, pictured below. Note that it becomes more and more negative, suggesting that the mean underestimation goes down as the severeity of the tail event increases. This is likely because of the way the skewness works out with the over and under estimation of the standard deviation.

But is this necessarily good news? Not really. While the mean might be tending towards less underestimation, the proportion of underestimated risk stays relatively constant.

While the mean value of the underestimation given the risk is underestimated shoots up.

From these graphs, we can see that the average magnitude of the underestimation is increasing very quickly, while the proportion of underestimation stays relatively constant. It is actually increasing faster than it looks because the y-axis is the log of the ratio. Combined with the fact that the proportion of underestimation is staying constant, this implies the distribution is getting flatter and more dispersed, opening the possibility of catastrophic loss. The mean underestimation likely understates the actual damage that would result, as only one extremely bad underestimation can send the firm bankrupt, along with possibly the rest of the industry.

These graphs show with great detail at how insufficient risk measurement techniques like VaR or even ES are. Small probabilities are hard to estimate, even in a normal, Black-Swan free world. But add some grey swans, fragile balance sheets, and large banks, it's a time bomb that's just waiting to explode.

The Danger of Promises

Promises are powerful, so don't make a promise you can't keep

Evan Soltas had an interesting post on the power of promises in the context of currency bands and monetary policy, but we should also remember that banks shouldn't try to make promises they can't keep. We all should be very worried when we see graphs that show sudden decreases in volatility, as Evan shows in his post. Whenever policymakers suppress volatility, we need to wonder where those pressures have gone, and why they have disappeared. More often then not, manufactured stability leads to calm periods punctuated by sudden change; they become Type 2 extremistan regimes, as pictured below:

So why did the currency peg work? The peg only makes sense in the context of Scott Sumner's argument that the currency peg is a form of monetary policy commitment. The undervalued currency increases aggregate demand, thereby filling the output gap. However, it should be observed that, given the undervalued currency, maintaining the peg leads eventually to above trend NGDP growth and an economy that's running "hot". However, the "hot" economy would call the currency peg into question. This is a classic example of Mundell's policy trilemma. The SNB cannot pursue an exchange rate peg, independent monetary policy, and capital mobility at the same time. Conditions are stable now only because the exchange rate peg matches the objective of an independent monetary policy. However, once the output gap starts to narrow the credibility of the exchange rate peg will be questioned.

This is where the expectations channel starts creating weird dynamics. If the market expects the SNB to pursue monetary policy that prioritizes internal conditions, then the market should expect that currency to appreciate in the future as the output gap is filled. However, because of inertial central bank policy, this will only occur when the currency is undervalued to such an extent that it is no longer feasible for the central bank to maintain the peg. At that point, we should expect to see a very sudden adjustment as the SNB comes under fire from speculators. This then creates a vicious loop, as the SNB's attempts to maintain the peg only increase the supply of currency, of which speculators buy increasing amounts as they now know that the currency will appreciate.

When the SNB is forced to rebalance the currency, it will shake up markets as it unwinds its large balance sheet of foreign assets. Since the bank would have been accumulating these assets for a long time, their sudden liquidation is likely to be a "fat tail" event, which, on one hand may not cause that much damage, but on the other hand may cause positive feedback loops to devastate markets in unknown ways.

While this analysis is in the specific context of the Swiss central bank, this argument has implications for NGDP targeting in developing countries as well. The problem with the SNB's currency peg is that it prioritizes one objective (exchange rate) over all others (including NGDP). However, when domestic politics rears its head, a internal measures such as NGDP will end up trumping external measures such as the exchange rate. The crisis arises from a sudden reversal in priorities.

This exchange rate-NGDP tension is very important for developing economies. If these countries pursue capital mobility, then they may need to compromise part of their monetary policy to maintain an exchange rate band. Thus, this is another source of possible fragility in a NGDP target, as other objectives, such as the exchange rate, suddenly come to the forefront.

Edit: Evan Soltas gave me a further explanation on how the "peg" is really a floor, as well as an explanation of the general macro conditions in Switzerland. I failed to take the time to analyze them, so the conclusions are slightly different. There's still possibilities of non-linear dynamics, and those are explained in the comment thread. I've also written a new post reflecting back of these issues here.

A Look Back Through the Lens of Complexity: "Bigger is Better"

When fears about the Euro were not always so well developed

Recent developments in Europe have created fears that Greece will soon exit the Euro. The spectacular way that the Euro has failed has led some to wonder why the Euro came together in the first place. Interestingly, the concern about a systemic Eurozone crisis did not really register when the Greek crisis started. Perhaps there were some musing that the crisis could spread, but it seemed that people believed that there would be a rescue package and that the crisis would pass. Size would come to the rescue, and an economy larger than that of the United States, the EU would end the threat of financial contagion.

If only they were right.

We now see massive capital outflows from the periphery and a frighteningly fast flight to quality within the Eurozone. But wasn't size supposed to blunt these impacts? While this narrative of "strength through fragility" seems hopelessly naive now, it's interesting to note that even in May/June of 2010 Foreign Affairs published an article by Richard Rosecrance extolling the benefits of size and larger currency arrangements in an era of turbulent capital flows. Rereading the essay, many of the key passages remind us how hindsight is 20-20, and that the narrative of the day can often push policy in the wrong direction.

The essay starts with a description of the Asian financial crises that roiled markets in the late 1990's as a justification for larger economic zones and currency areas;

But eventually the trading-state model ran into unexpected problems. Japanese growth stalled during the 1990s as U.S. growth and productivity surged. Many trading states were rocked by the Asian financial crisis of 1997-98, during which international investors took their money and went home. Because Indonesia, Malaysia, Thailand, and other relatively small countries did not have enough foreign capital to withstand the shock, they had to go into receivership. As Alan Greenspan, then the U.S. Federal Reserve chair, put it in 1999, "East Asia had no spare tires." Governments there devalued their currencies and adopted high interest rates to survive, and they did not regain their former glory afterward.

Russia, meanwhile, fell afoul of its creditors. And when Moscow could not pay back its loans, Russian government bonds went down the drain. Russia's problem was that although its territory was vast, its economy was small. China, India, and even Japan, on the other hand, had plenty of access to cash and so their economies remained steady. The U.S. market scarcely rippled. 

Small trading states failed because the assumptions on which they operated did not hold. To succeed, they needed an open international economy into which they could sell easily and from which they could borrow easily. But when trouble hit, the large markets of the developed world were not sufficiently open to absorb the trading states' goods. The beleaguered victims in 1998 could not redeem their positions by quick sales abroad, nor could they borrow on easy terms. Rather, they had to kneel at the altar of international finance and accept dictation from the International Monetary Fund, which imposed onerous conditions on its help. In the aftermath of the crisis, the small trading states vowed never to put themselves in a similar position again, and so they increased their access to foreign exchange through exports. Lately, they have proposed forming regional trade groups to get larger economically, by negotiating a preferential tariff zone in which to sell their goods and perhaps a currency zone in which to borrow cash.

The story of the Eurozone has shown us that increasing lending through large currency zones is a fool's errand. Private capital flows to the periphery destroyed the PIGS' competitiveness even though they were being quite fiscally conservative. The ability to borrow easily in good times has actually worsened situations, as at the first sign of debt troubles capital can quickly move out of the periphery states, again forcing them to "kneel at the altar of international finance and accept dictation from the International Monetary Fund."  Comically, Greece has been forced to accept "onerous conditions" from Germany for their help in the bailouts. While larger currency areas may secure cheaper funding in the short run, long run capital prospects don't improve.  In a sense, the larger currency area is a form of manufactured stability. It allows volatility to be restrained in good times, only to become fearsome in times of crisis. It's a blowup strategy, with all of the risk in the far left tail.

Moreover, a common currency area actually prevents a country from pursuing the other solution to sudden financial shocks: higher trade in goods. While the Asian economies faced the problem that "the large markets of the developed world were not sufficiently open to absorb the trading states' goods", Greece is facing the exact opposite problem. Past capital flows have left the economy severely overvalued and other markets may be willing to buy Greek goods, if only Greece could devalue its currency!  To worsen the problem, fears about Greece's debt situation lowers global equity values, further reducing global demand for Greek goods! A common currency area takes away from the external devaluation adjustment mechanism and therefore only aggravates the problems that small countries face in financial crises.

The fact that there is no adjustment mechanism makes the larger market available to each state less useful.  According to Rosencrance:

The 27 states that now compose the European Union will soon be accompanied by almost ten others, making Europe stretch from the Atlantic to the Caucasus. Member states have benefited from participating in an enlarged market extending beyond their national borders. The absence of tariffs in the EU allows greater cross- border commercial cooperation, which promotes specialization and efficiency and provides consumers in the member states with cheaper goods for purchase. Over time, as economists such as Andrew Rose and Jeffrey Frankel have shown, such trade zones increase their members' trade volume and GDP growth. There are also administrative advantages: southern and eastern European states with less advanced economies have found help and tutelage from veteran EU members and have not been allowed to fail (even if their fiscal policies have been reined in). 

But when Germany is running a massive current account surplus vis-a-vis virtually every other member of the Eurozone, the cross-border commercial cooperation is a joke. The cheaper German goods for purpose are only that way because of past capital flows that rendered periphery states uncompetitive. Given the horrendous costs of internal devaluation and the low likelihood that it would restore problems with capital structure, any possible microeconomic efficiency from trade is being swamped by disastrous levels of youth unemployment and civil unrest. When there's no transfer union, these asymmetric effects of trade flows on the different countries become incredibly important. We can no longer say "Europe is benefiting", we instead see the periphery on the verge of a full-fledged financial contagion.

It is also interesting to note that Rosencrance also makes an institutional argument here. Through the interaction of the "responsible" core states with the "irresponsible" periphery states, the periphery states will be brought up to the core states' level of institutional maturity. However, large capital flows promoted by a common currency rendered these kinds of supply side reforms unnecessary, and this institutional shift never happened. And as the Eurozone drama is unfolding, it is quite possible that some periphery states will be allowed to fail as the "help and tutelage from veteran EU members" abandons them.

In addition to these new harsh economic realities, the political realities of the situation don't seem to match up with Rosencrance's arguments either. According to the essay:

The peaceful expansion of trade blocs today, moreover, is likely to bring outsiders in rather than keep them out. It has done so in Europe and to some degree in North America and Asia as well. Self-sufficient trade blocs are impossible and will not be sought after. The key to a successful trade group, in fact, is that as it grows, it attracts sellers from the outside. 

What would China, India, and Japan do if the United States and the EU formed a trade partnership? They would not find an Asian pact a satisfactory rejoinder to the transatlantic combination. Since the major markets of the world are located in Europe and North America, Asian exporting nations would have to continue to sell to them. And if Japan eventually joined the partnership, the stakes for China and India would rise. China and India might not be significantly challenged if they could substitute domestic sales for exports. But even they, as big as they are, could not do so entirely. However important Chinese consumption becomes, it will not be able to sop up all the goods that China currently exports to technologically advanced and luxury markets in Europe, the United States, and Japan. To avoid falling behind, Beijing and New Delhi would need a continuing association with markets elsewhere.

What all this means is that the patterns of global politics and economics that have prevailed for the last half millennium are increasingly outmoded. During that period, eight out of the 11 instances of a new great power's rise led to a "hegemonic war." With a potential Chinese challenge looming in the 2020s, the odds would seem stacked in favor of conflict once again, and in other eras it would have made sense to bet on it. 

Yet military conflict is not likely to occur this time around, because even if political power sometimes repels, today economic power attracts. The United States does not need to fight rising challengers such as China or India or even to balance one off against another. It can use its own market capacity, combined with that of Europe, to draw surging protocapitalist states into its web. 

During the Cold War, the economic force of the West eventually surpassed and subverted even the heavy industrial growth of the Soviet economy. In the 1980s, the attractions of North Atlantic, Japanese, and even South Korean capitalism were a critical factor in Soviet leader Mikhail Gorbachev's decision to renew his country's economic and political system -- and end the Cold War. They also helped stimulate Deng Xiaoping's reforms in China after 1978.
Now that the formula for capitalist economic success has become widely understood and been replicated, Western economic magnetism will stem not just from the triumphs of individual economies but from their development as an increasingly integrated group. The expansion and agglomeration of economies in Europe -- and perhaps also across the Atlantic -- will serve as a beacon for isolated successes such as those in Asia.

The argument here seems to be that larger scale economic integration would be a virtuous cycle and therefore serve to moderate international conflict. As a result of integration in certain regions, other states will be forced to integrate with them, creating a unified global trade and financial regime. However, if large economic regime are important to this process, doesn't this just heighten the fragility embedded in the international system? This is especially vivid for the Euro now as, if anything, the collapse of the Eurozone would severely discredit the argument that open, integrated, western economies are the correct way forward. The failure of agglomeration would serve as a deterrent to the "isolated successes such as those in Asia."

A look back on such essays about the Euro serves as a reminder on how limited our capacity for prediction really is, and how the common narrative at any given time can hide the fragilities that persist in complex systems. The reasons behind large scale political developments such as the EMU are often opaque and create economic regimes whose faults are only revealed to us later. It is for this reason that our awareness of fundamental fragilities within economies is incredibly important. We do not know what we are truly doing, so we must strive towards a system that is robust to our errors.

Chinese Economic Slowdown: A Reminder of How Little We Know

For an economy of such complexity and opacity, how could one not be afraid?

With most of the developed world stuck in economic doldrums and the Eurozone quickly falling apart, developing countries like China have been a key source of economic growth. However, with recent developments in China, there's a substantial fear that Chinese growth could suffer a hard landing and go through the worse period of the crisis-inducing lower rate of less than 7%.

I won't offer any predictions on how this will evolve, but rather I find that this is another instance of how fragility and uncertainty are incredibly important to the evaluation of a macroeconomics. Much like my analysis on Chinese housing, the real concern is how far the left tail can go and how little we know about it. In addition to what we think we know, we need to be very aware of possible domains that contain unknown unknowns. One key area is the extent of the feedback mechanisms that link the economy together. Once we know about them, they seem obvious. The problem is trying to figure them out.

Housing is rapidly unwinding, inventories are overflowing, and government is unlikely to respond due to credibility issues. The slowdown in housing and shipbuilding is disrupting steel production. Internationally, steel production may cause Australia to slow down substantially. Domestically, the drop in steel production reduces electricity demand. Lower electricity demand causes defaults on coal contracts. Weakening export production means problems in providing Yuan liquidity as capital takes its flight to quality away from China. And on top of these interactions, shadow financial firms are folding as a result of the unsustainability of their ponzi schemes. The shadow banking internal link is particularly fearsome because there's a risk of a major systemic crisis. From Chovanec:

The concern in China is that — like that tornado — a drop in the local property market, or a decline in exports, could hit all borrowers at once, overwhelming the local credit guarantee company and leaving the banks high and dry. The risk is exacerbated by the fact that many credit guarantee companies were capitalized with loans from the same banks whose other loans they are guaranteeing. In effect, banks are insuring themselves, or each other, and would still end up holding the bag on loan losses that are supposedly insured. (It would be interesting to know how such “guaranteed” loans are treated when regulators perform their much-vaunted stress tests on Chinese banks. I suspect these loans are considered loss-proof, because they are “insured.”)

None of these connections are meant to be predictive; rather they are meant to show the limited capacity of prediction. These linkages were not publicized in earlier articles; now that they are uncovered they can leave the domain of unknown unknowns into that of known unknowns. The quote also hints at how common regulatory approaches such as stress tests fail spectacularly in these unclear, systemic conditions. How much risk is captured in these loans, when the webs connecting the official and shadow banks are so complex? Moreover, even if one knew about these connections, shoddy paperwork would prevent understanding of the magnitude of these connections. The holes in the data also should lead one to weight the possibility of a crisis more as the risk of substantially worse data is asymmetric. There's only a slight risk that the data overstates the crisis, whereas the magnitude of an understatement is unknown and can be extremely high.

This opacity in the Chinese economy leaves me unconvinced of arguments that the strength of the government is enough to prevent any crisis from spreading. To reverse the extent of the slowdown, there would need to be a massive reform in banking, corporate governance, and the structure of government in the society. For as much as the Chinese are a pragmatic lot that are willing to pursue institutional reform, the short-term does not look good. Investment has fallen too much, and consumption spending has fallen as well. Attempts at unwinding commodity bubbles such as those in copper may result in large unknown impacts as a result of copper's role in financing deals. Monetary policy also seems very uncertain given the complex network of repo/RRR cuts/interest rate controls/loan to deposit ratio requirements that severely distort what one would consider regular open market operations for easing. Capital flows look to be partially reversing on count of lower exports and higher demand for oil. Local currency loans grew nearly 300% from 2008 to 2009, adding massively to fragility. And fuhghedabout optimal hedging; how do we even know the probabilities? The way housing is unwinding rapidly is also concerning for the possibility of government intervention as land transactions have been an important source of revenue for governments. The Telegraph UK article has an important warning against the thought that government firewalls could stop a crisis:

The property correction is deemed benign because it is planned. Premier Wen Jiabao wishes to forces down prices as a social welfare policy. Yet did the Fed not slam on the brakes in 1928 to choke an asset boom? Did the Bank of Japan not do likewise in 1990, only to find that boom-bust deflation has its own fiendish momentum? Once you let credit rise by 100pc of GDP in five years – as China has, more than in those US or Japanese episodes – you are at the mercy of powerful forces.
Something odd is now happening. The People's Bank said new loans fell from $160bn (£99.5bn) in March to $108bn in April. Non-conventional lending seized up altogether. Trust lending fell by 96pc, bankers' acceptance bills by 90pc. This is astonishing data.
It may not be as easy for Beijing to turn the tap back on again. Loan demand has been falling for months. Banks are offering credit. Companies are refusing to take it. This is the old Japanese story of pushing on a string, or the European story today.

Given the precarious financial system, it's not unthinkable that there can be self-fulfilling prophecies that can overwhelm any kind of government stopgap. Once government measures start to fail, confidence in subsequent policies evaporates, the music stops and market participants scramble for chairs. Beware manufactured stability. Just because it's coming from a developing country doesn't make the warning any less true.

P.S. This gloominess hides an asymmetry in my personal bets on Chinese growth. In any given month, China is likely to grow at a moderate risk with a non-negligible chance of a catastrophic decline.  There isn't some convex set of possibilities: either it's medium good or really bad.  With all the possible feedback loops, a small  but critical perturbation is likely to cause much larger impacts.

Grexit Opacity: Why Are We Predicting Again?

Do we know how much we don't know?

(Photo credit from Reuters)

With the recent Greek elections, there's been a lot of talk about a possible Greek exit (or the cutely named "Grexit") from the Euro.  What has struck me about the situation is how much of it is still "up in the air".  You know, with  €436M dollar bonds lying around to be paid and high levels of uncertainty on what the EUR/USD exchange rate will end up as.  Much of it will be dependent on the political resolution in Greece, but also on political resolve in core countries such as Germany and France.

But time is running out.  If the breakup is going to work, it needs to be a surprise, but with European integration as is it's hard to imagine how Greece would be able to prevent capital flight.  Moreover, if they decide to break from the Euro, the other periphery countries would have a great incentive to leave the Euro as well, leaving only Germany to deal with the loss of so much Euro denominated debt.  Immediate losses could easily reach 400 billion in initial bank losses.

To me, this all illustrates how little we truly know about the way highly interconnected and leveraged economies work.  How do you build a model in which economic conditions in Greece endogenously create the political conditions over many rounds, endogenously weakening the political situation in Germany to create a contagious financial crisis over Europe?  There is no analytically tractable way to solve that system!  Moreover, we know about these factors now that there's been so much turmoil, but how was one supposed to be able to forecast all these issues beforehand?  I also see this as a worrying issue about NGDP targeting.  While, in the long run, NGDPLT makes sense, the real question is how the credibility is established in the short run.  If unanchored expectations can have so much impact on the interpretation of debt, it seems terrifying that so much of the global economy would become pinned on the monetary decisions of a few.  It just creates a whole new host of unknown possibilities.  We can barely work with the unknowns that we know about; I dread to think of the unknown unknowns that still remain.

The presence of those unknowns is also asymmetric.  There is likely nothing left that will manage to make Greece substantially better.  Had there been a quick fix, it would have already been tried.  Even if Germany decides to massively shift to increase Eurozone NGDP (which would also make Germany overheat by a large amount), it wouldn't be a cure-all; serious debt and supply-side issues would still remain.  We're reasonably certain of how good it can get; we have no idea how bad the left tail can go.

We can't continue down the path of development like this.  Antifragile solutions need to be found.

VaR She Blows!

It would be VaR-y funny if it weren't for the fact it's so VaRy dangerous

How do we know if an asset is risky, and how do we measure it?  One of the most common metrics is something called VaR, or value at risk.  From RiskMetrics:

VaR (Value-at-Risk) is the loss in a future period associated with a given quantile or confidence interval. For example, if there is a 5% chance our portfolio will lose more than USD 1mm over the next day, then we would say the one day 5% quantile (or 95% confidence) VaR is USD 1mm.

At first glance, this measure might seem reasonable.  The most common confidence level for VaR is actually 99%, which could naturally lead one to wonder why that wouldn't be enough.  But look at the definition carefully.  VaR only tells you the probability that the cost is more than a certain number; it gives you no idea how much money could be loss.  Of course, there's a wide variety of criticisms of VaR based on it's Gaussian methodology, but what I want to discuss here is how VaR functionally ignores high impact events!  VaR tells you nothing about the nature of a portfolio because everything in the tails becomes opaque and unknowable.  This creates an incentive to collect pennies on the train track because the impact of the train isn't in the confidence interval.  We get blowup strategies, like the one shown below:

VaR breaks down in a particularly tragic manner when the market under stress.  Systemic crises are rare, but due to the nature of our financial system, they're definitely something we need to worry about.  As an alternative, the Basel commission is considering an old alternative: Expected Shortfall (ES).  The difference, again from RiskMetrics:

VaR as a measure of the quantile of the P&L distribution has a history that extends back to at least the 1980s. The publication of the RiskMetrics Technical Document in 1994 established VaR's dominance over standard deviation as a measure of portfolio risk, particularly for portfolios with optionality. 

Expected shortfall incorporates more information than VaR. VaR tells you the loss at a particular quantile q. It therefore tells you nothing about what the distribution looks like below q. Expected shortfall gives the average loss in the tail below q. 

This is particularly important for portfolios that are short optionality. For such portfolios, as the market falls, losses accelerate. So VaR may look mild, but the average loss given that at least VaR is lost may be very large. 

Another major reason for preferring expected shortfall to VaR has to do with portfolio optimization. Portfolios with optimal VaR often exploit a VaR defect: its lack of subadditivity. In effect, an optimal VaR portfolio is likely to find a low VaR, high expected shortfall portfolio.

In short, VaR tells you the risk that you lose big.  ES tells you how much you are likely to lose if you lose big.

This is definitely an improvement, but how the designers know what the ES is?  How would they even calculate it?  The methodology behind the metrics are still eerily similar: they both rely on a normal world and reasonably predictable events.  But even in that world, a slight miscalibration in your standard deviation can create huge gaps in your perceptions of risk.  Considering how sensitive probability calculations are in the tails, why do we even try?  In the testimony of Richard Bookstaber, he is optimistic about the future of metrics like VaR:

I remember a cartoon that showed a man sitting behind a desk with a name plate that read ‘Risk Manager’. The man sitting in front of the desk said, “Be careful? That’s all you can tell me, is to be careful?” Stopping with the observation that extreme events can occur in the markets and redrawing the distribution accordingly is about as useful as saying “be careful.”  A better approach is to accept the limitations of VaR, and then try to understand the nature of the extreme events, the market crises where VaR fails. If we understand the dynamics of market crisis, we may be able to improve risk management to make it work when it is of the greatest importance.      

Perhaps you may one day be able to understand those dynamics, but given the fragility of your tools, wouldn't it be better to create robustness?  Small probabilities are near impossible in Gaussian worlds, much less worlds shaped by low probability, high impact Black Swans.

These Aren't Normal Times

But they sure are power (law)ful!

Surprise, surprise, normal distributions don't describe life very well!  This hardly seems like a controversial claim, but it's nice to see that the argument is being brought to such a non-technical outlet like NPR.

One of my personal favorite arguments against the bell curve is the fact that it's very sensitive to small miscalibrations in standard deviation, especially when we start to talk about 4, 5 sigma events.  Goldman Sachs is even skilled enough to manage a 25-sigma several days in a row.  Comically:

According to Goldman’s mathematical models, August, Year of Our Lord 2007, was a very special month. Things were happening that were only supposed to happen once in every 100,000 years. Either that … or Goldman’s models were wrong (Bonner, 2007b).  

In excel, I compared the probability of an event with a z-score of less than -10, and then compared it to the probability of an event with a z scores 1% above and below that.  If the z-score was actually 1% less than that, then the original probability was an overestimate.  If the z-score was actually 1% higher, the original probability would have been an underestimate.  I calculated the under and overestimate ratios for a range of errors, going from 0% to 10%.  If the original probability calculation was an overestimate, the ratio was greater than 1.  If the original probability calculation was an underestimate, the ratio was less than 1.  The results were predictably monstruous:


The average ratio gives you an idea of how bad the probability estimation would be if you invested half of your portfolio in an investment for which you overestimated the probability, and the other half in one that underestimated the probability.  This illustrates how ludicrous attempts to measure these small risks are.  Even a "reasonably small" 5% uncertainty in the standard deviation can lead to an underestimation by a factor of about 55.

Taleb uses this as an apriori mathematical reason to reject the bell curve.  If a standard deviation measurement has gaussian uncertainty, and if that uncertainty has gaussian uncertainty, when they compound they turn into power law volatility (Go down to "Errors, robustness, and the fourth quadrant")


The world is volatile.  Therefore, we need to build our models and the world they shape to grow, not suffer, from volatility.

Take-the-Best Statistical Model

Why do we do multiple regression?

Multiple regression is the workhorse of econometrics.  Almost every empirical paper in economics relies on it, and it also forms the basis for a large majority of political science research.  But is this a valid model for prediction?  How sensitive are the results?

As it turns out, the answer is "very".  Multiple regression is sensitive to a host of issues, including normality, linearity, and low error data.  But if the real world doesn't always fit these assumptions, why do we try to use the model to predict the real world?  One thing to notice when reading the empirical papers is that they often tell you that a certain coefficient is statistically significant, but rarely does one see a given confidence interval for that coefficient.  Of course, listing confidence intervals for every coefficient, especially when there are so many, is quite cumbersome.  Yet this convenient omission often leads us to be too confident about our estimates.  How much do we really know?

Behavioral economists have long criticized this model of human decision making because there's no feasible way that we can run a regression in our head and then make a decision.  At least, I know I don't.   Although many of my friends may have used excel spreadsheets to decide where to go to college, they did not end up basing their decision on some kind of complex regression model.  It's just too computationally intractable for everyday use.

Furthermore, one key problem of multiple regression is ecological validity.  We know that the regression model predicts the sample pretty well, but does it predict the future with any accuracy?  Especially if the future is highly variable and uncertain, why should we trust our Gaussian methods that are highly sensitive to outliers? According to Gerd Gigerenzer, the most accurate rules are often not the high powered intensive statistics methods.  Rather, fast and frugal algorithms that actually limit the information they evaluate can create more accurate results.

One of the prototypical fast and frugal algorithms Gigerenzer describes is Take the Best.  While multiple regression would look at all the data and perform various tests on individual data's contribution to the dependent variable, Take the Best does a sequential evaluation of a list of key determinants.  For example, multiple regression would decide between two restaurants by looking at all the data: food quality, wait time, location, parking spots.  It would then weight each input carefully according to an equation, and then look at the results of the equations for the two restaurants.  Whichever restaurant returns the higher value is the restaurant that's chosen.  On the other hand, Take the Best would look at whether the food quality gap is high.  If so, then pick the restaurant with better food.  If the gap is not high enough, move on to the next rule and repeat this simple process.  Computationally, this would require M+1 evaluations, which is in linear time and computationally quite tractable.

Gigerenzer applied this heuristic to predicting Chicago high school dropout rates.  Given two high schools and all the associated statistics: attendance rate, proportion of low income students, social science test scores, and more, what's the most accurate way to predict which high school had a higher dropout rate?   Gigerenzer and his fellow researchers took half the population of schools and built a Take the Best model and a multiple regression model to explain the data.  No surprise, multiple regression did better, predicting over 70% of the pairs correctly while Take the Best only managed around 65%.  Yet when the two models were tested on the other half of the population, Take the Best had about a 60% accuracy rate and multiple regression barely manged around the low 50%'s.

Surprising?  Gigerenzer in Gut Feelings explains:

But why did ignoring information pay in this case? High school dropout rates are highly unpredictable-in only 60 percent of the cases could the better strategy correctly predict which school had the higher rate, (Note that 50 percent would be chance.) Just as a financial adviser can produce a respectable explanation for yesterday's stock results, the complex strategy can weigh its many reasons so that the resulting equation fits well with what we already know. Yet, as Figure 5-2 clearly shows, in an uncertain world, a complex strategy can fail exactly because it explains too much in hindsight. Only part of the information is valuable for the future, and the art of intuition is to focus on that part and ignore the rest. A simple rule that relies only on the best clue has a good chance of hitting on that useful piece of information.

Looking backwards can hurt; you might end up blindsided by what the future can hold. The data might show trends that are only valid for the sample, and not the population as a whole.  Your results won't be ecologically valid if all the data is taken into consideration.  Especially since correlations change substantially over time, Gaussian methods are more likely to offer the pretense of knowledge than knowledge itself.

This has massive policy implications.  From Gigerenzer:

According to the complex strategy, the best predictors for a high dropout rate were the school`s percentage of Hispanic students, students with limited English, and black students-in that order. In contrast, Take the Best ranked attendance rate first, then writing score, then social science test score. On the basis of the complex analysis, a policy maker might recommend helping minorities to assimilate and supporting the English as a second language program. The simpler and better approach instead suggests that a policy maker should focus on getting students to attend class and teaching them the basics more thoroughly. Policy, not just accuracy, is at stake.

Yet with these policy issues at stake, it's surprising that fast and frugal algorithms aren't used more in economics research.  One disadvantage of take the best is that it doesn't give much quantitative accuracy.  It only tells which value is higher, but not by how much.  But how much does that matter?  While multiple regression may give more statistically significant coefficients, do we really have the power to tune the economy that much?  Even the best of natural experiments don't result in parameters that don't change through time.  Romer and Romer beautifully estimate tax elasticity, but how arrogant would a person need to be to build our entire tax policy based on an estimation from an almost 80 year old data set?  DSGE quantitative accuracy is such a joke that peripheral ad-hoc models are needed to make them even somewhat useful.

These alternative statistical tools are likely to add much to our insight of models.  The development of robust heuristics will be critical in a complex world, in which calculation becomes increasingly difficult and Gaussian methods increasingly fragile.

Nominal GDP Targeting and Complexity

The Complexity View

I've recently started rereading passages of The Black Swan: The Impact of the Highly Improbable and I find it fascinating. The prose is fluid, and the arguments are powerful. Much of the book mocks economic theory, as models tend to minimize the role of large shocks that defy normal distributions.  In the book, Taleb inserts the following chart that shows how much these "outliers" influence the stock market.

Taleb places the blame for these large swings in the market on the shoulders of the Federal Reserve.  He argues that stabilization policy actually makes the economy more fragile, making it more likely to go down in a dramatic fashion once the "big one" hits.  He sums up this argument in the following quote from a section titled "Beware Manufactured Stability" in a supplementary essay.

...fear of volatility, leading to interference with nature to impose "regularity" makes us more fragile across so many domains.  Preventing small forest fires sets the grounds for more extreme ones; giving out antibiotics when it is not very necessary makes us more vulnerable to severe epidemics... 

Which brings me to another organism: economic life. Our aversion to variability and desire for order and our acting on it has helped precipitate severe crises... Another thing we saw in the 2008 debacle: the U.S. government (or, rather, the Federal Reserve) had been trying for years to iron out the business cycle, making us exposed to a severe disintegration. This is the sort of reasoning I have against "stabilization" policies and manufacturing a nonvolatile environment ...

In a sense, the reduction of volatility in the Great Moderation was only an illusion of stability.  We were, as Taleb would say, "sitting on a pile of dynamite," unaware of the risk that lay underneath.

Impact on NGDP Targeting Policy

This kind of critique seems rather damning against nominal GDP targeting.  The typical analysis of NGDP targeting hinges on the assertion that low volatility implies high stability.  But what if this isn't true?  What if these times of low volatility are just times of high fragility?  Some analysis of the arguments for NGDP targeting even suggest mechanisms by which this is the case.  Debt problems are waved away because NGDP is stable, financial opacity becomes a non-issue because monetary policy compartmentalizes it,  perceptions of "safe" assets  change because expectations of nominal growth are maintained.  Stable expectations permit these innovations because agents can plan ahead, allowing for higher growth.

However, this higher efficiency comes at the cost of redundancy.  Taleb jokes in an interview with Russ Roberts that:

An economist would never design a human being with two lungs and two kidneys. It's wasteful. Deadweight loss.  

He follows up with:

So, the opposite of spare parts would be debt. And nature doesn't like debt. Nature likes redundancies. This mechanism of overreaction is redundancy.

And this is what terrifies me about NGDP targeting.  The incredibly stable regime creates an environment in which redundancy is eschewed in favor of fragility.  Perhaps it would be better to have a more resilient economy that wouldn't be able to accumulate as much capital, but one that has lower levels of debt.  The cost of a mistake in an NGDP targeting world would be incredible.  Even if, theoretically, under a stable monetary regime, there are no demand-side recessions, would you be willing to bet the stability of the entire global financial system on it?  Even if it were true, can you guarantee the Fed will be able to maintain a "stable monetary regime" for perpetuity?

I'm not trying to say the current monetary system is ideal; the dismal employment numbers firmly reject that view.  But when we look onto NGDP targeting as the solution to the global economic malaise, we need to be careful that we don't put all of our eggs into one basket.  NGDP targeting is an incredible tool for monetary policy; but it can't be a panacea for all of these troubles.  

This critique of NGDP targeting brings up another key issue for the design of policy.  Optimal policy has to do more than maximize welfare, it must also be robust to errors.  While in the game playing, platonic world of models NGDP targeting should create incredible reductions in volatility and instability, what are the possible effects on global fragility?  Policy engineering needs to take into account Murphy's Law: "If anything can go wrong, it will."  The only question is how we prepare.

NGDP Targeting: What if it fails?

Recently in the economics blogosphere, the monetary paradigm of nominal GDP level targeting (NGDPLT) has been gaining steam.  NGDP targeting takes a departure from the classic regime of inflation targeting by the growth rate in NGDP, allowing for balance between employment and inflation.  This then leads to a wide variety of benefits, as the new regime is robust to supply shocks, can craft stable expectations of overall future growth, and can reduce fears of any specific industry going through a crisis.  It's particularly attractive for financial crises, as if NGDP growth is stable, previously sustainable levels of debt are less likely to become unsustainable.  If the economy's productive capacity is constant, there's no reason for it to be less able to service its debt.

However, this view seems almost too simplistic.  Even though the US economy was incredibly stable during the over the 20 year Great Moderation, it all came down to a screeching halt with the 2008 financial crisis.  Similarly, even though Britain managed to stay out of a major recession for 16 years, NGDP fell by about 4.7% during the crisis.  Given that there had been such a long legacy of stability, how did expectations suddenly become unanchored?  Even if the US Federal Reserve made a bad policy decision at that point to focus on oil prices and other supply shocks to the detriment of nominal stability, why did the expectations of prudent policy in the future not "solve back" the concerns?  In the end, the crisis culminated into the worst disruption since the Great Depression: hardly a desired result for a responsible regime.

The unhappy ending in 2008 seems to suggest that responsible policy can break down into chaos given a large enough of an exogenous shock.  This problem is very close to what Nicolas Nassim Taleb discusses in The Black Swan: in exchange for low volatility, the economy goes along with high fragility, such that one large shock can cause non-linear, disproportionate harm.  So in the end, the question is about credibility.  How is it established?  How is it maintained?  If decades of prudent monetary policy were not enough to anchor expectations, why should we expect the Federal Reserve to be considered "credible" when the next large financial bubble appears?  If shadow banking markets start to grow shadows and systemic risk goes through the roof, why should we expect the Federal Reserve to be considered "credible"?  Especially if non-monetary factors as posited by Bernanke play a key role in recessions, why would nominal stability be enough?  And when everything crashes down, how will we deal with the mess of debt and contracts that were only sustainable under the old regime?  NGDP targeting seems to play on circular logic.  Boost aggregate demand to hold the expectation; with the expectation there's no need to boost aggregate demand.

So when a policy maker messes up, and lets a NGDP crisis unfold, the crisis emerges.  This is where the black swan hides, cloaked by the rhetoric of stable expectations and the "perfect" monetary policy.