The Costly Mistake of Misjudging Luck from Skill

By Moritz Seibert, edited by Niels Kaastrup-Larsen   Following up on a great H2 2014, most trend following CTAs continued to produce strong gains during Q1 this year. Good performance tends to attract new investments, i.e., investors are inclined to chase returns as they attribute recent outperformance to manager skill. Discounting the role of randomness in managers’ track records and disregarding the warnings on many factsheets, a large number of investors seem to assume that past performance is a good predictor of future performance. Unlike Q1 2015, CTA returns during the summer have been relatively volatile and mostly negative. Return-chasing investors may therefore be tempted to sell their investments for all the wrong reasons at the wrong point in time. This article explains why.

Separating skill from luck: the importance of sample size and time

Trading results include both skill and luck. When analyzing small sample sizes, an outperforming trader could either be a skilled trader or an unskilled trader with a lucky streak. Likewise, an underperforming trader may be suffering through a period of bad luck or indeed be lacking an edge. Separating skill from luck isn’t easy.

Case A: Pure luck (e.g., rolling a fair die, tossing a fair coin)

The tossing of a fair coin results in a 50/50 win-loss ratio for heads vs. tails.

Case B: More luck than skill

Without deploying any particular skill, the above 50/50 coin toss ratio can be increased (say, to 90/10) by selling far out-of-the money (OTM) put options or CDS, for example. Most investors prefer high winning percentages over low winning percentages because it feels good to win. However, it requires investors to turn a blind eye to the large downside risks hidden in the tail of a negatively skewed return distribution. Drawdowns for these strategies are rare, but often disastrous when they occur. In this example, when chasing recent returns, the short sample period may cause investors to incorrectly conclude that skill is the dominating return factor.

Case C: More skill than luck

Many skilled traders have winning percentages which are worse than 50/50. For example, the long-term winning percentage of most trend following CTAs is about 40%. Compared to selling OTM puts, this 40/60 ratio doesn’t feel so good because investors lose more often than they win while drawdowns are the norm rather than the exception. But, since trend following trading systems have a proven edge, these drawdowns are seldom disastrous. When chasing recent returns, the short sample period may cause investors to overlook the presence of skill in the trader’s track record.   Returning to case A, when tossing a coin many times (e.g., 1,000 tosses), the regression to the mean will cause the actual win-loss ratio to be very close to 50/50. Tossing the coin a mere 10 times may produce a very different outcome due to the randomness in small sample sizes. Hence, luck dominates the outcome in the short-run, but the actual probabilities will dominate it in the long-run.   Regarding case B, a manager may claim to outperform the DAX by 4% each year while staying 90% correlated to the index and realizing approximately the same volatility (assume 20%). In an attempt to achieve this result, the manager decides to sell far OTM DAX put options on a monthly basis. We can conduct a t-test to approximate how much data the investor will have to examine to see through the randomness in the manager’s return stream: To be 95% confident that the manager’s outperformance is indeed the result of an edge, the investor would have to analyze more than for 13.4 years of performance data. Not all managers have that long a track record, and most investors don’t have that much time. The issue here is a low signal to noise ratio in the manager’s trading results. It may cause the investor to detect skill where in fact none is present.   With respect to case C, consider the example of a proprietary trend following “Golden Cross” trading system. During the past 15 years, this system has produced 41% winning trades and 59% losing trades. However, the average win in dollar terms is 2.55 times larger than the average loss, and thus the system’s PL expectancy per trade is (41%*$2.55) – (59%*$1) = $0.45. From a probability standpoint it’s not at all unlikely for this trading system to have six losing trades in a row (or even more). It becomes clear that even though the system tends to earn $0.45 per trade on average in the long-run, pure bad luck can cause a large drawdown in the short-run. In contrast, the option selling system in case B may have a negative PL expectation in the long run, but is unlikely to be in a drawdown in the short-run. Chasing returns tends to be an expensive exercise for investors. Once the investment is made, many outperforming managers tend to perform much worse than their recent performance record would suggest. That’s especially true when selecting the best performing managers from a very large group of managers. The larger the group, the greater the chance that the best performing managers were simply lucky and vice versa. The more pair of dice you roll, the greater the probability of rolling snake eyes.  

With respect to trend following CTAs, the following two alternatives are likely to produce better outcomes:

• Instead of chasing returns, investors should stay invested for longer periods of time. CTAs can educate their clients about the importance of a long-term commitment, for instance by creating alternative paths from their historical track records (combining slices of past returns in different ways to show that periods of underperformance can very well be the result of pure randomness).
• From experience, investing in proven CTAs when they are in a drawdown is a great thing to do.

All trading results are a combination of skill and luck, but decomposing the outcome into these two categories isn’t easy. It’s impossible if the sample size is too small. By placing too much emphasis on recent returns, investors are at risk of allocating money to unskilled lucky traders while disregarding skilled but temporarily unlucky traders. This is a costly mistake.