In partnership with:
CME Group

Best of TTU – Track Records vs Simulations.. What’s Best?

For those of us who have been on the manager side for a while, you'll know that investors love to analyse decades of performance data before deciding which manager they are going to invest with, and that makes a lot of sense.  But we also come across investors who subsequently redeem based on just a year or two after they invested, which is usually just down to bad luck and unfortunate timing.  This does not seem logical, but it does relate to the short conversation I want to share with you today, where I discuss the role and importance of track-records & back-tests with Scot Billington.  We also ended up discussing an interesting twist to their research, which led them to abandon taking any Short Trades in their model.  So enjoy these unique insights from Scot, and if you would like to listen to the full conversation, just go to Top Traders Unplugged Episode 25 & Episode 26.

The Long side vs the Short side

Niels:  Now, track record...we've touched a little bit upon the track record. What I'd love to do is to ask you how one should read your track record because we all know that strategies evolve over time, and, therefore, in a sense, one could say that actually a track record...sure it shows that you survived. It shows that you have had some innovation, but I think sometimes people maybe get fooled to believe that a track record is a great indication of what the future is going to look like because they really don't know what changes have happened along the way in the model. So in some ways one could say that maybe it's better to look at the backtest of the current model when you look at a manager, maybe that would say more about the future, I don't know, but I'd love to hear about what your view and what your observation is about your own track record, because you mentioned the short side of things, and I know there was a period where I think you didn't take any short trades at all, so I'd love to hear your philosophical view about short trades, because I don't think many people realize that there is a big difference between the long side and the short side in terms of success and profitability, but also generally maybe putting that into context about your own track record and why you made the changes along the way?


Scot:  Well I think you make an excellent point about track records. I think people look at them as some kind of a one loss record in a sporting event, and they miss the enormous amounts of variance and randomness that happened to have lined up and occurred to produce whatever monthly return is shown, or daily return. I think that most groups don't do enough qualitative analysis of the trading method, and simply crunch performance numbers as though they're the end all be all of what the future is going to be. You make an interesting point. I would probably argue that a backtest of the current running model is probably the best, except that it is also going to...if you are hoping to be a 5 or 10 year investor, there're going to be future changes. And so what I might say is, as we've progressed, our changes have been successful, and therefore perhaps one would conclude that our future changes would also be successful. That's a different discussion. Our first major change to the model was in early 2002, was basically eliminating short trades and using a volatility filter for long trades.

'Niels: So why did you make a change to eliminate Short trades back in 2002? Scot: Well, a Short trade is bounded by zero, so it's going to face several hurdles'

Niels:  Were the short trades the cause of your drawdown back in the beginning?

Scot:  No, no.

Niels:  So why did you make a change to eliminate short trades back in 2002?

Scot:  Well, short trades, and this is particularly appropriate in our timeframe, on a shorter timeframe I don't think this would not hold as much. Let's imagine in our timeframe we're trying to hold a good winner for, let's say over a year. A short trade is's certainly bounded by 0, and in reality, in a lot of these things it's bounded by something north of 0, right? But I'll give you even to 0, the point is still the same. So a short trade is going to's going to face several hurdles.

The first is if you took every sell signal of silver at $9 and I took every buy signal of silver at $9, and I'm looking for...our trading system is trend following, it's built around large deviation winners and outlying large moves, all of your trades are going to be limited to make $9 and mine aren't. Furthermore, let's imagine we had two good trades, you sold yours at $9, and it moved $5 down to $4, and I bought mine at $9 and it moved $5 and it's at $14, now how are we sitting?  - because you are in a terrible position. You have virtually nothing that can be made, probably a lot of risk in the position in that you've had a large and rapid move, and you're seeing an intra-trade de-compounding effect. Because now a 10% move for you is only $.40. While I've had the exact opposite happen. I'm getting a compounding effect, meaning I got to enter in a volatility and even a percentage move, where at $9 and now I'm into $14, and every dollar I go up... a 10% move for me now is $1.40.


So if you imagine that, these rare moves (however you want to define them) come along equally infrequently. You and I have the same chance of one of these rare moves occurring. In your rare move, just for this trade example, you had to risk $.50 in silver to make $6? So you could have a 12 to 1 winner at best, and that's riding $9 down to $3 and not only is the market at $3, but your exit has got to get to $3. You see what I mean. You could catch some rolls in there, but we have to both agree that would be a huge winner short. My buy at $9...well silver has gone to $50 twice, right, so for the same $.50 risk and equally as likely an outcome, I'm going to make, let's say I got out at $45, 68X my initial risk. Well, start doing the math. You make 12X, I make 68X. You are going to have to's going to have to happen 6 times as often for you to tie me. Does that seem likely?

Now that doesn't mean in a 3 year 2008 you're going to murder me, right? This is where people, in a three-year period shorts could do way better, and honestly, in a 10 year period they could do a little better. That gets a little less likely because a lot of these markets are reciprocals of each other: long dollar is short Euro and Swiss, but there could be a long term in time in which there was a large decline in overall prices in which shorts might do better, much better. Shorts might make 30% a year while longs lost 5%. This gets back into the qualitative and quantitative of it. Qualitatively, I don't think my point...there's no point against me. Logically, if we sat in a courtroom and argued it, I'm killing you on that argument, you see what I mean? I think we both know if we both took 10,000 in trades and you took all shorts and I took all longs in what I just described, you can't beat me. You're just done for, but that doesn't mean in a 5 year period that your track record might not be better than mine. That's a good insight into how we try to look at research, and how we try to look at trading, and trading from a strategic standpoint, or even a philosophical standpoint, and I would say strategic, and how we might attach a research problem. What we really evolved into is that shorts and longs need to be treated differently. Initially, when I buy at $9 and you sell at $9, those trades are very similar for the first dollar or two move, they're very similar. It's only as they progress that yours gets worse and worse and worse while mine, conversely, is getting better, and better, and better. So I need to appropriately weight the fact that my long trades are ultimately going to be a lot better, but I can also maybe make some adjustments to how I handle a trade intra-trade to mirror the way a trade evolves. Does that make sense? As the years have passed, what we have attempted to do is just get better at how we trade short, because there are shorts, over time, that have proven to be the best diversifier versus longs. They're part of the portfolio, and have been for years, but I'd like to think (obviously we wouldn't have made the changes if we didn't think they were better) that we've gotten better at understanding how short trades work and why they're worse than longs and how that evolves, and that we've created tactics to minimize the weakness of short trades.

'What we think we've gotten better at along the way is: 'I've got a basic Trend Following model, and it generates signals.  Which of those signals are worth taking risk on, and which aren't? Without trying to predict if a trend will ensue, and how far and large it might be."

So, we're not big into trying to predict where the market is going to go, we don't think that we can. We're not big into...we don't think we have any magic pixie dust that tells me when a trend is going to start, but I do know this, if I get a signal long, and you some other time have the same signal, same risk short, I know that mine is better. That I know for a fact, and you see what I mean? I'm not trying to guess, oh well, Hillary Clinton is going to get elected, and that's going to do this and that. People can do that, and maybe they can do that effectively, I don't know, but I can tell you for sure my long signal is better. Now what I would argue is you're going to have to get awfully good at predicting future worldwide elections and results and all that to overcome that my long signals are way better, because they are unbounded and they're going to compound in the trade. You see what I mean? So what we think we've gotten better at, along the way is, I've got a basic trend following model and it generates signals. Which of those signals are worth taking risk on and which aren't without trying to predict if a trend will ensue, and how far or how large a trend might ensue. Does that make sense? Like I mentioned, a volatility filter. If, again I'll use silver. Let's imagine I get a long signal in silver at $9, but the recent volatility has been very high, and so through whatever metrics we use, I'd have to risk $.90. Then there's some other point in time you get the same signal at $9 but recent volatility has been low, and you're only going to have to risk $.45. So the first thing that we know is if we're risking the same amount of money on each trade, if I do 5 contracts, you're going to get to do 10. Well, let's imagine the same trend comes along, you're going to make twice as much money, and we knew that at the inset. We knew that for a fact. I'm going to need twice as large a trend account to make the same amount of money as you. By definition, a trend that's twice as large...if trend A is twice as large as trend B, trend A has to be more rare, right? So those are the kinds of improvements we like to think that we've made on our model, as well as diversifying...there's some advantage to diversifying entry points, diversifying exit points. It isn't that one is better than another, sometimes one is going to better, sometimes the other is going to be better. The main thing is that we try to achieve a robust, or even a smoothing effect to that.

maxresdefault (3)

"...that would be like the Bellagio closing down all its Blackjack tables except for the one that had done the best in the last hour. So trading more markets is a huge advantage, assuming that they are diversified enough."

Niels:  We talk about timeframe as maybe being something that has influenced returns in the last few years, and we talk about whether the short-term traders have done better than the longer term, and some instances we've certainly seen examples of longer term traders having done better than short term, but actually I want to throw in one more thing in this discussion, and that is sector allocation, because in my mind, I think one could argue that sector allocation either by design of by default have made a huge impact in recent years. You could say that because large managers, by default have had to be more focused on bonds and equities, they perhaps have made money more by luck than by skill, or is that too harsh?

Scott:  Well their outperformance in a short timeframe...any outperformance in a short time frame is 100% luck. So yeah, if you were comparing us to a 10 billion dollar manager that in a 3 year period, or even a 5 year period, it's pretty much going to be how did bonds and currencies trend versus commodities. That's going to be a whole deciding factor in 3 years. Again, this is in 10 years the idea would be that those things would have started to cancel each other out - cotton and coffee did awesome this year, and then the Euro currency did awesome this next year, and then those start to cancel out and then the real strategic differences between the two start to show up. But if you look at it scientifically, the fact is that if I had a certain trading edge, the more times I can apply that edge the better. So, on a future expected basis, trading more markets is without question more desirable than trading fewer, although, just as you said, and that's again why the investment horizon of an allocator and how they look at things can be so misleading. If I only traded stock indexes, I would have crushed it the last couple of years. But that would be a foolish way to trade. Because then I'm saying oh, well stock indexes adhere to this anomaly that I have, not other markets. Or if other markets do adhere to it, but I don't trade them, why wouldn't I want to get...that would be like the Bellagio closing down all its blackjack tables except for the one that had done the best in the last hour. That's foolhardy. So trading more markets is a huge advantage, assuming that they are diversified enough. Trading more markets is an advantage, and it's only an advantage in that I get more instances in a timeframe. In a given year, or 3 years, or 5 years, yes, some sector...if we're talking trend following, or whatever it is...some sector is by definition something has to have done the best. This gets back to my earlier point. When you then go sort the managers by their 3 year track record, who's going to be at the top? Whoever happens to specialize in that sector. Does that mean they're more likely to make money in the future?

Niels:  Nope.

Scot:  Well maybe, if that is where you want to make your stand, then have at it. It's pretty questionable.

Niels:  But it's interesting. I find it fascinating the way you talk about timeframe and you talk about these long-term timeframes, and I can imagine it's not easy to get investors to share that horizon, so to speak.

Scot:  If you want to take ''not easy" out, and replace it with "impossible" then you would be correct, but that's the business that I've chosen. I know full and well that the next time we get hot for 24 months will attract a lot of money, that's a fact; and will keep attracting money as long as we "stay hot"; and then the next time we have a drawdown, which will come, that's guaranteed, and it will last for X amount of period, we'll lose 1/3 of the money that we had. That's going to be...that's just a fact of the matter. We attempt, we try, we talk about these things with our clients, but it is probable that the Homosapien is not particularly hardwired very well for trading.