231 Systematic Investor Series ft. Richard Brennan – February 19th, 2023

[00:00:00] Intro

You’re about to join Niels Kaastrup-Larsen on a raw and honest journey into the world of systematic investing and learn about the most dependable and consistent yet often overlooked investment strategy. Welcome to the Systematic Investor Series.

[00:00:23] Niels

Welcome and welcome back to this week’s edition of the Systematic Investor Series with Richard Brennan and I, Niels Kaastrup-Larsen, where each week, we take the pulse of the global markets through the lens of a rules-based investor. If you’re new to the show, I hope that today’s episode will trigger your curiosity enough to check out the back catalog and listen to past episodes that you may have missed like my conversation with Mark last week, where we discussed among other things, the difference between long-sided and short-sided trades as well as the difference between short-term and long-term volatility and how trend following navigates some of these different parameters.

I would encourage you to listen to the midweek episode, where this week, I spoke to Jacob Shapiro, Roger Hirst about both the current geopolitical landscape as well as how that translates into the global macro environment that we’re experiencing at the moment. Of course, not least, I hope you’re enjoying our new CTA series where Alan and I have managed to line up pretty much the decision makers at most of the largest CTA firms in the world for some very meaningful conversations on many of the most important topics. I don’t think that it’s been done before in terms of rounding up all of these people more or less at the same time to give a snapshot of our industry at this point in time.

So I hope you’re enjoying this and appreciate all the time and access we’ve been given from these true industry leaders. So when you’re done listening to Rich and I today, perhaps you will find some time to check out these different series. Rich, it is wonderful to be back with you. It’s been a few weeks. You and I are having an interesting session so far because you have technical audio issues; I have a cough, a lurking cough. So I hope despite our challenges today, people will enjoy the content, which I think they will because it really is diving into some of the really important aspects of what we do.

But before we dive into all of that, I have to hear what it feels like enjoying hopefully still some nice warm summer weather in Australia.

[00:02:34] Richard

Warm, Niels, is sort of an understatement. We’re just now entering a heatwave. Our good neighbors across the Tasman, in New Zealand, just got slammed in the North Island by a large cyclone, which has done considerable damage over there. Over here, in Queensland particularly, we’re just about to enter a heatwave for a large number of days. So it’s hot over here, Niels. Very, very hot.

[00:03:07] Niels

Yeah, good to hear. We’re also going to be tackling some questions that we have that came in so I think there’ll be plenty of stuff to go through. I thought that because, instead of our usual market wrap, Rich, because you are in the trenches, and you really do pay attention to many of the things that are being released in our CTA series. I know we did this last time where we only had a couple of managers where we had released the episodes, but I’d love to hear what your thoughts are so far as we have navigated through many more of these managers and their differences and similarities. So why don’t you talk a little bit about what you’ve heard so far from these conversations.

[00:03:55] Richard

First of all, Niels, congratulations to the amazing work that you and Alan have put into this series. I must admit, I am addicted to this series and as soon as they come out, I’m on to them. You’ve had a large number of them, so I know that Alan and you probably have had many sleepless nights getting these out, but they’re fascinating.

As we mentioned last time when we spoke, I think at that stage we’d only covered Man AHL and Harold with Transtrend. But subsequent to that, now we’ve heard from Campbell & Company, we’ve heard from ROW Asset Management, Lynx, we’ve heard from Aspect Capital with Marty Lueck, and we’ve of course most recently had Graham Capital, which is the most recent one I’ve listened to. So it certainly gave me an understanding of the dispersion that exists within our industry.

As far as the philosophy regarding what trend is considered to be and as far as the methods deployed to target trend and the other methods deployed by many managers to target different arbitrage opportunities that they see, it certainly gives you significant dispersion amongst some managers. I’ve seen this myself when I’m looking at the correlations between the managers that we’ve described here, but when you listen to what they say, you can understand why there is a degree of dispersion. The good thing is we’re not all doing the same thing, which I think is a very good thing for our industry.

But of course, I’ve got my favorites amongst that group, and my favorite so far have been Harold, of course, from Transtrend. I love him simply because he thinks out of the box, and he certainly keeps me on my toes when I have my interpretation of trend and then he throws out this totally different interpretation which I haven’t heard of before. It makes me recognize that there’s a very creative mind there in Harold, and I love his work.

My favorite so far really, though, has been Marty Lueck with Aspect Capital, because I noticed with the other managers, they were not just strictly concentrating on trend. There was a lot of multi-strategy in a lot of their programs. DUNN, of course, was traditional trend, which I love, but Lynx, ROW, Campbell & Company, they were certainly going more into what I regard as a multi-strat area, which is interesting.

Of course, I’m a fan of 100% trend only. So when Marty started bringing the focus back to trend, I loved that. There were certain things that he said in the podcast, like, ‘trend being the medicine we all need.’ When he said that, it really struck home to me and I loved it, because I’ve always regarded trend as something that is difficult psychologically to apply. It’s not an easy game. It’s the hardest game in town to apply psychologically because it goes against what your brain is telling you. But that is why I see there is a significant edge in trend. So when Marty says this is the medicine we all need, I’m just there shaking my head in approval and nodding and saying, “Yes, totally agree with you.” But what do you think, Niels?

[00:07:40] Niels

Yeah, you’re absolutely right. Of course, what we’re doing is we’re talking to people who are a member of the CTA index, not the trend index so, of course, there will be managers doing more things. We may come to this towards the end of our conversation today as well, but I think that there is definitely a tendency where more firms, maybe than (I wouldn’t say) suspected, but there are definitely a lot of firms that are in our peer group that (you’re right) have essentially broadened out the strategy to become more absolute return rather than just trend, perhaps in recognition of the fact that they see more client demand for these type of strategies. So I agree with your observations.

I also of course liked Marty Lueck about his medicine analogy. I actually like my own Marty, so to speak, Marty Begin from DUNN saying when asked about dynamic position sizing, which of course is often a point of discussion on the podcast, that even though we are using dynamic position sizing, he said there’s absolutely nothing wrong with also finding success in doing it in a different way. So I actually thought that was very good of him to confirm that. So anyways, it has been interesting. There’s a lot of really good ones.

Actually, interestingly enough, one thing that did surprise me (you mentioned the latest one that we just published with Graham Capital), it did actually come as a surprise to me that given their size, which is multibillion dollar, that they are trading even fewer markets than we are at DUNN. So that I thought was a surprise, and of course, his argument for not necessarily believing in the fact that you have to trade many more markets to deliver that.

But let’s leave that for now because that is in itself a topic that we often end up discussing. So I’m going to leave that for now, but I do appreciate your insights. Before we jump into a little bit of a trend following round here, I take it, despite the heatwave, the battleship is doing okay?

[00:10:04] Richard

Yes, the battleship is doing fine, Niels. So really though, January and February, so far, have managed to balance each other out so I’m sitting at a breakeven position. January was slightly down but February was slightly up. Notably, in February, the thing that was driving that up was the orange juice trade. So, a magnificent trend in orange juice, which is sort of helping the rest of the portfolio. But really, January and early February have been what I regard as stagnation months. There’s no clear trending opportunity really in the markets that I trade, but it’s just great to be able to survive and not start building drawdowns. I don’t like building drawdowns. So if I can stagnate when there’s no trends, that’s better to me than a situation where I experience a slight build in drawdown.

[00:11:02] Niels

Yeah, and of course, I know you’ve talked about orange juice on social media a fair bit. It’s not a market that I follow that closely. I think larger managers probably can’t really trade orange juice in any meaningful way anyways, but of course it’s always interesting to see what’s going on and some of these outlier moves.

I do also notice that some people are coming up with creative ideas as to how they could actually get out of their March contract a little bit sooner than normal so they can realize this profit, which is, of course, not strictly 100% systematic the way I understand it. But anyways, there’s also a little bit of human element into these things. But I think generally speaking, you’re right. We’ve seen a little bit of correction so far in January of this year coming back now.

If I look at the numbers, pretty much all the CTA indices are coming back now positively. So, BTOP50 is up for the month of February, SocGen CTA Index is up for the month, and even the Trend Index, I believe, and they are sort of hovering around plus/minus 50 basis points for the year. Which is quite interesting in a sense, because there have been some quite large moves both in one and the other direction, so to speak, in some of the major asset classes like equities and bonds. I think, generally speaking, after a strong year last year, I think actually CTAs and trend followers have come out handling this uncertainty pretty well, frankly. So that’s what we see going on here.

All right. Let’s move on. There’s a question that came in from Jordan. Jordan writes, “When entering into a position, do you set initial stop loss as a certain ATR or percent of equity and actually take the minimum (I don’t know if he means divided by the maximum or just maximum) between the initial stop and the highest or lowest price from the past x amount of days or do you move the stop loss up to say x ATR movement?”

I’m not sure I fully understand the question from you Jordan, but I don’t know if you did get the gist of it, Richard.

[00:13:19] Richard

Look, I didn’t really take much out of it, but I’ll just explain what I do. So with my approach, I will adopt a set order; defined dollar risk per trade at any point in time. Let’s say that my realized balance of my equity, at that particular point in time (so this is the realized balance, not the total equity), I might say, let’s risk 50 basis points (so 0.5% of that realized balance which might translate to say $500 per trade).

Once I know that figure of the $500 per trade, I then define my stops using an ATR with a multiple. So, my ATR will look back 25 days. The ATR I’m using is fairly short term, looking back over a lookback of 25 days. It will define the average true range over that 25-day period and then will apply a multiple to it; like four or five times multiple to it to define the position where I place my stop.

That distance from the entry, at that point in time, to that position of the stop must equal $500. So then that defines the position size that I applied for that particular entry, and I know that at entry, I’m risking, as defined, dollar risk of $500 at that maximum adverse excursion. What I’ll tend to do is then I’ll adopt the trailing stop from entry going forward which might progressively adopt slightly looser pants. So where I’m using a 25 period ATR and 5 multiple for my initial stop, I might allow my trailing stop to be a bit less aggressive than that and open up the ability to breathe more or I might trail it at, say, 5 ATR, and so as the profits are moving in our direction, the trailing stop moves behind that at a distance of 5 ATR looking at the next 25 days multiplied by five period lookback or might be slightly looser pants, but that’s how I do it.

[00:15:32] Niels

I’m sure that’s going to be helpful for Jordan to understand the way you apply it for sure.

A final question before we move into your topics. It’s a question from Mark. Mark writes in, “Thanks so much for your podcast. I discovered your show last year, when I started with trend following. I can’t tell you how much you along with your co-hosts have helped me clear my head in terms of how to approach the mystical art of trend following.” So we appreciate that Mark. “I have one question that I would love the group to clarify if possible which is about lookbacks. In all the literature and discussions, the entry lookback period is always greater than the exit lookback period i.e. daily timeframe 100 days in and 75 days out. But through my backtests, especially within the equities and commodity space, it seems it’s the other way around. The exit periods are longer than the entry periods, say for example, 100 in and 250 out, which have historically given the maximum profit. I would be very grateful if someone could please confirm if having the longer exit period over the entries is valid or if this approach is problematic and not robust enough. Thank you again for an amazing show.”

So he is right in his observation that usually that’s how we talk about these breakout models at least. What are your thoughts? Have you ever tried to do it the other way around?

[00:17:04] Richard

Look, I understand where Mark is coming from because I think when we talk about these loose pants models, this is ideally what we’re wanting. We’re wanting a fairly tight initial stop with an exit point from that initial stop if price immediately goes against us on entry. But when we start building profits, in my models, I’m prepared to risk a bit more of that current equity with that unrealized equity in striving for these outliers. So to allow a greater degree of lookback in your exits does make a degree of sense. The only thing I would say is that I’d just be aware of the fact that your initial stop and your trailing stop, I don’t believe you want them to go backwards.

What I mean there is that if price moves in your favor, I prefer a trailing stop that moves you towards breakeven and beyond as you’re progressing into profits, rather than potentially going the other way and leaving yourself open for not only the complete absence of profit on that trending move but quite a significant loss on exit. So if you’ve got a very long lookback for your exit, say a 200-day exit and your entry is 100-day entry, that’s really suggesting that you’re going backwards. You’re allowing your trend models to go backwards into loss territory by 100 bars effectively of lookback, which I think I want to avoid with my model. So I hope that makes sense.

[00:18:39] Niels

Yeah, absolutely. I’m also just thinking about the whole concept and I’m wondering if you would even get very many trades if you had that kind of configuration. Maybe historically, depending on which markets Mark is looking at, maybe there could be some arguments that because we’ve been in a big bull market, for example, for equities, maybe having such loose pants would be okay, but I’m also thinking that in a different environment, that maybe it’s not part of the backtest, that that would hurt you much more than you would think.

But anyway, the point is just to do your testing, get comfort with whatever it is. Actually, we’re going to talk about testing today, aren’t we? So maybe I should just shut up and move straight into your points and Mark will get all the answers, I hope. Because we are actually going to talk about the quantitative analysis of price data in order to develop the trading models that we have and what we need to be aware of when we do these tests and strategy selections and how we avoid overfitting, but actually also how we avoid underfitting which could be a another issue when we apply these models to historical data.

So Rich has again very kindly prepared some really good talking points where he will take us through this, the difference between overfitting and underfitting, what it really means, some of the negative symptoms we have a strategy when it’s being developed, and how also backtesting, we can slip into maybe a process that doesn’t make us aware of some of the challenges that we may face later on. If we don’t understand these pitfalls, it’s very hard to be successful in our industry.

That’s actually very interesting because we were recording yesterday with AQR, it’s not released yet. I’m pretty sure that was one of the things we discussed as being… I can’t remember if it was AQR or maybe it was CFM, which we recorded with yesterday as well, two very interesting managers.

I think actually now that I think about it, it was from CFM. I think Phil, who we spoke to said that is actually one of the hardest things, in their mind, that it’s really hard to avoid the overfitting in the research process. So it is very timely, Rich, that you have brought up these issues, so why don’t I just let you dive into it, and as usual, I will do my very best to keep up with you.

[00:21:29] Richard

Okay, Niels. What we’re going to explore in this session are the terms what overfit means and what underfit means, and their importance to a quantitative modeler. So overfit systems and underfit systems are negative systems arising from the way we use price data to develop, or train our models, and select the models that we wish to apply in the future. This process can build into it this concept of overfitting or underfitting, if we don’t know what we’re dealing with. So before we get deeper into the discussion, we probably need to take a small detour first and understand what we mean by signals in price data and what we mean by noise in a data series.

So first, the signal. When we analyze price data, we’re typically looking for some pattern in that data that positively correlates with the systems we use to extract profits from that signal. So the particular pattern, therefore, is referred to as a signal. For example, a trend follower, they’re looking for a signal in the price data that represents a directionally trending price series or a divergent price series, either long or short. They therefore deploy a simple trend following model to extract a profitable outcome from that price signal. A mean reverter, on the other hand, they’re looking for signals in the price data that comprise a repeating oscillation of price about an equilibrium, or a convergent price series and then they deploy a mean reverting model that is used to extract profit from that convergent price signal.

Notice in this discussion we’re having, we are looking at a price pattern in the data and how the models we use positively correlate with that price pattern. So if we can access and successfully exploit the opportunities associated with a price signal, that means our systems are positively correlating with that price signal in the market data. If we find our models negatively correlate with that price pattern in the data, we find we have a loss. So this is where we start seeing that the interaction that occurs between how our systems interact with the price data creates this positively correlated outcome called a profitable trade. So the constraints in our system define the opportunities that we can extract from that price data.

Let’s look at a trend following system. In a trend following system, we have a single entry, an initial stop, and a trailing stop, but no profit target. The initial stop and the trailing stop set aside constraints, in that model, that when we apply that model to a price series that is a directionally trending price series, we see that we can exploit the opportunities provided that those constraints are aligned in the same direction as that directional trending price series. We can extract profits from that price series. So it’s the correlation that exists between the systems and the price data. What that means is that when we apply models to price data, we need to (and this is where the term), we need to fit the models to the price data.

So in this context, you can see that any successful trading system must be optimally fit to the signal it is targeting to generate optimal profits from that signal. Our definition of a price signal is intricately bound to the nature of the system we’ll be using to extract that price opportunity. That describes what signal is.

So, let’s look at what noise is. Noise relates to all other price patterns that interfere with that primary signal that we are seeking in that price data through our systems. Many of us assume that noise relates to simply random price patterns that do not offer any enduring exploitability, but that’s not strictly the case.

For a trend follower, who is seeking a divergent price series, noise can also refer to not only a random pattern, but also a convergent price pattern. I hope that makes sense. When we apply a trend following model to a convergent price pattern, we experience a negative correlation between the trend following model and the convergent price pattern and this is experienced as a whipsaw, where that converge of price pattern interferes with the ability of our trend following systems to extract a profit from that price pattern. However, when we apply a trend following model to a divergent price pattern, we experience a positive correlation to produce a profitable result.

So in a sense, a trend follower’s desired signal is referred to as noise by a mean reverting trader, and a mean reverter’s price signal is referred to as noise by a trend following trader. Does that make sense?

Notice, again, how we’re focusing on the relationship that exists between the patterns in the price data and the nature of the systems we deploy to extract opportunities from that price data. That’s important, that connection that exists between him.

A trend follower uses a design solution in their models that seeks to positively correlate with their trending price signal in the price data. So we apply this stop, this trailing stop, leave profit potential unbounded with the understanding that once these systems interact with a directionally trending price signal in the data, that the trend following models can then extract sufficient profit from that signal.

So using optimally fit trend following models, we interrogate price data. When we’re adopting a trend following model to price data, what we want to see in that price data is what we call a high signal to noise ratio. In other words, what we’re looking for is where the signals, which are the directionally trending price series in that price data, relate to these materially trending price series, and then more frequent relative to the background noise that’s also in their price data. So having a high signal to noise ratio in the price data means that our trend following models then have significant opportunity to derive more profits than losses when exploiting that signal.

When trend following models basically are embedded in a noisy price data series with only a few signals, we must expect those trend following models to perform poorly, because, in that context, we are applying our trend following models to a data series which has a low signal to noise ratio. So therefore, we should expect our systems to underperform in there. Obviously, we’re saying, all right, we’re trying to target these signals in the price data. These signals are directionally trending price series. But what we’re saying, we’re saying that there’s this concept called signals and there’s a concept called noise.

Noise is anything that interferes with that signal. But then a lot of people say, “Well, the signal then, are we saying that that is correlated to our performance?” Well, not necessarily, because that signal itself might be the result of a random directionally trending price series or it might be the result of an enduring bias or an enduring bias of momentum in that price series that is a real trending price series that can extend into the future. So, even though we have a signal, we’ve got to be aware that that signal could be random or it could be real. The key point here is that what we’re doing is if we are therefore only focusing on trading the signals in that price data, and those signals could be random or real, what we are actually doing, however, is we are reducing the chance of our systems to participate in the balance of price data that is noise.

What that’s saying is that if I’m only focusing on the signals in that price data with my trend following models, the result of that is that I’m not participating in all of these other price patterns that are noise, and the consequence of that is that I find that my trade outcomes have a higher signal to noise ratio because I’m focusing only on the signals in that price data. This is a very important point. If I’m only focusing on the signals in that price data, that means that I’m not therefore using trend following models to target noise in that price series, I’m only focusing on the signals in that price series. What that does is it’s reducing your trade sample size because you’re only exploiting those signals which are directionally material trending price series.

So, because I’m only focusing on the signals, this method of exclusion of other price patterns actually eliminates a lot of that noise associated with those interfering price patterns to produce this higher signal to noise ratio in your trade outcome. This means that when you talk about fitting your models to signals in that data, that process of fitting is focusing its intent on extracting the actual signals from their price data as opposed to the accompanying noise. So we’re getting near to what does overfit and underfit mean.

There was a really good movie that Carl Sagan wrote about in his book Contact. Do you remember that, Niels, where I think it had Jodie Foster in it, where she used these radio telescopes to basically search for extraterrestrial intelligence out there in the cosmos?

[00:32:18] Niels

Did she find any white balloons?

[00:32:20] Richard

Well, she never found white balloons, but she would sit there day after day listening to the background noise of what the radio telescopes were putting through her headphones, and it would just be noise. But then on one night, she was sitting there, she suddenly heard something that clearly could be distinguished from the background noise. It was this repeating signal of a significantly different amplitude and sine wave to the accompanying noise that was this continuous audible hiss in the background. They found that when they extracted that signal from the noise, they found that it contained information about how to build new technology that would allow them to visit the galaxies of this alien intelligence and they gave them the code through the signals of the data. So that’s the way to look at signals and noise.

A signal is this price pattern that we want to target, so a trend follower wants to target a trending price series in that price data. Only by focusing on that trending price series, you’re excluding a large amount of alternative price patterns that are noise to that signal. But then, when we isolate that signal with our trend following models, we’ve got to be aware that there still could be randomness in that signal. It could be a random materially directional price trend as opposed to a causative price trend that ideally is what we’re after. So, that therefore sort of covers the signal to noise. Now that we understand what are we dealing with there, we can understand that if there is no exploitable signal in that price data, then our systems should immediately degrade or stagnate over time.

Now, this is contrary to what many traders want. They want their models to be profitable all the time, but that’s not the nature of a price series. There is typically both signal and noise in a price series. So profitable outcomes over the long term can really only be generated when an enduring signal into the future is being exploited by our trading models. This is important as it gives us a way to see whether our systems are overfit or not. In other words, if there are no trends in a price series, our systems should not be expected to produce profitable outcomes. If they do produce profitable outcomes when there are no trending price signals in that data, we can be very suspect that those models are overfit to noise as opposed to signal.

This is very important to understand. So this gives us sort of a slight insight into a possible technique that we could deploy, which we’ll refer to later as a visual method, which is a way to identify the possibility that our trade outcomes are overfit to noise, but we’ll get to that later.

But we need to understand that it is exceedingly unlikely that we will produce profitable outcomes over the long term if we simply trade a noisy price series. Those that believe that we can derive long term profits from a noisy price series are fooling themselves. This is really likened to the belief in perpetual motion machines.

We must have that signal in the data to be able to derive sustainable outcomes in the long term as far as profits. Of course, there’s always a chance of being profitable from trading a noisy series, but we’re after an enduring signal that is not simply a chance event or a luck event.

[00:36:03] Niels

Yeah, it’s very interesting and I think it kind of highlights a little bit the complex task that we have, which will bring me to a point at the very end of this conversation, I think. Another dimension, which I don’t know if you’re going to come to, is that we talk about price series, but many managers, of course, actually apply the same rules across many, many different markets. So here we’re dealing with multiple different price series that we also want to fit into a cloud of choices. Let’s put it that way. We’re not trying to find the parameters per combination, of course. We’re trying to find a cloud of different combinations that can essentially exploit the signal and hopefully leave the noise out, so to speak.

Historically, I think there’s been experiments done by people like Tom Basso or maybe even David Harding, who have talked about doing studies where they use random entries and then see whether they can make that work, to some extent, maybe not as good as non-random entries. I don’t know. But how does that fit into all of this, Rich?

[00:37:21] Richard

Yeah. So that Tom Basso experiment is a very interesting one because it’s often touted as being a random trend following system being applied to the data, and lo and behold, we find that that random trend following system works. Now, there is a bit of a caveat in there because it’s not purely a random system. If you could imagine, what Tom did in that experiment was, yes, he used a random entry, but he also used an initial stop and a trailing stop which was non-random.

Now, the constraints applied by that initial stop and that trailing stop meant that that system, despite having that random entry, the design of that system configured it to be able to capture asymmetry in a trending price series. So when he applied that random entry condition with an initial stop and a trailing stop, lo and behold, we found it was profitable, because the actual design of the system itself was capable of exploiting asymmetry in that price series. Does that make sense?

[00:38:30] Niels

Well, it makes sense to me. It doesn’t mean it makes sense to everyone. But what it does, to me at least, is to essentially highlight the fact that there are two elements to some extent. There are the parameter choices that we end up with, so to speak, but there’s an overriding thing as well, which is the process. So even if you get one of these parameters not quite perfectly, so to speak, you can actually still make it work. I guess that’s what this Tom Basso experiment proved to some extent, so I’d love for you to explain that further if you want.

[00:39:14] Richard

Once again, we’re drawn to this. It’s a relationship that exists between the patterns in the price data and the nature of the system as we deploy to extract that opportunity from that price data. So with Tom Basso, we had the random entry, we had the initial stop and the trailing stop, but that asymmetrical configuration of that design, when overlaid and interacting with our price series, you could see that it still captures trends in that price series when they interact together. So this is where it’s not really a random outcome. It’s a very excellent method to show how Tom could use a random entry where people would think, “Oh, goodness. Surely, the entry is important.”

Yes, I actually believe the entry is very important. But I think what Tom showed or demonstrated with his model is that even with a random entry, provided you have the necessary asymmetry in your models, it could profitably exploit opportunities in price data for trend following. So what this is doing is that, all right, we’ve got to look at what the system does and then what the price data does.

So now we can talk about the terms overfitting, underfitting, or optimally fitting in this context. But before we quite get there, to understand these terms overfitting and underfitting, first we’ve got to be aware that the two factors at play with the performance of any trading strategy is a sum of two components.

Firstly, the ability of your trading strategy to extract an edge from the market data. Now this relates to what we call the intrinsic power of the trend following system. So how well does your trend following system do at capitalizing a real edge that resides in the market data which is possibly likely to repeat in the future? In other words, how optimally fit is that strategy to extract enduring signals from that market data? That’s one component. But we’ve inevitably always got this other component which is associated with simply luck. The other component found in our performance, which is either good luck or bad luck. In other words, the performance results attributed to trading noise.

Now, it might be that our best returning or best performing return streams, once we’ve undertaken a backtest, are actually the results of 100% luck. We don’t know whether that best performing strategy is actually because of some causally real enduring signal in that data that we’ve successfully exploited, or whether it’s just been a random outcome of luck. So, this is still a problem that exists. When we come to the terms overfit, underfit, etc., it’s still very hard to put our hands on the fundamental problem. How do we separate good luck from real signal? This is a real issue here.

When we talk about an optimally fit trading system, an optimally fit trading system is where we preferentially respond to the signals in that price data and not respond to the noise in that price data. Ideally, this is what any trend follower is looking for, having an optimally fit system. So we can be more confident that when we apply this system to future market data, then the majority of signals in that future market data will be exploited by our models. That’s an optimally fit trading system. An overfit trading system is where we extract both signal and noise from that price series as it responds to those price patterns that simply lead to profitable outcomes, and we know that profitable outcomes can be a result of real signals or luck.

So an overfit system does not discriminate in only extracting valid signals from that price series. It also responds to price patterns that simply lead to these profitable outcomes, which could be noise in that data series. Therefore, this is a problem we find in many data mining processes. For example, if we ask our data mining process to look for outcomes in our backtesting that deliver, say, a minimum compound annual growth rate and maximum drawdown, a minimum profit factor, and then we allow that data mining process to find its own solution that meets those criteria, we inevitably find that it trains itself to the noise and the signal in that price data to achieve those objectives. That typically delivers what we call an overfit outcome.

Yes, it achieves those objectives, but it’s doing it by exploiting luck in the noise as well as possible signals. We never know which. So we want to always avoid the ability of our data mining to basically simply go out and forage by itself, without any direction by us in defining how it should look for that signal in the price data. If we allow these data mining programs to do it themselves, they’re going to bring back these solutions to us which are overfit. What we find when we apply an overfit system to a future price series, it quickly degrades over time. There is no enduring edge in that trading system. So that’s an overfit trading system.

An underfit trading system, well, that’s less of a sin. An underfit model is one in which the trading system fails to optimally extract the signal from a price series and leaves a lot of the signal in that price series unexploited. So an underfit strategy does not necessarily degrade in the future, but we are very uncertain in the efficiency of that model in extracting a sufficient edge in the future if markets are trending. So, despite the inefficiency of underfit systems, it’s always preferable to have an underfit model as opposed to an overfit model when developing trend following systems. This is particularly relevant when we’re talking about targeting outliers, Niels, which are unique in nature.

So if we have a trading model that is specifically configured to extract opportunities from historic trends, that might not be the best way to go because the trends of the future might be different to the trends of the past. So we want to lean towards (especially in relation to outliers, which are anomalies; they’re exotic, they’re unique in nature, they are very rarely repeated in exact form or visual form or whatever), we want to actually attack them with an underfit model in preference to an overfit model. If we apply a model that is overfit to historical trends, we could find that the nature of these trends change in the future and we might be avoiding a lot of these future trends.

[00:46:35] Niels

Yeah. When you talk about these different types of fitting, so to speak, it reminds me of listening to the central banks, at the moment, about a hard landing, a soft landing or no landing. I’m not sure that’s a good analogy, but I guess it really does show you the challenge, so to speak.
The other thing, and I don’t know if that’s part of what you want to explain in the next part, but I guess it also raises the question a little bit in terms of how do we handle the data; the data in itself before we start applying our different types of rules and selections, etc.?

If you start doing something to the data, that can also change the outcome significantly, I guess. It’s actually one of the things that I’ve come to appreciate maybe a bit more as we speak to a lot of these brand names in our industry, that there’s a lot of things, there’s a lot of moving parts and it’s not just about buying the 50-day high or selling the 25-day low. There’s a lot of other things going on in order to be successful across so many different environments, across so many different markets, etc., etc. So anyways, I didn’t want to completely disrupt your flow so by all means, continue, Rich.

[00:48:01] Richard

That’s right, Niels. What you were saying blows me away as well, because it’s not easy. We know it’s not easy. There’s lots of considerations. How our models work, the price data we use, how clean is that price data, all of these things. There are so many things involved and if you don’t take them seriously, you can find that the outcomes you get from your process of testing are totally invalid. It’s so easy to do that. So we’ve just got to be aware of all of these things, particularly aware of what it means to have an overfit system, because inevitably, when we undertake backtests and we’re looking for the best models or the best performance results from that model, this is where another very important thing we need to consider comes into this overfitting term, and this is a selection bias. I’ll just explain a selection bias.

Selection bias enters a trading process as soon as we’ve got to select from competing alternatives, select outcomes from competing alternatives. I’ll explain an example. So let’s say we’ve got two people like Joe and Sam, who are two traders, and I’ve asked them to create a trend following model using the same 30-year historic data but I’ve reserved the last five years of that historic data as unseen data because what I’m going to do is once Joe and Sam come back to me with their models, I’m going to test their models on that unseen data with which they haven’t trained their models on, they’ve never seen that data. This is unseen to them.

So Joe and Sam, they develop their trading models and then they come back to me and then I apply that five years of unseen data to those models and I find out that Sam’s model outperforms Joe’s. Now so far, that’s fine. We’ve got Sam’s model outperforming Joe’s. So we currently have an unbiased result where we would conclude that by trading both models, Sam’s model outperforms Joe’s model. So we decide to use Sam’s model going forward, not Joe’s model. But as soon as we’ve made this decision, we in fact introduce bias into the process, and this bias has come from the fact that by deciding to trade Sam’s model rather than Joe’s model, we’ve introduced selection bias into the process.

This is sort of different to an unbiased decision where we decide to trade both Sam and Joe’s models, we’ve now decided to trade Sam’s model, and this is where this bias comes into the process. So the moment we introduce competition into a selection process and choose one strategy over another, we introduce selection bias. Because we always have to select from alternative models, it’s impossible to avoid this bias, but it’s very important to be aware of this bias because just the method of selection could mean that we’re actually selecting random models or non-random models. I’ll explain what I mean.

We mentioned that the performance outcomes of our trading strategy can arise from two competing components. There’s the outcome arising from the real edge that resides in the price data and the outcome arising from the noise in that price data whose result is a game of chance, you win some you lose some. But the problem with selection bias is that when we are presented with an alternative, let’s say we’ve gotten an array of 10 different return streams and we choose the best return stream, we could be choosing the random model or the result achieved from trading random noise.

The models that are applicable to the real causative drivers in their price series, we missed that entirely because we’ve chosen the most profitable outcome of the selection from alternatives; this selection bias has interfered with that process. So we need a way to eliminate the luck that exists in our performance results as a profitable appealing equity curve may simply be the result of overfitting and not a result arising from extracting an enduring signal from the price data. So now we can talk about some of the methods I deploy to reduce the chances of my models being overfit.

[00:52:40] Niels

Okay, cool. So just maybe to summarize, in one sense, you’re saying here that we can do all our backtests but it’s not as easy as just saying, “Let’s just pick the best model.” I will even take it a step further, Rich (and again, I don’t want you to derail from what you wanted to say), but I think actually, in order to appreciate maybe the complexity of not all trend following methodologies (because clearly we know some people who try and keep it super simple and that’s fine), when I look at some of the stuff that I’m exposed to, I’m imagining, with the large amount of different parameter combinations, that you can also enhance the overall outcome by combining selections that not in themselves are the best, but are the best when they are combined.

Maybe it’s also kind of why you select different systems, because one system in itself is not necessarily the best at all times, but when you combine them with different systems, they actually create and form something that is stronger. Am I going in the right direction here?

[00:53:58] Richard

Well, you’ve given away a bit of a clue that I use. So look, I’ll say the methods that I use to reduce overfitting. Now, this is different to what a convergent trader uses because typically, conventional or traditional methods to reduce overfitting typically involve the use of Monte Carlo methods, walk forward methods. However, they’re not appropriate for our trend following landscape because Monte Carlo and walk forward are good for when there are these consistent signals in the price series that are being exploited.

You want to see, for instance, in a walk forward matrix where you divide your price series up into intervals, you want to see solid performance in each of those intervals, and unfortunately, trend following doesn’t apply to that consistent trend in each of those intervals. As we know, trends are sporadic, are few in nature, occur here, occur there or whatever. They don’t line up in this consistent repeating signal in the future. They’re all over the place. So, we need a different process to ensure that our models aren’t overfit.

The principles I use, there’s about five of them, but the first thing I do is insist on what I call a design first logic. So the application of a design first logic is essential for my system development. What I mean there is I apply these golden rules first and these golden rules are defined by logic as opposed to statistics. So the logic is used to define a system design that is first capable of exploiting the desired edge I’m seeking from that price data. So in other words, I’m seeking to exploit trends in that price series. I therefore, first, want to apply design logic to develop these golden rules about a system that is capable of fully exploiting a trending price series.

So this differs from the data mining processes where you assume no design principles from the get go and you let your data mining set to some objectives. We’ve already discussed this, the compound annual growth rate has got to be this, the drawdown has got to be this, the profit factor has got to be this. It goes away and it iterates through lots of different possible design outcomes to come to a model that meets those criteria, but that model might be overfit, as we’ve talked about before. But when we are applying a design first logic, we are immediately eliminating the chance of being overfit from this iteration process you see with typical traditional data mining methods, which simply come up with a solution that have got no rhyme or reason.

If we know the logic or the design logic that we want to use to extract that signal from the data or trending price series for the data, we can then do a very important thing later on, which is what I call a visual mapping process, between how our designs respond to that price data, we’ll get to that soon. So this design first logic is the very first step of reducing my chances of having an overfit solution.

[00:57:24] Niels

Yeah, which is something I think we talk about quite a lot, which is really the philosophy or certain things we just wouldn’t want not to do when we talk about trend following, like cut your losses short and stuff like that.

[00:57:44] Richard

Yes. So once you’ve got those design first logical rules developed like cut your losses short, let your profits run, that automatically says to you we need an initial stop, we need a trailing stop or some form of mechanism. It’s placing these design constraints immediately before we apply any form of statistical assessment about where those things should be placed. It’s creating what I call the constrained boundaries of your system, the logical constraint boundaries. So logic first and then validate the statistical reasoning to that logic.

The second step I do to reduce my chances to have overfit models is I choose simple models with few parameters. In other words, loose pants models. So with models that only adopt a few parameters, they tend to be less over-optimized to historic market data.

So when we apply simple trend following models to historic market data, they have the ability to capture opportunities arising from a greater variation, impossible trend form. If we have more complex trend following models, they tend to be more prescriptive in nature, and therefore, they only attack a particular class of trend. But if we are applying simple design models that are loose pants, we can attack a greater variety of different forms of trend. This is very important when it comes to targeting outliers which have got a lot of different types of visual form. But having more precise models through the inclusion of more parameters in our trading models significantly reduces also the sample size of the signals that we can exploit from the market data.

More precise models therefore overfit to a particular form of signal and they lose their ability to exploit alternative patterns that can still arise in a trending price series yet are missed by those prescriptive models. So we actually want to underfit to any single possible trend form to allow us to capture or optimally capture a great variety of different forms of printed form. This therefore gives us a far larger sample size to work with by using simple models with fewer parameters. So these simple models, these loose pants models, give us high sample size and the ability to attack a lot of different forms of directionally trending price series.

[01:00:20] Niels

Yeah, absolutely. As we slowly start to wrap up on this, how do you then use the visual mapping in your decision-making process in order to target the models that give you the best chance of finding outliers?

[01:00:31] Richard

Yeah. Imagine we get an outcome. We’ve developed through our testing process, we’ve got 10 possible models that we want to consider, and then I’m looking at those models and I’m saying, “Which one am I going to consider?” Now, most people will use some form of statistical method to select their models. Which one has the best MA ratio? Which one has the best compound annual growth rate? Which one has the best profit factor? I don’t do that. What I use is a visual mapping process. I know that in those 10 models that I’ve got presented in front of me, some of them are going to be overfit. You can’t avoid it. It’s a horrible thing for quantitative models. It just pops up everywhere. So I know some of them are going to be overfit.

To select the best model, what I do is I map. I visually map the trade outcomes of that model. So you know when we develop an equity curve with our particular model, that equity curve, what I do is I align the equity curve or the time series of that equity curve to the time series of the price data which I’ve used for that model and then I can see at each point or each step in that price series how that equity curve is responding to each step in that price series. This is this visual mapping process where I’m mapping the trade performance of the equity curve against the price series. So then, I can say, all right, well, there clearly is an outlier in that price series and I look down immediately below to see how I performed on that outlier.

If I see a positively rising equity curve, I know that there is a correlated relationship between that outlier and that equity curve. Then what I do is I look to a period of price series which is awful for a trend follower. There’s no way you could develop profits here. I then visually map that down to the performance of my equity curve. If I see my equity curve going up when it should be stagnating or going down, that’s overfit. That’s a big strike. If I find that the equity curve is stagnating or deteriorating, there’s a tick. So as you can see, this process of visually mapping your trade performance against the equity curve, actually, you might get 10 ticks for one model, you might get 8 ticks for another model, you might get 9 ticks for another model.

It’s that visual mapping that makes me say this one here is the least overfit because it correlates with how I know that I’ve designed my logics. I’ve used my design first logic, I’ve used my simple parameters, I can tell in the price series when there’s a trending price series. So there’s a lot of logic entailed here, which avoids me from getting into a complicated statistical riddle trying to use performance metrics to say which one is the least overfit. I’m using a visual mapping process that ties back to the desired logic that I used for my models. Does that make sense?

[01:03:40] Niels

Yeah, it does make sense, Rich? Rich, are you able to summarize all of this? In the interest of time.

[01:03:50] Richard

I’ve only got two more steps. The next step I use is a multi-market process. So, as you and I have talked about Niels, we use a trend following model and our assumption is that trend following model should be applicable to all of the markets in our diversified portfolio. So we’re saying that trend following models should not be specifically selected to the characteristics of an individual market. We are looking for that trend following model to have universal application across the entire portfolio. So what that does is we have a problem in our trend following world when targeting outliers in that for a single market over a 30-year history, there might only be 10 trades we’ve done targeting the outliers in that price series, so that therefore is a very low sample size.

That’s a real conundrum for us. How do we achieve a large sample size with our process to give us confidence that is a valid process? Well, we use that model across all of our different data histories of our diversified portfolio, which therefore significantly increases the sample size. So if we’re trading 60 markets as you do, that’s 60 times the 10 trades for a single market. On average, that’s about 600 trades. You’ve upped the market sample or the data sample with your universal model. What that does is it says, by using a multi-market testing approach, I’m eliminating my chance to be overfit to a single market and I am maximizing my chance to be fit to the characteristics of multi-markets.

So that is another way of ensuring that your model hasn’t been over trained to the characteristics of the data of a single market. It’s saying that by having a universally applicable model over alternate market histories, this is going to be similar when we look into the future, because in the future, we’ve got an infinite array of possible future paths. How do I represent this set of backtesting contexts by looking at history? Well, I present it with an array of alternate possible history. So in other words, the different markets in my diversified portfolio. I’ve got 60 of them, there’s 60 alternative histories, if it passes all of them, I know that it’s not going to be overfit.

The very last thing, Niels, we can wrap it up here. But the other thing is a lot of people say, “Well, Richard, do you use out of sample data to do a final test on whether your models are overfit or not?” And I say with a big flat grin, “No.” I use the factors that we’ve discussed on this podcast, those four principal factors, as a way to reduce the chances of being overfit. Because any separation of data into out-of-sample data, that means I’m losing valuable data. I want all of my data to be applied to my backtesting process. The reason is that I am less concerned with the actual sample size itself as opposed to the need to ensure that your trend following models can operate across as many different market regimes as you can possibly muster.

So to me, the robustness of a trend following model is less about sample size and more about what exposure that trend following model has had to a variety of different possible market regimes. So in summary, Niels, I use a design first logic. We ensure or I ensure that I adopt simple models with few parameters. I use a visual mapping process, I use extensive multi-market testing, and I use all the data. So that sums up my process to reduce the propensity to have overfit systems. So there you go.

[01:07:36] Niels

It’s super useful, Rich. I know it’s a little bit nerdy sometimes when you and I go into the trenches here, but I actually think it is very interesting. So thanks for taking us through your process.

It is interesting in relation to what I’m learning as well during this CTA series with Alan, because we’re talking to managers that from the outside can look seemingly similar to some extent, maybe from an allocator point of view. But then actually, once you dig into some of the questions, you understand small subtle differences in the questions that we’ve discussed, whether it’s something as simple as the number of markets you want to trade, whether it’s liquid versus alternative markets as well, the importance of volatility, the importance of correlations, and as we talked about earlier today, even down to the way they treat the data before the models apply the rule. So I think that’s interesting.

When I take a step back from all of this, from what I’m hearing, what’s fascinating about this is I’m almost hearing two different conversations about trend following. I’m hearing on Twitter or I’m seeing on Twitter, often referred to trend following as being super easy, one entry, one exit, one stop loss. But then when I hear you explain this, even though I know you apply to the simplicity of it, but when I hear you explain this, but also of course when I speak to these firms who have done it for decades, who have been successful, who have delivered some of the best track records but also with significant amounts of money under management, not to minimize the impact of having a lot of money under management, they’re certainly not describing trend following as being easy.

I think this is also what came across just listening to your process. It is not easy. I don’t know if we’re doing ourselves a little bit of a disservice sometimes to make it sound like it’s easy, because I think the logic has been, and I meet this to some extent when I have my conversations with investors and potential investors because it’s most almost like they feel well, trend following is this, and some of them just refer to it as a factor or premia, that it’s so easy that you shouldn’t even really pay for it. It’s almost like it should be free. Of course, I couldn’t disagree with that more. And then, of course, we have our good friend from the replicator world where it’s kind of, “Oh, it’s so easy that I can replicate and give you 90% of the gross returns at low cost.”

Now there’s a new guy coming along saying, “Well, I can do twice that. I can just double the leverage and do it for half the price even though I’m not even sure what the real price is in some of these products.” I’ve even come across books that have recently been published, where they compare DUNN’s track record to simple breakout systems and moving average systems, but where I feel when I see this, it’s not really comparing apples to apples. First of all, they’re comparing some kind of backtest with no fees to a manager who’s done it for 40 plus years with fees. I don’t think you can really do that, frankly.

Also, I see some of the assumptions they put in where they say, “Well, in my model, I put $2.50 commission and exchange fees because it’s so cheap nowadays.” Well, you know, it wasn’t that cheap for 40 years. It might have cost you $25 or $50, 20 years ago, to do this. How do you account for this?

So I guess just newsflash, some of this stuff is, to me, completely wrong and kind of misleading the investors to believe that what we do is simple and it’s easy. A lot of this narrative that I come across really goes against my own experience for 30 plus years, that it’s not easy. It requires some of the smartest and some extraordinarily talented people, from research to execution, to deliver these above average returns for decades in an ever changing world.

[01:12:13] Richard

People say I’ve got no hair on my head, Niels. So all of that loss of hair from my head is because it’s not easy.

[01:12:20] Niels

It’s not easy. It’s not easy. You’re absolutely right, Rich. Sometimes I wonder why we talk about it in those terms, because it’s not (well, I could be completely wrong, of course), but it’s not my experience and it’s not the experience of the people I talk to either.

[01:12:38] Richard

It’s not mine either. I worry that it’s a derogatory statement being flipped out ways, to tell you the honest truth. Where this ever came from, I don’t know. But to refer to our process as simple is sort of like a backhand sort of oh, yes, trend following is very simple or whatever. But anyone who says that, as soon as they say that, I know that they are not trend followers, because if they were trend followers, they’d know that it’s not a simple exercise.

[01:13:05] Niels

But it is fascinating because at the end of the day (and this is the thing), anyone can have their own opinion, but they can’t have their own data, right? The data tells us that this is one of the most, if not the most, robust investment strategy out there.

I don’t want to take away any of the points that’s coming up in the CTA series, but I think there were some very, very interesting insights from Phil at CFM about what they found when they looked at CTA strategies, or maybe it was AQR. Again, I sometimes get some of these mixed up. But one of those two talked about how they had looked at the CTA space and the trend specific space, and to hear what they found and where the value comes from – fascinating.

So I’ll leave that as it is for now because people really should listen to these conversations themselves and you and I can pick this up. Before we end up though, Rich, we are coming to the point where I think soon, we’ll be able to publish the January Trend Report. But also, it’s a year-end kind of stuff. So usually at year-end, there’s a few things that are happening, in terms of some of the indices we put together. Of course, last year was an interesting year. There was a lot of dispersion in the returns among managers. Many managers did well. Some, I’m completely surprised, were to the downside, but I think it’s fair to say that the process is coming along and that there are some interesting findings, in some respect, to the new set of indices that we’re going to publish soon.

[01:14:51] Richard

Yes, Niels. I’m looking forward to that. I’m just waiting on a bit more data to come in and then I’ll have most of that content out. We’ve got a change in some of the serenity selection programs and some of the indexes have changed, SG Trend has had a small update, I wasn’t able to get BTOP50 composition yet. I think they’re a bit late in telling us who they have included in their index for 2023 but there should be some change there. So I’m looking forward to having a look at that. Of course, last year was a bit of a surprise to us with the performance of the four programs, which typically, they therefore get excluded from this process this year due to the poor performance they experienced last year, unfortunately.. But yeah, it’s an exciting year ahead and looking forward to getting that report out.

[01:15:44] Niels

Yeah, absolutely. We’ll bring that out as soon as we can. Let’s wrap up our conversation today. As always, Rich, thanks so much for all the preparation you put into this. For those who also appreciate all the hard work that goes into it, please go to iTunes or Amazon, Spotify, wherever you listen to your podcast, even to YouTube, if that’s where you watch them, please leave a rating and review. They really help not just in our confidence that we’re actually doing something that people find useful, but it certainly does help more people find the podcast.

Next week, although I did announce this last week and that was a mistake, but next week, I am joined by Nick Baltas, Head of R&D over at Goldman Sachs. I just got the weeks confused here. So looking forward to that, and of course, this will be your chance, if you would send us some questions for him to answer. He’s obviously written many articles over the years on trend following but also on many other things. So, do send us some questions if you would like us to tackle them.

I think that’s it for now. From Rich and me, thanks ever so so much for listening. We look forward to being back with you next week. Until next time, take care of yourself and take care of each other.

[01:17:01] Ending

Thanks for listening to the Systematic Investor podcast series. If you enjoy this series, go on over to iTunes and leave an honest rating and review and be sure to listen to all the other episodes from Top Traders Unplugged. If you have questions about systematic investing, send us an email with the word “question” in the subject line to info@toptradersunplugged.com, and we’ll try to get it on the show.

Remember, all the discussion that we have about investment performance is about the past and past performance does not guarantee or even infer anything about future performance. Also understand that there’s a significant risk of financial loss with all investment strategies and you need to request and understand the specific risks from the investment manager about their products before you make investment decisions.

Thanks for spending some of your valuable time with us and we’ll see you on the next episode of the Systematic Investor.