Does selling backtests work?

On 15 August 2024 - tagged evaluation, machine-learning

Sometimes we are approached by quants with a great backtest. They either want to sell their strategy or ask us to implement it and give them a share of the profits. That means we have to believe that the strategy is profitable and invest the time to implement it. Then we can allocate some capital into the strategy and split the profits with the quant.

But from our experience, the strategy is rarely profitable.

Can you recognize an overfitted strategy? #

The basic rule of algo-trading is that you can very easily get great backtest results. You just tune the model on historical data until it works or use some optimization technique to find a good set of parameters. This surprisingly often works even with a few parameters. This is called overfitting.

Overfitting is a common problem, that can be detected with a simple test. You can use some out-of-sample data to evaluate the model. However, if you're not the one who trained the model, you can't know if it's overfitted or not, you cannot know if it was tuned on some data or not. Even if the quant used an out-of-sample split in the backtest code, it doesn't mean that the model wasn't trained on the out-of-sample data before or that the quant simply didn't tune the parameters, hyperparameters, data processing etc. until the model was performing well even on the out-of-sample data.

Turns out any out-of-sample data simply become in-sample after you play with the model long enough.

Conclusion #

Want a simple conclusion? Never trust a backtest you didn't overfit yourself :). If you have to accept backtests from someone else, you should always check the source code and maybe ask about the development process.

In the end, it's all about trusting each other. Strategy portfolio managers have to trust the quant that the model is not overfitted and the quant has to trust his strategy portfolio manager that he will actually pay out the profit share and not just steal the idea.

By the way, if you have a great (overfitted) backtest to sell, feel free to reach out to me on Twitter or Farcaster.

New posts are announced on twitter @jan_skoda or @crypto_lake_com, so follow us!