Backtesting: the five ways a good-looking equity curve lies to you
Any rule can be made profitable on past data. The discipline is knowing the specific ways a backtest flatters itself, and testing for each one deliberately.
Key takeaways
- – Optimising parameters on the same data you test on guarantees a good result and predicts nothing
- – Backtests routinely assume fills that were never available at that price
- – Survivorship and look-ahead bias creep in quietly and inflate everything downstream
A backtest is a claim about what a rule would have done. It is not evidence that the rule works, because it is trivially easy to produce a rising equity curve from any dataset if you are allowed to keep adjusting until one appears. Knowing the specific failure modes is what separates a test that informs from one that flatters.
1. Overfitting
Every parameter you tune – lookback length, threshold, filter – is a chance to fit noise. With enough parameters, a strategy can be made to describe the past almost perfectly and the future not at all. The defence is to hold back data the strategy never sees during development, and to prefer fewer parameters over more. A rule with two settings that works acceptably is more likely to survive than one with nine that works beautifully.
2. Look-ahead bias
Using information that was not available at the moment of the decision. It creeps in subtly: a daily bar’s close used to trigger an entry at that same close, an economic figure applied on the date it refers to rather than the date it was published, a revised data series used as though the original print had been available. Each of these quietly hands the strategy knowledge of the future.
3. Survivorship bias
Testing on the instruments that still exist today excludes everything that was delisted, merged or collapsed. For equities this systematically removes the worst outcomes. For crypto it removes a great many tokens that went to zero. Any result derived from a universe defined by present-day existence is optimistic by construction.
4. Unrealistic execution
Backtests assume fills. Real markets have spreads that widen exactly when your signal fires, slippage on stops, gaps through levels, and limited depth at the quoted price. A strategy trading frequently on small edges is the most exposed here, because costs scale with turnover while the edge does not. Model spread and slippage pessimistically; if the edge disappears, it was never there.
5. Regime dependence
A rule tested only through a trending period will look excellent and will fail in a range. Test across regimes deliberately – trending, ranging, high volatility, low volatility – and look at performance within each rather than only in aggregate. A strategy that makes all its money in one eighteen-month window is a bet that the window returns.
What a useful test looks like
Out-of-sample data held back from the start. Costs modelled conservatively. Results reported by regime and by year, not as a single number. Drawdown examined as closely as return – because the drawdown is what you would actually have had to sit through, and most abandoned strategies were abandoned during one that the summary statistics made look small.
