Overfitting

Tuning a strategy so closely to past data that it fits noise rather than a real edge. It shines in the backtest and fails in the live market.

Overfitting happens when a strategy is designed, or optimised, so closely around one specific historical period that it learns that period’s coincidences instead of a repeatable pattern. Financial data contain a lot of noise and little signal, so with enough parameters you can always find a combination that would have been splendid in the past. That does not mean it will work in the future.

The basic mechanism is an excess of degrees of freedom. A 20-period moving average is one rule; adding an hour filter, a volatility threshold, a day-of-week condition and three exit parameters produces thousands of combinations. The more tests you run, the more likely some will look good by pure luck. This is the multiple comparisons problem.

A numerical example: if each individual test has a 5% chance of looking significant by accident, the probability that at least one of 20 independent tests does so is 1 − 0.95 to the power 20 ≈ 64.2%. With only 10 tests it is already 40.1%. Someone who optimises hundreds of combinations and keeps the best is almost always looking at the luck that disguises itself best as a result.

The warning signs are clear: equity curves that are too smooth; results that change a lot when you move a parameter by 10%; few trades behind many rules; performance that collapses outside the optimisation sample; and rules you cannot explain with any economic logic. A good strategy usually behaves similarly across a wide zone of parameters, not on a single peak.

To fight it, reduce the number of parameters and rules, use long samples with different regimes, reserve out-of-sample data you do not touch until the end, apply walk-forward validation, and test robustness with small perturbations of parameters, costs and data. If the system only survives with zero costs and exact values, it is probably overfitted.

A very common error is repeating out-of-sample validation again and again, adjusting after each failure: in practice that sample becomes part of the training set. Another is relying on a single asset or period. Testing the logic on several related instruments is a more reliable robustness check than optimising on one.

In risk management, assume live performance will be below the backtest. Start with minimum size, compare real metrics with the expected ones, and decide in advance how large a deviation will make you stop the system.

More terms