Why backtests lie
trading · backtesting · measurement
Zero commission
A backtest with no commission reports gross profit, not net. On one currency-pair system, commission consumed 46% of the gross; ranking by profit factor was effectively blind, and the ranking moved onto net pips.
Optimistic stop fills
Tick simulation usually books the stop exactly at the level. Real fills slip, and measurement on real ticks showed that the optimism costs money on stop and trail exits. The difference is small, but across a few hundred stop exits it removed close to half of one rule's measured result.
Validating on the tuning data
A parameter optimised over the whole window explains only the past. The minimum is to split the period in half and reject anything that works in only one half. Measurement against a calibrated random baseline is required: many "signals" are no better than a coin flip.