Why most backtests lie

Nobody sets out to build a dishonest backtest.

That’s the uncomfortable part. The classic backtest failures aren’t frauds. They’re self-deceptions: mistakes that flatter the researcher, made by people who genuinely wanted the answer to be yes. Which means you can’t screen for them by looking for villains. You screen for them by knowing the four ways a simulation learns to lie, because at least one of them is hiding inside almost every too-good chart you’ll ever be shown.

Lie one: overfitting, or the coin that flipped ten heads

A toy illustration, no markets involved. Hand a fair coin to 1,024 people and ask each to flip it ten times. On average, one person flips ten heads in a row. Interview that person and you’ll get a method: the wrist snap, the breathing, the mindset. The coin is still fair.

Strategy research recreates this constantly. Test enough variations of an idea, different lookbacks, different filters, different thresholds, and something will look brilliant on historical data by chance alone. The more combinations you try, the more certain you are to find a “winner,” and the less that winner means. The backtest hasn’t discovered how markets work. It has memorized how one stretch of history happened to unfold.

The tell is fragility. If the idea only works with the 14-day setting and falls apart at 12 or 16, you’ve probably found the lucky coin, not a durable pattern.

What honest testing does: keeps count of everything that was tried, distrusts complexity, and reserves data the strategy never got to study. That last part matters enough that it gets its own article.

Lie two: survivorship bias, or the winners’ yearbook

Run a stock strategy over twenty years of history using today’s large-cap index members as your universe, and you’ve rigged the game before the first simulated trade. Every company in the sample is one that survived, and usually thrived, long enough to be in the index now. The bankruptcies, the delistings, the slow fades: missing. You tested on the winners’ yearbook and concluded that everybody turned out great.

The same bias operates one level up, on strategies themselves. A publication that only ever shows you the systems that reached publication is a survivorship machine. You’re seeing the bottom of the funnel and being invited to believe it’s the whole funnel.

What honest testing does: uses point-in-time universes that include the dead, and shows the whole funnel. Around here, the second half of that is called the kill log.

Lie three: look-ahead bias, or tomorrow’s newspaper

A backtest peeks whenever a simulated decision uses information that didn’t exist at the simulated moment. Some peeks are obvious, like using a day’s closing price to decide a trade executed at that same close. Most are sneaky. Economic numbers get revised after release. Earnings land after hours. Databases get cleaned up years later, so history looks tidier in the file than it felt in the room.

Peeking is astonishingly powerful. A tiny leak of future information can turn pure noise into an apparently beautiful edge, which is exactly why it’s hard to catch: the results look wonderful, and everyone in the room wants to believe wonderful results.

What honest testing does: reconstructs what was knowable, and when. As-reported data, realistic reporting delays, and fills at the next price that was actually available rather than the convenient one.

Lie four: free trading, or costs that round to zero

Every real trade pays a toll: commissions, the gap between buyers’ and sellers’ prices, the market moving against you between decision and fill. A backtest that skips those costs isn’t simulating trading so much as simulating telepathy. And because the toll is charged per trade, the faster a strategy trades, the bigger this lie compounds. Plenty of simulated ideas turn out, on close inspection, to be nothing but the cost-free harvesting of costs.

What honest testing does: charges realistic costs on every simulated trade, then asks whether anything is left. Slippage gets its own article in this series. It has earned one.

The posture that fixes all four

Notice what the four lies share. None of them require bad intent. They just require a researcher who wants the idea to live and a process with no adversary in it. So the fix is a posture before it’s a technique: your job is to kill the idea, and the idea earns belief only by surviving. Held-out data. Honest costs. A running count of everything you tried. A public record of what died.

That’s the standard we hold our own work to, and it’s why every backtest we publish ships with its assumptions attached and the required hypothetical-performance disclaimer at full size, every time. Simulated results are simulated. A backtest that can’t show you how it might be lying isn’t research yet, just a story with axes on it.