What we mean by backtest honesty

A backtest is a simulation of how a trading idea would have behaved in the past. It’s the core research tool in systematic trading, and it’s also one of the easiest things in finance to make lie. Pick a flattering window, skip the trading costs, tune the settings until history cooperates, and almost anything looks brilliant. Then the idea meets a live market it has never seen, and the brilliance tends to evaporate.

We can’t fix the genre. What we can do is publish our own rules and invite you to hold us to them. Every test that ever appears in this letter follows these:

• Costs count. Results include realistic trading costs. An idea that only works when execution is free doesn’t work.
• Drawdowns are shown. If we show you how a test did, you also see how bad it got along the way. No equity-curve glamour shots with the ugly parts cropped out.
• No cherry-picked windows. Tests run on the data we planned to test before we saw the results, not on the slice that flatters the idea.
• Simulated is labeled simulated. Live is labeled live. The two are never blended into one chart or one track record. Ever.
• Failures get published. Most ideas die in testing. Those deaths are the most useful thing we produce, so they go in the letter instead of a drawer.

One more rule that isn’t about method: we trade our own capital using the same strategies we discuss. Skin in the game is why the rules above exist, and it also means we have a financial interest in the things we write about. You should know that every time you read us, so we’ll repeat it every time.

Even an honest backtest is still a simulation, with real limits. Whenever we show one, the required hypothetical-performance disclaimer will sit right next to it, at full size. Regulation asks for that, and we’d do it anyway.

Issue #001 lands August 1. If you want it in your inbox, the signup form is one field.