Doctrine

How we backtest

Most published backtests are not wrong because the arithmetic is wrong. They are wrong because the procedure that produced them let information leak backwards. This page states the rules we hold ourselves to, including the ones that make results look worse.

1. Real engine only

Signals are executed through the same engine that would trade them live. Not a vectorized approximation, not a spreadsheet that multiplies a position vector by a return vector.

Vectorized backtests are fast and they are useful for early screening, but they systematically flatter results because they quietly assume things the market does not provide. They tend to assume you transact at the close you just observed, that size is always available at that price, that orders never partially fill, and that the position you wanted is the position you got. Every one of those assumptions is a small loan taken from the future, and they compound.

Running through the real engine forces the uncomfortable specifics:

• Order types are declared. Market, limit, stop — with the behaviour that actually follows from each, including limit orders that do not fill and stops that gap through their trigger.
• Bar timing is honest. A signal computed from a bar's close cannot transact at that same close. Execution happens on subsequent available liquidity, because that is what happens in practice.
• Costs are charged. Commissions, spread, and a slippage model — applied on every fill, not netted at the end as an afterthought. Costs are stated as parameters so a reader can judge whether they are plausible for the instrument and size in question.
• Corporate actions are handled at the data layer. Splits and dividends are adjusted in the bar data, deliberately and visibly, rather than left to produce phantom gaps that a naive system reads as signal.

2. Parameters are frozen before the window opens

A parameter chosen after seeing the result it produces is not a parameter, it is a fitted constant. The lookback that worked, the threshold that cleaned up the drawdown, the stop that happened to sit just below the worst wick in the sample — each of these is a degree of freedom spent on the test data.

So the rule is procedural, not aspirational: the parameter set is written down and frozen before the evaluation window is run. If a parameter is subsequently changed, the run is a new experiment on a new window, and the earlier result does not survive as evidence for the new configuration.

We also count how many configurations were tried. The more variants tested against the same data, the more the best-looking result is explained by selection alone. A strategy presented as "the one that worked" out of a large search is making a much weaker claim than its equity curve suggests, and that search count belongs in the report.

3. Survivorship and hindsight bias

These are the two failure modes that quietly destroy the credibility of most retail backtesting, and neither is fixable by being careful in the strategy code. Both are properties of the data and the procedure.

Survivorship

A universe assembled from instruments that exist today has already had its failures removed. Backtesting a stock strategy against the current index constituents means testing against a set pre-selected for having survived — the delistings, bankruptcies, and acquisitions are simply absent. The result is a bias upward that no amount of out-of-sample discipline corrects, because the missing observations were never available to be tested.

This is why the dataset layer is treated as part of the methodology rather than as plumbing. A point-in-time universe that includes securities which later ceased to exist is a precondition for an equity backtest to mean anything.

Hindsight

Hindsight bias is broader than look-ahead in the code. It includes the strategy ideas themselves. We know which decade momentum worked in. We know which crashes happened and roughly when. A strategy designed in 2026 and tested on 2010–2020 data is being designed by someone who already knows the answer, and no amount of clean coding removes that. The designer is a source of leakage.

The technical forms are handled mechanically — data is only visible to the strategy as of the timestamp at which it would have been available, including the revision lag on macro series that get restated after first publication. But the conceptual form cannot be handled mechanically. It can only be handled by refusing to treat in-sample results as evidence, which leads directly to the next rule.

4. Forward paper records are the referee

A backtest is a filter, not a verdict. Its legitimate function is to reject systems that fail on data they have already seen. Passing establishes only that a strategy is not obviously broken.

The evidence that counts is the forward record: the strategy runs on data that did not exist when its parameters were frozen, on a paper account, with its trades logged as they happen. That record is kept whether or not it is flattering, and it is the number we hold ourselves to when a strategy's backtest and its forward performance disagree. When they disagree, the forward record wins and the backtest is treated as having been overfit.

This connects to the prediction ledger described on the home page. The same principle governs both: a claim recorded before the outcome, with stated resolution criteria, and scored when it resolves. Everything else is retrospective storytelling.

5. What we will not publish

No performance figures appear on this site, and none will until there is a forward record that justifies them. Where an example is needed to explain a method, it will be labelled as illustrative and will not be presented as something achieved.

Any hypothetical or simulated result we ever do show carries the limitations set out in the disclaimer: simulated results are prepared with the benefit of hindsight, have inherent limitations, and no representation is made that any account will achieve comparable outcomes. See also the simulation page for what resampling can and cannot establish.