Backtest, out-of-sample, dry run and live are four different kinds of evidence, from weakest to strongest.
A backtest shows what rules would have done on data the designer could see. Only live trading shows what actually happened with real fills.
Before reading any return, find out which class it is. Never add returns from different classes together.
A test uses information that was not yet available at the moment of the simulated decision.
It is the commonest way a backtest becomes better than reality — through revised data, a bar's close used before it closed, or today's index members applied to the past.
For every input, ask when it could first have been known, and use that time rather than the time the event happened.
Data held back while a system is designed, and used once to test it.
Every time held-out data influences a design change, it stops being out-of-sample and quietly becomes part of the backtest.
Decide the test before you look, and record how many times a data set has been used.
A price bar is closed when its time window has ended. Until then its high, low and close can still change.
A reading taken on a forming bar can flip within minutes. A signal that appears and disappears inside a bar cannot be tested honestly.
Base decisions and backtests on closed bars only, and act after the bar that produced the signal.
How many instruments take part in a move, rather than how far an index moves.
An index can rise on a handful of large names while most constituents fall. Narrow moves tend to be more fragile than broad ones.
Compare the index move with the share of instruments above their own trend, and always check the denominator.
Studying only the instruments that still exist today.
Delisted, failed or merged instruments drop out, so the remaining set looks better than the opportunity set really was.
Test on the universe as it was at each date, including what later disappeared.
The fall in value from a previous peak to a later trough, and the time taken to recover.
Two strategies with the same return can demand very different endurance. Depth and duration of losses decide whether a system can be followed in practice.
Read maximum drawdown and its duration next to every return figure.
How much of a result, or a portfolio's risk, comes from a few positions, trades or events.
A result carried by one trade, or a portfolio carried by one exposure, is fragile even if the headline looks diversified.
Check the share from the largest few contributors and how correlated the holdings are, not only how many there are.
An absent value — a price, a fundamental, a holding's sector — is unknown, not neutral.
Filling a gap with a default makes a reading look complete while changing its meaning.
Keep unknowns visible, and judge how much of the picture a conclusion actually rests on.
General explanations, not recommendations to buy, sell or hold anything.
Performance MethodologyInvestment Intelligence is in an invitation-only beta for a small number of evaluators.
