Why most backtests lie

You tweak a footprint-imbalance entry on five years of NQ data. Version 1 loses money. Version 7 breaks even. Version 23 — imbalance ratio 3.5, two stacked levels, entries only after 9:45 ET, stop 14 ticks — shows a smooth equity curve and an annualised Sharpe ratio of 1.4. You feel like you found something. The uncomfortable truth: you may have found nothing at all except the luckiest of 23 coin-flip sequences.

A backtest (kiểm thử lịch sử) is a simulation of how a set of rules would have traded on past data. It is not evidence that the rules will make money. It is evidence only if the test was designed so that luck, hindsight and selection could not have produced the same result. This module (M10) is about that design. This first lesson is the catalogue of how it goes wrong.

Chart first: the four ways a backtest lies

                        WHAT THE BACKTEST SHOWS
                                  |
     +-------------+--------------+---------------+---------------+
     |             |                              |               |
 OVERFITTING   LOOK-AHEAD                   SURVIVORSHIP     DATA SNOOPING
 rules bent    the rule "knew"              the sample only  many ideas tried,
 to fit noise  something it could           kept what        only the winner
               not have known yet           survived         is reported
     |             |                              |               |
 dies out of   impossible fills /           edge vanishes    "significant"
 sample        too-good entries             on full universe by chance

1. Overfitting (quá khớp). Every parameter you add is a knob the optimiser can turn to fit the past. Five years of NQ one-minute bars contain a lot of noise, and noise has shape. Give a rule enough knobs and it will memorise that shape. Out of sample the noise is different, so the memorised rule fails. Warning signs: many parameters, a “spiky” optimisation surface where performance collapses if one parameter moves one notch, and rules with suspiciously specific conditions (“only on Tuesdays when the overnight range is between 38 and 41 points”).

2. Look-ahead bias (thiên lệch nhìn trước). The rule uses information that was not available at the moment of the decision. Classic order-flow versions:

• using the session’s POC or value area to filter an entry made at 10:00 — the final POC is only known at the close;
• using a bar’s close to trigger an entry “at the open of that same bar”;
• a daily filter computed from the full day’s high/low;
• resampled data where a 5-minute bar is stamped with its start time but contains the next five minutes.

Look-ahead usually shows up as a backtest that looks too good — a win rate of 80%+ on a simple rule should make you suspicious before it makes you excited.

3. Survivorship and selection bias (thiên lệch sống sót và chọn mẫu). The sample only contains what survived. In equities this means testing on today’s index members, which excludes companies that went bankrupt. In futures the contract itself survives, but the same bias appears in other forms: testing only the “clean” days you saved screenshots of, discarding sessions where the data feed broke (often the most volatile ones), or building a back-adjusted continuous contract and forgetting that roll days behaved differently.

4. Data snooping (đào bới dữ liệu). Many ideas are tried on the same data, and only the best one is reported. Each test on its own might be honest. The search is what lies. Halbert White’s “Reality Check” (Econometrica, 2000) was built exactly for this problem: testing whether the best model from a search really beats a benchmark, or just won the lottery among many candidates.

The arithmetic of trying many things

Suppose none of your ideas has any edge. Test each one at the usual 5% significance level. The chance that a single test falsely looks “significant” is 5%. The chance that at least one of N independent tests does is 1 − 0.95ᴺ:

Twenty variants of an entry model is a normal afternoon of tinkering. At that point it is more likely than not that something will look significant purely by chance.

This is the core of Harvey, Liu & Zhu (2016), “…and the Cross-Section of Expected Returns” (Review of Financial Studies 29(1), 5–68). They counted hundreds of published return “factors” and argued that, because the whole profession has been mining the same data, a new factor should clear a t-statistic above 3.0, not the traditional 2.0. A t of 2 is a fine bar for one pre-planned test. It is far too low for the best of hundreds.

Bailey, Borwein, López de Prado & Zhu (2014), “Pseudo-Mathematics and Financial Charlatanism” (Notices of the AMS 61(5), 458–471), made the same point in backtest language. If a strategy has zero true Sharpe ratio, the best of N tried variants still has an expected in-sample Sharpe ratio that grows with N. Using the formula from their paper for the expected maximum of N independent trials, the best of about 45 variants is expected to show an annualised Sharpe ratio of about 1.0 over 5 years of data — with a true Sharpe of zero. They call the data length needed to rule this out the minimum backtest length. Five years of data is only enough to defend roughly 45 independent tries at a Sharpe of 1. Most retail optimisers run thousands.

Worked example: auditing the “version 23” backtest

Back to the opening. Write down what was actually done:

Every row is a reason the 1.4 Sharpe is inflated. None of them proves the idea has no edge. Together they mean the backtest is not evidence yet. The fix is not to give up. The fix is to (a) correct the look-ahead bug, (b) put back the missing days, (c) add realistic costs (M10.02), (d) record N honestly and deflate the result (M10.04), and (e) judge it on data the search never touched (M10.03, M10.06).

Common mistake: Thinking “I only tested one strategy” because you only kept one. The tests you ran and threw away still count. The number that matters is how many configurations you looked at — including the ones you eyeballed on a chart and rejected without formally running. If you don’t log them, you will always underestimate N.

Recap:

• A backtest is a simulation, not evidence. It becomes evidence only when the design rules out overfitting, look-ahead, survivorship and data snooping.
• With N independent tries at the 5% level, the chance of at least one false discovery is 1 − 0.95ᴺ: 64% at N = 20.
• Harvey, Liu & Zhu argue for t > 3 once many tests have been run. Bailey et al. show the best of ~45 zero-edge variants is expected to show a Sharpe of about 1 over 5 years.
• Always record how many variants you tried. That number is the input to every correction in this module.

Quiz: You test 10 independent variants of a delta-divergence entry on ES, each at the 5% level, and one comes out “significant” with t = 2.3. A friend says, “t above 2 means it’s real.” What is wrong with the claim? Answer: With 10 tries and no true edge, the chance of at least one false “significant” result is 1 − 0.95¹⁰ ≈ 40%. So a single t = 2.3 among ten tries is weak evidence. It is below the t > 3 bar that Harvey, Liu & Zhu suggest for searched results, and it needs a test on fresh data before you trust it. Explanation: the threshold for significance must rise with the number of tests. A t of 2 is meant for one pre-planned test, not the best of a search.

Further reading:

• Harvey, Liu & Zhu (2016), “…and the Cross-Section of Expected Returns”, Review of Financial Studies 29(1). Free working-paper version on SSRN (also NBER w20592).
• Bailey, Borwein, López de Prado & Zhu (2014), “Pseudo-Mathematics and Financial Charlatanism”, Notices of the AMS 61(5). The minimum backtest length argument.
• White (2000), “A Reality Check for Data Snooping”, Econometrica 68(5). The classic statistical test for “best of many” results.

Prerequisite: M7.10 (validating a method before trusting it) · Next: M10.02 Costs & slippage in ticks · Connects to: M10.04 (Deflated Sharpe Ratio), M10.06 (lockbox discipline)

Educational content only — not financial advice. Futures trading involves substantial risk of loss.