Validating a method before trusting it

Sunday evening, 18:00 ET (06:00 ICT Monday, winter), the Globex open. A trader posts in a chat: “I backtested my NQ rejection short. Sharpe 2.1.” What the post leaves out: the backtest covers one year, and the trader tried 40 combinations of stop, filter and timeframe to get there. Nobody lied. The procedure did it: run 40 variants and keep the best, and you have picked the luckiest draw. On one year of daily returns, the best of 40 variants with no skill at all is expected to show a Sharpe of about 2.2 Definition. This 2.1 tells you almost nothing yet. This lesson is the decontamination chamber every method passes through before it touches real size.

Key term

Validation (kiểm chứng) means testing a fully written method on data and in conditions that did not shape it, before you trade it at real size Definition. Its enemy is backtest overfitting (quá khớp): rules fitted to the noise of the sample you searched, so they describe the past and not the market.

Term you need: a confidence interval (khoảng tin cậy) is the range of true values your sample cannot rule out. The maths is owned by EV3.01; this lesson only applies it.

Chart first: the luck bar rises with every try

How it works

Three stages, three different questions

Stage 1: backtest (kiểm thử lịch sử). Replay at least 100 setups on historical charts, logging entry, stop and target as if live, with no peeking at the next bar. It tests whether the logic had an edge in that sample. It cannot test your execution, your slippage or whether you would really take every signal. Count the calendar, too: some setups fire only a handful of times a year (one trader’s pre-registered public study counted 4 to 10 a year for many taught confluences; a count, not an edge test), so 100 examples can mean ten or more years of history. Screen time alone cannot validate a rare setup.

Stage 2: forward sim (giao dịch mô phỏng). At least 50 trades in real time with the full routine. It tests whether you can execute the method: the boredom, the slow weeks, the drawdown. Many methods die here, and that is the stage working. One limit: simulated fills are kinder than live ones, above all for resting limit orders, which a simulator may fill on a touch. How much kinder is not yet measured (MKT2.03).

Stage 3: small live. One MNQ or MES, at least 50 trades. It tests whether the method survives real money and real costs. The gap between sim and live results, the “emotion tax”, gets its first estimate here. With 50 trades per stage that estimate is very rough (see the worked example).

The graduation gates are a house rule

Our gates are conjunctive: positive expectancy after costs at the stage, adherence of at least 85%, and drawdown within the lines in your plan. Miss one and you stay at the stage. There is no stage 4 until stage 3 passes. The thresholds are a policy choice, not a measured optimum House rule.

Be honest about what a passed gate means. With 50 trades, a 50% win rate carries an interval of about ±14 percentage points: the true rate could be anywhere from about 36% to 64% Definition. For a typical payoff (wins of 1.5R, losses of 1R, about 45% winners), 60 trades pin expectancy down only to about ±0.3R. So a positive result at 50 or 60 trades means “no sign of disaster”, not “edge proven”. The gates decide whether you may keep risking small, and nothing more.

Every variant you try is a trial

Each stop distance, filter or session window you tried was a trial, and the best of them is selected, not observed. The Deflated Sharpe Ratio (tỷ số Sharpe giảm phát, Bailey and López de Prado, 2014) corrects for this. It asks: given how many variants were tried, how long the sample is and how fat-tailed the returns are, how likely is it that the true Sharpe is above zero? Definition

You do not need the formula here (M10.04 has it), only its shape. The luck bar rises with the number of tries and falls with the length of the sample. With independent variants and no skill:

• best of 40 on one year of data: expected Sharpe about 2.2, so a reported 2.1 is luck about half the time (51%);
• best of 40 on three years: expected about 1.3, so a 2.1 would be luck only about 0.5% of the time;
• one variant chosen in advance, Sharpe 1.2 on one year: luck about 12% of the time.

So a Sharpe of 1.2 from one pre-chosen rule is stronger evidence than 2.1 from the best of 40 on the same year Definition. Real variants are correlated, which makes the effective number of tries smaller than your count; fat tails push the luck bar higher. The two cut in different directions, which is why the full correction exists.

The classic holdout (“it worked on 2023 too”) fails when the holdout was looked at during development, which is the usual case when one person builds and tests alone. Our lab hit this in its own work. In a study of 226 pre-registered pre-market level tests on 721 sessions, none survived correction for the number of tests, and one interaction that looked significant in the discovery half (+4.0 pp, interval +0.6 to +7.7) reversed in the holdout half (−5.6 pp, interval −9.8 to −1.4) Measured: no edgeNasdaq index. That was measured on QQQ as a Nasdaq proxy; NQ futures are extrapolated. In the lab, an idea counts as measured only after its rule is frozen and run once on a fixed forward window it has never seen (M10.06 teaches that protocol).

Validated for which weather?

Market regimes are real: on stock markets, calm and turbulent states with different volatility and correlation persist and switch (Ang and Bekaert, 2002) MeasuredOther market. A method validated mostly in one state may bleed in another, not because it broke but because its edge, if any, lived in that state. The practical step is a record, not a model: write down the conditions of your validation sample (volatility level, trend or range days, event days) and re-validate when live conditions fall outside them House rule. How to sort regime variables by what they can tell you is owned by REG1.02.

Myth / hypothesis: “A statistical regime model (a Hidden Markov Model, for example) tells you which setup to trade tomorrow.” Regimes exist, but using such a model to time a single day has no edge shown in our lab’s notes Lore. It is a hypothesis you can test, not a filter to trust.

What “validated” certifies: the method had an edge over those samples, under those conditions, measured with that much noise. Not future profits, not other regimes, not your execution next month. Validation is a licence to risk small, nothing more.

What the data says

Worked example: one method through the pipeline

Synthetic numbers, no real account. A trader validates an NQ rejection-short method.

Stage 1. 120 historical setups, expectancy +0.45R after costs, interval about ±0.22R. Before testing, the trader wrote down three stop placements to compare; with only three pre-chosen variants and a result this far from zero, the selection penalty is small. Logic: an edge in this sample.

Stage 2. 60 sim trades, +0.38R (interval about ±0.31R), adherence 88%. The main leak is early exits; they are tagged and fixed. Executable.

Stage 3. 55 live trades on 1 MNQ, +0.31R after costs (interval about ±0.33R). The trader is tempted to write “emotion tax = 0.07R, measured”. It is not measured. The interval on the difference between two samples this size is about ±0.46R, so all 55 trades show is that the gap is not huge. Every gate passed, so the house rule allows 2 MNQ.

Six months later comes a −12R month. No panic: the trader resamples the validation trade list (RSK1.03) and finds months this deep among the simulated paths, so a month like this sits inside the method’s known weather. The trader also checks the conditions log: the month was a high-volatility stretch the validation sample barely covered, so the method goes back to Stage 2 rules for that regime until 50 fresh trades say otherwise. That is what validation buys: not profits, but calibrated trust.

Prop-firm note. Evaluation and funded-account fills are simulated, so a sim fill rate is not a live fill rate (MKT2.03). Building a method to harvest simulator fills is prohibited conduct at the firm (PROP1.04); check the firm’s current help center before you trade.

Common mistake: Validating on the data you developed on, then calling it “out-of-sample”. If you looked at 2024 while choosing parameters and then “confirm” on 2024, that is in-sample with extra steps. True out-of-sample means data the development process never touched, which is why Stage 2 (forward sim on a future nobody has seen) is the real filter.

Common mistake: Reading 50 positive trades as proof of an edge. That is treating a noisy sample as a measurement: at 50 trades the win-rate interval is about ±14 pp, wide enough to hold a coin flip.

Common mistake: Counting only the variants you kept. Every stop, filter or timeframe you tried and dropped is still a trial. Keep a trial log from day one, or your Sharpe has no denominator.

Drill: count your trials, size your sample

Take one method you trade or plan to trade.

1. List every variant you have tried on it: stop distances, filters, timeframes, sessions. The count is your N.
2. Count its logged trades, n, and its win rate, p.
3. Compute the 95% win-rate interval: 1.96 × √(p × (1 − p) / n).
4. Decide which stage the method is really at, and write the date and conditions of your sample.

What we don’t know yet

• How much better sim fills are than live fills for NQ and MNQ orders of retail size: not yet measured (MKT2.03).
• Whether a forward sim of 50 trades predicts live results for the same trader: not yet measured.

Recap

• Three stages, backtest (logic), forward sim (execution), small live (money and costs), each with conjunctive house gates; no skipping.
• Every variant you tried is a trial: the best of many is biased upward, and the Deflated Sharpe Ratio corrects for the number of tries, the sample length and fat tails.
• A passed gate at 50 trades means “no sign of disaster”: the win-rate interval is still about ±14 pp.
• Record the conditions your method was validated in, and re-validate when live conditions leave them.

Further reading

• The Deflated Sharpe Ratio (Bailey and López de Prado, 2014): correcting the Sharpe ratio for the number of trials, sample length and non-normal returns.
• Deflating the Sharpe Ratio (slides): the accessible version, with backtest overfitting and why holdouts fail.
• Advances in Financial Machine Learning, Ch. 1 (López de Prado): a research checklist for strategy testing.
• Ang, A. and Bekaert, G. (2002), “International asset allocation with regime shifts”, Review of Financial Studies 15(4): regimes in equity markets.
• Order Flow, Measured (public practitioner study): one trader’s pre-registered study, cited here only for how rarely many taught confluences fire; its intervals are not clustered by session and it counts as Plausible at most.
• Our lab: 226 pre-market level tests against placebo levels on 721 sessions (QQQ/SPY proxies); none survived correction, and a discovery-half interaction reversed in the holdout.

Connects to: M7.09 (the plan being validated) → M7.11 (the review that keeps it validated). The maths behind the intervals is in EV3.01; the formal corrections are in M10.01 and M10.04.

Educational content only — not financial advice. Futures trading involves substantial risk of loss.