ULTIMA
Home /Orderflow /M10 · Quant Validation

Lockbox discipline: hold-out, pre-registration & the research log

M10.06Quant ValidationProfessional≈ 12 minIn production

Learning objectives

  • Set aside a lockbox (hold-out) data set and explain why it may be evaluated exactly once.
  • Write a one-page pre-registration of the hypothesis, rules, costs and pass/fail criteria before running the final test.
  • Keep a research log that records every variant tried, so that N for M10.04 and M10.05 is honest.
  • Decide in advance what you will do if the lockbox test fails.
INTERACTIVE ORDER-FLOW SIMULATIONUse the timeline scrubber and interactive controls below the chart
M7.10 — BACKTEST OVERFITTING & DEFLATED SHARPE VISUALIZER

Multiple-Testing Haircut: Reported Sharpe vs Deflated Sharpe Across N = 1 → 1000 Trials

Multiple-Testing Haircut: Reported Sharpe vs Deflated Sharpe Across N = 1 → 1000 Trials — Interactive figure
0%
WHAT TO SEELOW SELECTION BIAS (N ≤ 10): Hypothesis-driven auction rule tested once retains nearly all of its reported Sharpe ratio.

M10.06 — Lockbox discipline · Quant Validation

This lesson is in production. The outline below shows what it will cover. The full lesson will ship with templates and a quiz with answer explanations.

Hook

  • The second time you look at a hold-out set, it stops being a hold-out set.

Chart first: the data budget

  • Timeline: development data (free to explore) → validation (walk-forward, M10.03) → lockbox (touched once, at the end).
  • What counts as “touching”: looking at its equity curve, its summary statistics, or even a chart of that period while designing rules.

Pre-registration

  • One dated page, written before the lockbox run: hypothesis, exact rules, contract, session, cost model (M10.02), metric, pass threshold (for example DSR, or net expectancy in ticks), and the decision you will take on a pass or a fail.
  • The idea comes from clinical trials and experimental science, adapted here for a solo trader.

The research log

  • Every idea, variant and parameter sweep, with date and outcome, including ideas rejected by eye.
  • Supplies the honest N and V[SR] used by the Deflated Sharpe Ratio (M10.04) and CSCV (M10.05).
  • Builds on the journaling habit from M7.03.

When the lockbox fails

  • Accept the result. Do not re-run “small fixes” on the same lockbox. A new lockbox needs new data, from forward paper trading or simply from time passing.

Common mistake (planned)

  • “I only looked at the lockbox to check the code ran.” Any look counts.

Sources

  • Bailey & López de Prado (2014), “The Deflated Sharpe Ratio”, Journal of Portfolio Management 40(5), on recording all trials. SSRN
  • Harvey, Liu & Zhu (2016), “…and the Cross-Section of Expected Returns”, Review of Financial Studies 29(1), on the cost of unreported tests. SSRN

Prerequisite: M10.05 Probability of Backtest Overfitting · Next: M10.07 Testing an order-flow idea properly

Educational content only — not financial advice. Futures trading involves substantial risk of loss.

Saved in this browser (account sync comes at launch).

Not financial advice. Educational content only. Futures trading involves substantial risk of loss.