ULTIMA
Home /Orderflow /M10 · Quant Validation

Probability of Backtest Overfitting (CSCV)

M10.05Quant ValidationProfessional≈ 18 minIn production

Learning objectives

  • Explain what the Probability of Backtest Overfitting (PBO) measures — how often the in-sample winner ranks at or below the median out of sample.
  • Walk through Combinatorially Symmetric Cross-Validation (CSCV) step by step on a matrix of strategy-variant returns.
  • Interpret the outputs — the logit distribution, PBO and performance degradation.
  • Know the method's assumptions and limits, and when to use it alongside the Deflated Sharpe Ratio.
INTERACTIVE ORDER-FLOW SIMULATIONUse the timeline scrubber and interactive controls below the chart
M7.10 — BACKTEST OVERFITTING & DEFLATED SHARPE VISUALIZER

Multiple-Testing Haircut: Reported Sharpe vs Deflated Sharpe Across N = 1 → 1000 Trials

Multiple-Testing Haircut: Reported Sharpe vs Deflated Sharpe Across N = 1 → 1000 Trials — Interactive figure
0%
WHAT TO SEELOW SELECTION BIAS (N ≤ 10): Hypothesis-driven auction rule tested once retains nearly all of its reported Sharpe ratio.

M10.05 — Probability of Backtest Overfitting (CSCV) · Quant Validation

This lesson is in production. The outline below shows what it will cover. The full lesson will ship with a worked example and a quiz with answer explanations.

Hook

  • You kept a log of all N variants (M10.01, M10.04). CSCV turns that log into one number: how likely is it that your selection process picks an overfit strategy?

Chart first: the T × N performance matrix

  • Rows are time periods and columns are strategy variants. Split the rows into S equal blocks.
  • Every combination of S/2 blocks forms an in-sample set, and the remaining blocks form the out-of-sample set (“combinatorially symmetric”).

The CSCV procedure

  • For each combination, pick the best variant in-sample, find its relative rank out of sample, and convert the rank to a logit λ.
  • PBO = the share of combinations where λ ≤ 0, meaning the IS winner lands at or below the OOS median.
  • Companion diagnostics: performance degradation (IS vs OOS scatter) and the probability of a loss out of sample.

Worked example (planned)

  • A small illustrative matrix of opening-range variants (for example 8 blocks → 70 combinations), with the computed PBO and what it implies.

Limits & assumptions

  • CSCV needs the full set of variants you actually tried. It is sensitive to the block count S and assumes the blocks are comparable (regime caveat).
  • PBO evaluates the selection procedure, not one strategy’s true edge.

Common mistake (planned)

  • Running CSCV only on the shortlisted “good” variants, which removes the very losers the method needs.

Sources

Prerequisite: M10.04 Multiple testing & the Deflated Sharpe Ratio · Next: M10.06 Lockbox discipline

Educational content only — not financial advice. Futures trading involves substantial risk of loss.

Saved in this browser (account sync comes at launch).

Not financial advice. Educational content only. Futures trading involves substantial risk of loss.