Quantitative Validation Framework
Combinatorial purged cross-validation with a deflated-Sharpe correction, built to catch strategies that only look profitable by luck. Nineteen strategies tested on seven years of real Databento data.
A backtest will tell you almost anything you want to hear. Test a rule on one slice of history, tune it until the equity curve looks good, and you have manufactured a result that will not survive the next year of data. Worse, if you test eighteen strategies, at least one will look excellent purely by luck — the way one coin in a room of coin-flippers comes up heads ten times running. The framework exists to tell those apart from real skill.
The parts that carry the weight.
Combinatorial purged cross-validation
Instead of testing on one slice of history, the framework generates C(10,2) = 45 different fold combinations. Each fold is purged of look-ahead bias and embargoed at its boundaries to prevent serial-correlation leakage. The output is 45 independent Sharpe estimates — a real distribution rather than a single guess.
Probabilistic Sharpe Ratio
PSR(SR*) is the probability that the true Sharpe exceeds a threshold, given sample size, skewness, and kurtosis. It corrects for the fact that a fat-tailed strategy carrying the same Sharpe as a well-behaved one is dramatically less certain.
Deflated Sharpe Ratio — the gate
DSR shifts PSR's benchmark up to account for selection bias across every strategy tested. When you test N strategies, the best-looking one is inflated by extreme-value statistics; DSR computes the expected maximum Sharpe under the null and asks whether the observed Sharpe clears it. DSR ≥ 0.95 deploys. Nothing has.
Live-realism harness
A separate layer checks whether an edge survives real execution friction rather than idealized fills, because the gap between a backtest's assumptions and a broker's behavior is where most paper edges quietly die.
Why it's built this way.
- 01
Purging and embargoing are not optional details. Without them, overlapping folds leak information across the boundary and every Sharpe estimate is quietly correlated with its neighbors.
- 02
The gate is agreed before the numbers are seen. A threshold chosen after looking at results is not a threshold, it is a rationalization.
- 03
Nineteen strategies produced eighteen shelved and one with insufficient data. That is the finding, not a failure of the framework — the whole point was to learn it here rather than with money.