Catching the liesbacktests tell.
Trading strategies can look profitable in a backtest and still lose money in the real world. I built a system that catches the fakes before any capital is at risk. Across the 19 strategies I've fed it, not one false positive reached live capital: 18 were shelved and one had too few trades to judge.Under the hood: combinatorial purged cross-validation with deflated Sharpe correction, tested across 19 strategies and 1.6M bars of real market data.
False positives caught
19
Real-money losses
$0
Catch rate
100%
Years of real data
7yr
Step 01 · Hopeful
Nineteen strategies, all promising.
Each curve is a strategy's backtested equity, climbing. On the surface, they all look like edges.
Step 02 · Cross-validation
Then the folds sweep.
Combinatorial purged cross-validation tests each across many time slices; the overfit ones grey out.
Step 03 · Deflation
And the Sharpe deflates.
The Deflated Sharpe Ratio corrects for how many strategies were tried. Most apparent edge was luck.
Step 04 · Verdict
19 strategies, 7 years of real Databento data.
Almost nothing survives. The interactive proof is below, replay any strategy yourself.
What you're about to see
Every red verdict here is a win. Each red strategy looked profitable in early tests and would have lost real money if deployed. Across all 19, the gate let none through: 18 were shelved and one had too few trades to judge. Exactly what it's designed to do.
Live replay
Real validation runs. Animate any strategy.
Pick a strategy below and watch the testing happen. Each bar that animates in is a different slice of market history the strategy was tested on: green if it made money, pink if it lost. The numbers, the verdict, and every result are from real tests on 7 years of market data.
Strategy
Verdict
SHELVE
Do not trade. The strategy was profitable in only 29% of the 45 time slices we tested, and once we account for the fact that we tried 18 strategies, the apparent edge looks like statistical noise, not real skill.
Without the gate
Would have been deployed and lost
-$75.5K
Losing in 32/45 time slices tested across 7 years
With the gate
Caught at validation, never deployed
$0 risked
Filtered before any real money was at stake
CPCV fold execution
0/45 folds completed
Cumulative test P&L
Summed across folds from break-even
One thing that confuses people: this line can drop below what a real account could ever lose, past negative $50,000 and beyond. That's because it isn't one account's balance. The same stretch of market history is replayed across 45 overlapping test windows, and this curve adds up the wins and losses from all of them. So if a strategy loses money in most windows, those losses stack on top of each other and the total can fall far lower than a single $50,000 account would ever go. Read it as "how badly did this idea fail across every test," not "how much money was in the account."
Case Study
Watch the gate catch a false positive.
Bot 16, a pairs-trading strategy on the S&P and Nasdaq futures, looked highly profitable over 3 months of real market data. Then 7 years of data told the real story. The strict test (DSR) flagged it on day one. Without it, this strategy would have lost real money.
What it looked like
3 months of data
“Looked like a winner. But the strict test (DSR) was already flashing zero, a quiet warning that the other numbers might be misleading.”
What it actually was
7 years of data
“The strict test was right. With more data, the 'edge' disappeared. Without validation, this strategy would have lost $75K of real capital.”
The lesson
A strategy looking profitable in a short backtest isn't enough. The strict test asks the harder question: “Given that I tried 18 different strategies, is it actually surprising that one of them looks this good? Or is this just the lucky winner of an 18-shot lottery?” When the strict test says zero, the strategy is noise, no matter how good the other numbers look. Trust the math over the gut. Always.
Methodology
Three layers of statistical rigor.
Each layer catches a different kind of false positive. The combination is what separates real skill from luck.
Layer 1
Combinatorial Purged Cross-Validation
Instead of testing the strategy on one slice of market history, I tested it on 45 different overlapping slices. Now I can see how consistent the result actually is, not just whether one lucky window made it look good.
For the technically curious
Technically: the framework generates C(10,2) = 45 different fold combinations. Each fold is purged of look-ahead bias and embargoed at boundaries to prevent serial-correlation leakage. The result is 45 independent Sharpe estimates instead of one: a real distribution, not a single guess.
n_folds = C(N, k) • paths = C(N-1, k-1)Layer 2
Probabilistic Sharpe Ratio
Asks the question: 'how confident can I be that the strategy actually has an edge, given the data I have?' Higher = more confident.
For the technically curious
Technically: PSR(SR*) is the probability that the true Sharpe exceeds SR*, accounting for sample size, skewness, and kurtosis. It corrects for the fact that a fat-tailed strategy with the same Sharpe as a normal one carries dramatically more uncertainty.
PSR(SR*) = Φ[ (SR − SR*) × √(n − 1) / √(1 − γ₃·SR + (γ₄−1)/4·SR²) ]Layer 3
Deflated Sharpe Ratio
This is the strict test. If I try 18 different strategies, at least one will look amazing purely by luck, like flipping coins and finding one that came up heads 10 times in a row. DSR adjusts for that. It asks: 'is this strategy still impressive after accounting for the 18 lottery tickets I bought?'
For the technically curious
Technically: DSR is PSR's benchmark shifted up to account for selection bias from testing multiple strategies. When you test N strategies, the best-looking one is inflated by extreme-value statistics. DSR computes the expected maximum Sharpe under the null and asks if your observed Sharpe clears that bar. This is the gate. DSR ≥ 0.95 deploys.
SR* ≈ V × [(1−γ)·Φ⁻¹(1−1/N) + γ·Φ⁻¹(1−1/(N·e))]Seen enough? Let's talk.
Every project on the homepage has its own breakdown like this one. Explore the rest, or reach out if you have something worth building.