Deflated Sharpe Ratios: Start with the Selection Problem

The question

If you try many strategy variants and show only the best backtest, what does its Sharpe ratio tell a reader? Less than it seems. The winner was selected using the same returns on which it is being judged.

The deflated Sharpe ratio of Bailey and López de Prado addresses selection bias, sample length, and non-normal returns. Before applying that method, it helps to see the selection problem directly. The experiment below does not calculate a deflated Sharpe ratio.

A small experiment you can run

We generate 100 independent, zero-mean strategies. Each has 252 simulated daily returns for selection and another 252 for a holdout period. The returns are Gaussian with unit standard deviation. There are no costs, autocorrelation, changing volatility, or genuine signals. We annualize each sample Sharpe using 252 trading days and choose the strategy with the highest Sharpe in the selection period.

from math import sqrt
from random import Random
from statistics import mean, stdev

rng = Random(7)

def sharpe(returns):
    return sqrt(252) * mean(returns) / stdev(returns)

trials = []
for _ in range(100):
    train = [rng.gauss(0, 1) for _ in range(252)]
    holdout = [rng.gauss(0, 1) for _ in range(252)]
    trials.append((sharpe(train), sharpe(holdout)))

selected = max(trials, key=lambda pair: pair[0])
print(f"Best training Sharpe: {selected[0]:.2f}")
print(f"Same trial, holdout Sharpe: {selected[1]:.2f}")

With Python’s standard library and this seed, the output is:

Best training Sharpe: 2.38
Same trial, holdout Sharpe: 0.13

The 2.38 is an observed result from this particular simulated selection period, not an estimated true Sharpe. Every strategy’s true Sharpe is zero by construction. The holdout value is also noisy; a single holdout cannot prove that a strategy has or lacks an edge. Change the seed and both numbers will change.

What this does and does not show

The experiment shows why the number of attempted variants matters. It does not establish a universal penalty for 100 trials. Real variants often share signals and returns, so treating them as 100 independent trials would be wrong. Real returns may also be skewed or heavy-tailed, and a short track record adds uncertainty.

For a real study, record all trials, their dependence, sample length, return distribution, transaction costs, and the rule used to select the winner. Keep a genuinely untouched evaluation period. If you report a formal deflated Sharpe ratio, use the full method and disclose its inputs; do not apply a fixed subtraction to an annualized Sharpe.

Bailey and López de Prado’s The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality is on our reading list. Read it for the statistical test and its assumptions.