Module 18: Backtesting & Strategy Evaluation

Introduction: Judging Strategies Like a Statistician

Backtesting is the process of evaluating a trading strategy on historical data. It is the financial equivalent of out-of-sample model validation — but with far more ways to go wrong. A beautifully backtested strategy that earned 40% per year with a Sharpe of 3.0 might be nothing more than an artifact of overfitting, data snooping, and biased simulation. This module equips you to evaluate strategies with the same rigor you would apply to a statistical study.

1. The Sharpe Ratio: Finance's Signal-to-Noise Ratio

1.1 Definition

where R_p is the portfolio return, R_f is the risk-free rate, μ_excess is the mean excess return, and σ_excess is the standard deviation of excess returns.

1.2 Interpreting the Sharpe Ratio

1.3 Annualizing the Sharpe Ratio

This scaling assumes returns are independently and identically distributed (IID). The √252 factor comes from the fact that the variance of the sum of n IID variables scales as n, so the standard deviation scales as √n, while the mean scales as n. Thus the ratio of mean to standard deviation scales as √n.

1.4 Statistical Significance of the Sharpe Ratio

Under the null hypothesis that the true Sharpe ratio is zero, the estimated Sharpe ratio follows approximately:

This means you need years of data to confirm a genuine Sharpe ratio. With daily data:

2. The Sortino Ratio: Penalizing Only Downside

2.1 Motivation

The Sharpe ratio treats upside and downside volatility equally. But investors do not care about upside volatility — they only fear losses. The Sortino ratio replaces total standard deviation with downside deviation.

3. Maximum Drawdown: A Path-Dependent Risk Measure

3.1 Definition

Maximum drawdown (MDD) is the largest peak-to-trough decline in portfolio value over the entire backtest period.

3.2 Why Drawdown Matters More Than Volatility

Volatility is a statistical abstraction; drawdown is a lived experience. A strategy with 15% annualized volatility sounds moderate, but if it experiences a 45% drawdown, most investors will panic-sell at the bottom. Key relationships:

• For a random walk with annual volatility σ, the expected maximum drawdown over T years is approximately σ × √(2T × ln(T)).
• Drawdowns are path-dependent: two return series with identical means and variances can have very different drawdown profiles.
• Drawdown duration (how long it takes to recover) matters as much as drawdown depth.

3.3 Calmar Ratio

The Calmar ratio rewards returns and penalizes drawdowns. A Calmar above 1.0 is considered good; above 2.0 is excellent. Like the Sharpe ratio, very high Calmar ratios in a backtest should trigger suspicion of overfitting.

4. Complete Performance Metrics Reference

5. The Multiple Testing Problem in Strategy Search

5.1 The Setup

A quantitative researcher tests N candidate strategies and reports the best one. Even if all N strategies have zero true alpha, the best one will have a positive in-sample Sharpe simply due to chance. The expected maximum Sharpe ratio of N zero-alpha strategies is:

where n is the number of return observations. Testing 1,000 strategies on 10 years of daily data: E[max SR] ≈ √(2 × 6.9) / √2520 ≈ 0.074 daily ≈ 1.18 annualized. A purely random strategy search will produce Sharpe ratios above 1.0.

5.2 Haircut Sharpe Ratios

The idea is simple: given the number of strategies tested (N), adjust the observed Sharpe ratio downward to account for selection bias. Two approaches:

Bonferroni Correction

Benjamini-Hochberg (FDR Control)

Rather than controlling the probability of any false discovery (FWER), control the expected proportion of false discoveries among those declared significant. This is less conservative and more appropriate when you expect some strategies genuinely have alpha.

Python
import numpy as np
from scipy import stats
from statsmodels.stats.multitest import multipletests

def haircut_sharpe(observed_sharpe, n_strategies, n_obs, method="bonferroni"):
    """
    Adjust Sharpe ratio for multiple testing.

    Parameters
    ----------
    observed_sharpe : float or array, annualized Sharpe ratios
    n_strategies : int, total number of strategies tested
    n_obs : int, number of return observations
    method : str, 'bonferroni' or 'fdr_bh'

    Returns
    -------
    Adjusted significance assessment
    """
    # Convert annualized Sharpe to t-statistic
    t_stats = np.atleast_1d(observed_sharpe) * np.sqrt(n_obs / 252)

    # Compute p-values (two-sided)
    p_values = 2 * (1 - stats.t.cdf(np.abs(t_stats), df=n_obs - 1))

    # Apply multiple testing correction
    reject, adjusted_p, _, _ = multipletests(p_values, method=method)

    return {
        "observed_sharpe": observed_sharpe,
        "t_stat": t_stats,
        "raw_p": p_values,
        "adjusted_p": adjusted_p,
        "significant": reject,
    }

# Example: Test the best of 500 strategies
n_strategies = 500
n_obs = 252 * 10  # 10 years of daily data

# Suppose the best strategy has Sharpe = 1.8
result_bonf = haircut_sharpe(1.8, n_strategies, n_obs, "bonferroni")
result_bh = haircut_sharpe(1.8, n_strategies, n_obs, "fdr_bh")

print(f"Observed Sharpe: 1.8")
print(f"Raw p-value: {result_bonf['raw_p'][0]:.6f}")
print(f"Bonferroni adjusted p: {result_bonf['adjusted_p'][0]:.6f}")
print(f"  Significant? {result_bonf['significant'][0]}")
print(f"BH (FDR) adjusted p: {result_bh['adjusted_p'][0]:.6f}")
print(f"  Significant? {result_bh['significant'][0]}")

# Minimum Sharpe needed for significance
for n_strat in [1, 10, 100, 500, 1000, 10000]:
    # Bonferroni: need p < 0.05 / n_strat
    adjusted_alpha = 0.05 / n_strat
    min_t = stats.t.ppf(1 - adjusted_alpha / 2, df=n_obs - 1)
    min_sharpe = min_t / np.sqrt(n_obs / 252)
    print(f"N={n_strat:>5d} strategies -> min Sharpe: {min_sharpe:.2f}")

6. Backtest Overfitting

6.1 In-Sample vs Out-of-Sample Degradation

The hallmark of overfitting is a large gap between in-sample (IS) and out-of-sample (OOS) performance. A strategy that earns Sharpe 2.5 in-sample but Sharpe 0.3 out-of-sample has learned the noise in the training period.

6.2 The Probability of Backtest Overfitting (PBO)

Bailey et al. (2014) introduced a formal method to estimate the probability that a backtested strategy is overfit. The procedure:

1. Partition the data into S blocks.
2. Generate all combinatorially possible train/test splits using S/2 blocks for training and S/2 for testing.
3. For each split, find the optimal strategy in-sample and measure its OOS performance.
4. PBO = fraction of splits where the IS-optimal strategy has negative OOS Sharpe.

A PBO above 50% means the backtest is more likely overfit than not.

7. Walk-Forward Analysis: The Gold Standard

7.1 Procedure

1. Divide the historical period into sequential windows.
2. For each window, use the first portion for training (parameter fitting) and the remainder for testing.
3. Advance the window by one step and repeat.
4. Concatenate all OOS test periods to form a continuous OOS equity curve.
5. Compute performance metrics only on the OOS portion.

8. Common Backtesting Pitfalls

8.1 Look-Ahead Bias

Using information that was not available at the time of the trading decision. Examples:

• Using adjusted closing prices that incorporate future stock splits or dividends.
• Filtering the stock universe using future information (e.g., excluding stocks that later go bankrupt).
• Using fundamental data before its actual release date (earnings are reported weeks after quarter-end).

8.2 Survivorship Bias

Example: If you backtest a strategy on the current S&P 500 constituents going back 20 years, you are only testing on companies that survived 20 years. Companies that went bankrupt (Lehman Brothers, Enron, WorldCom) are excluded, removing the worst outcomes and biasing returns upward.

8.3 Transaction Cost Neglect

A strategy that trades frequently looks great without costs but may be deeply unprofitable after commissions, slippage, and market impact. Key cost components:

8.4 Overfitting to a Specific Period

A strategy optimized on 2010–2020 may have learned patterns specific to the post-GFC low-rate, low-volatility, QE-driven bull market. It will likely fail in a different regime (rising rates, high inflation, bear market). This is the non-stationarity problem from Module 16 applied to strategy design.

9. Python: Complete Momentum Strategy Backtest

Python
import numpy as np
import pandas as pd
import yfinance as yf
from scipy import stats

# ──────────────────────────────────────────────
# Step 1: Download data
# ──────────────────────────────────────────────
spy = yf.download("SPY", start="2005-01-01", end="2023-12-31")
returns = spy["Adj Close"].pct_change().dropna()

# ──────────────────────────────────────────────
# Step 2: Simple momentum strategy
#   Long SPY when 50-day return > 0, else flat (cash)
# ──────────────────────────────────────────────
lookback = 50
momentum_signal = returns.rolling(lookback).mean().shift(1)  # Shifted to avoid look-ahead
position = (momentum_signal > 0).astype(float)

# Strategy returns (assume 5bps transaction cost per trade)
trades = position.diff().abs()
tc_per_trade = 0.0005
strategy_returns = position * returns - trades * tc_per_trade
strategy_returns = strategy_returns.dropna()

# Benchmark: buy and hold
benchmark_returns = returns.loc[strategy_returns.index]

# ──────────────────────────────────────────────
# Step 3: Compute all performance metrics
# ──────────────────────────────────────────────
def compute_metrics(returns, name="Strategy", rf_annual=0.02):
    """Compute comprehensive backtest metrics."""
    rf_daily = rf_annual / 252
    excess = returns - rf_daily
    n = len(returns)

    # Basic stats
    ann_return = returns.mean() * 252
    ann_vol = returns.std() * np.sqrt(252)

    # Sharpe ratio
    sharpe = excess.mean() / excess.std() * np.sqrt(252)

    # Sortino ratio
    downside = excess[excess < 0]
    downside_vol = np.sqrt((excess.clip(upper=0)**2).mean()) * np.sqrt(252)
    sortino = excess.mean() * 252 / downside_vol if downside_vol > 0 else np.nan

    # Maximum drawdown
    cum_returns = (1 + returns).cumprod()
    running_max = cum_returns.cummax()
    drawdown = (cum_returns - running_max) / running_max
    max_dd = drawdown.min()

    # Calmar ratio
    calmar = ann_return / abs(max_dd) if max_dd != 0 else np.nan

    # Win rate
    win_rate = (returns > 0).mean()

    # Statistical significance
    t_stat = excess.mean() / (excess.std() / np.sqrt(n))
    p_value = 2 * (1 - stats.t.cdf(abs(t_stat), df=n-1))

    # Skewness and kurtosis
    skew = returns.skew()
    kurt = returns.kurtosis()

    # Maximum drawdown duration
    dd_duration = 0
    max_dd_duration = 0
    for d in drawdown:
        if d < 0:
            dd_duration += 1
            max_dd_duration = max(max_dd_duration, dd_duration)
        else:
            dd_duration = 0

    return pd.Series({
        "Ann. Return": f"{ann_return:.2%}",
        "Ann. Volatility": f"{ann_vol:.2%}",
        "Sharpe Ratio": f"{sharpe:.3f}",
        "Sortino Ratio": f"{sortino:.3f}",
        "Max Drawdown": f"{max_dd:.2%}",
        "Calmar Ratio": f"{calmar:.3f}",
        "Win Rate": f"{win_rate:.2%}",
        "Skewness": f"{skew:.3f}",
        "Kurtosis": f"{kurt:.3f}",
        "t-statistic": f"{t_stat:.3f}",
        "p-value": f"{p_value:.4f}",
        "Max DD Duration": f"{max_dd_duration} days",
    }, name=name)

# Compare strategy vs benchmark
metrics = pd.DataFrame({
    "Momentum": compute_metrics(strategy_returns, "Momentum"),
    "Buy & Hold": compute_metrics(benchmark_returns, "Buy & Hold"),
})
print(metrics)

# ──────────────────────────────────────────────
# Step 4: Apply multiple testing correction
#   Suppose we tested 20 lookback periods (10, 20, ..., 200)
# ──────────────────────────────────────────────
from statsmodels.stats.multitest import multipletests

lookbacks = range(10, 210, 10)
sharpes = []
pvals = []

for lb in lookbacks:
    sig = returns.rolling(lb).mean().shift(1)
    pos = (sig > 0).astype(float)
    tr = pos.diff().abs()
    strat_ret = (pos * returns - tr * tc_per_trade).dropna()

    excess = strat_ret - 0.02/252
    sr = excess.mean() / excess.std() * np.sqrt(252)
    t = excess.mean() / (excess.std() / np.sqrt(len(excess)))
    p = 2 * (1 - stats.t.cdf(abs(t), df=len(excess)-1))
    sharpes.append(sr)
    pvals.append(p)

# Apply BH correction
reject_bh, adjusted_p, _, _ = multipletests(pvals, method="fdr_bh")

results = pd.DataFrame({
    "Lookback": list(lookbacks),
    "Sharpe": sharpes,
    "Raw p-value": pvals,
    "BH adjusted p": adjusted_p,
    "Significant (BH)": reject_bh,
})
print("\n=== Multiple Testing Correction ===")
print(results.to_string(index=False))

10. Chapter Summary