Module 16: Machine Learning in Finance

Introduction: Where Statistics Meets Silicon

Machine learning in finance is not a revolution — it is an extension of the statistical methods you already know, applied to a domain where the signal-to-noise ratio is brutally low and the data-generating process is non-stationary. This module will equip you with the practical knowledge to apply ML to financial data without falling into the traps that have claimed countless quantitative strategies. The central theme: what works in Kaggle competitions fails in financial markets, and understanding why is essential.

1. Feature Engineering from Financial Data

1.1 The Raw Materials

Financial data comes in several forms, each requiring different feature engineering approaches:

1.2 Lag Features

The simplest and most important features are lagged values of the target and related variables. If you are predicting tomorrow’s return, features include today’s return, the return from 2 days ago, 5 days ago, and so on. This is the ML equivalent of an AR model.

1.3 Rolling Statistics

Rolling windows capture time-varying properties of the return distribution:

• Rolling mean (momentum): moving average of returns over 5, 20, 60, 252 days.
• Rolling volatility: standard deviation of returns over a lookback window.
• Rolling skewness and kurtosis: higher-order moments that capture tail behavior.
• Rolling correlation: time-varying co-movement between assets.
• Rolling z-score: (current value − rolling mean) / rolling std — mean-reversion signal.

1.4 Cross-Asset Features

Financial markets are interconnected. Features from related assets often contain more predictive information than the target asset’s own history:

• Yield curve slope (10Y − 2Y Treasury) predicts equity returns.
• VIX (implied volatility) predicts future realized volatility.
• Credit spreads (corporate minus Treasury yields) signal economic stress.
• Currency strength indices signal global risk appetite.

1.5 Python: Feature Engineering Pipeline

Pythonimport numpy as np
import pandas as pd
import yfinance as yf

def create_features(ticker, start="2015-01-01", end="2023-12-31"):
    """Build ML features from price data."""
    data = yf.download(ticker, start=start, end=end)
    df = pd.DataFrame()

    close = data["Adj Close"]
    volume = data["Volume"]
    returns = close.pct_change()

    # Target: next-day return (what we predict)
    df["target"] = returns.shift(-1)

    # Lag features (returns)
    for lag in [1, 2, 3, 5, 10, 20]:
        df[f"ret_lag_{lag}"] = returns.shift(lag)

    # Rolling statistics (use .shift(1) to avoid look-ahead)
    for window in [5, 20, 60]:
        df[f"rolling_mean_{window}"] = returns.rolling(window).mean().shift(1)
        df[f"rolling_std_{window}"] = returns.rolling(window).std().shift(1)
        df[f"rolling_skew_{window}"] = returns.rolling(window).skew().shift(1)
        df[f"rolling_kurt_{window}"] = returns.rolling(window).kurt().shift(1)

    # Momentum features
    for period in [5, 20, 60, 252]:
        df[f"momentum_{period}"] = (close / close.shift(period) - 1).shift(1)

    # Volatility ratio (short-term vs long-term)
    df["vol_ratio"] = (
        returns.rolling(5).std() / returns.rolling(60).std()
    ).shift(1)

    # Volume features
    df["volume_sma_ratio"] = (volume / volume.rolling(20).mean()).shift(1)

    # Rolling z-score (mean reversion signal)
    df["z_score_20"] = (
        (close - close.rolling(20).mean()) / close.rolling(20).std()
    ).shift(1)

    return df.dropna()

features = create_features("SPY")
print(f"Features shape: {features.shape}")
print(f"Feature names: {[c for c in features.columns if c != 'target']}")

2. Walk-Forward Validation: The Only Valid Approach

2.1 Why K-Fold Cross-Validation Is Wrong for Time Series

Standard k-fold cross-validation randomly shuffles data into folds. This is catastrophic for time series because:

• Temporal leakage: Training on future data to predict the past violates causality.
• Autocorrelation leakage: Nearby observations are correlated; shuffling puts correlated points in both train and test sets.
• Regime leakage: The model learns future regime characteristics and applies them to past predictions.

2.2 Walk-Forward Schemes

2.3 Embargo and Purging

Even with walk-forward splitting, information can leak through features that span the train/test boundary. If you use a 20-day rolling mean as a feature, the last training observation’s feature overlaps with the first test observation’s feature.

• Embargo: Drop a gap of observations between train and test sets.
• Purging: Remove training observations whose feature computation window overlaps with any test observation.

2.4 Python: Walk-Forward Framework

Pythonimport numpy as np
import pandas as pd
from sklearn.metrics import mean_squared_error, r2_score

def walk_forward_cv(X, y, model_fn, train_size=504, test_size=21,
                     embargo=10, expanding=False):
    """
    Walk-forward cross-validation with embargo.

    Parameters
    ----------
    X : DataFrame of features
    y : Series of targets
    model_fn : callable that returns a fitted model (accepts X_train, y_train)
    train_size : int, number of training observations (for rolling window)
    test_size : int, number of test observations per fold
    embargo : int, gap between train and test
    expanding : bool, if True use expanding window instead of rolling
    """
    results = []
    n = len(X)

    # Generate fold boundaries
    test_starts = range(train_size + embargo, n - test_size + 1, test_size)

    for test_start in test_starts:
        # Define train and test indices
        if expanding:
            train_start = 0
        else:
            train_start = test_start - embargo - train_size

        train_end = test_start - embargo
        test_end = min(test_start + test_size, n)

        X_train = X.iloc[train_start:train_end]
        y_train = y.iloc[train_start:train_end]
        X_test = X.iloc[test_start:test_end]
        y_test = y.iloc[test_start:test_end]

        # Fit and predict
        model = model_fn(X_train, y_train)
        y_pred = model.predict(X_test)

        # Store results
        fold_result = {
            "fold_start": X_test.index[0],
            "fold_end": X_test.index[-1],
            "mse": mean_squared_error(y_test, y_pred),
            "r2": r2_score(y_test, y_pred),
            "n_train": len(X_train),
            "n_test": len(X_test),
        }
        results.append(fold_result)

    return pd.DataFrame(results)

# Example usage
from sklearn.linear_model import Ridge

def ridge_model(X_train, y_train):
    model = Ridge(alpha=1.0)
    model.fit(X_train, y_train)
    return model

# Using features from Section 1
X = features.drop(columns=["target"])
y = features["target"]

cv_results = walk_forward_cv(X, y, ridge_model, expanding=True)
print(f"Mean OOS R^2: {cv_results['r2'].mean():.6f}")
print(f"Std  OOS R^2: {cv_results['r2'].std():.6f}")
print(f"Fraction positive R^2: {(cv_results['r2'] > 0).mean():.2%}")

3. The Overfitting Epidemic in Financial ML

3.1 Why Finance Is Uniquely Prone to Overfitting

The overfitting problem in finance is far more severe than in other ML domains. Here is why:

3.2 The Signal-to-Noise Problem Quantified

For financial returns:

• R² = 1%: SNR = 0.0101 (signal is ~1% of noise)
• R² = 0.1%: SNR = 0.001 (one part signal per thousand parts noise)
• R² = 5%: SNR = 0.053 (this would be exceptional performance)

4. Random Forests and Gradient Boosting

4.1 Why Tree-Based Models for Finance?

Tree-based ensemble methods have several properties that make them well-suited for financial data:

• Non-linear interactions: Financial relationships are often non-linear (e.g., volatility regimes).
• Feature selection built in: Trees naturally ignore irrelevant features.
• Robust to outliers: Splits are based on ranks, not magnitudes.
• No scaling required: Unlike linear models, trees don’t need standardized inputs.

4.2 Random Forest vs Gradient Boosting

4.3 Critical Hyperparameters for Financial Data

In finance, you must tune aggressively toward simplicity:

• max_depth: Use 3–5. Deep trees memorize noise.
• n_estimators: Use early stopping on a temporal validation set.
• learning_rate (boosting): Low values (0.01–0.05) with many rounds.
• min_samples_leaf: High values (50+) prevent fitting to small pockets of noise.
• max_features: Use sqrt(n_features) for random forests (the default is correct).
• subsample: 0.7–0.8 reduces overfitting in boosting.

Pythonfrom sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import Ridge, Lasso
import lightgbm as lgb

# Define model factories (conservative hyperparameters)
def make_ridge(X_train, y_train):
    m = Ridge(alpha=10.0)
    m.fit(X_train, y_train)
    return m

def make_rf(X_train, y_train):
    m = RandomForestRegressor(
        n_estimators=200,
        max_depth=4,
        min_samples_leaf=50,
        max_features="sqrt",
        random_state=42,
    )
    m.fit(X_train, y_train)
    return m

def make_lgbm(X_train, y_train):
    m = lgb.LGBMRegressor(
        n_estimators=300,
        max_depth=3,
        learning_rate=0.02,
        min_child_samples=50,
        subsample=0.8,
        colsample_bytree=0.8,
        verbose=-1,
    )
    m.fit(X_train, y_train)
    return m

# Compare all models using walk-forward CV
models = {
    "Ridge": make_ridge,
    "RandomForest": make_rf,
    "LightGBM": make_lgbm,
}

comparison = {}
for name, model_fn in models.items():
    cv = walk_forward_cv(X, y, model_fn, expanding=True)
    comparison[name] = {
        "Mean R2": cv["r2"].mean(),
        "Std R2": cv["r2"].std(),
        "% Positive R2": (cv["r2"] > 0).mean(),
        "Mean MSE": cv["mse"].mean(),
    }
    print(f"{name}: R2 = {cv['r2'].mean():.6f} +/- {cv['r2'].std():.6f}")

comparison_df = pd.DataFrame(comparison).T
print("\nModel Comparison:")
print(comparison_df)

5. Feature Importance vs Feature Selection

5.1 Methods for Assessing Feature Importance

Pythonfrom sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

# Fit a model on the full training period
split_idx = int(len(X) * 0.8)
X_train, X_test = X.iloc[:split_idx], X.iloc[split_idx:]
y_train, y_test = y.iloc[:split_idx], y.iloc[split_idx:]

rf = RandomForestRegressor(
    n_estimators=200, max_depth=4,
    min_samples_leaf=50, random_state=42
)
rf.fit(X_train, y_train)

# Method 1: Impurity-based importance
mdi_importance = pd.Series(rf.feature_importances_, index=X.columns)
print("Top 10 features (MDI):")
print(mdi_importance.nlargest(10))

# Method 2: Permutation importance (on test set!)
perm_imp = permutation_importance(rf, X_test, y_test, n_repeats=30,
                                   random_state=42)
perm_importance = pd.Series(perm_imp.importances_mean, index=X.columns)
print("\nTop 10 features (Permutation):")
print(perm_importance.nlargest(10))

# Compare the two methods
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 6))
mdi_importance.nlargest(15).plot.barh(ax=ax1, title="MDI Importance")
perm_importance.nlargest(15).plot.barh(ax=ax2, title="Permutation Importance")
plt.tight_layout()
plt.show()

6. The Multiple Testing Problem

6.1 The Danger of Strategy Mining

Suppose you test 1,000 trading strategies and find that 50 have statistically significant out-of-sample Sharpe ratios (p < 0.05). How many are true discoveries? If none of the strategies have real alpha, you would expect 1000 × 0.05 = 50 false positives. The significant strategies could all be false discoveries.

6.2 Correction Methods

Pythonfrom scipy import stats
from statsmodels.stats.multitest import multipletests

# Simulate the multiple testing problem
np.random.seed(42)
n_strategies = 1000
n_true_alpha = 10  # Only 10 strategies have real alpha
n_days = 252

# Generate strategy returns
sharpe_true = 0.8   # True Sharpe for real strategies
strategy_returns = np.random.randn(n_days, n_strategies) * 0.01

# Add genuine alpha to first 10 strategies
daily_alpha = sharpe_true / np.sqrt(252) * 0.01
strategy_returns[:, :n_true_alpha] += daily_alpha

# Compute t-statistics for each strategy
t_stats = np.mean(strategy_returns, axis=0) / (np.std(strategy_returns, axis=0) / np.sqrt(n_days))
p_values = 2 * (1 - stats.t.cdf(np.abs(t_stats), df=n_days-1))

# Without correction
naive_discoveries = (p_values < 0.05).sum()
naive_true = (p_values[:n_true_alpha] < 0.05).sum()
print(f"Naive (no correction): {naive_discoveries} discoveries, {naive_true} true")

# Bonferroni correction
bonf_reject, bonf_pval, _, _ = multipletests(p_values, method="bonferroni")
print(f"Bonferroni: {bonf_reject.sum()} discoveries, "
      f"{bonf_reject[:n_true_alpha].sum()} true")

# Benjamini-Hochberg (FDR control)
bh_reject, bh_pval, _, _ = multipletests(p_values, method="fdr_bh")
print(f"BH (FDR): {bh_reject.sum()} discoveries, "
      f"{bh_reject[:n_true_alpha].sum()} true")

7. Non-Stationarity: The Fundamental Enemy

7.1 Why Models Decay

Even a well-validated ML model will degrade over time because the data-generating process in finance is non-stationary. The relationships between features and returns change as:

• Market microstructure evolves (algorithmic trading, regulation).
• Macroeconomic regimes shift (rate hike cycles, QE, recessions).
• Profitable patterns get arbitraged away by other market participants.
• New instruments, new correlations, new sources of risk emerge.

7.2 Strategies for Handling Non-Stationarity

8. Why Deep Learning Often Underperforms in Finance

8.1 The Counterintuitive Reality

Despite remarkable success in computer vision and NLP, deep learning often underperforms simpler models like Ridge regression or random forests on tabular financial data. The reasons are structural:

1. Low SNR: Neural networks have immense capacity to fit noise. With 99%+ noise, they will.
2. Limited data: 20 years of daily data is ~5,000 observations. Neural networks need orders of magnitude more.
3. Non-stationarity: Deep learning assumes the training distribution matches the test distribution. In finance, it does not.
4. Tabular data: Neural networks lack the inductive biases (spatial locality for CNNs, sequential structure for RNNs) that make them powerful on images and text. Financial features are tabular, where tree-based models dominate.
5. Hyperparameter sensitivity: The architecture and training choices dwarf the signal in the data.

8.2 Where Deep Learning Does Help

Deep learning can add value in finance for specific applications where its strengths align with the data structure:

• NLP on financial text: Sentiment analysis of earnings calls, news, SEC filings. Transformer models excel here because the data is genuinely textual.
• High-frequency data: Limit order book modeling, where the data is truly massive (millions of events per day) and has spatial structure.
• Generative models: Simulating realistic market scenarios for stress testing (GANs, VAEs).
• Reinforcement learning: Optimal execution (minimizing market impact of large trades).

9. Comprehensive Model Comparison

Pythonimport numpy as np
import pandas as pd
from sklearn.linear_model import Ridge, Lasso
from sklearn.ensemble import RandomForestRegressor
import lightgbm as lgb

def full_model_comparison(X, y):
    """Compare multiple models on financial returns prediction."""

    model_configs = {
        "Ridge (alpha=1)": lambda Xt, yt: Ridge(1).fit(Xt, yt),
        "Ridge (alpha=100)": lambda Xt, yt: Ridge(100).fit(Xt, yt),
        "Lasso (alpha=0.001)": lambda Xt, yt: Lasso(0.001).fit(Xt, yt),
        "RF (depth=3)": lambda Xt, yt: RandomForestRegressor(
            n_estimators=100, max_depth=3,
            min_samples_leaf=50, random_state=42
        ).fit(Xt, yt),
        "RF (depth=6)": lambda Xt, yt: RandomForestRegressor(
            n_estimators=100, max_depth=6,
            min_samples_leaf=20, random_state=42
        ).fit(Xt, yt),
        "LGBM (conservative)": lambda Xt, yt: lgb.LGBMRegressor(
            n_estimators=200, max_depth=3, learning_rate=0.02,
            min_child_samples=50, verbose=-1
        ).fit(Xt, yt),
        "LGBM (aggressive)": lambda Xt, yt: lgb.LGBMRegressor(
            n_estimators=500, max_depth=8, learning_rate=0.05,
            min_child_samples=10, verbose=-1
        ).fit(Xt, yt),
    }

    results = {}
    for name, model_fn in model_configs.items():
        cv = walk_forward_cv(X, y, model_fn, expanding=True)
        results[name] = {
            "OOS R2 (mean)": cv["r2"].mean(),
            "OOS R2 (std)": cv["r2"].std(),
            "OOS R2 (median)": cv["r2"].median(),
            "% Folds R2>0": (cv["r2"] > 0).mean() * 100,
        }

    results_df = pd.DataFrame(results).T.round(6)
    print(results_df.to_string())
    return results_df

comparison = full_model_comparison(X, y)

10. Chapter Summary

Financial ML is statistics with extreme constraints. Here are the essential mappings: