Phase 18 - Lesson 18.5

Capstone 5 - Statistical-Learning Research Pipeline

Build a time-aware backtest with regularization, honest transaction costs, and explicit model risk - and refuse to overfit.

⏱ 140 min● Research🔗 Prereqs: Phases 11, 12, 17

↖ Phase 18 hub

Builds on: Integrates Statistical Learning (Phase 11), Time-Series (Phase 12), and Research Engineering (Phase 17).
Leads to: This pipeline’s discipline (time-aware CV, costs, OOS) is the backbone of the final capstone (18.8).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Design a time-aware (walk-forward) validation scheme that prevents look-ahead leakage.
Apply regularization and explain the bias–variance trade-off it controls.
Incorporate realistic transaction costs into a strategy’s net performance.
Quantify model risk via out-of-sample degradation and multiple-testing awareness.
Explain why in-sample performance is not evidence and design an honest test.

Key Vocabulary

Walk-forward validation: Training on a past window and testing on the next, rolling forward - the time-series analogue of cross-validation.

Data leakage: Information from the test period (or future) contaminating training, inflating apparent performance.

Regularization: Penalizing model complexity (e.g. ridge/lasso \(\lambda\)) to reduce variance at the cost of some bias.

Transaction cost: The cost of trading (spread, commission, slippage) that erodes gross returns; often decisive.

Multiple testing: Trying many strategies inflates the chance one looks good by luck; demands a higher bar or correction.

Model risk: The risk that the model is wrong or overfit, so realized performance falls short of the backtest.

Intuition & Motivation

Intuition

A statistical-learning research pipeline is where good intentions go to die. The failure mode is almost always the same: the model looks brilliant in-sample and evaporates live. Four disciplines prevent this: time-aware validation (never let the future leak into the past), regularization (don’t fit noise), transaction costs (a signal that can’t survive costs isn’t a signal), and model-risk awareness (you tried many things; most ‘edges’ are luck).

The honest stance: in-sample performance is not evidence. Any sufficiently flexible model fits the past. The only thing that counts is out-of-sample, net-of-cost performance, evaluated in a way that respects the arrow of time - and even then, discounted for how many strategies you tested. This capstone is as much about intellectual honesty as code.

The brief

Build a walk-forward backtest for a simple predictive signal, with ridge regularization, a realistic per-trade cost, and an out-of-sample evaluation. Deliverable: code, an in-sample-vs-out-of-sample comparison, a cost-sensitivity curve, and a paragraph honestly discussing model risk and multiple testing. No profitability is promised or implied.

Required theory recap

Bias–variance (Phase 11): test error \(=\text{bias}^2+\text{variance}+\text{irreducible}\); regularization trades a little bias for less variance.
Ridge (Phase 11): \(\hat\beta=\arg\min\|y-X\beta\|^2+\lambda\|\beta\|^2\); \(\lambda\) chosen by time-aware CV.
Time-series validation (Phase 12): walk-forward / expanding window; never shuffle time.
Costs & capacity: net return \(=\text{gross}-\text{turnover}\times\text{cost}\); costs scale with trading frequency.

Time-aware validation: the arrow of time is not optional

Standard k-fold cross-validation shuffles rows - catastrophic for time series, because it trains on the future to predict the past. Use walk-forward: fit on \([t_0,t_1)\), test on \([t_1,t_2)\), roll forward. Every prediction uses only data available before it (equation 17.7). Any feature computed with future information (a full-sample mean, a look-ahead label) is leakage and must be purged.

\[\text{train }[t_0,t_1)\ \to\ \text{test }[t_1,t_2)\ \to\ \text{train }[t_0,t_2)\ \to\ \text{test }[t_2,t_3)\ \to\ \cdots\] (18.11)

Regularization and the bias–variance trade-off

With many candidate predictors and noisy financial data, an unregularized fit chases noise (high variance). Ridge shrinks coefficients toward zero, accepting a little bias for a large variance reduction - usually improving out-of-sample error. The penalty \(\lambda\) is itself chosen by walk-forward CV, never by looking at the test set.

Worked Example - Why in-sample R² is not evidence

Fit a flexible model on in-sample data: \(R^2_{\text{in}}=0.30\) - looks great.

Walk-forward out-of-sample: \(R^2_{\text{out}}=0.01\) (or negative). The in-sample fit was mostly overfitting.

The gap \(R^2_{\text{in}}-R^2_{\text{out}}\) is a direct measure of overfitting/model risk.

Now subtract costs: with turnover 200%/yr at 10 bps, net drops further. A signal must survive both the OOS gap and costs.

Honest conclusion: report OOS net performance with its standard error; if it’s within noise of zero, say so.

Transaction costs: the signal killer

Gross returns flatter than net. A per-unit-turnover cost \(c\) (spread + slippage + commission) turns a beautiful gross curve into a flat or negative net one when the strategy trades a lot:

\[r^{\text{net}}_t = r^{\text{gross}}_t - c\cdot \text{turnover}_t,\qquad \text{turnover}_t=\sum_i|w_{i,t}-w_{i,t-1}|.\] (18.12)

Key Idea

Always plot net performance against a range of cost assumptions. A strategy that is only profitable at 0 bps is not a strategy; the cost-sensitivity curve is often more informative than the headline Sharpe.

Model risk and multiple testing

If you test 100 strategies at the 5% level, ~5 will look ‘significant’ by pure luck. Backtest Sharpe ratios are inflated by the search that produced them. Guard with: a pre-registered hypothesis, a higher significance bar, out-of-sample holdout you touch once, and honest reporting of how many variants you tried. This is the deepest form of model risk.

▶ Reference architecture for the pipeline

pipe/features.py -> build_features(prices) # PIT only, no leakage pipe/model.py -> ridge_fit(X,y,lam); predict(...) pipe/cv.py -> walk_forward(splits); choose lam by OOS score pipe/backtest.py -> positions -> gross, net (subtract costs, 18.12) pipe/report.py -> in-vs-out table, cost curve, #variants tested

Features are strictly point-in-time (17.6); \(\lambda\) is chosen on validation folds, never the final test; the report states the multiple-testing context honestly.

Common Mistakes to Avoid

Using k-fold CV that shuffles time - trains on the future, guaranteeing look-ahead leakage.
Standardizing/normalizing features using full-sample statistics (leaks the test distribution).
Tuning \(\lambda\) (or anything) on the test set, then reporting that test score as out-of-sample.
Reporting gross returns and omitting transaction costs - the most common way to overstate an edge.
Ignoring that you tried 50 variants; the best one’s backtest Sharpe is selection-biased upward.
Treating in-sample \(R^2\) or Sharpe as evidence the strategy works.

Quant Practitioner Tips

Validate walk-forward; every number a prediction uses must predate it (17.7).
Fit all preprocessing (means, scalings, \(\lambda\)) on the training fold only; apply to the test fold.
Subtract realistic costs and show a cost-sensitivity curve, not a single-point net Sharpe.
Keep a final holdout you evaluate exactly once; report how many strategies you tested.
Report OOS net performance with a standard error; if it’s within noise of zero, say so plainly.

Interactive: walk-forward validation with costs, no leakage

Implement the walk-forward split and a net-return calculation. The tests check that no split uses future data and that costs reduce returns as turnover rises.

Knowledge Check

Q1 Medium

Why is ordinary shuffled k-fold cross-validation invalid for a trading signal?

It is too slow

It trains on future data to predict the past, leaking look-ahead information

It uses too little data

It cannot compute Sharpe

Q2 Medium

A strategy has a great gross Sharpe but a flat net Sharpe once costs are included. The correct conclusion is:

Report the gross Sharpe; costs are a detail

The edge does not survive realistic costs, so it is not tradeable as-is

Increase leverage

Lower the cost assumption until it works

Q3 Hard

You tested 40 signal variants and report the best one’s backtest Sharpe of 1.5. The honest caveat is:

None needed

Its Sharpe is inflated by multiple testing/selection; the out-of-sample value is likely much lower

It is guaranteed to work live

Multiple testing improves Sharpe

Practical Exercise

You are reviewing a colleague’s backtest claiming a Sharpe of 2.1. Write a due-diligence checklist of at least six questions whose answers would determine whether you believe it, covering leakage, validation, costs, and model risk. For each, state what a bad answer looks like.

▶ Show full solution

A due-diligence checklist (bad answers in parentheses):

Was validation walk-forward or shuffled? (Bad: shuffled k-fold - look-ahead leakage.)
Were features and scalings computed point-in-time, or using full-sample statistics? (Bad: full-sample means/normalization - leakage.)
Were hyperparameters (\(\lambda\), thresholds) tuned on the test set? (Bad: yes - the ‘OOS’ score is contaminated.)
Is the Sharpe gross or net of realistic costs, and what is the turnover? (Bad: gross only, high turnover - likely flat net.)
How many strategies/variants were tried before this one? (Bad: dozens, no correction - selection-biased Sharpe.)
Is the universe survivorship-free and point-in-time? (Bad: today’s index members - survivorship bias, Capstone-adjacent 17.6.)
What is the out-of-sample degradation and the standard error of the Sharpe? (Bad: no OOS holdout, no error bar.)

If several answers are ‘bad’, the 2.1 is almost certainly an artifact. A trustworthy claim survives all of these with net, out-of-sample, leakage-free numbers reported with uncertainty - and still promises nothing about the future.

After the reveal, answer for yourself: Which single question, answered badly, would most completely invalidate the result on its own?

Lesson Summary

A statistical-learning research pipeline earns trust only through discipline: time-aware walk-forward validation that forbids look-ahead leakage, regularization to manage the bias–variance trade-off, realistic transaction costs (net \(=\) gross \(-\,c\cdot\) turnover) with a sensitivity curve, and explicit model-risk and multiple-testing accounting. In-sample performance is not evidence; only leakage-free, net-of-cost, out-of-sample results reported with uncertainty are - and even those promise nothing about the future.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: Why must trading-strategy validation be walk-forward rather than shuffled k-fold?
A: Because shuffling breaks the time order and lets the model train on future data to predict the past - look-ahead leakage that inflates performance. Walk-forward trains on a past window and tests on the next, preserving the arrow of time.

Q: Give the net-return formula and why the cost-sensitivity curve matters.
A: \(r^{\text{net}}=r^{\text{gross}}-c\cdot\text{turnover}\). The curve matters because a strategy profitable only at zero cost is not tradeable; showing net performance across a range of cost assumptions reveals whether the edge is real or an artifact of ignoring costs.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 18.4 Next: 18.6 →