Quant Academy Math · Probability · Quant Finance
Phase 11 - Lesson 11.3

Model Assessment: Cross-Validation and Model Selection

Why training error lies, how cross-validation estimates out-of-sample error honestly, and how AIC/BIC and the one-standard-error rule pick complexity.

⏱ 50 min● Intermediate🔗 Prereqs: 11.1, 11.2
↖ Phase 11 hub
Builds on: 11.1 (bias-variance) and 11.2 (in-sample vs out-of-sample) set this up.
Leads to: CV is the tool that chooses \(\lambda\) in ridge/lasso (11.4) and every hyperparameter after.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Training error
Average loss on the same data used to fit; optimistically low, decreasing with complexity.
Test error
Expected loss on independent data from the same distribution; the quantity we actually care about.
Optimism
The gap \(\E[\text{test}]-\E[\text{train}]\); grows with model complexity and with fitting noise.
K-fold CV
Split data into \(K\) folds; fit on \(K-1\), validate on the held-out fold, rotate, average.
LOOCV
Leave-one-out CV: \(K=n\); nearly unbiased but high variance and expensive.
AIC / BIC
Analytic scores balancing fit against parameter count; BIC penalizes complexity more heavily as \(n\) grows.
One-standard-error rule
Choose the simplest model whose CV error is within one standard error of the minimum.

Intuition & Motivation

Intuition
If you grade yourself on the exact problems you studied, you will look brilliant - and learn nothing about how you will do on the real exam. Training error is that self-graded score. Every extra parameter lets the model memorize a little more noise, so training error keeps falling even as the model gets worse at the only thing that matters: new data. The gap between the two is optimism, and it widens with complexity.

Cross-validation is the honest exam. Hide part of the data, fit on the rest, grade on what was hidden, and rotate so every point gets a turn as the test. The averaged score estimates test error without needing a separate hold-out you cannot afford. In finance the catch is subtle: the data are ordered in time and serially dependent, so shuffling them lets the future leak into the past - a beautiful CV curve built on cheating.

Why training error is optimistic

Fit \(\hat f\) by minimizing loss on the training set. The same optimization that reduces error also fits the noise \(\varepsilon\) in those specific points. For squared loss the expected optimism is provably positive and proportional to how strongly fitted values co-move with the responses:

\[\E[\text{err}_{\text{test}}]-\E[\text{err}_{\text{train}}]=\frac{2}{n}\sum_{i=1}^n \Cov(\hat y_i, y_i)=\frac{2}{n}\,\sigma^2\,\text{df},\] (11.6)

where the effective degrees of freedom \(\text{df}=\tfrac1{\sigma^2}\sum_i\Cov(\hat y_i,y_i)\) equals \(p\) for linear regression with \(p\) parameters and \(\tr(H)\) more generally. More flexibility means more optimism.

K-fold cross-validation

Partition the \(n\) rows into \(K\) roughly equal folds. For each fold \(k\), fit on the other \(K-1\) folds and record the loss on fold \(k\). The CV estimate is

\[\text{CV}(\hat f)=\frac1n\sum_{i=1}^n L\big(y_i,\ \hat f^{-\kappa(i)}(x_i)\big),\] (11.7)

where \(\kappa(i)\) is the fold containing point \(i\) and \(\hat f^{-\kappa(i)}\) is the model trained without that fold. Each point is predicted by a model that never saw it.

Worked Example - Choosing K: the bias-variance tradeoff of CV itself
1
Small \(K\) (e.g. 5): each training set is much smaller than \(n\), so CV overestimates test error a bit (pessimistic bias), but the folds are fairly independent so the average is low variance and cheap.
2
Large \(K=n\) (LOOCV): training sets are size \(n-1\), so almost unbiased for the error at full \(n\); but the \(n\) models are nearly identical, their errors are highly correlated, so the average has high variance - and it costs \(n\) fits.
3
Practical default: \(K=5\) or \(10\) trades a little bias for much lower variance and cost.
4
In time series, replace random folds with forward-chaining (train on the past, test on the next block) to respect ordering.

Analytic penalties: Cp, AIC, BIC

When refitting many times is costly, in-sample scores add an explicit complexity penalty to the training loss:

\[C_p=\overline{\text{err}}+\frac{2\,\text{df}}{n}\hat\sigma^2,\qquad \text{AIC}=-2\,\ell+2p,\qquad \text{BIC}=-2\,\ell+p\log n.\] (11.8)

AIC targets predictive accuracy (asymptotically like LOOCV); BIC targets recovering the true model and, because its penalty \(p\log n\) grows with \(n\), selects more parsimonious models and is consistent for model identity. In finance, with weak signals and large \(n\), BIC's heavier penalty guards against spurious factors.

Key Idea
The one-standard-error rule: among models whose CV error is within one standard error of the best, pick the simplest. It deliberately errs toward parsimony because the minimum of a noisy CV curve is itself uncertain - and simpler models generalize and survive regime change better.

Interactive: K-fold CV picks polynomial degree

Common Mistakes to Avoid
  • Selecting features or standardizing using the whole dataset before splitting - the test folds then leak into training (see 11.7).
  • Using random K-fold on time-series data; shuffling lets the future predict the past. Use forward-chaining.
  • Picking the exact CV minimum and ignoring its uncertainty; the one-SE rule exists because that minimum is noisy.
  • Comparing AIC across models fit on different data sets or with different responses - the likelihoods are not comparable.
  • Running dozens of model variants and reporting only the best CV score - that is selection bias / backtest overfitting.
Quant Practitioner Tips
  • Nest your CV: an outer loop estimates performance, an inner loop tunes hyperparameters, so tuning never sees the test fold.
  • For time series always validate forward in time; a purged, embargoed split removes leakage from overlapping labels.
  • Report the CV standard error, not just the mean; use the one-SE rule to prefer the simpler model.
  • Count how many configurations you tried - the more you search, the more the winning score is inflated by luck.

Knowledge Check

Q1 Medium
Training error is a biased estimate of test error because:
It uses too few data points
The same fit that lowers it also fits the noise in the training points
It ignores the intercept
It is computed with the wrong loss
Q2 Medium
Compared with 10-fold CV, leave-one-out CV generally has:
Higher bias, lower variance
Lower bias, higher variance
Lower bias and lower variance
Identical behavior
Q3 Medium
Which practice most directly causes an optimistic, leaked CV score in finance?
Using K=10 instead of K=5
Standardizing features using statistics from the entire dataset before splitting
Using squared-error loss
Reporting the CV standard error

Practical Exercise

You have 2000 daily returns and want to select the lag order \(p\) of a return-predicting regression. (a) Why is random K-fold inappropriate here? (b) Describe a leakage-free validation scheme. (c) You tried \(p=1,\dots,20\) and report the best CV Sharpe. What bias remains and how would you quantify it?

▶ Show full solution

(a) Returns are time-ordered and serially/volatility dependent. Random folds put future observations in the training set used to predict past ones (look-ahead), and overlapping information across nearby days leaks across the split, inflating the score.

(b) Use forward-chaining / walk-forward CV: train on \([1,t]\), test on \([t+1,t+h]\), roll forward. Add a purge (drop training points whose label window overlaps the test) and an embargo (skip a gap after each test block) to kill leakage from autocorrelation.

(c) Selecting the best of 20 configurations is a maximum over noisy estimates - selection bias / backtest overfitting: the reported best CV Sharpe is upward biased. Quantify it by (i) a nested/outer test set never used for selection, (ii) a deflated Sharpe ratio that adjusts for the number of trials, or (iii) a permutation/bootstrap null to see how large a ‘best of 20’ Sharpe arises by chance.

After the reveal, answer for yourself: Every model you compare is a coin you flip; report how many coins you flipped, or the winner will disappoint out of sample.

Lesson Summary

Training error is optimistically low by \(2\sigma^2\text{df}/n\), so honest assessment needs held-out data. K-fold cross-validation estimates test error by predicting each point from a model that never saw it; small K is cheaper and lower variance, LOOCV is nearly unbiased but high variance. AIC/BIC/Cp are analytic alternatives, and the one-standard-error rule favors parsimony. In finance, respect time ordering and count your trials, or leakage and selection bias will fool you.

Formula Sheet Additions

Optimism
\[\E[\text{err}_{\text{test}}]-\E[\text{err}_{\text{train}}]=\tfrac{2}{n}\sigma^2\,\text{df}\]
Grows with effective degrees of freedom; the reason training error lies.
BIC
\[\text{BIC}=-2\ell+p\log n\]
Heavier penalty than AIC; consistent for the true model, favors parsimony.
Error Log Checklist
  • Did I fit every preprocessing step inside the CV loop, not before it?
  • For time series, did I validate forward with purge/embargo instead of shuffling?
  • Did I use the one-SE rule rather than the raw CV minimum?
  • Did I record how many configurations I searched and adjust for it?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: Why does training error understate test error, and by how much for linear regression?
A: Optimism \(=2\sigma^2\text{df}/n\); for linear regression \(\text{df}=p\), so the gap is \(2p\sigma^2/n\).
Q: State the bias-variance tradeoff in choosing K for K-fold CV.
A: Small K: pessimistic bias (smaller training sets) but low variance/cost. LOOCV: nearly unbiased but high variance from correlated near-identical fits.
Q: What does the one-standard-error rule prescribe and why?
A: Pick the simplest model within one SE of the minimum CV error, because that minimum is itself uncertain and parsimony generalizes better.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

K-fold CV
Fit on \(K-1\) folds, test on the held-out fold, rotate, average; each point predicted by a model that never saw it.
AIC vs BIC
AIC penalty \(2p\) (predictive, like LOOCV); BIC penalty \(p\log n\) (consistent, more parsimonious).
Optimism
\(2\sigma^2\text{df}/n\): how much training error undershoots test error; grows with complexity.

Completion Checklist

Confidence / mastery rating
Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

← 11.2Next: 11.4 →