Phase 11 - Lesson 11.7

Unsupervised Learning, Dimensionality Reduction, and Financial-Data Pitfalls

PCA and clustering as structure-finders - and the research-integrity essentials: data leakage, look-ahead bias, and backtest overfitting.

⏱ 60 min● Advanced🔗 Prereqs: 11.1-11.6, 4.x SVD

↖ Phase 11 hub

Builds on: PCA rests on the SVD (Phase 4/5); every earlier lesson's validation warnings culminate here.
Leads to: These pitfalls govern every strategy backtest in Phases 14-18; leakage recurs in time-series (12.6).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Derive principal components as the eigenvectors of the covariance / SVD of the data.
Explain k-means clustering, its objective, and its non-convex local-minimum behavior.
Define data leakage and give the exact preprocessing pattern that prevents it.
Distinguish look-ahead bias, survivorship bias, and backtest overfitting, with financial examples.
Quantify how multiple testing inflates a backtest's apparent Sharpe ratio.

Key Vocabulary

PCA: Principal Component Analysis: orthogonal directions of maximal variance, the eigenvectors of the covariance matrix.

Explained variance ratio: The share \(\lambda_j/\sum_k\lambda_k\) of total variance captured by component \(j\).

k-means: Partition points into \(K\) clusters minimizing within-cluster squared distance to centroids; non-convex, local optima.

Data leakage: Information from outside the training fold (or from the future) contaminating the fit, inflating measured performance.

Look-ahead bias: Using data not available at decision time - a special, time-flavored leakage.

Survivorship bias: Studying only assets that survived, silently dropping the delisted/defaulted ones and overstating returns.

Backtest overfitting: Selecting a strategy by searching many variants, so the winner's performance is largely luck (selection bias).

Intuition & Motivation

Intuition

Unsupervised learning looks for structure with no labels to chase. PCA asks: along which few directions does the data vary most? It rotates to those axes and lets you keep a handful, compressing a correlated cloud into its essential shape - the backbone of factor models and risk decomposition. Clustering asks: which points group together? Both are exploratory and both are seductive: they will happily find ‘structure’ in pure noise.

Which is why this lesson's real subject is research integrity. Financial data punish carelessness. If you scale or select features using the whole dataset before splitting, the test set has already whispered to the model - leakage. If a rule uses tomorrow's information today - look-ahead bias. If you study only the funds that survived - survivorship. And if you try a thousand strategies and report the best - backtest overfitting. Each produces a beautiful backtest and a losing live account. Knowing the mathematics of these traps is the difference between a quant and a curve-fitter.

PCA: directions of maximal variance

Center the \(n\times p\) data \(X\) (subtract column means). The first principal direction maximizes the projected variance:

\[w_1=\argmax_{\|w\|=1} \ w^\top \hat\Sigma\, w,\qquad \hat\Sigma=\tfrac1{n-1}X^\top X,\] (11.16)

whose solution is the top eigenvector of \(\hat\Sigma\) (eigenvalue = the variance along it). Subsequent components are the remaining eigenvectors, orthogonal and ordered by variance. Equivalently, from the SVD \(X=UDV^\top\), the columns of \(V\) are the principal directions and \(d_j^2/(n-1)\) the variances. Keeping the top \(r\) gives the best rank-\(r\) approximation (Eckart-Young). In finance the leading components of a return covariance are recognizable market and sector factors.

Worked Example - Reading a scree plot

Compute eigenvalues \(\lambda_1\ge\dots\ge\lambda_p\) of \(\hat\Sigma\).

The explained-variance ratio of component \(j\) is \(\lambda_j/\sum_k\lambda_k\).

Plot ratios vs \(j\) (the scree plot); keep components before the ‘elbow’ where added variance flattens.

Caveat: on returns, the first component is almost always ‘the market’; PCA axes are statistical, not economically labelled, and rotate as the sample window moves - a non-stationarity warning.

k-means clustering

k-means minimizes total within-cluster squared distance \(\sum_{k}\sum_{i\in C_k}\|x_i-\mu_k\|^2\) by alternating assign (each point to its nearest centroid) and update (each centroid to its cluster mean). The objective is non-convex, so the result depends on initialization - restart many times and keep the best, or use k-means++ seeding. You must choose \(K\) (elbow/silhouette), and the method presumes roughly spherical, equal-size clusters - often false for financial data.

The pitfalls: where quant research dies

Now the core of the lesson. Each trap below turns an honest question into a self-deception.

Definition - Data leakage

Any flow of information from the validation/test data (or from the future) into model fitting. The cardinal sin is preprocessing - scaling, imputation, feature selection, PCA - fit on the whole dataset before the train/test split. The fix is a strict pipeline: fit every transform on the training fold only, then apply it to the test fold.

Look-ahead bias

Using data that was not yet available at the moment of the decision. Examples: trading on a closing price with a signal that used that same close; using restated/as-of fundamentals instead of point-in-time data; aligning an economic release to its reference date rather than its (later) publication date. Always ask: would I truly have known this then?

Survivorship bias

Building a universe from today's surviving names silently deletes the delisted, merged, and defaulted ones - exactly the losers. Mean returns, factor premia, and drawdowns are all flattered. Use a point-in-time universe that includes assets as they existed historically.

Backtest overfitting, quantified

Suppose you try \(N\) independent strategies, each with a true Sharpe of 0 and an estimated Sharpe that is noisy with standard error \(\approx 1/\sqrt{T}\) (in the same annualized units, over \(T\) years). The maximum of \(N\) such noise draws grows roughly like

\[\E\big[\max_{1\le i\le N}\widehat{SR}_i\big]\ \approx\ \frac{\sqrt{2\log N}}{\sqrt{T}},\] (11.17)

so with \(N=1000\) trials over \(T=10\) years the best purely-random strategy shows an apparent Sharpe near \(\sqrt{2\log 1000}/\sqrt{10}\approx 1.2\) - entirely spurious. This is why a headline Sharpe is meaningless without the trial count; the deflated Sharpe ratio corrects for \(N\), and honest research pre-registers hypotheses and reserves an untouched out-of-time set.

Key Idea

The order of operations is the whole game: split first, then fit every transform on the training data only. Reverse that order - even for something as innocent as standardizing - and your test error is a fiction.

Interactive: measure the leakage from scaling before the split

Common Mistakes to Avoid

Standardizing, imputing, or running PCA on the full dataset before splitting - the most common and most invisible leak.
Selecting features by their correlation with the target on all data, then ‘validating’ on a subset of the same data.
Backtesting on a survivorship-biased universe (today's index members) and reporting the inflated premium.
Aligning data to reference dates rather than publication dates, trading on information you could not have had.
Reporting the best of many strategies with no trial-count adjustment - the winner is mostly luck (eq. 11.17).

Quant Practitioner Tips

Wrap all preprocessing in a pipeline that is fit inside each CV fold; never touch test statistics during fitting.
Use point-in-time datasets (as-reported, delisting-inclusive) to kill survivorship and look-ahead bias at the source.
For overlapping labels, purge training samples near the test window and add an embargo gap (Lopez de Prado).
Track your trial count and deflate the Sharpe; pre-register the hypothesis and keep one out-of-time set truly untouched.
Treat PCA factors as unlabeled and unstable - re-estimate on rolling windows and expect the axes to drift (non-stationarity).

Knowledge Check

Q1 Medium

You standardize all features using the mean and standard deviation of the entire dataset, then split into train/test. This causes:

No problem if you use the same scaler

Data leakage: test statistics influenced the training pipeline

Only a numerical rounding issue

Underfitting

Q2 Medium

Trading a signal that is computed from today's closing price, executed at today's close, exhibits:

Survivorship bias

Look-ahead bias

Selection bias

No bias

Q3 Hard

You test 1000 zero-alpha strategies over 10 years and report the best Sharpe. Its expected value is roughly:

About 1.2, entirely from luck

Exactly the true Sharpe of the best strategy

Negative

Practical Exercise

A colleague reports a cross-sectional equity strategy with in-sample Sharpe 2.5. On inspection: features were standardized on the full sample, the universe is the current index, and 300 feature combinations were screened. (a) Name each specific bias present. (b) Explain the mechanism by which each inflates the Sharpe. (c) Redesign the study to be leakage-free and state how you would report the result honestly.

▶ Show full solution

(a) Leakage (full-sample standardization), survivorship bias (current-index universe), and backtest overfitting / selection bias (300 combinations, best reported).

(b) Full-sample scaling leaks future distributional information into every training fold, tightening fit; the surviving-names universe drops the delisted losers, mechanically lifting mean return and cutting drawdown; screening 300 variants and keeping the max is a maximum over noise, adding roughly \(\sqrt{2\log 300}/\sqrt{T}\) of spurious Sharpe.

(c) Use a point-in-time, delisting-inclusive universe; fit all preprocessing inside a walk-forward CV with purge and embargo; freeze the feature set / hypothesis before touching a final out-of-time hold-out; report the number of configurations tried and a deflated Sharpe, plus performance on the untouched hold-out. If the honest, out-of-time Sharpe is, say, 0.4, that - not 2.5 - is the number to trust.

After the reveal, answer for yourself: The gap between 2.5 and the honest figure is not model error; it is research-process error, and it is the most expensive kind.

Lesson Summary

PCA finds orthogonal directions of maximal variance (eigenvectors of the covariance / SVD) and compresses correlated data; k-means partitions points by within-cluster distance but is non-convex and assumption-laden. The lesson's core is integrity: leakage (preprocessing before splitting), look-ahead bias (future data), survivorship bias (surviving universe), and backtest overfitting (best of many trials, apparent Sharpe \(\approx\sqrt{2\log N}/\sqrt{T}\)). Split first, fit transforms on training data only, use point-in-time data, and report your trial count - or your backtest is fiction.

Formula Sheet Additions

PCA objective

\[w_1=\argmax_{\|w\|=1} w^\top\hat\Sigma w\]

Top eigenvector of the covariance; variance captured = eigenvalue.

Overfit Sharpe bound

\[\E[\max_i \widehat{SR}_i]\approx\sqrt{2\log N}/\sqrt{T}\]

Apparent best-of-\(N\) Sharpe from pure luck; deflate by \(N\).

Error Log Checklist

Did I split BEFORE any scaling, imputation, feature selection, or PCA?
Is every timestamp a publication (available) time, not a reference time?
Does my universe include delisted/defaulted names (no survivorship)?
Did I record the number of configurations tried and deflate the Sharpe?
Is there a final out-of-time set I have never looked at?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: State the PCA objective and its solution.
A: Maximize projected variance \(w^\top\hat\Sigma w\) over unit \(w\); the solution is the top eigenvector of \(\hat\Sigma\) (from the SVD, the top right singular vector).

Q: Give the exact preprocessing rule that prevents leakage.
A: Split first; fit every transform (scaling, imputation, selection, PCA) on the training fold only, then apply it to validation/test.

Q: How large an apparent Sharpe can arise from testing many zero-alpha strategies?
A: About \(\sqrt{2\log N}/\sqrt{T}\); e.g. \(N=1000\), \(T=10\) gives roughly 1.2 - purely spurious, corrected by a deflated Sharpe.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

PCA

Eigenvectors of the covariance / right singular vectors; keep top components by explained-variance ratio; axes drift on returns.

Leakage rule

Split first; fit all preprocessing on train only. Full-sample scaling/selection = leakage.

Backtest overfitting

Best of \(N\) trials has apparent Sharpe \(\approx\sqrt{2\log N}/\sqrt{T}\); report trial count, deflate, keep an untouched hold-out.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 14 - Ch. 14: unsupervised learning - PCA, k-means, and the dangers of finding structure in noise.
Time Series Analysis (James Hamilton, 1994) foundational - Hamilton 18 - Non-stationarity and spurious relationships motivating point-in-time, forward-only validation.

← 11.6 Next: 12.1 →