Phase 4 - Lesson 4.7

The Singular Value Decomposition and Applications (PCA, Covariance)

The factorization every matrix admits, and how it powers principal component analysis of asset returns.

⏱ 60 min● Advanced🔗 Prereqs: 4.6

Builds on: Spectral theorem and PSD covariance from 4.6.
Leads to: Low-rank approximation and PCA recur in statistical learning (Phase 11) and risk modeling (Phase 14).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

State the singular value decomposition and relate singular values to eigenvalues of \(A^\top A\).
Interpret the SVD geometrically as rotate–stretch–rotate.
Derive PCA as the eigendecomposition of the covariance matrix (or SVD of centered data).
Implement power iteration to extract the top eigenvector/principal component.

Key Vocabulary

Singular value decomposition: \(A=U\Sigma V^\top\) with orthogonal \(U,V\) and nonnegative diagonal \(\Sigma\).

Singular value: A diagonal entry \(\sigma_i\ge0\) of \(\Sigma\); \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\).

Principal component: An eigenvector of the covariance matrix; a direction of maximal variance in the data.

Explained variance: The eigenvalue \(\lambda_i\) of \(\Sigma\), the variance captured by component \(i\).

Low-rank approximation: The best rank-\(k\) fit \(A_k=\sum_{i\le k}\sigma_i u_i v_i^\top\) (Eckart–Young).

Power iteration: Repeated multiplication \(x\leftarrow Ax/\lVert Ax\rVert\) converging to the dominant eigenvector.

Intuition & Motivation

Intuition

The spectral theorem is beautiful but only for symmetric matrices. The SVD extends the same idea to every matrix, even non-square ones: any linear map is a rotation, then an axis-aligned stretch by the singular values, then another rotation. The largest singular value is the maximum stretch the matrix can apply. For data, this is PCA: the directions of largest stretch of the centered data matrix are the directions of largest variance - the principal components. In finance those are the dominant risk factors driving a basket of returns, and the singular values squared tell you how much variance each explains.

The singular value decomposition

Theorem - SVD

Every real \(m\times n\) matrix factors as \(A=U\Sigma V^\top\) with \(U\) (\(m\times m\)) and \(V\) (\(n\times n\)) orthogonal, and \(\Sigma\) diagonal with entries \(\sigma_1\ge\sigma_2\ge\dots\ge0\). The columns of \(V\) (right singular vectors) are eigenvectors of \(A^\top A\); the columns of \(U\) (left singular vectors) are eigenvectors of \(AA^\top\); and \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\).

Geometrically \(A\) maps the unit sphere to an ellipsoid whose axes are the \(u_i\) with lengths \(\sigma_i\). The rank of \(A\) is the number of nonzero singular values, and \(\sigma_1=\max_{\lVert x\rVert=1}\lVert Ax\rVert\) is the operator (spectral) norm.

Best low-rank approximation

Truncating the SVD gives the optimal compression:

\[A_k=\sum_{i=1}^{k}\sigma_i\,u_i v_i^\top\ =\ \arg\min_{\operatorname{rank}(B)\le k}\lVert A-B\rVert_F,\] (4.3)

the Eckart–Young theorem. Keeping the top \(k\) singular triplets captures the most structure per unit of storage - the mathematics behind image compression, latent factor models, and denoising.

PCA from the covariance matrix

Let the rows of a centered data matrix \(X\) (\(N\) observations, \(d\) features) be demeaned returns. The sample covariance is \(\Sigma=\tfrac1{N-1}X^\top X\), symmetric PSD. Its spectral decomposition \(\Sigma=Q\Lambda Q^\top\) defines PCA:

The eigenvectors \(q_i\) (columns of \(Q\)) are the principal components - orthogonal directions of decreasing variance.
The eigenvalues \(\lambda_i\) are the variances along them; \(\lambda_i/\sum_j\lambda_j\) is the fraction of variance explained.
Projecting data onto the top few components \(Z=XQ_k\) gives a low-dimensional summary retaining most variance.

Equivalently, PCA is the SVD of the centered data: \(X=U\Sigma V^\top\) gives principal components \(V\) and variances \(\sigma_i^2/(N-1)\) - numerically preferable to forming \(X^\top X\) (Phase 5).

Worked Example - PCA of a two-asset return covariance

Take \(\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}\) with eigenpairs \(\lambda_1=3,q_1=\tfrac1{\sqrt2}(1,1)\) and \(\lambda_2=1,q_2=\tfrac1{\sqrt2}(1,-1)\).

First PC is the common-mode \((1,1)/\sqrt2\) explaining \(3/(3+1)=75\%\) of total variance.

Second PC is the spread \((1,-1)/\sqrt2\) explaining the remaining \(25\%\).

A one-factor risk model would keep only the first PC, approximating both assets as driven by a single market factor - the empirical starting point of factor investing.

Interactive: power iteration for the top component

Common Mistakes to Avoid

Confusing singular values with eigenvalues of \(A\); singular values are \(\sqrt{\lambda(A^\top A)}\) and are always nonnegative.
Running PCA on un-centered (or un-scaled) data - PCA needs demeaned columns, and standardization matters when units differ.
Interpreting the largest principal component as ‘the most important asset’; it is a direction (a portfolio), not a single asset.
Forgetting power iteration finds only the dominant eigenvector; deflation or a full solver is needed for the rest.

Quant Practitioner Tips

Prefer the SVD of centered data over eigendecomposing \(X^\top X\) - it avoids squaring the condition number.
The scree plot of \(\lambda_i\) (or \(\sigma_i^2\)) shows how many factors to keep; a few usually explain most variance in returns.
Power iteration converges at rate \((\lambda_2/\lambda_1)^k\) - fast when the top eigenvalue is well-separated.

Knowledge Check

Q1 Easy

The singular values of \(A\) are:

The eigenvalues of \(A\)

The square roots of the eigenvalues of \(A^\top A\)

Always equal to 1

The diagonal entries of \(A\)

Q2 Medium

In PCA, the fraction of variance explained by the i-th principal component is:

\(\sigma_i\)

\(\lambda_i/\textstyle\sum_j\lambda_j\)

\(1/d\)

\(\lambda_i^2\)

Q3 Hard

The best rank-\(k\) approximation of \(A\) in Frobenius norm is:

Its first k rows

\(\sum_{i\le k}\sigma_i u_i v_i^\top\)

The diagonal of \(A\)

Any k columns of \(A\)

Practical Exercise

A centered return matrix has sample covariance \(\Sigma=\begin{bmatrix}4&2\\2&4\end{bmatrix}\). (a) Find the principal components and their explained-variance fractions. (b) If you keep only the first PC, what fraction of total variance is retained, and what is the interpretation?

▶ Show full solution

(a) Eigenvalues from \(\lambda^2-8\lambda+12=0\): \(\lambda=6,2\) (trace 8, det 12 check out). Eigenvectors \(q_1=(1,1)/\sqrt2\) for \(\lambda_1=6\) and \(q_2=(1,-1)/\sqrt2\) for \(\lambda_2=2\). Explained fractions \(6/8=75\%\) and \(2/8=25\%\).

(b) Keeping PC1 retains \(75\%\) of total variance. Interpretation: a single common factor - the equally-weighted combination \((1,1)/\sqrt2\) - drives three-quarters of the joint variability of the two assets; the remaining \(25\%\) is the idiosyncratic spread between them. This is the one-factor reduction underlying factor risk models.

After the reveal, answer for yourself: Because the two variances are equal and the correlation is positive, the principal axes are forced to 45° - symmetry pins the components.

Lesson Summary

The SVD \(A=U\Sigma V^\top\) factors every matrix into rotate–stretch–rotate, with singular values \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\) and optimal low-rank truncation (Eckart–Young). PCA is the spectral decomposition of the covariance (or SVD of centered data): eigenvectors are principal components, eigenvalues are the variances they explain, and power iteration extracts the dominant one - the leading risk factor of a return basket.

Formula Sheet Additions

SVD & PCA

\[A=U\Sigma V^\top,\quad \Sigma_{\text{cov}}=Q\Lambda Q^\top,\ \text{var frac}=\lambda_i/\textstyle\sum_j\lambda_j\]

Singular values, principal components, and explained variance.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: How are singular values related to eigenvalues?
A: The singular values of A are the nonnegative square roots of the eigenvalues of A^T A (equivalently AA^T).

Q: How is PCA obtained from a covariance matrix?
A: Eigendecompose the (centered-data) covariance Σ=QΛQ^T: eigenvectors are principal components, eigenvalues are variances, and λ_i/Σλ_j is the explained-variance fraction.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

Numerical Linear Algebra (Trefethen & Bau, SIAM, 1997) foundational - Lec. 4-5 - Lec. 4-5: the singular value decomposition, geometry, and low-rank approximation.
The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - Ch. 14 - Ch. 14: principal components analysis and its link to the SVD of the data matrix.

← 4.6 Next: 5.1 →