Quant Academy Math · Probability · Quant Finance
Phase 4 - Lesson 4.7

The Singular Value Decomposition and Applications (PCA, Covariance)

The factorization every matrix admits, and how it powers principal component analysis of asset returns.

⏱ 60 min● Advanced🔗 Prereqs: 4.6
↖ Phase 4 hub
Builds on: Spectral theorem and PSD covariance from 4.6.
Leads to: Low-rank approximation and PCA recur in statistical learning (Phase 11) and risk modeling (Phase 14).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Singular value decomposition
\(A=U\Sigma V^\top\) with orthogonal \(U,V\) and nonnegative diagonal \(\Sigma\).
Singular value
A diagonal entry \(\sigma_i\ge0\) of \(\Sigma\); \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\).
Principal component
An eigenvector of the covariance matrix; a direction of maximal variance in the data.
Explained variance
The eigenvalue \(\lambda_i\) of \(\Sigma\), the variance captured by component \(i\).
Low-rank approximation
The best rank-\(k\) fit \(A_k=\sum_{i\le k}\sigma_i u_i v_i^\top\) (Eckart–Young).
Power iteration
Repeated multiplication \(x\leftarrow Ax/\lVert Ax\rVert\) converging to the dominant eigenvector.

Intuition & Motivation

Intuition
The spectral theorem is beautiful but only for symmetric matrices. The SVD extends the same idea to every matrix, even non-square ones: any linear map is a rotation, then an axis-aligned stretch by the singular values, then another rotation. The largest singular value is the maximum stretch the matrix can apply. For data, this is PCA: the directions of largest stretch of the centered data matrix are the directions of largest variance - the principal components. In finance those are the dominant risk factors driving a basket of returns, and the singular values squared tell you how much variance each explains.

The singular value decomposition

Theorem - SVD
Every real \(m\times n\) matrix factors as \(A=U\Sigma V^\top\) with \(U\) (\(m\times m\)) and \(V\) (\(n\times n\)) orthogonal, and \(\Sigma\) diagonal with entries \(\sigma_1\ge\sigma_2\ge\dots\ge0\). The columns of \(V\) (right singular vectors) are eigenvectors of \(A^\top A\); the columns of \(U\) (left singular vectors) are eigenvectors of \(AA^\top\); and \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\).

Geometrically \(A\) maps the unit sphere to an ellipsoid whose axes are the \(u_i\) with lengths \(\sigma_i\). The rank of \(A\) is the number of nonzero singular values, and \(\sigma_1=\max_{\lVert x\rVert=1}\lVert Ax\rVert\) is the operator (spectral) norm.

Best low-rank approximation

Truncating the SVD gives the optimal compression:

\[A_k=\sum_{i=1}^{k}\sigma_i\,u_i v_i^\top\ =\ \arg\min_{\operatorname{rank}(B)\le k}\lVert A-B\rVert_F,\] (4.3)

the Eckart–Young theorem. Keeping the top \(k\) singular triplets captures the most structure per unit of storage - the mathematics behind image compression, latent factor models, and denoising.

PCA from the covariance matrix

Let the rows of a centered data matrix \(X\) (\(N\) observations, \(d\) features) be demeaned returns. The sample covariance is \(\Sigma=\tfrac1{N-1}X^\top X\), symmetric PSD. Its spectral decomposition \(\Sigma=Q\Lambda Q^\top\) defines PCA:

Equivalently, PCA is the SVD of the centered data: \(X=U\Sigma V^\top\) gives principal components \(V\) and variances \(\sigma_i^2/(N-1)\) - numerically preferable to forming \(X^\top X\) (Phase 5).

Worked Example - PCA of a two-asset return covariance
1
Take \(\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}\) with eigenpairs \(\lambda_1=3,q_1=\tfrac1{\sqrt2}(1,1)\) and \(\lambda_2=1,q_2=\tfrac1{\sqrt2}(1,-1)\).
2
First PC is the common-mode \((1,1)/\sqrt2\) explaining \(3/(3+1)=75\%\) of total variance.
3
Second PC is the spread \((1,-1)/\sqrt2\) explaining the remaining \(25\%\).
4
A one-factor risk model would keep only the first PC, approximating both assets as driven by a single market factor - the empirical starting point of factor investing.

Interactive: power iteration for the top component

Common Mistakes to Avoid
  • Confusing singular values with eigenvalues of \(A\); singular values are \(\sqrt{\lambda(A^\top A)}\) and are always nonnegative.
  • Running PCA on un-centered (or un-scaled) data - PCA needs demeaned columns, and standardization matters when units differ.
  • Interpreting the largest principal component as ‘the most important asset’; it is a direction (a portfolio), not a single asset.
  • Forgetting power iteration finds only the dominant eigenvector; deflation or a full solver is needed for the rest.
Quant Practitioner Tips
  • Prefer the SVD of centered data over eigendecomposing \(X^\top X\) - it avoids squaring the condition number.
  • The scree plot of \(\lambda_i\) (or \(\sigma_i^2\)) shows how many factors to keep; a few usually explain most variance in returns.
  • Power iteration converges at rate \((\lambda_2/\lambda_1)^k\) - fast when the top eigenvalue is well-separated.

Knowledge Check

Q1 Easy
The singular values of \(A\) are:
The eigenvalues of \(A\)
The square roots of the eigenvalues of \(A^\top A\)
Always equal to 1
The diagonal entries of \(A\)
Q2 Medium
In PCA, the fraction of variance explained by the i-th principal component is:
\(\sigma_i\)
\(\lambda_i/\textstyle\sum_j\lambda_j\)
\(1/d\)
\(\lambda_i^2\)
Q3 Hard
The best rank-\(k\) approximation of \(A\) in Frobenius norm is:
Its first k rows
\(\sum_{i\le k}\sigma_i u_i v_i^\top\)
The diagonal of \(A\)
Any k columns of \(A\)

Practical Exercise

A centered return matrix has sample covariance \(\Sigma=\begin{bmatrix}4&2\\2&4\end{bmatrix}\). (a) Find the principal components and their explained-variance fractions. (b) If you keep only the first PC, what fraction of total variance is retained, and what is the interpretation?

▶ Show full solution

(a) Eigenvalues from \(\lambda^2-8\lambda+12=0\): \(\lambda=6,2\) (trace 8, det 12 check out). Eigenvectors \(q_1=(1,1)/\sqrt2\) for \(\lambda_1=6\) and \(q_2=(1,-1)/\sqrt2\) for \(\lambda_2=2\). Explained fractions \(6/8=75\%\) and \(2/8=25\%\).

(b) Keeping PC1 retains \(75\%\) of total variance. Interpretation: a single common factor - the equally-weighted combination \((1,1)/\sqrt2\) - drives three-quarters of the joint variability of the two assets; the remaining \(25\%\) is the idiosyncratic spread between them. This is the one-factor reduction underlying factor risk models.

After the reveal, answer for yourself: Because the two variances are equal and the correlation is positive, the principal axes are forced to 45° - symmetry pins the components.

Lesson Summary

The SVD \(A=U\Sigma V^\top\) factors every matrix into rotate–stretch–rotate, with singular values \(\sigma_i=\sqrt{\lambda_i(A^\top A)}\) and optimal low-rank truncation (Eckart–Young). PCA is the spectral decomposition of the covariance (or SVD of centered data): eigenvectors are principal components, eigenvalues are the variances they explain, and power iteration extracts the dominant one - the leading risk factor of a return basket.

Formula Sheet Additions

SVD & PCA
\[A=U\Sigma V^\top,\quad \Sigma_{\text{cov}}=Q\Lambda Q^\top,\ \text{var frac}=\lambda_i/\textstyle\sum_j\lambda_j\]
Singular values, principal components, and explained variance.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: How are singular values related to eigenvalues?
A: The singular values of A are the nonnegative square roots of the eigenvalues of A^T A (equivalently AA^T).
Q: How is PCA obtained from a covariance matrix?
A: Eigendecompose the (centered-data) covariance Σ=QΛQ^T: eigenvectors are principal components, eigenvalues are variances, and λ_i/Σλ_j is the explained-variance fraction.

Completion Checklist

Confidence / mastery rating
Personal notes
← 4.6Next: 5.1 →