Phase 4 - Lesson 4.6

The Spectral Theorem, Symmetric and Positive-Definite Matrices, Quadratic Forms

Why symmetric matrices are the best behaved of all: real eigenvalues, orthogonal eigenvectors, and the geometry of covariance.

⏱ 60 min● Advanced🔗 Prereqs: 4.4

↖ Phase 4 hub

Builds on: Diagonalization (4.4) and orthonormal bases (4.5).
Leads to: The SVD (4.7) extends this to non-symmetric matrices; PSD covariance underlies all of portfolio theory.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

State the spectral theorem for real symmetric matrices.
Classify a symmetric matrix as positive-(semi)definite via eigenvalues or pivots.
Diagonalize a quadratic form and read its geometry (principal axes).
Prove that any covariance matrix is symmetric positive-semidefinite.

Key Vocabulary

Symmetric matrix: \(A=A^\top\); the natural class for covariance, Hessians, and Gram matrices.

Spectral theorem: Every real symmetric matrix is \(A=Q\Lambda Q^\top\) with orthonormal \(Q\) and real \(\Lambda\).

Positive definite: Symmetric with \(x^\top Ax\gt 0\) for all \(x\neq0\); equivalently all eigenvalues \(\gt 0\).

Positive semidefinite: Symmetric with \(x^\top Ax\ge0\); eigenvalues \(\ge0\) (covariances live here).

Quadratic form: A function \(q(x)=x^\top Ax\) with symmetric \(A\); its level sets are ellipsoids when \(A\succ0\).

Covariance matrix: \(\Sigma=\E[(X-\mu)(X-\mu)^\top]\); symmetric PSD, encoding all variances and covariances.

Intuition & Motivation

Intuition

Symmetric matrices are the aristocrats of linear algebra: their eigenvalues are always real, and their eigenvectors can always be chosen mutually orthogonal. Geometrically that means a symmetric matrix acts by stretching space along a set of perpendicular axes - no shearing, no rotation-mixing. A quadratic form \(x^\top Ax\) then becomes a sum of squares in those axes, and its sign is decided by the eigenvalues. This is exactly the geometry of a covariance matrix: its eigenvectors are the principal risk directions and its eigenvalues are the variances along them - the foundation of PCA and portfolio risk.

The spectral theorem

Theorem - Spectral theorem (real symmetric)

If \(A=A^\top\) is real, there is an orthonormal basis of eigenvectors: \(A=Q\Lambda Q^\top\) with \(Q^\top Q=I\) and \(\Lambda=\operatorname{diag}(\lambda_1,\dots,\lambda_n)\), all \(\lambda_i\in\R\).

Proof

Two facts do the work. (i) Eigenvalues are real: if \(Av=\lambda v\) then \(\lambda\lVert v\rVert^2=\bar v^\top A v\) equals its own conjugate because \(A\) is real symmetric, forcing \(\lambda\in\R\). (ii) Eigenvectors for distinct eigenvalues are orthogonal: \(\lambda_i\langle v_i,v_j\rangle=\langle Av_i,v_j\rangle=\langle v_i,Av_j\rangle=\lambda_j\langle v_i,v_j\rangle\), so \((\lambda_i-\lambda_j)\langle v_i,v_j\rangle=0\). Repeated eigenvalues are handled by choosing an orthonormal basis within each eigenspace.

Definiteness and quadratic forms

Because \(x^\top A x=x^\top Q\Lambda Q^\top x=\sum_i \lambda_i\,y_i^2\) where \(y=Q^\top x\) are the coordinates in the eigenbasis, the sign of the quadratic form is governed entirely by the eigenvalues:

Eigenvalues	Classification	\(x^\top Ax\)
all \(\gt 0\)	positive definite	\(\gt 0\) for \(x\neq0\)
all \(\ge0\)	positive semidefinite	\(\ge0\)
mixed signs	indefinite	takes both signs
all \(\lt 0\)	negative definite	\(\lt 0\) for \(x\neq0\)

For a positive-definite \(A\) the level set \(x^\top Ax=1\) is an ellipsoid whose principal axes are the eigenvectors of \(A\) and whose semi-axis lengths are \(1/\sqrt{\lambda_i}\).

Application: covariance is symmetric PSD

Proposition - Covariance matrices are PSD

For a random vector \(X\) with mean \(\mu\), the covariance \(\Sigma=\E[(X-\mu)(X-\mu)^\top]\) is symmetric and positive-semidefinite.

Proof

Symmetry is immediate from the outer product. For any fixed vector \(w\), the scalar \(w^\top X\) is a portfolio return, and \(w^\top\Sigma w=\E\big[(w^\top(X-\mu))^2\big]=\Var(w^\top X)\ge0\). A variance is never negative, so \(\Sigma\succeq0\). It is strictly positive-definite unless some portfolio \(w^\top X\) is (almost surely) constant - i.e. unless the assets are perfectly collinear.

Key Idea

Portfolio variance is a quadratic form: \(\Var(w^\top X)=w^\top\Sigma w\ge0\). Positive-semidefiniteness of \(\Sigma\) is the mathematical statement that no portfolio can have negative variance.

Worked Example - Diagonalize a covariance and read the risk axes

Let \(\Sigma=\begin{bmatrix}2&1\\1&2\end{bmatrix}\) (two assets, equal variance, positive covariance). Eigenvalues \(\lambda=3,1\) (from 4.3).

Eigenvectors \(q_1=\tfrac1{\sqrt2}(1,1)\) (\(\lambda=3\)) and \(q_2=\tfrac1{\sqrt2}(1,-1)\) (\(\lambda=1\)), orthonormal as promised.

Interpretation: the common-mode portfolio \((1,1)/\sqrt2\) carries the largest variance \(3\); the spread \((1,-1)/\sqrt2\) carries the smallest, \(1\).

Both eigenvalues are positive, so \(\Sigma\succ0\): every nonzero portfolio has strictly positive variance and the risk ellipse is elongated along the common-mode axis.

With a symmetric matrix (\(b=c\)) the two eigen-directions come out perpendicular - the visual signature of the spectral theorem. The unit circle maps to an ellipse whose axes are those eigenvectors, scaled by the eigenvalues.

Common Mistakes to Avoid

Claiming a non-symmetric matrix has orthogonal eigenvectors - the spectral theorem needs \(A=A^\top\).
Testing definiteness by checking only the diagonal entries; you need all eigenvalues (or all leading principal minors) positive.
Forgetting that a covariance can be singular (PSD not PD) when assets are collinear - then \(\Sigma^{-1}\) fails to exist.
Confusing positive entries with positive definiteness; a matrix can have all positive entries yet be indefinite.

Quant Practitioner Tips

Definiteness = eigenvalue signs; Sylvester’s criterion (all leading principal minors \(\gt 0\)) is a fast hand-check for PD.
The eigenvectors of a covariance matrix are the principal components; the eigenvalues are the variances they explain (Lesson 4.7).
A regularized covariance \(\Sigma+\epsilon I\) is guaranteed PD - the standard fix for a singular sample covariance before inverting.

Knowledge Check

Q1 Easy

The spectral theorem guarantees that a real symmetric matrix has:

Complex eigenvalues

An orthonormal basis of eigenvectors and real eigenvalues

No eigenvectors

Distinct eigenvalues only

Q2 Medium

A symmetric matrix has eigenvalues \(0\) and \(4\). It is:

Positive definite

Positive semidefinite but not definite

Indefinite

Negative definite

Q3 Hard

Why is every covariance matrix positive-semidefinite?

Its entries are positive

\(w^\top\Sigma w=\Var(w^\top X)\ge0\) for all \(w\)

It is diagonal

It is invertible

Practical Exercise

Let \(A=\begin{bmatrix}2&-1\\-1&2\end{bmatrix}\). (a) Show \(A\) is positive definite. (b) Describe the level set \(x^\top Ax=1\) (orientation and axis lengths).

▶ Show full solution

(a) Symmetric with \(\tr A=4\gt 0\) and \(\det A=4-1=3\gt 0\), so both eigenvalues are positive (their sum and product are positive). Explicitly \(\lambda=2\pm1=3,1\). Hence \(A\succ0\). Equivalently the leading minors \(2\gt 0\) and \(3\gt 0\) satisfy Sylvester’s criterion.

(b) In the eigenbasis \(x^\top Ax=3y_1^2+y_2^2=1\) is an ellipse. Principal axes are the eigenvectors \((1,-1)/\sqrt2\) (for \(\lambda=3\)) and \((1,1)/\sqrt2\) (for \(\lambda=1\)). Semi-axis lengths are \(1/\sqrt3\) (short, along the large-eigenvalue direction) and \(1\) (long). The ellipse is tilted 45° to the axes.

After the reveal, answer for yourself: Larger eigenvalue → steeper form → shorter semi-axis. The eigenvector with the largest eigenvalue is the direction of fastest increase.

Lesson Summary

The spectral theorem says every real symmetric matrix diagonalizes orthogonally, \(A=Q\Lambda Q^\top\), with real eigenvalues and perpendicular eigenvectors. Definiteness is read off those eigenvalues, and a quadratic form becomes a weighted sum of squares along principal axes. Covariance matrices are symmetric PSD because \(w^\top\Sigma w=\Var(w^\top X)\ge0\) - the bedrock of portfolio risk and PCA.

Formula Sheet Additions

Spectral decomposition

\[A=Q\Lambda Q^\top,\quad x^\top Ax=\sum_i\lambda_i y_i^2,\ y=Q^\top x\]

Symmetric = orthogonal stretch along principal axes.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: What does the spectral theorem state for real symmetric matrices?
A: A = Q Λ Q^T with Q orthonormal (eigenvectors) and Λ diagonal with real eigenvalues; eigenvectors are orthogonal.

Q: Why is a covariance matrix always positive-semidefinite?
A: Because w^T Σ w = Var(w^T X) ≥ 0 for every weight vector w; a variance cannot be negative.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

Linear Algebra Done Right (Sheldon Axler, 4th ed., 2024) current - Ch. 7 - Ch. 7: self-adjoint operators, the spectral theorem, positive operators.
The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - Ch. 3,14 - Ch. 3 & 14: covariance structure, quadratic forms, and positive-definiteness in statistics.

← 4.5 Next: 4.7 →