Quant Academy Math · Probability · Quant Finance
Phase 4 - Lesson 4.5

Inner-Product Spaces, Orthogonality, and Projections

Geometry for abstract vectors: angles, orthonormal bases, orthogonal projection, and the least-squares solution.

⏱ 55 min● Intermediate🔗 Prereqs: 4.2
↖ Phase 4 hub
Builds on: Linear maps and column spaces from 4.2.
Leads to: Projections power least squares (Phase 11) and orthonormal bases feed the spectral theorem (4.6) and QR (Phase 5).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Inner product
A map \(\langle\cdot,\cdot\rangle\) that is linear, symmetric, and positive-definite; generalizes the dot product.
Orthonormal set
Vectors that are mutually orthogonal and of unit norm: \(\langle q_i,q_j\rangle=\delta_{ij}\).
Orthogonal projection
The closest point \(\operatorname{proj}_U x\) in a subspace \(U\) to a given \(x\).
Gram–Schmidt
An algorithm turning any independent list into an orthonormal basis of the same span.
Orthogonal complement
\(U^\perp=\{v:\langle v,u\rangle=0\ \forall u\in U\}\); \(V=U\oplus U^\perp\).
Least squares
Minimizing \(\lVert Ax-b\rVert^2\) by projecting \(b\) onto the column space of \(A\).

Intuition & Motivation

Intuition
An inner product installs geometry - length, angle, perpendicularity - onto any vector space, even spaces of functions or random variables (where it becomes covariance). The single most useful operation it enables is projection: given a target you cannot reach exactly, find the closest reachable point. That is precisely regression: the fitted values are the projection of the data onto the span of the predictors, and the residual is orthogonal to that span. Gram–Schmidt is the tool that straightens a skewed basis into a clean orthonormal one so projections become trivial coordinate reads.

Inner products, norms, and Cauchy–Schwarz

An inner product \(\langle x,y\rangle\) is linear in each argument, symmetric, and positive-definite (\(\langle x,x\rangle\gt 0\) for \(x\neq0\)). It induces the norm \(\lVert x\rVert=\sqrt{\langle x,x\rangle}\) and the angle via \(\langle x,y\rangle=\lVert x\rVert\lVert y\rVert\cos\theta\). Cauchy–Schwarz \(|\langle x,y\rangle|\le\lVert x\rVert\lVert y\rVert\) follows from expanding \(\lVert x-ty\rVert^2\ge0\).

Orthogonal projection

Theorem - Projection onto a subspace
If \((q_1,\dots,q_k)\) is an orthonormal basis of \(U\), the orthogonal projection of \(x\) is \(\operatorname{proj}_U x=\sum_{i} \langle x,q_i\rangle\,q_i\). It is the unique closest point of \(U\) to \(x\), and the residual \(x-\operatorname{proj}_U x\) lies in \(U^\perp\).

For a general (not-necessarily-orthonormal) basis assembled into a matrix \(A\) with independent columns, the projection onto \(\operatorname{col}(A)\) is \(P=A(A^\top A)^{-1}A^\top\).

Gram–Schmidt

Worked Example - Orthonormalize two vectors
1
Take \(a_1=(1,1,0),\ a_2=(1,0,1)\). First normalize \(a_1\): \(q_1=\tfrac1{\sqrt2}(1,1,0)\).
2
Subtract the \(q_1\) component of \(a_2\): \(\langle a_2,q_1\rangle=\tfrac1{\sqrt2}\), so \(v_2=a_2-\tfrac1{\sqrt2}q_1=(\tfrac12,-\tfrac12,1)\).
3
Normalize: \(\lVert v_2\rVert=\sqrt{\tfrac14+\tfrac14+1}=\sqrt{3/2}\), giving \(q_2=\sqrt{\tfrac23}(\tfrac12,-\tfrac12,1)\).
4
Now \((q_1,q_2)\) is orthonormal and spans the same plane; projecting onto that plane is just \(\langle x,q_1\rangle q_1+\langle x,q_2\rangle q_2\).

Least squares as projection

The system \(Ax=b\) with more equations than unknowns is usually inconsistent (\(b\) is off the column space). The best we can do is minimize the residual norm \(\lVert Ax-b\rVert\). The minimizer makes the residual orthogonal to the column space, giving the normal equations:

\[A^\top A\,\hat x=A^\top b\ \Longrightarrow\ \hat x=(A^\top A)^{-1}A^\top b,\qquad \hat b=A\hat x=\operatorname{proj}_{\operatorname{col}(A)}b.\] (4.2)

This is ordinary least-squares regression: \(\hat b\) are the fitted values, the projection of the data onto the span of the predictors, and the residual is orthogonal to every predictor.

Interactive: implement Gram–Schmidt

Common Mistakes to Avoid
  • Forgetting to normalize after orthogonalizing - you then have an orthogonal but not orthonormal set.
  • Projecting onto a non-orthonormal basis with the simple dot-product formula; that only works when the basis is orthonormal.
  • Solving \(Ax=b\) directly when it is inconsistent instead of the normal equations \(A^\top A\hat x=A^\top b\).
  • Assuming the residual is small; least squares only guarantees it is orthogonal to the column space, not tiny.
Quant Practitioner Tips
  • In statistics the inner product of centered variables is their covariance; orthogonality means zero correlation.
  • Prefer QR (numerically stable Gram–Schmidt) over forming \(A^\top A\) directly - the latter squares the condition number (Phase 5).
  • Every regression is a projection: fitted values live in the predictors’ span, residuals are orthogonal to it.

Knowledge Check

Q1 Easy
The orthogonal projection of \(x\) onto \(U\) with orthonormal basis \((q_1,q_2)\) is:
\(x\)
\(\langle x,q_1\rangle q_1+\langle x,q_2\rangle q_2\)
\(q_1+q_2\)
\(\lVert x\rVert(q_1+q_2)\)
Q2 Medium
The least-squares solution of \(Ax=b\) (independent columns) satisfies:
\(Ax=b\)
\(A^\top A\hat x=A^\top b\)
\(A\hat x=0\)
\(\hat x=A^\top b\)
Q3 Medium
After Gram–Schmidt, the residual \(x-\operatorname{proj}_U x\) lies in:
\(U\)
\(U^\perp\)
the zero subspace only
the whole space with no structure

Practical Exercise

Fit a line \(y=\beta_0+\beta_1 t\) to the points \((0,1),(1,1),(2,3)\) by least squares. Set up \(A,b\), solve the normal equations, and report \(\hat\beta\).

▶ Show full solution

Design matrix and target: \(A=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix},\ b=\begin{bmatrix}1\\1\\3\end{bmatrix}\).

\[A^\top A=\begin{bmatrix}3&3\\3&5\end{bmatrix},\qquad A^\top b=\begin{bmatrix}5\\7\end{bmatrix}.\]

Solve \(\begin{bmatrix}3&3\\3&5\end{bmatrix}\hat\beta=\begin{bmatrix}5\\7\end{bmatrix}\). Determinant \(15-9=6\), so \(\hat\beta=\tfrac16\begin{bmatrix}5&-3\\-3&3\end{bmatrix}\begin{bmatrix}5\\7\end{bmatrix}=\tfrac16\begin{bmatrix}4\\6\end{bmatrix}=\begin{bmatrix}2/3\\1\end{bmatrix}\).

So \(\hat\beta_0=2/3,\ \hat\beta_1=1\): the fitted line is \(\hat y=\tfrac23+t\). The fitted values \((2/3,\,5/3,\,8/3)\) are the projection of \(b\) onto \(\operatorname{col}(A)\).

After the reveal, answer for yourself: The residual (1/3,-2/3,1/3) is orthogonal to both columns of A - check the dot products are zero.

Lesson Summary

An inner product gives any space geometry - length, angle, orthogonality - and its central operation is orthogonal projection onto a subspace. Gram–Schmidt manufactures orthonormal bases that make projection a coordinate read, and the normal equations \(A^\top A\hat x=A^\top b\) express least-squares regression as projecting the data onto the predictors’ span.

Formula Sheet Additions

Normal equations / projection
\[\hat x=(A^\top A)^{-1}A^\top b,\quad P=A(A^\top A)^{-1}A^\top\]
Least squares = orthogonal projection onto the column space.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: What geometric fact characterizes the least-squares solution?
A: The residual b - A x-hat is orthogonal to the column space of A, giving the normal equations A^T A x-hat = A^T b.
Q: What does Gram-Schmidt produce and why is it useful?
A: An orthonormal basis of the same span; with it, projections and coordinates reduce to simple inner products (the basis of QR).

Completion Checklist

Confidence / mastery rating
Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

← 4.4Next: 4.6 →