Phase 4 - Lesson 4.5

Inner-Product Spaces, Orthogonality, and Projections

Geometry for abstract vectors: angles, orthonormal bases, orthogonal projection, and the least-squares solution.

⏱ 55 min● Intermediate🔗 Prereqs: 4.2

Builds on: Linear maps and column spaces from 4.2.
Leads to: Projections power least squares (Phase 11) and orthonormal bases feed the spectral theorem (4.6) and QR (Phase 5).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

State the inner-product axioms and derive the induced norm and Cauchy–Schwarz inequality.
Construct an orthonormal basis via the Gram–Schmidt process.
Compute the orthogonal projection onto a subspace.
Derive the least-squares normal equations and interpret them geometrically.

Key Vocabulary

Inner product: A map \(\langle\cdot,\cdot\rangle\) that is linear, symmetric, and positive-definite; generalizes the dot product.

Orthonormal set: Vectors that are mutually orthogonal and of unit norm: \(\langle q_i,q_j\rangle=\delta_{ij}\).

Orthogonal projection: The closest point \(\operatorname{proj}_U x\) in a subspace \(U\) to a given \(x\).

Gram–Schmidt: An algorithm turning any independent list into an orthonormal basis of the same span.

Orthogonal complement: \(U^\perp=\{v:\langle v,u\rangle=0\ \forall u\in U\}\); \(V=U\oplus U^\perp\).

Least squares: Minimizing \(\lVert Ax-b\rVert^2\) by projecting \(b\) onto the column space of \(A\).

Intuition & Motivation

Intuition

An inner product installs geometry - length, angle, perpendicularity - onto any vector space, even spaces of functions or random variables (where it becomes covariance). The single most useful operation it enables is projection: given a target you cannot reach exactly, find the closest reachable point. That is precisely regression: the fitted values are the projection of the data onto the span of the predictors, and the residual is orthogonal to that span. Gram–Schmidt is the tool that straightens a skewed basis into a clean orthonormal one so projections become trivial coordinate reads.

Inner products, norms, and Cauchy–Schwarz

An inner product \(\langle x,y\rangle\) is linear in each argument, symmetric, and positive-definite (\(\langle x,x\rangle\gt 0\) for \(x\neq0\)). It induces the norm \(\lVert x\rVert=\sqrt{\langle x,x\rangle}\) and the angle via \(\langle x,y\rangle=\lVert x\rVert\lVert y\rVert\cos\theta\). Cauchy–Schwarz \(|\langle x,y\rangle|\le\lVert x\rVert\lVert y\rVert\) follows from expanding \(\lVert x-ty\rVert^2\ge0\).

Orthogonal projection

Theorem - Projection onto a subspace

If \((q_1,\dots,q_k)\) is an orthonormal basis of \(U\), the orthogonal projection of \(x\) is \(\operatorname{proj}_U x=\sum_{i} \langle x,q_i\rangle\,q_i\). It is the unique closest point of \(U\) to \(x\), and the residual \(x-\operatorname{proj}_U x\) lies in \(U^\perp\).

For a general (not-necessarily-orthonormal) basis assembled into a matrix \(A\) with independent columns, the projection onto \(\operatorname{col}(A)\) is \(P=A(A^\top A)^{-1}A^\top\).

Gram–Schmidt

Worked Example - Orthonormalize two vectors

Take \(a_1=(1,1,0),\ a_2=(1,0,1)\). First normalize \(a_1\): \(q_1=\tfrac1{\sqrt2}(1,1,0)\).

Subtract the \(q_1\) component of \(a_2\): \(\langle a_2,q_1\rangle=\tfrac1{\sqrt2}\), so \(v_2=a_2-\tfrac1{\sqrt2}q_1=(\tfrac12,-\tfrac12,1)\).

Normalize: \(\lVert v_2\rVert=\sqrt{\tfrac14+\tfrac14+1}=\sqrt{3/2}\), giving \(q_2=\sqrt{\tfrac23}(\tfrac12,-\tfrac12,1)\).

Now \((q_1,q_2)\) is orthonormal and spans the same plane; projecting onto that plane is just \(\langle x,q_1\rangle q_1+\langle x,q_2\rangle q_2\).

Least squares as projection

The system \(Ax=b\) with more equations than unknowns is usually inconsistent (\(b\) is off the column space). The best we can do is minimize the residual norm \(\lVert Ax-b\rVert\). The minimizer makes the residual orthogonal to the column space, giving the normal equations:

\[A^\top A\,\hat x=A^\top b\ \Longrightarrow\ \hat x=(A^\top A)^{-1}A^\top b,\qquad \hat b=A\hat x=\operatorname{proj}_{\operatorname{col}(A)}b.\] (4.2)

This is ordinary least-squares regression: \(\hat b\) are the fitted values, the projection of the data onto the span of the predictors, and the residual is orthogonal to every predictor.

Interactive: implement Gram–Schmidt

Common Mistakes to Avoid

Forgetting to normalize after orthogonalizing - you then have an orthogonal but not orthonormal set.
Projecting onto a non-orthonormal basis with the simple dot-product formula; that only works when the basis is orthonormal.
Solving \(Ax=b\) directly when it is inconsistent instead of the normal equations \(A^\top A\hat x=A^\top b\).
Assuming the residual is small; least squares only guarantees it is orthogonal to the column space, not tiny.

Quant Practitioner Tips

In statistics the inner product of centered variables is their covariance; orthogonality means zero correlation.
Prefer QR (numerically stable Gram–Schmidt) over forming \(A^\top A\) directly - the latter squares the condition number (Phase 5).
Every regression is a projection: fitted values live in the predictors’ span, residuals are orthogonal to it.

Knowledge Check

Q1 Easy

The orthogonal projection of \(x\) onto \(U\) with orthonormal basis \((q_1,q_2)\) is:

\(x\)

\(\langle x,q_1\rangle q_1+\langle x,q_2\rangle q_2\)

\(q_1+q_2\)

\(\lVert x\rVert(q_1+q_2)\)

Q2 Medium

The least-squares solution of \(Ax=b\) (independent columns) satisfies:

\(Ax=b\)

\(A^\top A\hat x=A^\top b\)

\(A\hat x=0\)

\(\hat x=A^\top b\)

Q3 Medium

After Gram–Schmidt, the residual \(x-\operatorname{proj}_U x\) lies in:

\(U\)

\(U^\perp\)

the zero subspace only

the whole space with no structure

Practical Exercise

Fit a line \(y=\beta_0+\beta_1 t\) to the points \((0,1),(1,1),(2,3)\) by least squares. Set up \(A,b\), solve the normal equations, and report \(\hat\beta\).

▶ Show full solution

Design matrix and target: \(A=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix},\ b=\begin{bmatrix}1\\1\\3\end{bmatrix}\).

\[A^\top A=\begin{bmatrix}3&3\\3&5\end{bmatrix},\qquad A^\top b=\begin{bmatrix}5\\7\end{bmatrix}.\]

Solve \(\begin{bmatrix}3&3\\3&5\end{bmatrix}\hat\beta=\begin{bmatrix}5\\7\end{bmatrix}\). Determinant \(15-9=6\), so \(\hat\beta=\tfrac16\begin{bmatrix}5&-3\\-3&3\end{bmatrix}\begin{bmatrix}5\\7\end{bmatrix}=\tfrac16\begin{bmatrix}4\\6\end{bmatrix}=\begin{bmatrix}2/3\\1\end{bmatrix}\).

So \(\hat\beta_0=2/3,\ \hat\beta_1=1\): the fitted line is \(\hat y=\tfrac23+t\). The fitted values \((2/3,\,5/3,\,8/3)\) are the projection of \(b\) onto \(\operatorname{col}(A)\).

After the reveal, answer for yourself: The residual (1/3,-2/3,1/3) is orthogonal to both columns of A - check the dot products are zero.

Lesson Summary

An inner product gives any space geometry - length, angle, orthogonality - and its central operation is orthogonal projection onto a subspace. Gram–Schmidt manufactures orthonormal bases that make projection a coordinate read, and the normal equations \(A^\top A\hat x=A^\top b\) express least-squares regression as projecting the data onto the predictors’ span.

Formula Sheet Additions

Normal equations / projection

\[\hat x=(A^\top A)^{-1}A^\top b,\quad P=A(A^\top A)^{-1}A^\top\]

Least squares = orthogonal projection onto the column space.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: What geometric fact characterizes the least-squares solution?
A: The residual b - A x-hat is orthogonal to the column space of A, giving the normal equations A^T A x-hat = A^T b.

Q: What does Gram-Schmidt produce and why is it useful?
A: An orthonormal basis of the same span; with it, projections and coordinates reduce to simple inner products (the basis of QR).

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 4.4 Next: 4.6 →