Phase 3 - Lesson 3.4

Unconstrained Optimization and the Second-Order Test

Finding and classifying critical points with the gradient and the Hessian, and why convexity makes it global.

⏱ 50 min● Intermediate🔗 Prereqs: 3.2

Builds on: Gradient and Hessian from 3.2.
Leads to: The Hessian test previews the definiteness ideas of Phase 4 and the convex programs of Phase 10.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Locate critical points by solving \(\nabla f=0\).
Classify critical points using the Hessian’s definiteness (second-order test).
Explain why a convex objective’s stationary point is a global minimum.
Interpret gradient descent as a discrete step toward a critical point.

Key Vocabulary

Critical point: A point where \(\nabla f=0\) (or the gradient fails to exist); a candidate extremum.

Positive definite: A symmetric matrix with \(x^\top Hx\gt 0\) for all \(x\neq 0\); equivalently all eigenvalues positive.

Saddle point: A critical point that is a min in some directions and a max in others; the Hessian is indefinite.

Second-order test: Classifies a critical point by the definiteness of the Hessian there.

Convex function: One whose graph lies below its chords; equivalently \(H\succeq 0\) everywhere for twice-differentiable \(f\).

Global minimizer: A point \(x^*\) with \(f(x^*)\le f(x)\) for all feasible \(x\).

Intuition & Motivation

Intuition

Optimization in several variables has the same skeleton as one variable: set the derivative to zero to find candidates, then use the second derivative to sort minima from maxima. The twist is that ‘second derivative’ is now a matrix, and its sign becomes definiteness. A new creature also appears - the saddle - a point that goes up one way and down another, invisible in one dimension.

The decisive property is convexity. If the Hessian is positive semidefinite everywhere, the bowl has no false bottoms: any point where the gradient vanishes is the global minimum. This single fact is why convex problems are the tractable heart of quantitative optimization.

First-order condition

At an interior extremum every partial must vanish, so \(\nabla f(x^*)=0\). This is necessary, not sufficient: maxima, minima, and saddles all satisfy it. Solving the system \(\nabla f=0\) yields the candidate list.

Second-order test via the Hessian

Theorem - Second-order sufficient conditions

Let \(x^*\) be a critical point of a twice-differentiable \(f\) with Hessian \(H=H_f(x^*)\). If \(H\) is positive definite, \(x^*\) is a strict local minimum; if negative definite, a strict local maximum; if indefinite (mixed-sign eigenvalues), a saddle. If \(H\) is only semidefinite the test is inconclusive.

In two variables there is a compact form. With \(H=\begin{bmatrix}f_{xx}&f_{xy}\\ f_{xy}&f_{yy}\end{bmatrix}\) let \(D=\det H=f_{xx}f_{yy}-f_{xy}^2\). Then:

\(D\gt 0\) and \(f_{xx}\gt 0\) → local minimum;
\(D\gt 0\) and \(f_{xx}\lt 0\) → local maximum;
\(D\lt 0\) → saddle point;
\(D=0\) → inconclusive.

Worked Example - Classify the critical points of \(f=x^3-3x+y^2\)

Gradient: \(\nabla f=(3x^2-3,\ 2y)\). Set to zero: \(y=0\) and \(x^2=1\), giving \((1,0)\) and \((-1,0)\).

Hessian: \(H=\begin{bmatrix}6x&0\\0&2\end{bmatrix}\), so \(D=12x\).

At \((1,0)\): \(D=12\gt 0\) and \(f_{xx}=6\gt 0\) → local minimum.

At \((-1,0)\): \(D=-12\lt 0\) → saddle point. The cubic in \(x\) creates exactly the up/down mix a saddle needs.

Convexity makes it global

Proposition - Convex stationary point is global

If \(f\) is convex (equivalently \(H\succeq 0\) everywhere) and \(\nabla f(x^*)=0\), then \(x^*\) is a global minimizer. If strictly convex, it is the unique one.

This is the reason a quadratic risk objective \(f(w)=\tfrac12 w^\top\Sigma w\) with covariance \(\Sigma\succeq 0\) is so friendly: solving \(\nabla f=\Sigma w=0\) (with constraints, next lesson) lands on the true optimum, no local traps.

Interactive: descend to the minimum

Steepest descent takes steps \(x_{k+1}=x_k-\eta\nabla f(x_k)\). On a well-conditioned bowl it heads almost straight in; raise the condition number and it zig-zags, motivating the preconditioning and Newton methods of Phase 10 (which multiply the gradient by \(H^{-1}\)).

Common Mistakes to Avoid

Declaring a critical point a minimum from \(D\gt 0\) alone - you also need \(f_{xx}\gt 0\) to rule out a maximum.
Treating \(\nabla f=0\) as sufficient; saddles satisfy it too.
Forgetting that \(D=0\) (semidefinite Hessian) is inconclusive; higher-order terms decide.
Assuming a local minimum is global without convexity - non-convex objectives have many basins.

Quant Practitioner Tips

Compute the Hessian once symbolically; for quadratics it is constant and the test is immediate.
Definiteness = eigenvalue signs. In 2-D the \((D,f_{xx})\) shortcut avoids eigenvalues entirely.
If your objective is a sum of a convex quadratic and a convex penalty, it is convex - a stationary point is the global optimum.

Knowledge Check

Q1 Easy

A critical point has Hessian with eigenvalues \(+3\) and \(-2\). It is:

A local minimum

A local maximum

A saddle point

Inconclusive

Q2 Medium

In 2-D with \(D=f_{xx}f_{yy}-f_{xy}^2\gt 0\) and \(f_{xx}\lt 0\), the critical point is a:

Local minimum

Local maximum

Saddle

Inflection

Q3 Medium

For a convex function, a point with \(\nabla f=0\) is:

Only a local minimum

A global minimum

Possibly a saddle

Never an extremum

Practical Exercise

Let \(f(x,y)=x^2+y^2-xy-3x\) (a stylized convex cost). Find all critical points and classify them with the second-order test. Is the minimizer global?

▶ Show full solution

Gradient: \(\nabla f=(2x-y-3,\ 2y-x)\). Set to zero: from the second equation \(x=2y\); substitute into the first: \(4y-y-3=0\Rightarrow y=1\), so \(x=2\). Single critical point \((2,1)\).

\[H=\begin{bmatrix}2&-1\\-1&2\end{bmatrix},\quad D=\det H=4-1=3\gt 0,\ f_{xx}=2\gt 0.\]

So \((2,1)\) is a strict local minimum. Since \(H\) is constant and positive definite (eigenvalues \(1\) and \(3\)), \(f\) is strictly convex, and the minimizer is the unique global minimum, with value \(f(2,1)=4+1-2-6=-3\).

After the reveal, answer for yourself: Because the Hessian was constant PD, we got global optimality for free - the quadratic case.

Lesson Summary

Unconstrained optima are found by solving \(\nabla f=0\) and classified by the Hessian’s definiteness: positive definite → min, negative definite → max, indefinite → saddle. Convexity (\(H\succeq 0\)) upgrades any stationary point to a global minimizer, the foundation of tractable quant optimization.

Formula Sheet Additions

2-D second-order test

\[D=f_{xx}f_{yy}-f_{xy}^2:\ D\gt 0,f_{xx}\gt 0\Rightarrow\text{min};\ D\lt 0\Rightarrow\text{saddle}\]

Definiteness via a determinant shortcut.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: What two conditions classify a 2-D critical point as a local minimum?
A: D=f_xx f_yy - f_xy^2 > 0 and f_xx > 0.

Q: Why does convexity guarantee a global minimizer?
A: A PSD Hessian everywhere means no false bottoms, so any stationary point is the global minimum (unique if strictly convex).

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 3.3 Next: 3.5 →