Phase 3 - Lesson 3.2

Partial Derivatives, Gradient, Jacobian, and Hessian

The full first- and second-order local picture of a multivariable function, and the objects that drive every optimizer.

⏱ 55 min● Intermediate🔗 Prereqs: 3.1

Builds on: Scalar fields and level sets from 3.1.
Leads to: The gradient and Hessian are the engine of optimization (3.4, 3.5) and gradient descent everywhere later.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Compute partial derivatives and assemble the gradient of a scalar field.
Derive the directional derivative and prove the gradient points in the direction of steepest ascent.
Build the Jacobian of a vector field and the Hessian of a scalar field.
Implement a numerical gradient and verify it against an analytic one.

Key Vocabulary

Partial derivative: The rate \(\partial f/\partial x_i\) with all other variables held fixed.

Gradient: The vector \(\nabla f=(\partial f/\partial x_1,\dots,\partial f/\partial x_n)\) of first partials.

Directional derivative: The rate \(D_u f=\nabla f\cdot u\) of change along a unit vector \(u\).

Jacobian: The \(m\times n\) matrix \(J_{ij}=\partial F_i/\partial x_j\) of a vector field \(F:\R^n\to\R^m\).

Hessian: The symmetric matrix \(H_{ij}=\partial^2 f/\partial x_i\partial x_j\) of second partials.

Total derivative: The best linear approximation \(f(a+h)\approx f(a)+\nabla f(a)\cdot h\); differentiability means this error is \(o(\lVert h\rVert)\).

Intuition & Motivation

Intuition

A partial derivative asks a one-dimensional question inside a many-dimensional world: ‘if I nudge only \(x_i\), how fast does \(f\) move?’ Stacking those answers gives the gradient, an arrow that points straight uphill. Its length is the steepest slope; its direction is where a hiker climbs fastest. Every gradient-based optimizer - and every neural-net trainer - is just walking opposite that arrow.

Second derivatives measure curvature. The Hessian packages all of them into a symmetric matrix whose eigenvalues will later tell us whether a critical point is a minimum, maximum, or saddle.

Partials and the gradient

For \(f(x,y)=x^2 y+\sin y\) the partials are \(\partial_x f=2xy\) and \(\partial_y f=x^2+\cos y\). The gradient collects them:

\[\nabla f(x,y)=\big(\,\partial_x f,\ \partial_y f\,\big)=\big(2xy,\ x^2+\cos y\big).\] (3.2)

Theorem - Steepest ascent

If \(f\) is differentiable at \(a\) with \(\nabla f(a)\neq 0\), then among all unit directions \(u\) the directional derivative \(D_u f(a)=\nabla f(a)\cdot u\) is maximized by \(u=\nabla f(a)/\lVert\nabla f(a)\rVert\), with maximal value \(\lVert\nabla f(a)\rVert\).

Proof

By Cauchy–Schwarz, \(\nabla f\cdot u\le\lVert\nabla f\rVert\lVert u\rVert=\lVert\nabla f\rVert\) with equality iff \(u\) is a positive multiple of \(\nabla f\). Hence the gradient direction is steepest ascent and \(-\nabla f\) is steepest descent. A corollary: \(\nabla f\) is orthogonal to the level set through \(a\), since moving along a level curve keeps \(f\) constant, giving \(\nabla f\cdot u=0\).

Jacobian and Hessian

For a vector field \(F=(F_1,\dots,F_m)\) the Jacobian stacks the gradients of each component as rows; it is the total-derivative matrix. For a scalar field the Hessian is the Jacobian of the gradient:

\[J_F=\begin{bmatrix}\nabla F_1\\ \vdots\\ \nabla F_m\end{bmatrix},\qquad H_f=\nabla(\nabla f)=\big[\partial_i\partial_j f\big].\] (3.3)

When the second partials are continuous, Clairaut’s theorem gives \(\partial_i\partial_j f=\partial_j\partial_i f\), so the Hessian is symmetric - the fact that makes the spectral theorem (Phase 4) applicable.

Worked Example - Gradient and Hessian of a quadratic form

Let \(f(x)=\tfrac12\,x^\top A x\) with \(A\) symmetric. Expand: \(f=\tfrac12\sum_{ij}A_{ij}x_i x_j\).

Partial: \(\partial_k f=\tfrac12\sum_j A_{kj}x_j+\tfrac12\sum_i A_{ik}x_i=(Ax)_k\) using symmetry, so \(\nabla f=Ax\).

Second partial: \(\partial_l\partial_k f=A_{kl}\), hence \(H_f=A\) everywhere.

This is the multivariable analogue of \(\tfrac{d}{dx}\tfrac12 a x^2=ax\) and \(f''=a\): quadratic forms have constant Hessian equal to their matrix.

Interactive: descend a bowl

The workbench walks \(x_{k+1}=x_k-\eta\,\nabla f(x_k)\) on an ill-conditioned bowl. Watch the zig-zag when the level ellipses are elongated (high condition number): steepest descent bounces across the narrow valley.

Interactive: a numerical gradient

Common Mistakes to Avoid

Forgetting to normalize \(u\) before computing \(D_u f=\nabla f\cdot u\); the formula needs a unit vector.
Swapping Jacobian rows and columns - \(J_F\) is \(m\times n\) with component gradients as rows.
Assuming \(\partial_i\partial_j f=\partial_j\partial_i f\) without continuity of the second partials (Clairaut needs it).
Treating the gradient as a slope (a number) rather than a vector; its length is the steepest slope.

Quant Practitioner Tips

Always sanity-check an analytic gradient against a central-difference gradient before trusting an optimizer.
For \(f=\tfrac12 x^\top A x+b^\top x\) memorize \(\nabla f=Ax+b\) and \(H=A\); most quant objectives are locally this.
The gradient is orthogonal to level sets - on a risk contour map it points straight across the ellipses toward higher loss.

Knowledge Check

Q1 Easy

At a point where \(\nabla f=(3,4)\), the maximum rate of increase of \(f\) per unit step is:

Q2 Medium

For \(f(x)=\tfrac12 x^\top A x\) with symmetric \(A\), the Hessian is:

\(2A\)

\(A\)

\(A^\top A\)

the zero matrix

Q3 Medium

The gradient of a scalar field at a point is always:

Tangent to the level set through that point

Orthogonal to the level set through that point

Zero at every point

Parallel to the x-axis

Practical Exercise

Let \(f(x,y)=x^2+xy+2y^2\). (a) Find \(\nabla f\) and the Hessian \(H\). (b) Give the direction of steepest ascent at \((1,1)\) and the maximal directional derivative there.

▶ Show full solution

(a) \(\partial_x f=2x+y,\ \partial_y f=x+4y\), so \(\nabla f=(2x+y,\ x+4y)\). The Hessian is constant:

\[H=\begin{bmatrix}2&1\\1&4\end{bmatrix}.\]

(b) At \((1,1)\), \(\nabla f=(3,5)\). Steepest ascent is the unit vector \((3,5)/\sqrt{34}\), and the maximal directional derivative is \(\lVert(3,5)\rVert=\sqrt{34}\approx 5.83\).

After the reveal, answer for yourself: The Hessian here is symmetric and, as you will verify in Phase 4, positive definite - a strictly convex bowl.

Lesson Summary

Partial derivatives assemble into the gradient (steepest-ascent arrow, orthogonal to level sets), vector fields differentiate to Jacobians, and scalar fields to symmetric Hessians. For quadratics \(\tfrac12 x^\top Ax\) the gradient is \(Ax\) and the Hessian is \(A\) - the template for every local model.

Formula Sheet Additions

Gradient of a quadratic

\[\nabla\big(\tfrac12 x^\top A x+b^\top x\big)=Ax+b,\quad H=A\]

The workhorse identity for optimization.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: Why does the gradient point in the direction of steepest ascent?
A: Because D_u f = grad f · u is maximized (Cauchy-Schwarz) when u aligns with grad f, giving value ||grad f||.

Q: What are the gradient and Hessian of (1/2) x^T A x for symmetric A?
A: Gradient Ax and Hessian A (constant).

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 3.1 Next: 3.3 →