Quant Academy Math · Probability · Quant Finance
Phase 3 - Lesson 3.2

Partial Derivatives, Gradient, Jacobian, and Hessian

The full first- and second-order local picture of a multivariable function, and the objects that drive every optimizer.

⏱ 55 min● Intermediate🔗 Prereqs: 3.1
↖ Phase 3 hub
Builds on: Scalar fields and level sets from 3.1.
Leads to: The gradient and Hessian are the engine of optimization (3.4, 3.5) and gradient descent everywhere later.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Partial derivative
The rate \(\partial f/\partial x_i\) with all other variables held fixed.
Gradient
The vector \(\nabla f=(\partial f/\partial x_1,\dots,\partial f/\partial x_n)\) of first partials.
Directional derivative
The rate \(D_u f=\nabla f\cdot u\) of change along a unit vector \(u\).
Jacobian
The \(m\times n\) matrix \(J_{ij}=\partial F_i/\partial x_j\) of a vector field \(F:\R^n\to\R^m\).
Hessian
The symmetric matrix \(H_{ij}=\partial^2 f/\partial x_i\partial x_j\) of second partials.
Total derivative
The best linear approximation \(f(a+h)\approx f(a)+\nabla f(a)\cdot h\); differentiability means this error is \(o(\lVert h\rVert)\).

Intuition & Motivation

Intuition
A partial derivative asks a one-dimensional question inside a many-dimensional world: ‘if I nudge only \(x_i\), how fast does \(f\) move?’ Stacking those answers gives the gradient, an arrow that points straight uphill. Its length is the steepest slope; its direction is where a hiker climbs fastest. Every gradient-based optimizer - and every neural-net trainer - is just walking opposite that arrow.

Second derivatives measure curvature. The Hessian packages all of them into a symmetric matrix whose eigenvalues will later tell us whether a critical point is a minimum, maximum, or saddle.

Partials and the gradient

For \(f(x,y)=x^2 y+\sin y\) the partials are \(\partial_x f=2xy\) and \(\partial_y f=x^2+\cos y\). The gradient collects them:

\[\nabla f(x,y)=\big(\,\partial_x f,\ \partial_y f\,\big)=\big(2xy,\ x^2+\cos y\big).\] (3.2)
Theorem - Steepest ascent
If \(f\) is differentiable at \(a\) with \(\nabla f(a)\neq 0\), then among all unit directions \(u\) the directional derivative \(D_u f(a)=\nabla f(a)\cdot u\) is maximized by \(u=\nabla f(a)/\lVert\nabla f(a)\rVert\), with maximal value \(\lVert\nabla f(a)\rVert\).
Proof
By Cauchy–Schwarz, \(\nabla f\cdot u\le\lVert\nabla f\rVert\lVert u\rVert=\lVert\nabla f\rVert\) with equality iff \(u\) is a positive multiple of \(\nabla f\). Hence the gradient direction is steepest ascent and \(-\nabla f\) is steepest descent. A corollary: \(\nabla f\) is orthogonal to the level set through \(a\), since moving along a level curve keeps \(f\) constant, giving \(\nabla f\cdot u=0\).

Jacobian and Hessian

For a vector field \(F=(F_1,\dots,F_m)\) the Jacobian stacks the gradients of each component as rows; it is the total-derivative matrix. For a scalar field the Hessian is the Jacobian of the gradient:

\[J_F=\begin{bmatrix}\nabla F_1\\ \vdots\\ \nabla F_m\end{bmatrix},\qquad H_f=\nabla(\nabla f)=\big[\partial_i\partial_j f\big].\] (3.3)

When the second partials are continuous, Clairaut’s theorem gives \(\partial_i\partial_j f=\partial_j\partial_i f\), so the Hessian is symmetric - the fact that makes the spectral theorem (Phase 4) applicable.

Worked Example - Gradient and Hessian of a quadratic form
1
Let \(f(x)=\tfrac12\,x^\top A x\) with \(A\) symmetric. Expand: \(f=\tfrac12\sum_{ij}A_{ij}x_i x_j\).
2
Partial: \(\partial_k f=\tfrac12\sum_j A_{kj}x_j+\tfrac12\sum_i A_{ik}x_i=(Ax)_k\) using symmetry, so \(\nabla f=Ax\).
3
Second partial: \(\partial_l\partial_k f=A_{kl}\), hence \(H_f=A\) everywhere.
4
This is the multivariable analogue of \(\tfrac{d}{dx}\tfrac12 a x^2=ax\) and \(f''=a\): quadratic forms have constant Hessian equal to their matrix.

Interactive: descend a bowl

The workbench walks \(x_{k+1}=x_k-\eta\,\nabla f(x_k)\) on an ill-conditioned bowl. Watch the zig-zag when the level ellipses are elongated (high condition number): steepest descent bounces across the narrow valley.

Interactive: a numerical gradient

Common Mistakes to Avoid
  • Forgetting to normalize \(u\) before computing \(D_u f=\nabla f\cdot u\); the formula needs a unit vector.
  • Swapping Jacobian rows and columns - \(J_F\) is \(m\times n\) with component gradients as rows.
  • Assuming \(\partial_i\partial_j f=\partial_j\partial_i f\) without continuity of the second partials (Clairaut needs it).
  • Treating the gradient as a slope (a number) rather than a vector; its length is the steepest slope.
Quant Practitioner Tips
  • Always sanity-check an analytic gradient against a central-difference gradient before trusting an optimizer.
  • For \(f=\tfrac12 x^\top A x+b^\top x\) memorize \(\nabla f=Ax+b\) and \(H=A\); most quant objectives are locally this.
  • The gradient is orthogonal to level sets - on a risk contour map it points straight across the ellipses toward higher loss.

Knowledge Check

Q1 Easy
At a point where \(\nabla f=(3,4)\), the maximum rate of increase of \(f\) per unit step is:
7
5
12
1
Q2 Medium
For \(f(x)=\tfrac12 x^\top A x\) with symmetric \(A\), the Hessian is:
\(2A\)
\(A\)
\(A^\top A\)
the zero matrix
Q3 Medium
The gradient of a scalar field at a point is always:
Tangent to the level set through that point
Orthogonal to the level set through that point
Zero at every point
Parallel to the x-axis

Practical Exercise

Let \(f(x,y)=x^2+xy+2y^2\). (a) Find \(\nabla f\) and the Hessian \(H\). (b) Give the direction of steepest ascent at \((1,1)\) and the maximal directional derivative there.

▶ Show full solution

(a) \(\partial_x f=2x+y,\ \partial_y f=x+4y\), so \(\nabla f=(2x+y,\ x+4y)\). The Hessian is constant:

\[H=\begin{bmatrix}2&1\\1&4\end{bmatrix}.\]

(b) At \((1,1)\), \(\nabla f=(3,5)\). Steepest ascent is the unit vector \((3,5)/\sqrt{34}\), and the maximal directional derivative is \(\lVert(3,5)\rVert=\sqrt{34}\approx 5.83\).

After the reveal, answer for yourself: The Hessian here is symmetric and, as you will verify in Phase 4, positive definite - a strictly convex bowl.

Lesson Summary

Partial derivatives assemble into the gradient (steepest-ascent arrow, orthogonal to level sets), vector fields differentiate to Jacobians, and scalar fields to symmetric Hessians. For quadratics \(\tfrac12 x^\top Ax\) the gradient is \(Ax\) and the Hessian is \(A\) - the template for every local model.

Formula Sheet Additions

Gradient of a quadratic
\[\nabla\big(\tfrac12 x^\top A x+b^\top x\big)=Ax+b,\quad H=A\]
The workhorse identity for optimization.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: Why does the gradient point in the direction of steepest ascent?
A: Because D_u f = grad f · u is maximized (Cauchy-Schwarz) when u aligns with grad f, giving value ||grad f||.
Q: What are the gradient and Hessian of (1/2) x^T A x for symmetric A?
A: Gradient Ax and Hessian A (constant).

Completion Checklist

Confidence / mastery rating
Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

← 3.1Next: 3.3 →