Phase 3 - Lesson 3.5

Constrained Optimization and Lagrange Multipliers

Optimizing subject to equality constraints, the geometry of parallel gradients, and the multiplier as a shadow price.

⏱ 55 min● Advanced🔗 Prereqs: 3.4

Builds on: Gradients (3.2) and unconstrained extrema (3.4).
Leads to: Lagrangian duality generalizes this to inequalities in Phase 10; the minimum-variance portfolio uses it directly.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

State the Lagrange condition \(\nabla f=\lambda\nabla g\) and explain its geometry.
Solve equality-constrained optimization problems using the Lagrangian.
Interpret the multiplier \(\lambda\) as the sensitivity of the optimum to the constraint level.
Derive the minimum-variance portfolio as a constrained quadratic program.

Key Vocabulary

Constraint set: The feasible region \(\{x: g(x)=c\}\) over which \(f\) is optimized.

Lagrange multiplier: The scalar \(\lambda\) linking objective and constraint gradients at an optimum.

Lagrangian: The function \(\mathcal{L}(x,\lambda)=f(x)-\lambda\,(g(x)-c)\) whose stationarity encodes the conditions.

Shadow price: The rate \(d f^*/dc=\lambda\) at which the optimal value improves as the constraint loosens.

Binding constraint: One satisfied with equality at the optimum, so its gradient enters the stationarity condition.

Minimum-variance portfolio: Weights minimizing \(w^\top\Sigma w\) subject to \(\mathbf{1}^\top w=1\).

Intuition & Motivation

Intuition

Picture the contour map of \(f\) and draw the constraint curve \(g=c\) on top. Walking along the constraint, \(f\) rises and falls; you stop improving exactly where the constraint curve is tangent to a level curve of \(f\). Tangency means the two curves share a normal direction, i.e. the gradients are parallel: \(\nabla f=\lambda\nabla g\). The multiplier \(\lambda\) is the proportionality constant, and it doubles as a price: it tells you how much the best attainable value would change if you relaxed the constraint by one unit.

The Lagrange condition

Theorem - Lagrange multipliers (equality constraint)

Let \(f,g\) be \(C^1\) and suppose \(x^*\) optimizes \(f\) on \(\{g=c\}\) with \(\nabla g(x^*)\neq 0\). Then there is a scalar \(\lambda\) with \(\nabla f(x^*)=\lambda\,\nabla g(x^*)\). Equivalently, \((x^*,\lambda)\) is a stationary point of the Lagrangian \(\mathcal{L}=f-\lambda(g-c)\).

The recipe: form \(\mathcal{L}\), set \(\nabla_x\mathcal{L}=0\) and \(\partial_\lambda\mathcal{L}=0\) (the latter just restores the constraint), and solve the combined system. With several constraints, add one multiplier each: \(\nabla f=\sum_k\lambda_k\nabla g_k\).

Worked Example - Closest point on a line / smallest sum of squares

Minimize \(f=x^2+y^2\) subject to \(g=x+y=1\) (a stylized fully-invested constraint).

Lagrangian \(\mathcal{L}=x^2+y^2-\lambda(x+y-1)\). Stationarity: \(2x=\lambda,\ 2y=\lambda\), so \(x=y\).

Constraint \(x+y=1\) gives \(x=y=\tfrac12\) and \(\lambda=1\).

Minimum value \(f=\tfrac14+\tfrac14=\tfrac12\). The multiplier \(\lambda=1=df^*/dc\): nudging the budget \(c\) from 1 raises the minimal cost at unit rate. Verify: \(f^*(c)=c^2/2\), so \(df^*/dc=c=1\).

The multiplier as a shadow price

Envelope theorem: at the optimum, \(\dfrac{d f^*}{dc}=\lambda\). In portfolio terms \(\lambda\) is a marginal cost/benefit - the improvement in the objective per unit relaxation of the constraint. This is why multipliers are called shadow prices and reappear as dual variables in Phase 10.

Application: the minimum-variance portfolio

Let \(\Sigma\succ 0\) be the covariance matrix of \(n\) assets’ returns and \(\mathbf 1\) the all-ones vector. The minimum-variance portfolio solves:

\[\min_w\ \tfrac12\,w^\top\Sigma w\quad\text{s.t.}\quad \mathbf 1^\top w=1.\] (3.5)

Worked Example - Solving the min-variance program

Lagrangian \(\mathcal L=\tfrac12 w^\top\Sigma w-\lambda(\mathbf 1^\top w-1)\). Gradient in \(w\): \(\Sigma w-\lambda\mathbf 1=0\).

Solve \(w=\lambda\,\Sigma^{-1}\mathbf 1\) (valid since \(\Sigma\succ 0\) is invertible).

Impose \(\mathbf 1^\top w=1\): \(\lambda\,\mathbf 1^\top\Sigma^{-1}\mathbf 1=1\), so \(\lambda=1/(\mathbf 1^\top\Sigma^{-1}\mathbf 1)\).

Therefore \(w^*=\dfrac{\Sigma^{-1}\mathbf 1}{\mathbf 1^\top\Sigma^{-1}\mathbf 1}\) with variance \(w^{*\top}\Sigma w^*=1/(\mathbf 1^\top\Sigma^{-1}\mathbf 1)=\lambda\).

The multiplier again equals the optimal variance - the shadow price of the budget constraint. This closed form is a cornerstone of Markowitz portfolio theory (Phase 14).

Common Mistakes to Avoid

Forgetting the constraint equation - \(\partial_\lambda\mathcal L=0\) must be part of the system, or you lose it.
Assuming \(\lambda\) has a fixed sign; for equality constraints it can be positive or negative.
Applying the method when \(\nabla g=0\) at the candidate (constraint qualification fails).
Interpreting a Lagrange stationary point as automatically a minimum; check second-order/convexity as in 3.4.

Quant Practitioner Tips

Set gradients proportional, then use the constraint to pin down \(\lambda\) - two steps, always.
Read \(\lambda\) as a price: it is the marginal value of loosening the constraint, the dual variable of Phase 10.
For quadratic objective + linear constraints the system is linear; solve it as one matrix equation (the min-variance closed form).

Knowledge Check

Q1 Easy

The Lagrange condition for minimizing \(f\) on \(g=c\) states that at the optimum:

\(\nabla f=0\)

\(\nabla f=\lambda\nabla g\)

\(f=g\)

\(\nabla g=0\)

Q2 Medium

The Lagrange multiplier \(\lambda\) at the optimum equals:

The constraint value c

The sensitivity \(df^*/dc\) of the optimal value to the constraint level

The objective value

Always 1

Q3 Hard

For \(\min\ \tfrac12 w^\top\Sigma w\) s.t. \(\mathbf 1^\top w=1\) with \(\Sigma\succ0\), the solution is:

\(w=\Sigma^{-1}\mathbf 1\)

\(w=\Sigma^{-1}\mathbf 1/(\mathbf 1^\top\Sigma^{-1}\mathbf 1)\)

\(w=\mathbf 1/n\)

\(w=\Sigma\mathbf 1\)

Practical Exercise

Maximize \(f(x,y)=xy\) subject to \(x+y=10\) (allocate a fixed budget of 10 across two uses to maximize their product). Solve with a multiplier and interpret \(\lambda\).

▶ Show full solution

Lagrangian \(\mathcal L=xy-\lambda(x+y-10)\). Stationarity: \(y=\lambda,\ x=\lambda\), so \(x=y\).

Constraint \(x+y=10\) gives \(x=y=5\) and \(\lambda=5\), with maximum \(f=25\).

Interpretation: \(\lambda=5=df^*/dc\). Increasing the budget from 10 to \(c\) gives optimum \(f^*=(c/2)^2=c^2/4\), so \(df^*/dc=c/2=5\) at \(c=10\). Each extra unit of budget adds about 5 to the product at the margin. The symmetric split \(x=y\) is the classic AM–GM result.

After the reveal, answer for yourself: Notice the second-order check: on the constraint line the product is a downward parabola, confirming a maximum.

Lesson Summary

Equality-constrained optima occur where the objective’s and constraint’s gradients are parallel, \(\nabla f=\lambda\nabla g\), encoded by stationarity of the Lagrangian. The multiplier \(\lambda\) is a shadow price, \(df^*/dc\). Applied to \(\min\ w^\top\Sigma w\) with a budget constraint it yields the minimum-variance portfolio \(w^*=\Sigma^{-1}\mathbf 1/(\mathbf 1^\top\Sigma^{-1}\mathbf 1)\).

Formula Sheet Additions

Min-variance portfolio

\[w^*=\frac{\Sigma^{-1}\mathbf 1}{\mathbf 1^\top\Sigma^{-1}\mathbf 1}\]

Lagrange applied to a quadratic risk objective with a budget constraint.

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: What is the geometric meaning of \(\nabla f=\lambda\nabla g\)?
A: At the constrained optimum the constraint curve is tangent to a level set of f, so their gradients (normals) are parallel.

Q: What does the Lagrange multiplier represent economically?
A: A shadow price: the marginal change df*/dc in the optimal value per unit relaxation of the constraint.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 3.4 Next: 4.1 →