Regularization: Ridge and Lasso
Shrinking coefficients on purpose - the closed-form ridge estimator, the sparsity of the lasso, and why bias buys out-of-sample accuracy.
Leads to: The same penalized-likelihood idea underlies penalized logistic regression and elastic nets.
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Derive the closed-form ridge estimator and interpret its shrinkage in the SVD basis.
- Explain why the lasso's \(\ell_1\) penalty produces exactly-zero coefficients while ridge does not.
- Relate the penalty \(\lambda\) to the bias-variance tradeoff and choose it by cross-validation.
- Compute the soft-thresholding solution for the orthonormal lasso.
- Justify regularization for low-signal, collinear financial predictors.
Key Vocabulary
- Ridge regression
- Least squares with an \(\ell_2\) penalty \(\lambda\|\beta\|_2^2\); shrinks coefficients toward zero, never exactly to zero.
- Lasso
- Least squares with an \(\ell_1\) penalty \(\lambda\|\beta\|_1\); performs shrinkage and variable selection (exact zeros).
- Regularization path
- The trajectory of \(\hat\beta(\lambda)\) as the penalty varies from 0 (OLS) to \(\infty\) (all zero).
- Soft-thresholding
- The operator \(S_\lambda(z)=\text{sign}(z)(|z|-\lambda)_+\); the lasso solution under orthonormal predictors.
- Shrinkage factor
- In the SVD basis, ridge multiplies the \(j\)th component by \(d_j^2/(d_j^2+\lambda)\in(0,1)\).
- Elastic net
- A blend \(\alpha\|\beta\|_1+(1-\alpha)\|\beta\|_2^2\) combining lasso sparsity with ridge stability under collinearity.
- Effective degrees of freedom
- For ridge, \(\sum_j d_j^2/(d_j^2+\lambda)\le p\); complexity falls smoothly as \(\lambda\) rises.
Intuition & Motivation
Ridge uses a round \(\ell_2\) spring: it shrinks everything smoothly and keeps all predictors. Lasso uses a diamond-shaped \(\ell_1\) constraint whose corners sit on the axes, so the solution often lands exactly on an axis - setting some coefficients to precisely zero. That geometry is why lasso selects variables and ridge merely shrinks them.
Ridge: penalized least squares with a closed form
Ridge solves, for a tuning parameter \(\lambda\ge0\) (predictors standardized, intercept unpenalized):
In the SVD \(X=UDV^\top\), the fitted values become \(\hat y=\sum_j u_j\,\frac{d_j^2}{d_j^2+\lambda}\,u_j^\top y\). Each principal direction is multiplied by a shrinkage factor \(d_j^2/(d_j^2+\lambda)\in(0,1)\): directions with small singular values \(d_j\) (the noisy, collinear directions) are shrunk hardest. Ridge is ‘soft PCA regression’.
Lasso: the \(\ell_1\) penalty and sparsity
Replace the \(\ell_2\) penalty with \(\ell_1\):
There is no closed form in general (the objective is convex but nondifferentiable at zero), but with orthonormal predictors (\(X^\top X=I\)) each coefficient decouples and the solution is soft-thresholding of the OLS estimate:
Small coefficients (below \(\lambda\)) are set to exactly zero; larger ones are pulled toward zero by \(\lambda\). Ridge, by contrast, multiplies by \(1/(1+\lambda)\) - it never zeroes anything. Geometrically the \(\ell_1\) ball has corners on the axes, so the first contact between the residual ellipse and the constraint ball tends to occur at a corner: a sparse solution.
| Property | Ridge (\(\ell_2\)) | Lasso (\(\ell_1\)) |
|---|---|---|
| Closed form | Yes: \((X^\top X+\lambda I)^{-1}X^\top y\) | No (convex program) |
| Sets coefficients to 0 | Never | Yes (variable selection) |
| Handles \(p\gt n\) | Yes | Yes (selects \(\le n\)) |
| Correlated predictors | Shares weight, stable | May pick one arbitrarily |
| Best when | Many small effects | Few strong effects (sparse truth) |
Interactive: ridge shrinkage in action
- Forgetting to standardize predictors before penalizing - the penalty is scale-dependent, so raw units silently reweight variables.
- Penalizing the intercept; it should stay free so shrinkage does not bias the overall level.
- Treating lasso's selected set as the ‘true’ variables; under correlation it picks one of a group almost arbitrarily and is unstable across resamples.
- Choosing \(\lambda\) by training error (which always prefers \(\lambda=0\)); it must come from cross-validation.
- Reading shrunk coefficients as unbiased effect sizes - they are deliberately biased and not valid for classical inference.
- Default to ridge when predictors are many and correlated (typical factor zoos); default to lasso when you believe few predictors truly matter.
- Use the elastic net when you want sparsity but predictors come in correlated groups - it keeps or drops groups together.
- Select \(\lambda\) with the one-SE rule on a CV curve for a more parsimonious, robust model.
- Ridge's closed form makes LOOCV essentially free via the hat-matrix diagonal - exploit it for fast tuning.
Knowledge Check
Practical Exercise
Take orthonormal predictors so \(X^\top X=I\) and let \(z=X^\top y\) be the OLS coefficients. (a) Show ridge gives \(\hat\beta_j=z_j/(1+\lambda)\). (b) Show lasso gives soft-thresholding \(\hat\beta_j=S_\lambda(z_j)\). (c) Contrast their treatment of a tiny coefficient \(z_j=0.01\) versus a large one \(z_j=5\) at \(\lambda=0.1\).
(a) With \(X^\top X=I\), ridge is \((I+\lambda I)^{-1}z=z/(1+\lambda)\) componentwise - uniform proportional shrinkage.
(b) The lasso objective decouples into \(\tfrac12(z_j-\beta_j)^2+\lambda|\beta_j|\). For \(\beta_j\gt 0\) the derivative \(\beta_j-z_j+\lambda=0\) gives \(\beta_j=z_j-\lambda\) (valid while positive); symmetrically for negative; and \(\beta_j=0\) when \(|z_j|\le\lambda\). That is \(S_\lambda(z_j)=\text{sign}(z_j)(|z_j|-\lambda)_+\).
(c) At \(\lambda=0.1\): ridge maps \(0.01\to0.0091\) and \(5\to4.545\) (same 1/1.1 factor - the big coefficient is shrunk in absolute terms much more). Lasso maps \(0.01\to0\) (killed) and \(5\to4.9\) (shifted by a constant \(\lambda\)). Lasso selects (zeros the noise) while lightly biasing the strong signal; ridge keeps everything but proportionally shrinks the strong signal harder.
Lesson Summary
Formula Sheet Additions
- Did I standardize predictors and leave the intercept unpenalized?
- Did I choose \(\lambda\) by cross-validation, not training error?
- Am I treating lasso's selection as stable when predictors are correlated (it is not)?
- Did I avoid interpreting shrunk coefficients as unbiased effect sizes?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: \((X^\top X+\lambda I)^{-1}X^\top y\); the \(\lambda I\) term makes the matrix positive definite even under collinearity or \(p\gt n\).
A: The \(\ell_1\) constraint ball has corners on the axes so solutions land there; the \(\ell_2\) ball is smooth so ridge only shrinks proportionally.
A: Increasing \(\lambda\) raises bias and lowers variance; pick the \(\lambda\) minimizing cross-validated error, often via the one-SE rule.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check
Source References
This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.
- The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 3.4 - Ch. 3.4: ridge, lasso, elastic net, SVD view of shrinkage, degrees of freedom.
- Additional Exercises for Convex Optimization (Boyd & Vandenberghe, 2016) current - Boyd 6 - Convexity and optimality of the \(\ell_1\)/\(\ell_2\)-penalized least-squares programs.