Regularization: Ridge and Lasso
Shrinking coefficients on purpose - the closed-form ridge estimator, the sparsity of the lasso, and why bias buys out-of-sample accuracy.
Leads to: The same penalized-likelihood idea underlies penalized logistic regression and elastic nets.
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Derive the closed-form ridge estimator and interpret its shrinkage in the SVD basis.
- Explain why the lasso's \(\ell_1\) penalty produces exactly-zero coefficients while ridge does not.
- Relate the penalty \(\lambda\) to the bias-variance tradeoff and choose it by cross-validation.
- Compute the soft-thresholding solution for the orthonormal lasso.
- Justify regularization for low-signal, collinear financial predictors.
Key Vocabulary
- Ridge regression
- Least squares with an \(\ell_2\) penalty \(\lambda\|\beta\|_2^2\); shrinks coefficients toward zero, never exactly to zero.
- Lasso
- Least squares with an \(\ell_1\) penalty \(\lambda\|\beta\|_1\); performs shrinkage and variable selection (exact zeros).
- Regularization path
- The trajectory of \(\hat\beta(\lambda)\) as the penalty varies from 0 (OLS) to \(\infty\) (all zero).
- Soft-thresholding
- The operator \(S_\lambda(z)=\text{sign}(z)(|z|-\lambda)_+\); the lasso solution under orthonormal predictors.
- Shrinkage factor
- In the SVD basis, ridge multiplies the \(j\)th component by \(d_j^2/(d_j^2+\lambda)\in(0,1)\).
- Elastic net
- A blend \(\alpha\|\beta\|_1+(1-\alpha)\|\beta\|_2^2\) combining lasso sparsity with ridge stability under collinearity.
- Effective degrees of freedom
- For ridge, \(\sum_j d_j^2/(d_j^2+\lambda)\le p\); complexity falls smoothly as \(\lambda\) rises.
Intuition & Motivation
Ridge uses a round \(\ell_2\) spring: it shrinks everything smoothly and keeps all predictors. Lasso uses a diamond-shaped \(\ell_1\) constraint whose corners sit on the axes, so the solution often lands exactly on an axis - setting some coefficients to precisely zero. That geometry is why lasso selects variables and ridge merely shrinks them.
Ridge: penalized least squares with a closed form
Ridge solves, for a tuning parameter \(\lambda\ge0\) (predictors standardized, intercept unpenalized):
In the SVD \(X=UDV^\top\), the fitted values become \(\hat y=\sum_j u_j\,\frac{d_j^2}{d_j^2+\lambda}\,u_j^\top y\). Each principal direction is multiplied by a shrinkage factor \(d_j^2/(d_j^2+\lambda)\in(0,1)\): directions with small singular values \(d_j\) (the noisy, collinear directions) are shrunk hardest. Ridge is ‘soft PCA regression’.
Lasso: the \(\ell_1\) penalty and sparsity
Replace the \(\ell_2\) penalty with \(\ell_1\):
There is no closed form in general (the objective is convex but nondifferentiable at zero), but with orthonormal predictors (\(X^\top X=I\)) each coefficient decouples and the solution is soft-thresholding of the OLS estimate:
Small coefficients (below \(\lambda\)) are set to exactly zero; larger ones are pulled toward zero by \(\lambda\). Ridge, by contrast, multiplies by \(1/(1+\lambda)\) - it never zeroes anything. Geometrically the \(\ell_1\) ball has corners on the axes, so the first contact between the residual ellipse and the constraint ball tends to occur at a corner: a sparse solution.
| Property | Ridge (\(\ell_2\)) | Lasso (\(\ell_1\)) |
|---|---|---|
| Closed form | Yes: \((X^\top X+\lambda I)^{-1}X^\top y\) | No (convex program) |
| Sets coefficients to 0 | Never | Yes (variable selection) |
| Handles \(p\gt n\) | Yes | Yes (selects \(\le n\)) |
| Correlated predictors | Shares weight, stable | May pick one arbitrarily |
| Best when | Many small effects | Few strong effects (sparse truth) |
Interactive: ridge shrinkage in action
- Forgetting to standardize predictors before penalizing - the penalty is scale-dependent, so raw units silently reweight variables.
- Penalizing the intercept; it should stay free so shrinkage does not bias the overall level.
- Treating lasso's selected set as the ‘true’ variables; under correlation it picks one of a group almost arbitrarily and is unstable across resamples.
- Choosing \(\lambda\) by training error (which always prefers \(\lambda=0\)); it must come from cross-validation.
- Reading shrunk coefficients as unbiased effect sizes - they are deliberately biased and not valid for classical inference.
- Default to ridge when predictors are many and correlated (typical factor zoos); default to lasso when you believe few predictors truly matter.
- Use the elastic net when you want sparsity but predictors come in correlated groups - it keeps or drops groups together.
- Select \(\lambda\) with the one-SE rule on a CV curve for a more parsimonious, robust model.
- Ridge's closed form makes LOOCV essentially free via the hat-matrix diagonal - exploit it for fast tuning.
Knowledge Check
Practical Exercise
Take orthonormal predictors so \(X^\top X=I\) and let \(z=X^\top y\) be the OLS coefficients. (a) Show ridge gives \(\hat\beta_j=z_j/(1+\lambda)\). (b) Show lasso gives soft-thresholding \(\hat\beta_j=S_\lambda(z_j)\). (c) Contrast their treatment of a tiny coefficient \(z_j=0.01\) versus a large one \(z_j=5\) at \(\lambda=0.1\).
(a) With \(X^\top X=I\), ridge is \((I+\lambda I)^{-1}z=z/(1+\lambda)\) componentwise - uniform proportional shrinkage.
(b) The lasso objective decouples into \(\tfrac12(z_j-\beta_j)^2+\lambda|\beta_j|\). For \(\beta_j\gt 0\) the derivative \(\beta_j-z_j+\lambda=0\) gives \(\beta_j=z_j-\lambda\) (valid while positive); symmetrically for negative; and \(\beta_j=0\) when \(|z_j|\le\lambda\). That is \(S_\lambda(z_j)=\text{sign}(z_j)(|z_j|-\lambda)_+\).
(c) At \(\lambda=0.1\): ridge maps \(0.01\to0.0091\) and \(5\to4.545\) (same 1/1.1 factor - the big coefficient is shrunk in absolute terms much more). Lasso maps \(0.01\to0\) (killed) and \(5\to4.9\) (shifted by a constant \(\lambda\)). Lasso selects (zeros the noise) while lightly biasing the strong signal; ridge keeps everything but proportionally shrinks the strong signal harder.
Lesson Summary
Formula Sheet Additions
- Did I standardize predictors and leave the intercept unpenalized?
- Did I choose \(\lambda\) by cross-validation, not training error?
- Am I treating lasso's selection as stable when predictors are correlated (it is not)?
- Did I avoid interpreting shrunk coefficients as unbiased effect sizes?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: \((X^\top X+\lambda I)^{-1}X^\top y\); the \(\lambda I\) term makes the matrix positive definite even under collinearity or \(p\gt n\).
A: The \(\ell_1\) constraint ball has corners on the axes so solutions land there; the \(\ell_2\) ball is smooth so ridge only shrinks proportionally.
A: Increasing \(\lambda\) raises bias and lowers variance; pick the \(\lambda\) minimizing cross-validated error, often via the one-SE rule.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check