Phase 11 - Lesson 11.5

Classification: Logistic Regression and LDA

Modeling a probability instead of a mean - the logit link and its MLE, linear discriminant analysis, and how the two linear boundaries relate.

⏱ 50 min● Intermediate🔗 Prereqs: 11.1, 11.2

↖ Phase 11 hub

Builds on: 11.1 (MLE) and 11.2 (linear models) are the direct scaffolding.
Leads to: Logistic loss reappears as the boosting surrogate and in neural-network output layers.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Formulate logistic regression via the logit link and write its log-likelihood.
Derive the logistic gradient and explain why the fit needs iterative optimization.
State the LDA model assumptions and derive its linear discriminant boundary.
Compare logistic regression and LDA on assumptions, robustness, and boundary shape.
Explain why accuracy is a misleading metric for rare-event financial classification.

Key Vocabulary

Logit / log-odds: The transform \(\log\frac{p}{1-p}\) mapping a probability to the whole real line; logistic regression makes it linear in \(x\).

Sigmoid: The inverse logit \(\sigma(z)=1/(1+e^{-z})\), squashing a linear score into \((0,1)\).

Log-likelihood (Bernoulli): \(\ell(\beta)=\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]\); concave, so its maximizer is unique.

LDA: Linear Discriminant Analysis: Gaussian class densities with a shared covariance, giving a linear decision boundary.

Decision boundary: The set where two classes are equally probable; linear for logistic and LDA, quadratic for QDA.

Bayes classifier: Assign to the class with the highest posterior \(\Prob(G=k\mid x)\); the theoretical error-minimizing rule.

Class imbalance: When one class is rare (defaults, crashes); accuracy is dominated by the majority and hides model quality.

Intuition & Motivation

Intuition

You cannot fit a straight line to a yes/no outcome - probabilities live in \([0,1]\), but a linear function ranges over all reals. The fix is to model the log-odds linearly: as the odds of the event swing from tiny to huge, the log-odds sweep smoothly from \(-\infty\) to \(+\infty\), exactly matching a linear predictor. Invert that link with the S-shaped sigmoid and you get a valid probability. Fit by asking, as always, which coefficients make the observed 0/1 labels most likely.

LDA reaches a linear boundary from the opposite direction. Assume each class is a Gaussian blob with the same shape (shared covariance), differing only in center. The optimal boundary between two equal-shape Gaussians is a straight line. Logistic regression models the boundary directly; LDA models the two clouds and derives the boundary. When the Gaussian assumption holds, LDA is more efficient; when it fails (fat tails, outliers), logistic regression is more robust.

Logistic regression: modeling the log-odds

For a binary label \(y\in\{0,1\}\) we posit \(\Prob(y=1\mid x)=p(x)=\sigma(x^\top\beta)\), equivalently a linear logit:

\[\log\frac{p(x)}{1-p(x)}=x^\top\beta,\qquad \sigma(z)=\frac{1}{1+e^{-z}}.\] (11.12)

Under independence the Bernoulli log-likelihood is \(\ell(\beta)=\sum_i\big[y_i\log p_i+(1-y_i)\log(1-p_i)\big]\) with \(p_i=\sigma(x_i^\top\beta)\).

Worked Example - The logistic gradient and why there is no closed form

Use \(\sigma'(z)=\sigma(z)(1-\sigma(z))\) and the chain rule on \(\ell\).

The score simplifies beautifully: \(\nabla\ell(\beta)=\sum_i (y_i-p_i)\,x_i=X^\top(y-p).\)

Setting it to zero gives \(X^\top(y-p)=0\) - but \(p=\sigma(X\beta)\) is nonlinear in \(\beta\), so no closed form exists.

Solve numerically: Newton's method here is iteratively reweighted least squares (IRLS), \(\beta\leftarrow\beta+(X^\top W X)^{-1}X^\top(y-p)\) with \(W=\diag(p_i(1-p_i))\).

Because \(-\ell\) is convex (the Hessian \(-X^\top W X\preceq0\)), the maximizer is unique when it exists; it fails to exist only under perfect separation, where coefficients diverge - a signal to regularize.

Linear discriminant analysis

LDA models each class density as Gaussian with a shared covariance: \(x\mid G=k\sim\Normal(\mu_k,\Sigma)\), with priors \(\pi_k\). By Bayes' rule the log-posterior-odds between classes \(k\) and \(\ell\) is linear in \(x\) because the quadratic \(x^\top\Sigma^{-1}x\) terms cancel:

\[\delta_k(x)=x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k,\qquad \hat G(x)=\argmax_k \delta_k(x).\] (11.13)

The boundary \(\{\delta_k=\delta_\ell\}\) is a hyperplane. If instead each class has its own covariance \(\Sigma_k\), the quadratic terms survive and you get QDA with a curved boundary - more flexible but more parameters to estimate.

Aspect	Logistic regression	LDA
Models	the boundary \(\Prob(y\mid x)\) directly	the class densities \(\Prob(x\mid G)\)
Assumes Gaussian \(x\)	No	Yes (shared \(\Sigma\))
Fit	MLE / IRLS (iterative)	Plug in means and pooled covariance (closed form)
Robust to outliers / fat tails	More	Less
Efficiency if Gaussian holds	Slightly lower	Higher

Key Idea

Both give linear boundaries but from different premises. LDA borrows strength from the Gaussian assumption to be more efficient in small samples; logistic regression makes no marginal assumption on \(x\) and so is safer under the fat-tailed, contaminated data typical of finance.

Interactive: logistic score and one Newton step

Common Mistakes to Avoid

Fitting ordinary least squares to a 0/1 label; predicted ‘probabilities’ then fall outside \([0,1]\) and the error variance is wrong.
Interpreting a logistic coefficient as a probability change; it is a change in log-odds, and \(e^{\beta_j}\) is an odds ratio.
Reporting accuracy on an imbalanced problem (1% defaults): predicting ‘never’ scores 99% accuracy and is useless.
Using LDA when classes are clearly non-Gaussian or have very different covariances - QDA or logistic is safer.
Ignoring perfect (quasi-)separation, which sends logistic coefficients to infinity; regularize or add a prior.

Quant Practitioner Tips

For rare events use ranking metrics (AUC), precision/recall, log-loss, or profit curves - not raw accuracy.
Calibrate probabilities before betting sizes on them; a good ranker can still be miscalibrated.
Penalized (ridge/lasso) logistic regression handles separation and high dimension gracefully - the same \(\lambda\) idea as 11.4.
LDA doubles as a supervised dimension-reduction tool (Fisher's discriminant directions), useful before plotting or clustering.

Knowledge Check

Q1 Medium

In logistic regression, a coefficient \(\beta_j\) is best interpreted as:

The change in probability per unit \(x_j\)

The change in log-odds per unit \(x_j\)

The class prior

The residual variance

Q2 Medium

Why does logistic regression require iterative optimization?

The likelihood is non-convex

The score equation \(X^\top(y-p)=0\) is nonlinear in \(\beta\) because \(p=\sigma(X\beta)\)

There are too many parameters

The data must be standardized first

Q3 Medium

LDA yields a linear boundary because:

It uses least squares

The shared covariance makes the quadratic \(x^\top\Sigma^{-1}x\) terms cancel between classes

The priors are equal

It ignores the covariance

Practical Exercise

A model flags market crashes; crashes are 2% of days. Your classifier attains 98% accuracy. (a) Show this can be achieved trivially. (b) Define precision and recall and explain what they reveal here. (c) Why might log-loss or AUC be better objectives, and how does class imbalance interact with the logistic threshold?

▶ Show full solution

(a) Predict ‘no crash’ every day: you are right on 98% of days, so 98% accuracy is the no-skill baseline. Accuracy is uninformative when one class dominates.

(b) Precision = TP/(TP+FP) = of the days you flagged, how many crashed; Recall = TP/(TP+FN) = of the crashes, how many you caught. A useful crash detector needs decent recall (catch crashes) at acceptable precision (few false alarms); the trivial classifier has recall 0.

(c) Log-loss rewards calibrated probabilities and heavily punishes confident wrong calls; AUC measures ranking skill independent of the threshold and of the base rate. With imbalance, the default 0.5 logistic threshold is usually wrong - you tune the threshold (or reweight classes) to trade precision against recall according to the cost of a missed crash versus a false alarm.

After the reveal, answer for yourself: In quant classification the metric is the strategy: choose the loss that matches the P&L consequence of each error type.

Lesson Summary

Classification models a probability, not a mean. Logistic regression makes the log-odds linear, \(\text{logit}\,p=x^\top\beta\), and fits by maximizing a concave Bernoulli likelihood whose score \(X^\top(y-p)=0\) needs Newton/IRLS. LDA assumes Gaussian classes with shared covariance and derives a linear boundary in closed form; distinct covariances give quadratic QDA. Both boundaries are linear but rest on different assumptions, and for rare-event finance accuracy must yield to AUC, precision/recall, and log-loss.

Formula Sheet Additions

Logit link

\[\log\frac{p}{1-p}=x^\top\beta,\quad p=\sigma(x^\top\beta)\]

Linear in log-odds; sigmoid maps back to a probability.

LDA discriminant

\[\delta_k(x)=x^\top\Sigma^{-1}\mu_k-\tfrac12\mu_k^\top\Sigma^{-1}\mu_k+\log\pi_k\]

Linear score; assign to the largest.

Error Log Checklist

Did I model a probability (logit) rather than regress a 0/1 with OLS?
Did I report a threshold-free / imbalance-aware metric (AUC, log-loss), not raw accuracy?
Did I check for separation and regularize if coefficients diverged?
Is the Gaussian/shared-covariance assumption plausible before I trust LDA?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: Write the logistic score equation and say why it needs iteration.
A: \(X^\top(y-p)=0\) with \(p=\sigma(X\beta)\); nonlinear in \(\beta\), so solved by Newton/IRLS despite convexity.

Q: What assumption makes LDA's boundary linear, and what breaks it?
A: A covariance \(\Sigma\) shared across classes makes the quadratic terms cancel; class-specific \(\Sigma_k\) (QDA) restores curvature.

Q: Why is accuracy a poor metric for rare-event classification?
A: A trivial majority predictor scores near the base rate (e.g. 98%); use AUC, precision/recall, or log-loss that reflect the costs of each error.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

Logistic regression

Linear log-odds \(\text{logit}\,p=x^\top\beta\); concave Bernoulli likelihood; fit by IRLS; score \(X^\top(y-p)\).

LDA vs QDA

Shared \(\Sigma\) -> linear boundary (LDA); class-specific \(\Sigma_k\) -> quadratic boundary (QDA).

Imbalance metrics

Use AUC, precision/recall, log-loss, and a tuned threshold; accuracy misleads when a class is rare.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 4 - Ch. 4: linear methods for classification - LDA/QDA, logistic regression, separating hyperplanes.
Probability: Theory and Examples (Rick Durrett, 5th ed.) current - Durrett 4 - Conditional distributions and Bayes rule underlying the posterior class probabilities.

← 11.4 Next: 11.6 →