Phase 11 - Lesson 11.1

Statistical Inference, Estimation, and the Bias-Variance Decomposition

What an estimator is, why maximum likelihood is the workhorse, and the single decomposition that governs every model you will ever fit.

⏱ 50 min● Intermediate🔗 Prereqs: 7.x probability, 4.x linear algebra

↖ Phase 11 hub

Builds on: Phase 7 gave you random variables, expectation, and the law of large numbers.
Leads to: Bias-variance recurs in regression (11.2), model selection (11.3), and regularization (11.4).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Define an estimator and distinguish its bias, variance, and mean-squared error.
Derive the maximum-likelihood estimator for the Gaussian mean and variance.
Prove the bias-variance decomposition of mean-squared error.
Interpret the prediction-error decomposition and its irreducible-noise floor.
Explain why an unbiased estimator is not always the one to prefer.

Key Vocabulary

Estimator: A function \(\hat\theta=T(X_1,\dots,X_n)\) of the data used to approximate an unknown parameter; itself a random variable.

Bias: The systematic error \(\E[\hat\theta]-\theta\); zero bias means correct on average across resamples.

Variance of an estimator: The spread \(\Var(\hat\theta)\) of the estimate across hypothetical repeated samples.

Mean-squared error: Expected squared distance to the truth, \(\E[(\hat\theta-\theta)^2]\); the standard risk under squared loss.

Likelihood: The data density viewed as a function of the parameter, \(L(\theta)=\prod_i f(x_i;\theta)\); the MLE maximizes it.

Consistency: A sequence of estimators with \(\hat\theta_n\to\theta\) in probability as \(n\to\infty\).

Irreducible error: The noise variance \(\sigma^2\) that no model can remove; the floor of prediction error.

Intuition & Motivation

Intuition

An estimator is a recipe you run on data. Because the data are random, the number it returns is random too: run the same recipe on a fresh sample and you get a slightly different answer. Two things can go wrong. The recipe can be systematically off - it aims at the wrong spot (bias). Or it can be jumpy - it scatters widely around wherever it aims (variance). Total expected error under squared loss is exactly bias-squared plus variance, and you almost never get to drive both to zero at once.

Maximum likelihood is the default recipe because it asks a clean question: which parameter value would have made the data I actually saw most probable? Under mild conditions it is consistent and, asymptotically, as precise as any unbiased estimator can be. But ‘asymptotically optimal’ is a promise about large samples and correct models - two things financial data rarely grant.

Estimators as random variables

Fix an unknown parameter \(\theta\) (a mean return, a volatility, a beta). An estimator \(\hat\theta\) is any function of the sample. Its quality is judged by its sampling distribution - how it behaves across the many samples you could have drawn. Three summaries dominate:

\[\text{bias}(\hat\theta)=\E[\hat\theta]-\theta,\qquad \Var(\hat\theta)=\E\big[(\hat\theta-\E[\hat\theta])^2\big],\qquad \text{MSE}(\hat\theta)=\E\big[(\hat\theta-\theta)^2\big].\] (11.1)

Theorem - Bias-variance decomposition of MSE

For any estimator with finite second moment, \(\text{MSE}(\hat\theta)=\text{bias}(\hat\theta)^2+\Var(\hat\theta).\) The two sources of error are additive and separately controllable.

Proof

Write \(\mu=\E[\hat\theta]\) and insert-and-cancel:

\[\E[(\hat\theta-\theta)^2]=\E[(\hat\theta-\mu+\mu-\theta)^2]=\E[(\hat\theta-\mu)^2]+2(\mu-\theta)\E[\hat\theta-\mu]+(\mu-\theta)^2.\]

The cross term vanishes because \(\E[\hat\theta-\mu]=0\). The remaining terms are \(\Var(\hat\theta)\) and \(\text{bias}^2\). \(\qquad\blacksquare\)

Maximum likelihood, concretely

Given data \(x_1,\dots,x_n\) from a density \(f(\cdot;\theta)\), the likelihood is \(L(\theta)=\prod_i f(x_i;\theta)\). We maximize its logarithm (sums beat products for calculus and numerics):

\[\hat\theta_{\text{MLE}}=\argmax_\theta \ \ell(\theta),\qquad \ell(\theta)=\sum_{i=1}^n \log f(x_i;\theta).\] (11.2)

Worked Example - MLE for a Gaussian mean and variance

Model \(x_i\sim\Normal(\mu,\sigma^2)\) i.i.d. The log-likelihood is \(\ell(\mu,\sigma^2)=-\tfrac n2\log(2\pi\sigma^2)-\tfrac1{2\sigma^2}\sum_i(x_i-\mu)^2.\)

Differentiate in \(\mu\): \(\partial_\mu\ell=\tfrac1{\sigma^2}\sum_i(x_i-\mu)=0\Rightarrow \hat\mu=\bar x\), the sample mean.

Differentiate in \(\sigma^2\): \(\partial_{\sigma^2}\ell=-\tfrac{n}{2\sigma^2}+\tfrac1{2\sigma^4}\sum_i(x_i-\hat\mu)^2=0\Rightarrow \hat\sigma^2=\tfrac1n\sum_i(x_i-\bar x)^2.\)

Note the divisor \(n\), not \(n-1\): the MLE of the variance is biased low because it reuses \(\bar x\) in place of the true \(\mu\).

Unbiased is not the same as good

The bias of \(\hat\sigma^2\) is \(\E[\hat\sigma^2]-\sigma^2=-\sigma^2/n\). The classic fix divides by \(n-1\) to get the unbiased \(s^2\). Yet \(s^2\) has larger MSE than a slightly-biased alternative for Gaussian data: shrinking the estimate toward zero trades a little bias for a bigger variance reduction. This is the first appearance of a theme - deliberately accepting bias to cut variance - that drives ridge and lasso in 11.4.

From parameters to predictions

In supervised learning the target is a prediction \(\hat f(x)\) of a response \(Y=f(x)+\varepsilon\) with \(\E[\varepsilon]=0,\ \Var(\varepsilon)=\sigma^2\). Averaging over both the noise and the training set that produced \(\hat f\), the expected squared prediction error at a fixed \(x\) decomposes as

\[\E\big[(Y-\hat f(x))^2\big]=\underbrace{\sigma^2}_{\text{irreducible}}+\underbrace{\big(\E[\hat f(x)]-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\Var(\hat f(x))}_{\text{variance}}.\] (11.3)

Key Idea

The \(\sigma^2\) floor is non-negotiable: with a signal-to-noise ratio as low as it is in daily returns, a model that looks ‘bad’ (R-squared near zero) may in fact be near the achievable limit. Chasing the floor is how backtests get overfit.

Interactive: watch bias and variance trade off

Common Mistakes to Avoid

Reporting a point estimate with no sense of its sampling variability - a number without a standard error is not inference.
Believing ‘unbiased’ means ‘best’; MSE, not bias, is what usually matters for prediction.
Using the \(n\)-divisor MLE variance and then plugging it into a \(t\)-statistic that assumes the \(n-1\) version.
Forgetting the irreducible \(\sigma^2\): no amount of modeling removes it, and pretending otherwise is how overfit backtests are born.

Quant Practitioner Tips

Always ask ‘variance across what?’ - across resampled histories, across bootstraps, across folds. Name the resampling and the standard error follows.
In low signal-to-noise regimes (returns), a slightly biased, lower-variance estimator usually wins out of sample. Shrink on purpose.
MLE is asymptotically efficient if the model is correct; under fat tails, robust or quasi-MLE estimators can beat it badly.
Report MSE, not just bias or variance, when comparing estimators - it is the quantity that actually decomposes.

Knowledge Check

Q1 Easy

An estimator has bias 0.2 and variance 0.09. Its mean-squared error is:

0.29

0.13

0.11

0.049

Q2 Medium

The maximum-likelihood variance \(\hat\sigma^2=\tfrac1n\sum(x_i-\bar x)^2\) is biased because:

It divides by \(n-1\)

It reuses \(\bar x\) in place of the true mean, removing one degree of freedom

The data are not Gaussian

Likelihood maximization always biases downward

Q3 Medium

In the prediction-error decomposition, the \(\sigma^2\) term represents:

Model bias you can remove with a bigger model

Estimation variance that shrinks with more data

Irreducible noise that no model can eliminate

A numerical rounding error

Practical Exercise

Let \(X_1,\dots,X_n\sim\Normal(\mu,\sigma^2)\) i.i.d. Consider the family \(\hat\sigma^2_c=c\sum_i(X_i-\bar X)^2\). (a) For which \(c\) is it unbiased? (b) Using \(\Var(\sum(X_i-\bar X)^2)=2(n-1)\sigma^4\), find the \(c\) minimizing MSE. (c) Comment on which is larger.

▶ Show full solution

(a) Since \(\E[\sum(X_i-\bar X)^2]=(n-1)\sigma^2\), unbiasedness needs \(c=1/(n-1)\) (the usual \(s^2\)).

(b) Let \(S=\sum(X_i-\bar X)^2\), so \(\E[S]=(n-1)\sigma^2\), \(\Var(S)=2(n-1)\sigma^4\), hence \(\E[S^2]=\Var(S)+\E[S]^2=(n^2-1)\sigma^4\). Then \(\text{MSE}(c)=\E[(cS-\sigma^2)^2]=c^2\E[S^2]-2c\sigma^2\E[S]+\sigma^4\). Setting the derivative to zero: \(c^\star=\dfrac{\sigma^2\E[S]}{\E[S^2]}=\dfrac{(n-1)}{(n^2-1)}=\dfrac1{n+1}.\)

(c) \(1/(n+1)\lt 1/(n-1)\): the minimum-MSE estimator shrinks harder than the unbiased one. Accepting a small downward bias buys a larger variance reduction - the recurring lesson of this phase.

After the reveal, answer for yourself: The minimum-MSE divisor is \(n+1\), the unbiased one is \(n-1\), and the MLE uses \(n\). All three are defensible under different loss criteria.

Lesson Summary

An estimator is a random function of the data judged by its sampling distribution. Under squared loss its risk splits exactly into bias-squared plus variance, and prediction error adds an irreducible noise floor \(\sigma^2\). Maximum likelihood is the default estimator - consistent and asymptotically efficient when the model is right - but unbiasedness is not the goal; minimizing MSE, often by deliberately shrinking, is.

Formula Sheet Additions

MSE decomposition

\[\text{MSE}(\hat\theta)=\text{bias}(\hat\theta)^2+\Var(\hat\theta)\]

The master identity behind every model-complexity tradeoff.

Prediction error

\[\E[(Y-\hat f)^2]=\sigma^2+\text{bias}^2+\Var(\hat f)\]

Adds the irreducible-noise floor to the estimator decomposition.

Error Log Checklist

Did I report a standard error, not just a point estimate?
Did I state what the variance is taken over (resamples/folds/bootstrap)?
Did I check whether an unbiased estimator actually has lower MSE than a shrunk one?
Did I acknowledge the irreducible \(\sigma^2\) floor rather than modeling into it?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: State the bias-variance decomposition of MSE and why the cross term vanishes.
A: \(\text{MSE}=\text{bias}^2+\Var\). Expanding around \(\mu=\E[\hat\theta]\), the cross term is proportional to \(\E[\hat\theta-\mu]=0\).

Q: Why is the MLE of the Gaussian variance biased, and by how much?
A: It divides by \(n\) and reuses \(\bar x\) for \(\mu\), so \(\E[\hat\sigma^2]=\tfrac{n-1}{n}\sigma^2\), a bias of \(-\sigma^2/n\).

Q: Give one reason to prefer a biased estimator.
A: It can have strictly lower MSE: a small bias may buy a large variance reduction, as with shrinkage estimators.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

Bias-variance decomposition

\(\text{MSE}(\hat\theta)=(\E[\hat\theta]-\theta)^2+\Var(\hat\theta)\). Additive; both usually cannot be zero at once.

MLE

Maximizes \(\ell(\theta)=\sum_i\log f(x_i;\theta)\); consistent and asymptotically efficient under a correct model.

Irreducible error

The noise variance \(\sigma^2\); the floor of prediction error that no model removes.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 2, 7 - Ch. 2 (statistical decision theory, EPE) and Ch. 7 (bias, variance, and model complexity).
Probability: Theory and Examples (Rick Durrett, 5th ed.) current - Durrett 2 - Weak law of large numbers and moment computations underlying estimator consistency.

← 10.5 Next: 11.2 →