Phase 11 - Lesson 11.1

Statistical Inference, Estimation, and the Bias-Variance Decomposition

What an estimator is, why maximum likelihood is the workhorse, and the single decomposition that governs every model you will ever fit.

⏱ 50 min● Intermediate🔗 Prereqs: 7.x probability, 4.x linear algebra

↖ Phase 11 hub

Builds on: Phase 7 gave you random variables, expectation, and the law of large numbers.
Leads to: Bias-variance recurs in regression (11.2), model selection (11.3), and regularization (11.4).

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Define an estimator and distinguish its bias, variance, and mean-squared error.
Derive the maximum-likelihood estimator for the Gaussian mean and variance.
Prove the bias-variance decomposition of mean-squared error.
Interpret the prediction-error decomposition and its irreducible-noise floor.
Explain why an unbiased estimator is not always the one to prefer.

Key Vocabulary

Estimator: A function \(\hat\theta=T(X_1,\dots,X_n)\) of the data used to approximate an unknown parameter; itself a random variable.

Bias: The systematic error \(\E[\hat\theta]-\theta\); zero bias means correct on average across resamples.

Variance of an estimator: The spread \(\Var(\hat\theta)\) of the estimate across hypothetical repeated samples.

Mean-squared error: Expected squared distance to the truth, \(\E[(\hat\theta-\theta)^2]\); the standard risk under squared loss.

Likelihood: The data density viewed as a function of the parameter, \(L(\theta)=\prod_i f(x_i;\theta)\); the MLE maximizes it.

Consistency: A sequence of estimators with \(\hat\theta_n\to\theta\) in probability as \(n\to\infty\).

Irreducible error: The noise variance \(\sigma^2\) that no model can remove; the floor of prediction error.

Intuition & Motivation

Intuition

An estimator is a recipe you run on data. Because the data are random, the number it returns is random too: run the same recipe on a fresh sample and you get a slightly different answer. Two things can go wrong. The recipe can be systematically off - it aims at the wrong spot (bias). Or it can be jumpy - it scatters widely around wherever it aims (variance). Total expected error under squared loss is exactly bias-squared plus variance, and you almost never get to drive both to zero at once.

Maximum likelihood is the default recipe because it asks a clean question: which parameter value would have made the data I actually saw most probable? Under mild conditions it is consistent and, asymptotically, as precise as any unbiased estimator can be. But ‘asymptotically optimal’ is a promise about large samples and correct models - two things financial data rarely grant.

Estimators as random variables

Fix an unknown parameter \(\theta\) (a mean return, a volatility, a beta). An estimator \(\hat\theta\) is any function of the sample. Its quality is judged by its sampling distribution - how it behaves across the many samples you could have drawn. Three summaries dominate:

\[\text{bias}(\hat\theta)=\E[\hat\theta]-\theta,\qquad \Var(\hat\theta)=\E\big[(\hat\theta-\E[\hat\theta])^2\big],\qquad \text{MSE}(\hat\theta)=\E\big[(\hat\theta-\theta)^2\big].\] (11.1)

Theorem - Bias-variance decomposition of MSE

For any estimator with finite second moment, \(\text{MSE}(\hat\theta)=\text{bias}(\hat\theta)^2+\Var(\hat\theta).\) The two sources of error are additive and separately controllable.

Proof

Write \(\mu=\E[\hat\theta]\) and insert-and-cancel:

\[\E[(\hat\theta-\theta)^2]=\E[(\hat\theta-\mu+\mu-\theta)^2]=\E[(\hat\theta-\mu)^2]+2(\mu-\theta)\E[\hat\theta-\mu]+(\mu-\theta)^2.\]

The cross term vanishes because \(\E[\hat\theta-\mu]=0\). The remaining terms are \(\Var(\hat\theta)\) and \(\text{bias}^2\). \(\qquad\blacksquare\)

Maximum likelihood, concretely

Given data \(x_1,\dots,x_n\) from a density \(f(\cdot;\theta)\), the likelihood is \(L(\theta)=\prod_i f(x_i;\theta)\). We maximize its logarithm (sums beat products for calculus and numerics):

\[\hat\theta_{\text{MLE}}=\argmax_\theta \ \ell(\theta),\qquad \ell(\theta)=\sum_{i=1}^n \log f(x_i;\theta).\] (11.2)

Worked Example - MLE for a Gaussian mean and variance

Model \(x_i\sim\Normal(\mu,\sigma^2)\) i.i.d. The log-likelihood is \(\ell(\mu,\sigma^2)=-\tfrac n2\log(2\pi\sigma^2)-\tfrac1{2\sigma^2}\sum_i(x_i-\mu)^2.\)

Differentiate in \(\mu\): \(\partial_\mu\ell=\tfrac1{\sigma^2}\sum_i(x_i-\mu)=0\Rightarrow \hat\mu=\bar x\), the sample mean.

Differentiate in \(\sigma^2\): \(\partial_{\sigma^2}\ell=-\tfrac{n}{2\sigma^2}+\tfrac1{2\sigma^4}\sum_i(x_i-\hat\mu)^2=0\Rightarrow \hat\sigma^2=\tfrac1n\sum_i(x_i-\bar x)^2.\)

Note the divisor \(n\), not \(n-1\): the MLE of the variance is biased low because it reuses \(\bar x\) in place of the true \(\mu\).

Unbiased is not the same as good

The bias of \(\hat\sigma^2\) is \(\E[\hat\sigma^2]-\sigma^2=-\sigma^2/n\). The classic fix divides by \(n-1\) to get the unbiased \(s^2\). Yet \(s^2\) has larger MSE than a slightly-biased alternative for Gaussian data: shrinking the estimate toward zero trades a little bias for a bigger variance reduction. This is the first appearance of a theme - deliberately accepting bias to cut variance - that drives ridge and lasso in 11.4.

From parameters to predictions

In supervised learning the target is a prediction \(\hat f(x)\) of a response \(Y=f(x)+\varepsilon\) with \(\E[\varepsilon]=0,\ \Var(\varepsilon)=\sigma^2\). Averaging over both the noise and the training set that produced \(\hat f\), the expected squared prediction error at a fixed \(x\) decomposes as

\[\E\big[(Y-\hat f(x))^2\big]=\underbrace{\sigma^2}_{\text{irreducible}}+\underbrace{\big(\E[\hat f(x)]-f(x)\big)^2}_{\text{bias}^2}+\underbrace{\Var(\hat f(x))}_{\text{variance}}.\] (11.3)

Key Idea

The \(\sigma^2\) floor is non-negotiable: with a signal-to-noise ratio as low as it is in daily returns, a model that looks ‘bad’ (R-squared near zero) may in fact be near the achievable limit. Chasing the floor is how backtests get overfit.

Interactive: watch bias and variance trade off

Common Mistakes to Avoid

Reporting a point estimate with no sense of its sampling variability - a number without a standard error is not inference.
Believing ‘unbiased’ means ‘best’; MSE, not bias, is what usually matters for prediction.
Using the \(n\)-divisor MLE variance and then plugging it into a \(t\)-statistic that assumes the \(n-1\) version.
Forgetting the irreducible \(\sigma^2\): no amount of modeling removes it, and pretending otherwise is how overfit backtests are born.

Quant Practitioner Tips

Always ask ‘variance across what?’ - across resampled histories, across bootstraps, across folds. Name the resampling and the standard error follows.
In low signal-to-noise regimes (returns), a slightly biased, lower-variance estimator usually wins out of sample. Shrink on purpose.
MLE is asymptotically efficient if the model is correct; under fat tails, robust or quasi-MLE estimators can beat it badly.
Report MSE, not just bias or variance, when comparing estimators - it is the quantity that actually decomposes.

Knowledge Check

Q1 Easy

An estimator has bias 0.2 and variance 0.09. Its mean-squared error is:

0.29

0.13

0.11

0.049

Q2 Medium

The maximum-likelihood variance \(\hat\sigma^2=\tfrac1n\sum(x_i-\bar x)^2\) is biased because:

It divides by \(n-1\)

It reuses \(\bar x\) in place of the true mean, removing one degree of freedom

The data are not Gaussian

Likelihood maximization always biases downward

Q3 Medium

In the prediction-error decomposition, the \(\sigma^2\) term represents:

Model bias you can remove with a bigger model

Estimation variance that shrinks with more data

Irreducible noise that no model can eliminate

A numerical rounding error

Practical Exercise

Let \(X_1,\dots,X_n\sim\Normal(\mu,\sigma^2)\) i.i.d. Consider the family \(\hat\sigma^2_c=c\sum_i(X_i-\bar X)^2\). (a) For which \(c\) is it unbiased? (b) Using \(\Var(\sum(X_i-\bar X)^2)=2(n-1)\sigma^4\), find the \(c\) minimizing MSE. (c) Comment on which is larger.

▶ Show full solution

(a) Since \(\E[\sum(X_i-\bar X)^2]=(n-1)\sigma^2\), unbiasedness needs \(c=1/(n-1)\) (the usual \(s^2\)).

(b) Let \(S=\sum(X_i-\bar X)^2\), so \(\E[S]=(n-1)\sigma^2\), \(\Var(S)=2(n-1)\sigma^4\), hence \(\E[S^2]=\Var(S)+\E[S]^2=(n^2-1)\sigma^4\). Then \(\text{MSE}(c)=\E[(cS-\sigma^2)^2]=c^2\E[S^2]-2c\sigma^2\E[S]+\sigma^4\). Setting the derivative to zero: \(c^\star=\dfrac{\sigma^2\E[S]}{\E[S^2]}=\dfrac{(n-1)}{(n^2-1)}=\dfrac1{n+1}.\)

(c) \(1/(n+1)\lt 1/(n-1)\): the minimum-MSE estimator shrinks harder than the unbiased one. Accepting a small downward bias buys a larger variance reduction - the recurring lesson of this phase.

After the reveal, answer for yourself: The minimum-MSE divisor is \(n+1\), the unbiased one is \(n-1\), and the MLE uses \(n\). All three are defensible under different loss criteria.

Lesson Summary

An estimator is a random function of the data judged by its sampling distribution. Under squared loss its risk splits exactly into bias-squared plus variance, and prediction error adds an irreducible noise floor \(\sigma^2\). Maximum likelihood is the default estimator - consistent and asymptotically efficient when the model is right - but unbiasedness is not the goal; minimizing MSE, often by deliberately shrinking, is.

Formula Sheet Additions

MSE decomposition

\[\text{MSE}(\hat\theta)=\text{bias}(\hat\theta)^2+\Var(\hat\theta)\]

The master identity behind every model-complexity tradeoff.

Prediction error

\[\E[(Y-\hat f)^2]=\sigma^2+\text{bias}^2+\Var(\hat f)\]

Adds the irreducible-noise floor to the estimator decomposition.

Error Log Checklist

Did I report a standard error, not just a point estimate?
Did I state what the variance is taken over (resamples/folds/bootstrap)?
Did I check whether an unbiased estimator actually has lower MSE than a shrunk one?
Did I acknowledge the irreducible \(\sigma^2\) floor rather than modeling into it?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: State the bias-variance decomposition of MSE and why the cross term vanishes.
A: \(\text{MSE}=\text{bias}^2+\Var\). Expanding around \(\mu=\E[\hat\theta]\), the cross term is proportional to \(\E[\hat\theta-\mu]=0\).

Q: Why is the MLE of the Gaussian variance biased, and by how much?
A: It divides by \(n\) and reuses \(\bar x\) for \(\mu\), so \(\E[\hat\sigma^2]=\tfrac{n-1}{n}\sigma^2\), a bias of \(-\sigma^2/n\).

Q: Give one reason to prefer a biased estimator.
A: It can have strictly lower MSE: a small bias may buy a large variance reduction, as with shrinkage estimators.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

Bias-variance decomposition

\(\text{MSE}(\hat\theta)=(\E[\hat\theta]-\theta)^2+\Var(\hat\theta)\). Additive; both usually cannot be zero at once.

MLE

Maximizes \(\ell(\theta)=\sum_i\log f(x_i;\theta)\); consistent and asymptotically efficient under a correct model.

Irreducible error

The noise variance \(\sigma^2\); the floor of prediction error that no model removes.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

← 10.5 Next: 11.2 →