Quant Academy Math · Probability · Quant Finance
Phase 7 - Lesson 7.5

The Laws of Large Numbers

Why sample averages converge to expectations - the weak law via Chebyshev, the strong law almost surely, and the honest limits of both.

⏱ 45 min● Advanced🔗 Prereqs: 7.2, 7.4
↖ Phase 7 hub
Builds on: 7.2 gave Chebyshev; 7.3-7.4 gave independence and transforms.
Leads to: The LLN justifies Monte Carlo estimation (Phase 13) and the frequentist meaning of probability.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Sample mean
\(\bar X_n=\tfrac1n\sum_{i=1}^n X_i\); the empirical average of \(n\) draws.
Convergence in probability
\(\bar X_n\to\mu\) if \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\to0\) for every \(\varepsilon\gt 0\).
Almost sure convergence
\(\Prob(\lim_n \bar X_n=\mu)=1\); the whole random sequence settles, not just its distribution.
Weak law (WLLN)
For i.i.d. with finite mean, \(\bar X_n\to\mu\) in probability.
Strong law (SLLN)
For i.i.d. with finite mean, \(\bar X_n\to\mu\) almost surely (Kolmogorov).
i.i.d.
Independent and identically distributed - the standard sampling assumption.

Intuition & Motivation

Intuition
The law of large numbers is why averaging works. Each draw is noisy, but the sample mean of \(n\) i.i.d. draws has variance \(\sigma^2/n\), which shrinks to zero: the average concentrates on the true mean. The weak law says the probability of being far off vanishes; the strong law says the sequence of averages actually converges for almost every infinite run - a statement about individual sample paths, not just their marginals. Both fail if the mean does not exist (Cauchy), a warning that fat tails can break the most basic averaging intuition.

Two laws, two modes of convergence

Theorem - Weak law of large numbers
If \(X_1,X_2,\dots\) are i.i.d. with \(\E|X_1|\lt \infty\) and mean \(\mu\), then \(\bar X_n\to\mu\) in probability: for all \(\varepsilon\gt 0\), \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\to0\).
Proof
Assume additionally \(\Var(X_1)=\sigma^2\lt \infty\) (the finite-variance case). Then \(\E[\bar X_n]=\mu\) and \(\Var(\bar X_n)=\sigma^2/n\). Chebyshev gives \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\le \dfrac{\sigma^2/n}{\varepsilon^2}\to0.\) (The general finite-mean case uses truncation and characteristic functions.)
Theorem - Strong law of large numbers (Kolmogorov)
If \(X_1,X_2,\dots\) are i.i.d. with \(\E|X_1|\lt \infty\) and mean \(\mu\), then \(\Prob\big(\lim_{n\to\infty}\bar X_n=\mu\big)=1\).

Almost-sure convergence is strictly stronger than convergence in probability: it constrains the entire tail of each realized sequence. The WLLN allows rare, ever-later excursions away from \(\mu\); the SLLN forbids them (except on a null set).

\[\Var(\bar X_n)=\frac{\sigma^2}{n}\ \Rightarrow\ \text{std error}=\frac{\sigma}{\sqrt n}\ \ (\text{the }1/\sqrt n\text{ rule}).\] (7.8)

Interactive: watch the sample mean converge

Run repeated sampling and watch \(\bar X_n\) settle toward \(\mu\) while the spread narrows like \(1/\sqrt n\).

Interactive: Monte Carlo estimate of pi

The LLN is the license for Monte Carlo: a sample average of an indicator converges to a probability, here \(\pi/4\).

Worked Example - Why Monte Carlo pricing converges
1
To price a derivative we estimate \(\theta=\E[g(X)]\) by \(\hat\theta_n=\tfrac1n\sum g(X_i)\) with i.i.d. simulated \(X_i\).
2
If \(\E|g(X)|\lt \infty\), the SLLN gives \(\hat\theta_n\to\theta\) almost surely - the estimator is consistent.
3
The error scales as \(\mathrm{sd}(g(X))/\sqrt n\): to halve the error you need \(4\times\) the paths.
4
This dimension-free \(1/\sqrt n\) rate is exactly why Monte Carlo beats grids in high-dimensional pricing.
5
But if \(g(X)\) has infinite variance (or mean), the guarantee weakens or vanishes - check integrability first.
Common Mistakes to Avoid
  • Assuming the LLN needs a finite variance - a finite mean suffices; variance only simplifies the proof and gives the rate.
  • Confusing convergence in probability with almost-sure convergence; the strong law is genuinely stronger.
  • Applying the LLN to Cauchy data: the mean does not exist, so \(\bar X_n\) does not converge at all.
  • Believing a larger \(n\) removes bias; the LLN removes variance, not systematic error in the estimator.
Quant Practitioner Tips
  • The \(1/\sqrt n\) rule is the single most useful back-of-envelope fact in simulation: quadruple work for double accuracy.
  • Before trusting an average, ask whether the underlying mean and variance are finite - fat tails invalidate both laws.
  • Variance-reduction methods (antithetics, control variates) keep the \(1/\sqrt n\) rate but shrink the constant \(\sigma\).

Knowledge Check

Q1 Easy
The weak law of large numbers asserts convergence of \(\bar X_n\) to \(\mu\) in the sense of:
Almost sure convergence
Convergence in probability
Convergence in distribution only
Uniform convergence
Q2 Medium
In the Chebyshev proof of the WLLN, the key fact is that \(\Var(\bar X_n)\) equals:
\(\sigma^2\)
\(n\sigma^2\)
\(\sigma^2/n\)
\(\sigma^2/\sqrt n\)
Q3 Hard
For i.i.d. Cauchy random variables the sample mean \(\bar X_n\):
Converges to 0 by the SLLN
Converges to the median
Does not converge; it stays Cauchy-distributed
Converges but slowly

Practical Exercise

You estimate a probability \(p\) by the fraction of successes in \(n\) i.i.d. Bernoulli(\(p\)) trials, \(\hat p_n\). (a) Give \(\E[\hat p_n]\) and \(\Var(\hat p_n)\). (b) Use Chebyshev to find \(n\) guaranteeing \(\Prob(|\hat p_n-p|\gt 0.01)\le0.05\) in the worst case.

▶ Show full solution

(a) \(\E[\hat p_n]=p\) and \(\Var(\hat p_n)=p(1-p)/n\).

(b) Chebyshev: \(\Prob(|\hat p_n-p|\gt 0.01)\le \dfrac{p(1-p)/n}{0.01^2}\). Worst case \(p(1-p)\le1/4\), so we need \(\dfrac{1/4}{n\cdot10^{-4}}\le0.05\), i.e. \(n\ge \dfrac{1/4}{0.05\times10^{-4}}=50{,}000.\)

So 50,000 trials suffice by the (conservative) Chebyshev guarantee. A CLT-based bound would demand far fewer (~9,600), illustrating how loose Chebyshev is - but it is distribution-free and rigorous.

After the reveal, answer for yourself: Recompute the CLT-based \(n\) using \(1.96\,\sqrt{p(1-p)/n}\le0.01\); why is it smaller?

Lesson Summary

The laws of large numbers formalize that sample averages of i.i.d. data converge to the mean: the weak law in probability (provable via Chebyshev with variance \(\sigma^2/n\)) and the strong law almost surely. They license Monte Carlo estimation with a \(1/\sqrt n\) error rate, but require a finite mean - fat-tailed data such as Cauchy break both.

Formula Sheet Additions

Standard error of the mean
\[\mathrm{sd}(\bar X_n)=\sigma/\sqrt n\]
Quadruple the sample to halve the error.
Error Log Checklist
  • Did I confirm the mean is finite before invoking any LLN?
  • Did I state which convergence mode (probability vs almost sure) I mean?
  • Did I remember the LLN reduces variance, not bias?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: Prove the WLLN under finite variance in two lines.
A: \(\Var(\bar X_n)=\sigma^2/n\); Chebyshev gives \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\le \sigma^2/(n\varepsilon^2)\to0\).
Q: How do the weak and strong laws differ?
A: Weak = convergence in probability (marginal); strong = almost-sure convergence of the whole sequence. The strong law is stronger.
Q: Why does the LLN fail for Cauchy data?
A: The Cauchy mean does not exist (integral diverges), so there is nothing to converge to; \(\bar X_n\) stays Cauchy.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

1/sqrt(n) rule
Standard error of a sample mean is \(\sigma/\sqrt n\); accuracy improves as the square root of effort.
SLLN vs WLLN
Strong = almost sure; weak = in probability; both need a finite mean.

Completion Checklist

Confidence / mastery rating
Personal notes
← 7.4Next: 7.6 →