Quant Academy Math · Probability · Quant Finance
Phase 7 - Lesson 7.5

The Laws of Large Numbers

Why sample averages converge to expectations - the weak law via Chebyshev, the strong law almost surely, and the honest limits of both.

⏱ 45 min● Advanced🔗 Prereqs: 7.2, 7.4
↖ Phase 7 hub
Builds on: 7.2 gave Chebyshev; 7.3-7.4 gave independence and transforms.
Leads to: The LLN justifies Monte Carlo estimation (Phase 13) and the frequentist meaning of probability.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Key Vocabulary

Sample mean
\(\bar X_n=\tfrac1n\sum_{i=1}^n X_i\); the empirical average of \(n\) draws.
Convergence in probability
\(\bar X_n\to\mu\) if \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\to0\) for every \(\varepsilon\gt 0\).
Almost sure convergence
\(\Prob(\lim_n \bar X_n=\mu)=1\); the whole random sequence settles, not just its distribution.
Weak law (WLLN)
For i.i.d. with finite mean, \(\bar X_n\to\mu\) in probability.
Strong law (SLLN)
For i.i.d. with finite mean, \(\bar X_n\to\mu\) almost surely (Kolmogorov).
i.i.d.
Independent and identically distributed - the standard sampling assumption.

Intuition & Motivation

Intuition
The law of large numbers is why averaging works. Each draw is noisy, but the sample mean of \(n\) i.i.d. draws has variance \(\sigma^2/n\), which shrinks to zero: the average concentrates on the true mean. The weak law says the probability of being far off vanishes; the strong law says the sequence of averages actually converges for almost every infinite run - a statement about individual sample paths, not just their marginals. Both fail if the mean does not exist (Cauchy), a warning that fat tails can break the most basic averaging intuition.

Two laws, two modes of convergence

Theorem - Weak law of large numbers
If \(X_1,X_2,\dots\) are i.i.d. with \(\E|X_1|\lt \infty\) and mean \(\mu\), then \(\bar X_n\to\mu\) in probability: for all \(\varepsilon\gt 0\), \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\to0\).
Proof
Assume additionally \(\Var(X_1)=\sigma^2\lt \infty\) (the finite-variance case). Then \(\E[\bar X_n]=\mu\) and \(\Var(\bar X_n)=\sigma^2/n\). Chebyshev gives \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\le \dfrac{\sigma^2/n}{\varepsilon^2}\to0.\) (The general finite-mean case uses truncation and characteristic functions.)
Theorem - Strong law of large numbers (Kolmogorov)
If \(X_1,X_2,\dots\) are i.i.d. with \(\E|X_1|\lt \infty\) and mean \(\mu\), then \(\Prob\big(\lim_{n\to\infty}\bar X_n=\mu\big)=1\).

Almost-sure convergence is strictly stronger than convergence in probability: it constrains the entire tail of each realized sequence. The WLLN allows rare, ever-later excursions away from \(\mu\); the SLLN forbids them (except on a null set).

\[\Var(\bar X_n)=\frac{\sigma^2}{n}\ \Rightarrow\ \text{std error}=\frac{\sigma}{\sqrt n}\ \ (\text{the }1/\sqrt n\text{ rule}).\] (7.8)

Interactive: watch the sample mean converge

Run repeated sampling and watch \(\bar X_n\) settle toward \(\mu\) while the spread narrows like \(1/\sqrt n\).

Interactive: Monte Carlo estimate of pi

The LLN is the license for Monte Carlo: a sample average of an indicator converges to a probability, here \(\pi/4\).

Worked Example - Why Monte Carlo pricing converges
1
To price a derivative we estimate \(\theta=\E[g(X)]\) by \(\hat\theta_n=\tfrac1n\sum g(X_i)\) with i.i.d. simulated \(X_i\).
2
If \(\E|g(X)|\lt \infty\), the SLLN gives \(\hat\theta_n\to\theta\) almost surely - the estimator is consistent.
3
The error scales as \(\mathrm{sd}(g(X))/\sqrt n\): to halve the error you need \(4\times\) the paths.
4
This dimension-free \(1/\sqrt n\) rate is exactly why Monte Carlo beats grids in high-dimensional pricing.
5
But if \(g(X)\) has infinite variance (or mean), the guarantee weakens or vanishes - check integrability first.
Common Mistakes to Avoid
  • Assuming the LLN needs a finite variance - a finite mean suffices; variance only simplifies the proof and gives the rate.
  • Confusing convergence in probability with almost-sure convergence; the strong law is genuinely stronger.
  • Applying the LLN to Cauchy data: the mean does not exist, so \(\bar X_n\) does not converge at all.
  • Believing a larger \(n\) removes bias; the LLN removes variance, not systematic error in the estimator.
Quant Practitioner Tips
  • The \(1/\sqrt n\) rule is the single most useful back-of-envelope fact in simulation: quadruple work for double accuracy.
  • Before trusting an average, ask whether the underlying mean and variance are finite - fat tails invalidate both laws.
  • Variance-reduction methods (antithetics, control variates) keep the \(1/\sqrt n\) rate but shrink the constant \(\sigma\).

Knowledge Check

Q1 Easy
The weak law of large numbers asserts convergence of \(\bar X_n\) to \(\mu\) in the sense of:
Almost sure convergence
Convergence in probability
Convergence in distribution only
Uniform convergence
Q2 Medium
In the Chebyshev proof of the WLLN, the key fact is that \(\Var(\bar X_n)\) equals:
\(\sigma^2\)
\(n\sigma^2\)
\(\sigma^2/n\)
\(\sigma^2/\sqrt n\)
Q3 Hard
For i.i.d. Cauchy random variables the sample mean \(\bar X_n\):
Converges to 0 by the SLLN
Converges to the median
Does not converge; it stays Cauchy-distributed
Converges but slowly

Practical Exercise

You estimate a probability \(p\) by the fraction of successes in \(n\) i.i.d. Bernoulli(\(p\)) trials, \(\hat p_n\). (a) Give \(\E[\hat p_n]\) and \(\Var(\hat p_n)\). (b) Use Chebyshev to find \(n\) guaranteeing \(\Prob(|\hat p_n-p|\gt 0.01)\le0.05\) in the worst case.

▶ Show full solution

(a) \(\E[\hat p_n]=p\) and \(\Var(\hat p_n)=p(1-p)/n\).

(b) Chebyshev: \(\Prob(|\hat p_n-p|\gt 0.01)\le \dfrac{p(1-p)/n}{0.01^2}\). Worst case \(p(1-p)\le1/4\), so we need \(\dfrac{1/4}{n\cdot10^{-4}}\le0.05\), i.e. \(n\ge \dfrac{1/4}{0.05\times10^{-4}}=50{,}000.\)

So 50,000 trials suffice by the (conservative) Chebyshev guarantee. A CLT-based bound would demand far fewer (~9,600), illustrating how loose Chebyshev is - but it is distribution-free and rigorous.

After the reveal, answer for yourself: Recompute the CLT-based \(n\) using \(1.96\,\sqrt{p(1-p)/n}\le0.01\); why is it smaller?

Lesson Summary

The laws of large numbers formalize that sample averages of i.i.d. data converge to the mean: the weak law in probability (provable via Chebyshev with variance \(\sigma^2/n\)) and the strong law almost surely. They license Monte Carlo estimation with a \(1/\sqrt n\) error rate, but require a finite mean - fat-tailed data such as Cauchy break both.

Formula Sheet Additions

Standard error of the mean
\[\mathrm{sd}(\bar X_n)=\sigma/\sqrt n\]
Quadruple the sample to halve the error.
Error Log Checklist
  • Did I confirm the mean is finite before invoking any LLN?
  • Did I state which convergence mode (probability vs almost sure) I mean?
  • Did I remember the LLN reduces variance, not bias?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers
Q: Prove the WLLN under finite variance in two lines.
A: \(\Var(\bar X_n)=\sigma^2/n\); Chebyshev gives \(\Prob(|\bar X_n-\mu|\gt \varepsilon)\le \sigma^2/(n\varepsilon^2)\to0\).
Q: How do the weak and strong laws differ?
A: Weak = convergence in probability (marginal); strong = almost-sure convergence of the whole sequence. The strong law is stronger.
Q: Why does the LLN fail for Cauchy data?
A: The Cauchy mean does not exist (integral diverges), so there is nothing to converge to; \(\bar X_n\) stays Cauchy.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

1/sqrt(n) rule
Standard error of a sample mean is \(\sigma/\sqrt n\); accuracy improves as the square root of effort.
SLLN vs WLLN
Strong = almost sure; weak = in probability; both need a finite mean.

Completion Checklist

Confidence / mastery rating
Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

← 7.4Next: 7.6 →