Phase 7 - Lesson 7.3

Independence, Conditional Probability, and Conditional Expectation

From elementary conditioning and Bayes to conditional expectation as a projection onto a sigma-algebra - the object that becomes the martingale.

⏱ 55 min● Advanced🔗 Prereqs: 7.2

↖ Phase 7 hub

Builds on: 7.2 established expectation and its inequalities.
Leads to: Conditional expectation given a filtration defines martingales (7.7) and the whole of stochastic calculus.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Define independence of events and random variables and test it on examples.
Apply Bayes’ theorem and the law of total probability to update beliefs.
Define conditional expectation \(\E[X\mid\mathcal{G}]\) and state its defining property.
Use the tower property and interpret conditional expectation as an \(L^2\) projection.

Key Vocabulary

Independent events: \(\Prob(A\cap B)=\Prob(A)\Prob(B)\); knowing one gives no information about the other.

Conditional probability: \(\Prob(A\mid B)=\Prob(A\cap B)/\Prob(B)\) for \(\Prob(B)\gt 0\).

Bayes’ theorem: \(\Prob(A\mid B)=\Prob(B\mid A)\Prob(A)/\Prob(B)\); inverts the direction of conditioning.

Conditional expectation: \(\E[X\mid\mathcal{G}]\): the \(\mathcal{G}\)-measurable random variable matching \(X\)’s integral on every \(\mathcal{G}\)-event.

Tower property: \(\E[\E[X\mid\mathcal{G}]]=\E[X]\); more generally \(\E[\E[X\mid\mathcal{G}]\mid\mathcal{H}]=\E[X\mid\mathcal{H}]\) for \(\mathcal{H}\subseteq\mathcal{G}\).

Sigma-algebra as information: A sub-sigma-algebra \(\mathcal{G}\subseteq\F\) models ‘what is known’; measurability w.r.t. \(\mathcal{G}\) means observable.

Intuition & Motivation

Intuition

Conditioning is updating. Elementary conditional probability rescales the world to the event you learned. Bayes turns ‘probability of data given hypothesis’ into ‘probability of hypothesis given data’ - the engine of inference. The deep object is conditional expectation given a sigma-algebra: think of \(\mathcal{G}\) as the information you have. \(\E[X\mid\mathcal{G}]\) is your best forecast of \(X\) using only that information - the orthogonal projection of \(X\) onto the \(\mathcal{G}\)-measurable functions in the least-squares sense. As \(\mathcal{G}\) grows over time, these forecasts evolve, and that evolution is a martingale.

Independence and Bayes

Definition - Independence

Events \(A,B\) are independent if \(\Prob(A\cap B)=\Prob(A)\Prob(B)\). Random variables \(X,Y\) are independent if \(\Prob(X\in A,Y\in B)=\Prob(X\in A)\Prob(Y\in B)\) for all Borel \(A,B\); equivalently the joint law factorizes.

For independent \(X,Y\): \(\E[XY]=\E X\,\E Y\) and \(\Var(X+Y)=\Var X+\Var Y\). The converse is false - uncorrelated need not mean independent.

Proposition - Total probability and Bayes

If \(\{B_i\}\) partition \(\Omega\) with \(\Prob(B_i)\gt 0\), then \(\Prob(A)=\sum_i\Prob(A\mid B_i)\Prob(B_i)\), and

\[\Prob(B_j\mid A)=\frac{\Prob(A\mid B_j)\,\Prob(B_j)}{\sum_i \Prob(A\mid B_i)\,\Prob(B_i)}.\] (7.5)

Conditional expectation given a sigma-algebra

Definition - Conditional expectation

Given \(X\) with \(\E|X|\lt \infty\) and a sub-sigma-algebra \(\mathcal{G}\subseteq\F\), the conditional expectation \(\E[X\mid\mathcal{G}]\) is the (a.s. unique) \(\mathcal{G}\)-measurable random variable \(Y\) such that

\[\int_G Y\,d\Prob=\int_G X\,d\Prob\qquad\text{for every }G\in\mathcal{G}.\] (7.6)

Existence follows from the Radon-Nikodym theorem. Two special cases anchor intuition: \(\E[X\mid\{\emptyset,\Omega\}]=\E X\) (no information), and if \(X\) is \(\mathcal{G}\)-measurable then \(\E[X\mid\mathcal{G}]=X\) (full information).

Key properties

Conditional expectation is linear, order-preserving, and satisfies:

Tower / iterated expectations: \(\mathcal{H}\subseteq\mathcal{G}\Rightarrow \E[\E[X\mid\mathcal{G}]\mid\mathcal{H}]=\E[X\mid\mathcal{H}]\); in particular \(\E[\E[X\mid\mathcal{G}]]=\E X\).
Taking out what is known: if \(Z\) is \(\mathcal{G}\)-measurable and bounded, \(\E[ZX\mid\mathcal{G}]=Z\,\E[X\mid\mathcal{G}]\).
Independence: if \(X\) is independent of \(\mathcal{G}\), then \(\E[X\mid\mathcal{G}]=\E X\).
\(L^2\) projection: among all \(\mathcal{G}\)-measurable \(Y\), \(\E[X\mid\mathcal{G}]\) minimizes \(\E[(X-Y)^2]\) - it is the best mean-square forecast.

Worked Example - Iterated expectations in action

Draw \(N\sim\)Poisson(\(\lambda\)) claims, each of independent size \(Y_i\) with mean \(\mu\); total loss \(S=\sum_{i=1}^N Y_i\).

Condition on \(N\): \(\E[S\mid N]=N\mu\), because with \(N\) fixed the sum has \(N\) i.i.d. terms of mean \(\mu\).

Apply the tower property: \(\E[S]=\E[\E[S\mid N]]=\E[N\mu]=\mu\,\E[N]=\mu\lambda.\)

This compound-Poisson mean is the backbone of insurance and operational-risk pricing - conditioning made a hard sum easy.

Interactive: Monte Carlo check of the tower property

Common Mistakes to Avoid

Confusing uncorrelated with independent: independence implies zero covariance, not conversely.
Treating \(\E[X\mid\mathcal{G}]\) as a number - it is a random variable (a function of the information).
Pulling a non-\(\mathcal{G}\)-measurable factor out of a conditional expectation.
Swapping the order in the tower property; you must condition down to the coarser sigma-algebra.

Quant Practitioner Tips

Whenever a sum has a random number of terms, condition on the count first - iterated expectations linearizes it.
Read \(\E[X\mid\mathcal{G}]\) as ‘the best forecast of \(X\) given what \(\mathcal{G}\) knows’; it is a projection.
The tower property is the single most-used identity in derivative pricing - today’s price is the conditional expectation of tomorrow’s.

Practice this in the Euler Lab

Computational problems that exercise exactly this technique. Each opens in the Euler Lab with a Python workbench, a progressive hint ladder, and answer checking. Tier A/B run at full scale in the browser.

Warm-up:
#59 XOR Decryption (4%, tier A) #79 Passcode Derivation (7%, tier A) #102 Triangle Containment (8%, tier A) #54 Poker Hands (9%, tier A)

Applied:
#938 Exhausting a Colour (16%, tier B) #816 Shortest Distance Among Points (17%, tier B) #323 Bitwise-OR Operations on Random In (18%, tier B)

Challenge:
#375 Minimum of Subsequences (41%, tier C) #666 Polymorphic Bacteria (41%, tier C)

124 Project Euler problems in total are mapped to this lesson. Open the Euler Lab to filter them all.

Knowledge Check

Q1 Medium

If \(X\) is independent of the sigma-algebra \(\mathcal{G}\), then \(\E[X\mid\mathcal{G}]\) equals:

\(X\)

\(0\)

\(\E[X]\) (a constant)

undefined

Q2 Easy

The tower property \(\E[\E[X\mid\mathcal{G}]]=\E[X]\) expresses that:

Conditioning changes the mean

Averaging the conditional forecasts recovers the unconditional mean

\(X\) must be \(\mathcal{G}\)-measurable

Conditional expectation is deterministic

Q3 Hard

Conditional expectation \(\E[X\mid\mathcal{G}]\) is characterized as the \(\mathcal{G}\)-measurable variable that:

Maximizes \(\E[XY]\)

Has the same integral as \(X\) over every set in \(\mathcal{G}\)

Equals \(X\) everywhere

Is always constant

Practical Exercise

A test for a disease is 99% sensitive and 95% specific; the disease prevalence is 0.5%. A random person tests positive. (a) Use Bayes to find the probability they actually have the disease. (b) Interpret.

▶ Show full solution

(a) Let \(D\) be disease, \(+\) a positive test. \(\Prob(+\mid D)=0.99\), \(\Prob(+\mid D^c)=0.05\), \(\Prob(D)=0.005\).

\[\Prob(D\mid +)=\frac{0.99\times0.005}{0.99\times0.005+0.05\times0.995}=\frac{0.00495}{0.00495+0.04975}\approx 0.0905.\]

(b) Despite a ‘99% accurate’ test, only about 9% of positives are true - because the base rate is tiny, false positives from the healthy 99.5% dominate. This is the base-rate fallacy, and it recurs in signal detection and alpha screening.

After the reveal, answer for yourself: How high must prevalence be before a positive test exceeds 50% posterior probability?

Lesson Summary

Independence factorizes joint laws; Bayes and total probability update beliefs. The central construct is conditional expectation given a sigma-algebra: the unique measurable variable matching \(X\)’s integrals on that information, equivalently the least-squares forecast. Its tower property and take-out rules are the algebra of information on which martingales and pricing are built.

Formula Sheet Additions

Bayes

\[\Prob(B_j\mid A)=\frac{\Prob(A\mid B_j)\Prob(B_j)}{\sum_i\Prob(A\mid B_i)\Prob(B_i)}\]

Invert the direction of conditioning.

Tower property

\[\mathcal{H}\subseteq\mathcal{G}\Rightarrow \E[\E[X\mid\mathcal{G}]\mid\mathcal{H}]=\E[X\mid\mathcal{H}]\]

Forecasts collapse to the coarser information.

Error Log Checklist

Did I keep \(\E[X\mid\mathcal{G}]\) as a random variable, not a number?
Did I check independence rather than mere zero correlation?
Did I only pull out factors that are \(\mathcal{G}\)-measurable?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: State the defining property of \(\E[X\mid\mathcal{G}]\).
A: It is the \(\mathcal{G}\)-measurable \(Y\) with \(\int_G Y\,d\Prob=\int_G X\,d\Prob\) for all \(G\in\mathcal{G}\); equivalently the \(L^2\) projection onto \(\mathcal{G}\)-measurable functions.

Q: Why does conditioning on the count simplify a compound sum?
A: Given \(N\), the sum has a fixed number of i.i.d. terms so \(\E[S\mid N]=N\mu\); the tower property then gives \(\E[S]=\mu\E[N]\).

Q: Does uncorrelated imply independent?
A: No. Independence implies zero covariance, but zero covariance can hold for dependent variables (e.g. \(X\) and \(X^2\) for symmetric \(X\)).

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

Conditional expectation

The \(\mathcal{G}\)-measurable least-squares forecast of \(X\); matches \(X\)’s integrals on \(\mathcal{G}\)-sets.

Tower property

\(\E[\E[X\mid\mathcal{G}]\mid\mathcal{H}]=\E[X\mid\mathcal{H}]\) for \(\mathcal{H}\subseteq\mathcal{G}\).

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

Probability: Theory and Examples (Rick Durrett, 5th ed.) current - Ch. 2,4 - Ch. 2 & 4: independence, conditional expectation via Radon-Nikodym.
Brownian Motion and Stochastic Calculus (Karatzas & Shreve, 2nd ed., GTM 113) foundational - Ch. 1 - Conditional expectation and filtrations as information.

← 7.2 Next: 7.4 →