Linear Regression: OLS, Geometry, Gauss-Markov, and Inference vs Prediction
The normal equations, least squares as an orthogonal projection, when OLS is best, and the crucial split between explaining and forecasting.
Leads to: Regularization (11.4) modifies these normal equations; CV (11.3) evaluates prediction.
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Derive the OLS estimator from the normal equations and from the projection viewpoint.
- Interpret least squares as orthogonal projection onto the column space via the hat matrix.
- State the Gauss-Markov theorem and the assumptions under which OLS is BLUE.
- Distinguish inference (coefficients, standard errors, confidence intervals) from prediction (out-of-sample accuracy).
- Compute an OLS fit numerically and verify it against the closed form.
Key Vocabulary
- Design matrix
- The \(n\times p\) matrix \(X\) of predictors (often with a leading column of ones for the intercept).
- Normal equations
- The linear system \(X^\top X\beta=X^\top y\) whose solution is the OLS estimate.
- Hat matrix
- The projection \(H=X(X^\top X)^{-1}X^\top\) mapping \(y\) to fitted values \(\hat y=Hy\).
- Residual
- The gap \(e=y-\hat y=(I-H)y\), orthogonal to every column of \(X\).
- BLUE
- Best Linear Unbiased Estimator: minimum variance among all linear unbiased estimators.
- Homoscedasticity
- Constant error variance \(\Var(\varepsilon_i)=\sigma^2\) across observations.
- Standard error
- The estimated standard deviation of a coefficient estimate, the basis of \(t\)-tests and intervals.
Intuition & Motivation
Two very different jobs share this machinery. Inference asks what the coefficients are and how sure we are - we need unbiasedness, standard errors, and honest assumptions. Prediction asks only whether \(\hat y\) is accurate on new data - biased coefficients are fine if they forecast better. Confusing the two is one of the most common and most expensive errors in applied quant work.
The model and the least-squares problem
Assume \(y=X\beta+\varepsilon\) with \(X\in\R^{n\times p}\) of full column rank and \(\E[\varepsilon]=0\). OLS chooses \(\beta\) to minimize the residual sum of squares:
Geometry: least squares is a projection
Fitted values are \(\hat y=X\hat\beta=X(X^\top X)^{-1}X^\top y=Hy\). The hat matrix \(H\) is the orthogonal projector onto \(\text{col}(X)\): it is symmetric, idempotent (\(H^2=H\)), and \(Hx=x\) for any column of \(X\). The residual \(e=(I-H)y\) lives in the orthogonal complement, so
Gauss-Markov: when OLS is best
Note what is not assumed: normality is not needed for BLUE (only for exact \(t\) and \(F\) distributions in finite samples). And ‘best linear unbiased’ leaves the door open - a biased nonlinear estimator (ridge, lasso) can beat OLS in MSE, which is precisely 11.4.
Inference: coefficients and their uncertainty
Under the Gauss-Markov assumptions plus Gaussian errors, \(\hat\beta\sim\Normal\big(\beta,\ \sigma^2(X^\top X)^{-1}\big)\). Estimating \(\sigma^2\) by \(\hat\sigma^2=\|e\|^2/(n-p)\) gives standard errors \(\mathrm{SE}(\hat\beta_j)=\hat\sigma\sqrt{[(X^\top X)^{-1}]_{jj}}\) and the \(t\)-statistic \(t_j=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\). This is the machinery of ‘is this factor significant?’
Prediction: a different target
For forecasting we do not care whether \(\hat\beta\) is unbiased; we care about \(\E[(y_0-x_0^\top\hat\beta)^2]\) at a new \(x_0\). This equals \(\sigma^2\) (irreducible) plus a model-variance term \(\sigma^2 x_0^\top(X^\top X)^{-1}x_0\) plus any bias-squared. Adding a slightly biased penalty can shrink the variance term and lower total prediction error.
| Question | Inference | Prediction |
|---|---|---|
| Goal | Interpret \(\beta\), test hypotheses | Accurate \(\hat y\) on unseen \(x_0\) |
| Needs unbiasedness? | Yes | No |
| Key output | SEs, \(t\)-stats, CIs | Out-of-sample error, CV score |
| Danger | Multicollinearity inflates SEs | Overfitting, look-ahead bias |
| Favors | OLS / GLS | Regularized / shrunk models |
Interactive: OLS from scratch
- Inverting \(X^\top X\) explicitly when predictors are collinear - it is near-singular; solve the system or use a QR/SVD instead.
- Reading a large coefficient as ‘important’ without its standard error; significance is \(\hat\beta_j/\mathrm{SE}\), not \(\hat\beta_j\).
- Reporting in-sample \(R^2\) as evidence of predictive skill; it never decreases as you add regressors.
- Assuming Gauss-Markov needs normality - it does not for BLUE; normality is only for exact finite-sample \(t\)/\(F\) inference.
- Interpreting a regression coefficient causally when predictors are correlated (omitted-variable bias).
- Decide up front: are you doing inference or prediction? The right model, diagnostics, and validation differ completely.
- For prediction, judge models by cross-validated error (11.3), never by in-sample fit.
- Financial predictors are often nearly collinear (many are lagged versions of each other); watch the condition number of \(X^\top X\).
- Heteroscedastic or autocorrelated errors (the norm in finance) break the naive SE formula - use robust (White/Newey-West) standard errors.
Knowledge Check
Practical Exercise
With \(\hat\beta=(X^\top X)^{-1}X^\top y\) and \(y=X\beta+\varepsilon\), \(\E[\varepsilon]=0\), \(\Var(\varepsilon)=\sigma^2 I\): (a) show OLS is unbiased; (b) derive \(\Var(\hat\beta)\); (c) explain why near-collinear predictors make individual coefficients unreliable but need not hurt prediction.
(a) \(\E[\hat\beta]=(X^\top X)^{-1}X^\top\E[y]=(X^\top X)^{-1}X^\top X\beta=\beta\). Unbiased.
(b) \(\hat\beta-\beta=(X^\top X)^{-1}X^\top\varepsilon\), so \(\Var(\hat\beta)=(X^\top X)^{-1}X^\top(\sigma^2 I)X(X^\top X)^{-1}=\sigma^2(X^\top X)^{-1}\).
(c) Collinearity makes \(X^\top X\) nearly singular, so \((X^\top X)^{-1}\) has huge entries and coefficient variances blow up - individual \(\hat\beta_j\) are unstable and can swap signs across samples. But the fitted values \(\hat y=Hy\) depend only on the column space, which is well-defined even when the basis is ill-conditioned; prediction can remain accurate. This is exactly why we shrink coefficients (ridge) when we only need \(\hat y\).
Lesson Summary
Formula Sheet Additions
- Did I check the condition number of \(X^\top X\) before trusting individual coefficients?
- Am I doing inference or prediction, and did I use the matching validation?
- Did I use robust SEs if errors look heteroscedastic or autocorrelated?
- Did I avoid reporting in-sample \(R^2\) as predictive evidence?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: \(\hat\beta=(X^\top X)^{-1}X^\top y\); fitted values \(\hat y=Hy\) are the orthogonal projection of \(y\) onto \(\text{col}(X)\).
A: OLS is minimum-variance among linear unbiased estimators (BLUE), assuming zero-mean uncorrelated homoscedastic errors; it does not require normality.
A: Near-collinear predictors: \((X^\top X)^{-1}\) blows up so coefficient variances explode, yet \(\hat y\) depends only on the column space and stays stable.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check