Linear Regression: OLS, Geometry, Gauss-Markov, and Inference vs Prediction
The normal equations, least squares as an orthogonal projection, when OLS is best, and the crucial split between explaining and forecasting.
Leads to: Regularization (11.4) modifies these normal equations; CV (11.3) evaluates prediction.
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Derive the OLS estimator from the normal equations and from the projection viewpoint.
- Interpret least squares as orthogonal projection onto the column space via the hat matrix.
- State the Gauss-Markov theorem and the assumptions under which OLS is BLUE.
- Distinguish inference (coefficients, standard errors, confidence intervals) from prediction (out-of-sample accuracy).
- Compute an OLS fit numerically and verify it against the closed form.
Key Vocabulary
- Design matrix
- The \(n\times p\) matrix \(X\) of predictors (often with a leading column of ones for the intercept).
- Normal equations
- The linear system \(X^\top X\beta=X^\top y\) whose solution is the OLS estimate.
- Hat matrix
- The projection \(H=X(X^\top X)^{-1}X^\top\) mapping \(y\) to fitted values \(\hat y=Hy\).
- Residual
- The gap \(e=y-\hat y=(I-H)y\), orthogonal to every column of \(X\).
- BLUE
- Best Linear Unbiased Estimator: minimum variance among all linear unbiased estimators.
- Homoscedasticity
- Constant error variance \(\Var(\varepsilon_i)=\sigma^2\) across observations.
- Standard error
- The estimated standard deviation of a coefficient estimate, the basis of \(t\)-tests and intervals.
Intuition & Motivation
Two very different jobs share this machinery. Inference asks what the coefficients are and how sure we are - we need unbiasedness, standard errors, and honest assumptions. Prediction asks only whether \(\hat y\) is accurate on new data - biased coefficients are fine if they forecast better. Confusing the two is one of the most common and most expensive errors in applied quant work.
The model and the least-squares problem
Assume \(y=X\beta+\varepsilon\) with \(X\in\R^{n\times p}\) of full column rank and \(\E[\varepsilon]=0\). OLS chooses \(\beta\) to minimize the residual sum of squares:
Geometry: least squares is a projection
Fitted values are \(\hat y=X\hat\beta=X(X^\top X)^{-1}X^\top y=Hy\). The hat matrix \(H\) is the orthogonal projector onto \(\text{col}(X)\): it is symmetric, idempotent (\(H^2=H\)), and \(Hx=x\) for any column of \(X\). The residual \(e=(I-H)y\) lives in the orthogonal complement, so
Gauss-Markov: when OLS is best
Note what is not assumed: normality is not needed for BLUE (only for exact \(t\) and \(F\) distributions in finite samples). And ‘best linear unbiased’ leaves the door open - a biased nonlinear estimator (ridge, lasso) can beat OLS in MSE, which is precisely 11.4.
Inference: coefficients and their uncertainty
Under the Gauss-Markov assumptions plus Gaussian errors, \(\hat\beta\sim\Normal\big(\beta,\ \sigma^2(X^\top X)^{-1}\big)\). Estimating \(\sigma^2\) by \(\hat\sigma^2=\|e\|^2/(n-p)\) gives standard errors \(\mathrm{SE}(\hat\beta_j)=\hat\sigma\sqrt{[(X^\top X)^{-1}]_{jj}}\) and the \(t\)-statistic \(t_j=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\). This is the machinery of ‘is this factor significant?’
Prediction: a different target
For forecasting we do not care whether \(\hat\beta\) is unbiased; we care about \(\E[(y_0-x_0^\top\hat\beta)^2]\) at a new \(x_0\). This equals \(\sigma^2\) (irreducible) plus a model-variance term \(\sigma^2 x_0^\top(X^\top X)^{-1}x_0\) plus any bias-squared. Adding a slightly biased penalty can shrink the variance term and lower total prediction error.
| Question | Inference | Prediction |
|---|---|---|
| Goal | Interpret \(\beta\), test hypotheses | Accurate \(\hat y\) on unseen \(x_0\) |
| Needs unbiasedness? | Yes | No |
| Key output | SEs, \(t\)-stats, CIs | Out-of-sample error, CV score |
| Danger | Multicollinearity inflates SEs | Overfitting, look-ahead bias |
| Favors | OLS / GLS | Regularized / shrunk models |
Interactive: OLS from scratch
- Inverting \(X^\top X\) explicitly when predictors are collinear - it is near-singular; solve the system or use a QR/SVD instead.
- Reading a large coefficient as ‘important’ without its standard error; significance is \(\hat\beta_j/\mathrm{SE}\), not \(\hat\beta_j\).
- Reporting in-sample \(R^2\) as evidence of predictive skill; it never decreases as you add regressors.
- Assuming Gauss-Markov needs normality - it does not for BLUE; normality is only for exact finite-sample \(t\)/\(F\) inference.
- Interpreting a regression coefficient causally when predictors are correlated (omitted-variable bias).
- Decide up front: are you doing inference or prediction? The right model, diagnostics, and validation differ completely.
- For prediction, judge models by cross-validated error (11.3), never by in-sample fit.
- Financial predictors are often nearly collinear (many are lagged versions of each other); watch the condition number of \(X^\top X\).
- Heteroscedastic or autocorrelated errors (the norm in finance) break the naive SE formula - use robust (White/Newey-West) standard errors.
Knowledge Check
Practical Exercise
With \(\hat\beta=(X^\top X)^{-1}X^\top y\) and \(y=X\beta+\varepsilon\), \(\E[\varepsilon]=0\), \(\Var(\varepsilon)=\sigma^2 I\): (a) show OLS is unbiased; (b) derive \(\Var(\hat\beta)\); (c) explain why near-collinear predictors make individual coefficients unreliable but need not hurt prediction.
(a) \(\E[\hat\beta]=(X^\top X)^{-1}X^\top\E[y]=(X^\top X)^{-1}X^\top X\beta=\beta\). Unbiased.
(b) \(\hat\beta-\beta=(X^\top X)^{-1}X^\top\varepsilon\), so \(\Var(\hat\beta)=(X^\top X)^{-1}X^\top(\sigma^2 I)X(X^\top X)^{-1}=\sigma^2(X^\top X)^{-1}\).
(c) Collinearity makes \(X^\top X\) nearly singular, so \((X^\top X)^{-1}\) has huge entries and coefficient variances blow up - individual \(\hat\beta_j\) are unstable and can swap signs across samples. But the fitted values \(\hat y=Hy\) depend only on the column space, which is well-defined even when the basis is ill-conditioned; prediction can remain accurate. This is exactly why we shrink coefficients (ridge) when we only need \(\hat y\).
Lesson Summary
Formula Sheet Additions
- Did I check the condition number of \(X^\top X\) before trusting individual coefficients?
- Am I doing inference or prediction, and did I use the matching validation?
- Did I use robust SEs if errors look heteroscedastic or autocorrelated?
- Did I avoid reporting in-sample \(R^2\) as predictive evidence?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: \(\hat\beta=(X^\top X)^{-1}X^\top y\); fitted values \(\hat y=Hy\) are the orthogonal projection of \(y\) onto \(\text{col}(X)\).
A: OLS is minimum-variance among linear unbiased estimators (BLUE), assuming zero-mean uncorrelated homoscedastic errors; it does not require normality.
A: Near-collinear predictors: \((X^\top X)^{-1}\) blows up so coefficient variances explode, yet \(\hat y\) depends only on the column space and stays stable.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check
Source References
This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.
- The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 3 - Ch. 3: linear regression, least squares geometry, Gauss-Markov, subset selection.
- Time Series Analysis (James Hamilton, 1994) foundational - Hamilton 8 - OLS algebra and finite-sample vs asymptotic inference for time-series regressors.