ARIMA and Forecasting
Differencing away a unit root, the Box-Jenkins loop, and how to produce point forecasts with honest, widening prediction intervals.
Leads to: Forecast-error variance recurs in the Kalman filter (12.5) and backtests (12.6).
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Define an ARIMA(\(p,d,q\)) model and explain the role of the differencing order \(d\).
- Test for a unit root and decide how many differences to take.
- Compute optimal linear forecasts and their error variance from an ARMA model.
- Show why forecast intervals widen with the horizon and converge to the unconditional variance.
- Explain why financial mean-forecasting is hard and easy to overfit.
Key Vocabulary
- ARIMA(\(p,d,q\))
- An ARMA(\(p,q\)) fitted to the series after differencing \(d\) times: \(\phi(L)(1-L)^d X_t=\theta(L)\varepsilon_t\).
- Integrated of order \(d\)
- A series \(I(d)\) that becomes stationary after \(d\) differences; a random walk is \(I(1)\).
- Differencing
- The transform \(\nabla X_t=X_t-X_{t-1}=(1-L)X_t\) that removes a unit root / linear trend.
- ADF test
- Augmented Dickey-Fuller: a hypothesis test whose null is a unit root (non-stationarity).
- h-step forecast
- The conditional expectation \(\hat X_{t+h\mid t}=\E[X_{t+h}\mid \mathcal F_t]\), the minimum-MSE predictor.
- Forecast-error variance
- The variance of \(X_{t+h}-\hat X_{t+h\mid t}\); grows with \(h\) toward the unconditional variance.
- Prediction interval
- A band, e.g. \(\hat X_{t+h\mid t}\pm1.96\,\sigma_h\), quantifying forecast uncertainty at horizon \(h\).
Intuition & Motivation
Forecasting from the fitted model is just conditional expectation: the best \(h\)-step forecast under squared loss is \(\E[X_{t+h}\mid\text{past}]\), which you build by iterating the recursion and setting future shocks to their mean of zero. The crucial honesty is in the interval: uncertainty compounds with the horizon. A one-step forecast is fairly tight; a twenty-step forecast for a stationary series fans out until it is no more informative than the unconditional mean. Reporting a point forecast without its widening band is the forecasting equivalent of a point estimate with no standard error.
From ARMA to ARIMA: differencing
If \(X_t\) has a unit root, \((1-L)X_t\) removes it. An ARIMA(\(p,d,q\)) posits that the \(d\)-th difference is a stationary, invertible ARMA:
Choose \(d\) by testing: the ADF test has a unit-root null, the KPSS test a stationarity null; use them together. Difference once, re-test, and stop as soon as the series is stationary. Log-transform first when the variance grows with the level (typical for prices). Beware over-differencing - it introduces a non-invertible MA unit root and inflates variance.
The Box-Jenkins loop
- Identify: transform to stationarity (log, difference); read ACF/PACF to propose \((p,d,q)\).
- Estimate: fit by MLE / conditional least squares; check that roots satisfy stationarity and invertibility.
- Diagnose: residuals must be white noise (ACF flat, Ljung-Box \(p\)-value large); if not, revise orders.
- Select & forecast: among adequate models pick the lowest AIC/BIC, then forecast with intervals - and validate out of time.
Optimal forecasts and their variance
For a general MA(\(\infty\)) representation \(X_t=\sum_k\psi_k\varepsilon_{t-k}\), the \(h\)-step forecast-error variance is \(\sigma_h^2=\sigma^2\sum_{j=0}^{h-1}\psi_j^2\), and the interval is \(\hat X_{t+h\mid t}\pm1.96\,\sigma_h\) under Gaussian shocks.
Interactive: iterate forecasts and watch the interval fan out
- Over-differencing (taking \(d\) too large); it creates a non-invertible MA unit root and needlessly inflates forecast variance.
- Reporting point forecasts with no interval, or with a constant interval that ignores horizon growth.
- Forecasting the level of an \(I(1)\) series with a stationary model’s saturating band - the true band grows like \(\sqrt h\).
- Choosing orders purely by in-sample AIC without out-of-time validation on financial data.
- Confusing a good in-sample fit with forecast skill; a random walk (forecast = last value) beats most elaborate models on prices.
- Use ADF and KPSS together; if they disagree, inspect the ACF and prefer the more parsimonious \(d\).
- Benchmark every forecast against the naive random-walk / last-value predictor - if you cannot beat it out of sample, you have no edge.
- For prices, model returns (already differenced logs) rather than levels; it sidesteps the unit root cleanly.
- Evaluate forecasts by out-of-time RMSE and interval coverage (are 95% intervals right ~95% of the time?), not in-sample fit.
Knowledge Check
Practical Exercise
You must forecast a stock price index. (a) Why fit ARIMA rather than ARMA, and what is a sensible \(d\)? (b) A colleague’s ARIMA(2,1,2) has in-sample RMSE far below a random walk but loses to it out of sample. Diagnose. (c) State how you would report a 10-day-ahead forecast honestly.
(a) The index has a unit root (prices wander), so ARMA on the level is invalid; ARIMA with \(d=1\) (equivalently modeling log-returns) makes the differenced series stationary. Usually \(d=1\) suffices; check with ADF/KPSS and avoid \(d\ge2\) unless clearly needed.
(b) Classic over-fitting: ARIMA(2,1,2) has enough parameters (and likely near-canceling AR/MA roots) to fit in-sample noise, but the true mean-predictability of returns is near zero, so it cannot beat the naive last-value forecast out of sample. The in-sample RMSE is optimistic (11.3); orders were chosen without out-of-time validation.
(c) Report the point forecast (for an \(I(1)\) level, close to the last value) with a prediction interval that widens with the horizon - for the differenced series the level-forecast band grows roughly like \(\sigma\sqrt{h}\). Include out-of-time RMSE versus the random-walk benchmark and the empirical coverage of the interval. If it does not beat the random walk, say so.
Lesson Summary
Formula Sheet Additions
- Did I test for a unit root and pick the smallest \(d\) that gives stationarity?
- Did I check for over-differencing (an MA unit root)?
- Did I report horizon-widening prediction intervals, not just points?
- Did I beat the random-walk benchmark out of time?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: It differences the series \(d\) times to remove unit roots; choose the smallest \(d\) making the series stationary (ADF/KPSS).
A: Stationary: saturates at the unconditional variance \(\sigma^2/(1-\phi^2)\). \(I(1)\) level: grows without bound like \(\sigma\sqrt h\).
A: The random walk / last-value forecast; if you cannot beat it out of sample you have no predictive edge.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check
Source References
This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.
- Time Series Analysis (James Hamilton, 1994) foundational - Hamilton 4-5 - Ch. 4-5: forecasting, ARIMA, unit roots and the mechanics of prediction-error variance.
- The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 7 - Prediction error, optimism, and out-of-sample evaluation applied to forecasting.