ARIMA and Forecasting
Differencing away a unit root, the Box-Jenkins loop, and how to produce point forecasts with honest, widening prediction intervals.
Leads to: Forecast-error variance recurs in the Kalman filter (12.5) and backtests (12.6).
Learning Objectives
Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.
- Define an ARIMA(\(p,d,q\)) model and explain the role of the differencing order \(d\).
- Test for a unit root and decide how many differences to take.
- Compute optimal linear forecasts and their error variance from an ARMA model.
- Show why forecast intervals widen with the horizon and converge to the unconditional variance.
- Explain why financial mean-forecasting is hard and easy to overfit.
Key Vocabulary
- ARIMA(\(p,d,q\))
- An ARMA(\(p,q\)) fitted to the series after differencing \(d\) times: \(\phi(L)(1-L)^d X_t=\theta(L)\varepsilon_t\).
- Integrated of order \(d\)
- A series \(I(d)\) that becomes stationary after \(d\) differences; a random walk is \(I(1)\).
- Differencing
- The transform \(\nabla X_t=X_t-X_{t-1}=(1-L)X_t\) that removes a unit root / linear trend.
- ADF test
- Augmented Dickey-Fuller: a hypothesis test whose null is a unit root (non-stationarity).
- h-step forecast
- The conditional expectation \(\hat X_{t+h\mid t}=\E[X_{t+h}\mid \mathcal F_t]\), the minimum-MSE predictor.
- Forecast-error variance
- The variance of \(X_{t+h}-\hat X_{t+h\mid t}\); grows with \(h\) toward the unconditional variance.
- Prediction interval
- A band, e.g. \(\hat X_{t+h\mid t}\pm1.96\,\sigma_h\), quantifying forecast uncertainty at horizon \(h\).
Intuition & Motivation
Forecasting from the fitted model is just conditional expectation: the best \(h\)-step forecast under squared loss is \(\E[X_{t+h}\mid\text{past}]\), which you build by iterating the recursion and setting future shocks to their mean of zero. The crucial honesty is in the interval: uncertainty compounds with the horizon. A one-step forecast is fairly tight; a twenty-step forecast for a stationary series fans out until it is no more informative than the unconditional mean. Reporting a point forecast without its widening band is the forecasting equivalent of a point estimate with no standard error.
From ARMA to ARIMA: differencing
If \(X_t\) has a unit root, \((1-L)X_t\) removes it. An ARIMA(\(p,d,q\)) posits that the \(d\)-th difference is a stationary, invertible ARMA:
Choose \(d\) by testing: the ADF test has a unit-root null, the KPSS test a stationarity null; use them together. Difference once, re-test, and stop as soon as the series is stationary. Log-transform first when the variance grows with the level (typical for prices). Beware over-differencing - it introduces a non-invertible MA unit root and inflates variance.
The Box-Jenkins loop
- Identify: transform to stationarity (log, difference); read ACF/PACF to propose \((p,d,q)\).
- Estimate: fit by MLE / conditional least squares; check that roots satisfy stationarity and invertibility.
- Diagnose: residuals must be white noise (ACF flat, Ljung-Box \(p\)-value large); if not, revise orders.
- Select & forecast: among adequate models pick the lowest AIC/BIC, then forecast with intervals - and validate out of time.
Optimal forecasts and their variance
For a general MA(\(\infty\)) representation \(X_t=\sum_k\psi_k\varepsilon_{t-k}\), the \(h\)-step forecast-error variance is \(\sigma_h^2=\sigma^2\sum_{j=0}^{h-1}\psi_j^2\), and the interval is \(\hat X_{t+h\mid t}\pm1.96\,\sigma_h\) under Gaussian shocks.
Interactive: iterate forecasts and watch the interval fan out
- Over-differencing (taking \(d\) too large); it creates a non-invertible MA unit root and needlessly inflates forecast variance.
- Reporting point forecasts with no interval, or with a constant interval that ignores horizon growth.
- Forecasting the level of an \(I(1)\) series with a stationary model’s saturating band - the true band grows like \(\sqrt h\).
- Choosing orders purely by in-sample AIC without out-of-time validation on financial data.
- Confusing a good in-sample fit with forecast skill; a random walk (forecast = last value) beats most elaborate models on prices.
- Use ADF and KPSS together; if they disagree, inspect the ACF and prefer the more parsimonious \(d\).
- Benchmark every forecast against the naive random-walk / last-value predictor - if you cannot beat it out of sample, you have no edge.
- For prices, model returns (already differenced logs) rather than levels; it sidesteps the unit root cleanly.
- Evaluate forecasts by out-of-time RMSE and interval coverage (are 95% intervals right ~95% of the time?), not in-sample fit.
Knowledge Check
Practical Exercise
You must forecast a stock price index. (a) Why fit ARIMA rather than ARMA, and what is a sensible \(d\)? (b) A colleague’s ARIMA(2,1,2) has in-sample RMSE far below a random walk but loses to it out of sample. Diagnose. (c) State how you would report a 10-day-ahead forecast honestly.
(a) The index has a unit root (prices wander), so ARMA on the level is invalid; ARIMA with \(d=1\) (equivalently modeling log-returns) makes the differenced series stationary. Usually \(d=1\) suffices; check with ADF/KPSS and avoid \(d\ge2\) unless clearly needed.
(b) Classic over-fitting: ARIMA(2,1,2) has enough parameters (and likely near-canceling AR/MA roots) to fit in-sample noise, but the true mean-predictability of returns is near zero, so it cannot beat the naive last-value forecast out of sample. The in-sample RMSE is optimistic (11.3); orders were chosen without out-of-time validation.
(c) Report the point forecast (for an \(I(1)\) level, close to the last value) with a prediction interval that widens with the horizon - for the differenced series the level-forecast band grows roughly like \(\sigma\sqrt{h}\). Include out-of-time RMSE versus the random-walk benchmark and the empirical coverage of the interval. If it does not beat the random walk, say so.
Lesson Summary
Formula Sheet Additions
- Did I test for a unit root and pick the smallest \(d\) that gives stationarity?
- Did I check for over-differencing (an MA unit root)?
- Did I report horizon-widening prediction intervals, not just points?
- Did I beat the random-walk benchmark out of time?
Retrieval Practice
Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.
A: It differences the series \(d\) times to remove unit roots; choose the smallest \(d\) making the series stationary (ADF/KPSS).
A: Stationary: saturates at the unconditional variance \(\sigma^2/(1-\phi^2)\). \(I(1)\) level: grows without bound like \(\sigma\sqrt h\).
A: The random walk / last-value forecast; if you cannot beat it out of sample you have no predictive edge.
Flashcards
Click to flip. These feed the site-wide spaced-repetition queue.
Completion Checklist
- I can explain the core ideas in my own words
- I worked the derivations/examples by hand
- I completed the interactive workbench(es)
- I passed the knowledge check