Phase 12 - Lesson 12.2

AR, MA, and ARMA Models

The three linear building blocks of stationary time series, their stationarity and invertibility conditions, and how to fit them.

⏱ 55 min● Intermediate🔗 Prereqs: 12.1

↖ Phase 12 hub

Builds on: 12.1 gave stationarity and the ACF/PACF identification signatures.
Leads to: ARIMA (12.3) adds differencing; GARCH (12.4) applies ARMA logic to variance.

Learning Objectives

Click a status chip to cycle: Not started → In progress → Studied → Practiced → Needs review → Mastered.

Write AR, MA, and ARMA models in lag-operator form and state their stationarity/invertibility conditions.
Derive the Yule-Walker equations linking AR coefficients to the ACF.
Explain the duality: a stationary AR is an infinite MA, and an invertible MA is an infinite AR.
Fit an AR model by least squares / Yule-Walker and interpret the coefficients.
Select ARMA orders using ACF/PACF and information criteria.

Key Vocabulary

AR(\(p\)): Autoregression: \(X_t=\sum_{i=1}^p\phi_i X_{t-i}+\varepsilon_t\); today is a linear function of its own past plus a shock.

MA(\(q\)): Moving average: \(X_t=\varepsilon_t+\sum_{j=1}^q\theta_j\varepsilon_{t-j}\); today is a weighted window of recent shocks.

ARMA(\(p,q\)): Combines both: \(\phi(L)X_t=\theta(L)\varepsilon_t\); parsimonious for series with both features.

Lag operator: \(L\) with \(LX_t=X_{t-1}\); polynomials \(\phi(L),\theta(L)\) encode the model compactly.

Stationarity condition: AR is stationary iff all roots of \(\phi(z)=0\) lie strictly outside the unit circle.

Invertibility: MA is invertible (writable as a convergent AR(\(\infty\))) iff all roots of \(\theta(z)=0\) lie strictly outside the unit circle.

Yule-Walker equations: Linear system \(\rho(k)=\sum_i\phi_i\rho(k-i)\) relating AR coefficients to the ACF; the basis of one estimator.

Intuition & Motivation

Intuition

There are two elementary ways a series can have memory. It can depend on its own recent values - today is mostly yesterday, decayed - which is autoregression. Or it can depend on recent shocks - a surprise this week still echoes for a few weeks and then is gone - which is a moving average. AR memory fades geometrically and never fully disappears; MA memory is finite, cut off after \(q\) lags. ARMA simply allows both mechanisms at once, often describing a series with far fewer parameters than a pure AR or MA would need.

The deep fact is a duality: a stable AR unrolls into an infinite MA of past shocks, and an invertible MA rolls up into an infinite AR of past values. ‘Stable’ and ‘invertible’ both mean the same thing algebraically - the roots of the relevant polynomial sit outside the unit circle, so the geometric series converge. That single condition is what separates a well-behaved forecastable model from an explosive or non-stationary one.

Lag-operator form

Let \(L\) denote the lag operator, \(LX_t=X_{t-1}\). Define \(\phi(L)=1-\phi_1 L-\dots-\phi_p L^p\) and \(\theta(L)=1+\theta_1 L+\dots+\theta_q L^q\). The three models are

\[\text{AR}(p):\ \phi(L)X_t=\varepsilon_t,\qquad \text{MA}(q):\ X_t=\theta(L)\varepsilon_t,\qquad \text{ARMA}(p,q):\ \phi(L)X_t=\theta(L)\varepsilon_t.\] (12.3)

Stationarity, invertibility, and duality

An AR(\(p\)) is (weakly) stationary iff every root of the characteristic polynomial \(\phi(z)=0\) lies strictly outside the unit circle (\(|z|\gt 1\)). Then \(X_t=\phi(L)^{-1}\varepsilon_t=\sum_{k\ge0}\psi_k\varepsilon_{t-k}\) is a convergent MA(\(\infty\)). Symmetrically, an MA(\(q\)) is invertible - expressible as a convergent AR(\(\infty\)) - iff the roots of \(\theta(z)=0\) lie outside the unit circle. Invertibility matters because forecasting needs to recover the unobservable shocks \(\varepsilon_t\) from observed data.

Worked Example - Deriving the Yule-Walker equations for AR(p)

Start from \(X_t=\sum_{i=1}^p\phi_i X_{t-i}+\varepsilon_t\) (mean-zero, stationary).

Multiply by \(X_{t-k}\) (\(k\ge1\)) and take expectations; since \(\E[\varepsilon_t X_{t-k}]=0\) for \(k\ge1\): \(\gamma(k)=\sum_{i=1}^p\phi_i\gamma(k-i)\).

Divide by \(\gamma(0)\): the Yule-Walker system \(\rho(k)=\sum_{i=1}^p\phi_i\rho(k-i)\), \(k=1,\dots,p\).

In matrix form \(R\phi=r\) with \(R_{ij}=\rho(i-j)\), \(r_k=\rho(k)\): solve for \(\phi\) by plugging in sample ACFs - the Yule-Walker estimator.

At \(k=0\) the same algebra gives the innovation variance \(\sigma^2=\gamma(0)\big(1-\sum_i\phi_i\rho(i)\big)\). Estimating AR by conditional least squares (regress \(X_t\) on its \(p\) lags) gives nearly identical results and extends easily to ARMA via nonlinear/MLE fitting.

Order selection

Combine three tools: (1) the ACF/PACF signatures from 12.1 - PACF cutoff at \(p\) for AR, ACF cutoff at \(q\) for MA; (2) information criteria (AIC for prediction, BIC for parsimony) over a grid of \((p,q)\); and (3) residual diagnostics - a fitted model’s residuals should be white noise (flat ACF, Ljung-Box \(p\)-value large). Prefer the simplest model whose residuals pass.

Key Idea

Parsimony is not aesthetic here: over-parameterized ARMA models have near-canceling AR/MA roots, unstable coefficients, and terrible out-of-sample forecasts. On low-signal financial data, favor small \((p,q)\) and validate forward in time.

Interactive: identify and fit AR

Now recover AR coefficients from data by least squares:

Common Mistakes to Avoid

Fitting an AR to a non-stationary series; coefficients drift and the \(|z|\gt 1\) root condition is violated - difference first (12.3).
Over-fitting ARMA orders; near-canceling roots give unstable, non-identified parameters and poor forecasts.
Forgetting invertibility: a non-invertible MA cannot be inverted to recover shocks, so its forecasts are ill-defined.
Trusting coefficient standard errors without checking that residuals are white noise (Ljung-Box).
Reading a strong in-sample fit as forecastability - on returns the true AR/MA structure is weak and easily overfit.

Quant Practitioner Tips

Identify with ACF/PACF, estimate a small grid, then confirm with residual white-noise tests - the Box-Jenkins loop.
Yule-Walker and conditional least squares agree in large samples; MLE is preferred for ARMA with an MA part.
Check the fitted roots: if an AR and MA root nearly cancel, drop an order - the model is over-specified.
For financial returns, expect tiny AR/MA coefficients; the action is usually in the variance (GARCH), not the mean.

Knowledge Check

Q1 Medium

An AR(\(p\)) process is stationary if and only if:

All \(|\phi_i|\lt 1\)

The roots of \(\phi(z)=0\) lie strictly outside the unit circle

The roots of \(\theta(z)=0\) lie inside the unit circle

The mean is zero

Q2 Medium

The Yule-Walker equations relate:

MA coefficients to the PACF

AR coefficients to the autocorrelations via \(\rho(k)=\sum_i\phi_i\rho(k-i)\)

The mean to the variance

Residuals to the forecast

Q3 Medium

A stationary AR(1) can equivalently be written as:

A finite MA(\(1\)

An infinite, convergent MA(\(\infty\)) of past shocks

A random walk

A non-invertible MA

Practical Exercise

Consider the MA(1) \(X_t=\varepsilon_t+\theta\varepsilon_{t-1}\), \(\Var(\varepsilon_t)=\sigma^2\). (a) Compute \(\gamma(0),\gamma(1),\gamma(h)\) for \(h\ge2\) and the ACF. (b) For what \(\theta\) is it invertible? (c) Show \(\theta\) and \(1/\theta\) give the same ACF, and explain why invertibility resolves the ambiguity.

▶ Show full solution

(a) \(\gamma(0)=\Var(\varepsilon_t+\theta\varepsilon_{t-1})=(1+\theta^2)\sigma^2\); \(\gamma(1)=\Cov(\varepsilon_t+\theta\varepsilon_{t-1},\varepsilon_{t-1}+\theta\varepsilon_{t-2})=\theta\sigma^2\); \(\gamma(h)=0\) for \(h\ge2\) (no shared shocks). So \(\rho(1)=\theta/(1+\theta^2)\), \(\rho(h)=0\) for \(h\ge2\) - the MA(1) ACF cutoff.

(b) Invertible iff the root of \(1+\theta z=0\), namely \(z=-1/\theta\), lies outside the unit circle, i.e. \(|\theta|\lt 1\).

(c) Replacing \(\theta\) by \(1/\theta\) leaves \(\rho(1)=\theta/(1+\theta^2)\) unchanged (the expression is symmetric under \(\theta\leftrightarrow1/\theta\)), so two different MA(1) models share an ACF and are observationally equivalent in second moments. Requiring \(|\theta|\lt 1\) (invertibility) selects the unique representation whose innovations can be recovered from the past - the one usable for forecasting.

After the reveal, answer for yourself: Invertibility is not a technicality: it is what makes the shock sequence, and hence the forecast, recoverable and unique.

Lesson Summary

AR, MA, and ARMA are the linear building blocks of stationary series, written compactly as \(\phi(L)X_t=\theta(L)\varepsilon_t\). Stationarity needs the AR roots outside the unit circle; invertibility needs the MA roots outside it, and the two conditions power the AR/MA duality (each is an infinite version of the other). The Yule-Walker equations and conditional least squares estimate AR coefficients from the ACF, and orders are chosen by ACF/PACF signatures, information criteria, and white-noise residual checks - kept parsimonious, especially for weak-signal financial data.

Formula Sheet Additions

ARMA lag form

\[\phi(L)X_t=\theta(L)\varepsilon_t\]

Compact model; roots of \(\phi\)/\(\theta\) give stationarity/invertibility.

Yule-Walker

\[\rho(k)=\textstyle\sum_{i}\phi_i\,\rho(k-i)\]

Solve \(R\phi=r\) for AR coefficients from the ACF.

Error Log Checklist

Did I confirm stationarity (AR roots) and invertibility (MA roots) of the fitted model?
Are the residuals white noise (flat ACF, Ljung-Box passes)?
Did I check for near-canceling AR/MA roots (over-specification)?
Did I keep the orders small and validate forward in time?

Retrieval Practice

Close the lesson and answer from memory before checking. This is deliberate, effortful recall - the single highest-yield study action.

▶ Show retrieval prompts & answers

Q: Give the stationarity condition for AR and the invertibility condition for MA.
A: AR stationary iff roots of \(\phi(z)=0\) lie outside the unit circle; MA invertible iff roots of \(\theta(z)=0\) lie outside the unit circle.

Q: Write the Yule-Walker equations and say what they estimate.
A: \(\rho(k)=\sum_i\phi_i\rho(k-i)\) for \(k=1,\dots,p\); solving \(R\phi=r\) gives the AR coefficients from the sample ACF.

Q: State the AR/MA duality.
A: A stationary AR is a convergent MA(\(\infty\)) of past shocks; an invertible MA is a convergent AR(\(\infty\)) of past values.

Flashcards

Click to flip. These feed the site-wide spaced-repetition queue.

AR/MA/ARMA

\(\phi(L)X_t=\theta(L)\varepsilon_t\); AR = past values, MA = past shocks, ARMA = both.

Root conditions

Stationary: AR roots \(|z|\gt 1\). Invertible: MA roots \(|z|\gt 1\).

Yule-Walker

\(\rho(k)=\sum_i\phi_i\rho(k-i)\); estimates AR coefficients from the ACF.

Completion Checklist

I can explain the core ideas in my own words
I worked the derivations/examples by hand
I completed the interactive workbench(es)
I passed the knowledge check

Confidence / mastery rating

Personal notes

Source References

This lesson synthesizes and paraphrases concepts from the sources below. No copyrighted text, problem sets, or solutions are reproduced. Return to the originals for full depth.

Time Series Analysis (James Hamilton, 1994) foundational - Hamilton 3-4 - Ch. 3-4: ARMA processes, lag operators, Yule-Walker, and estimation.
The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, 2nd ed.) current - ESL 3 - Time-series regression viewpoint and least-squares estimation of autoregressions.

← 12.1 Next: 12.3 →