Key Concepts in Time Series Models

Why This Matters

Time series models form the backbone of forecasting and dynamic analysis in quantitative methods. Whether you're predicting stock prices, analyzing economic indicators, or modeling climate patterns, understanding stationarity requirements, volatility dynamics, and multivariate relationships separates surface-level knowledge from genuine analytical skill.

Don't just memorize model acronyms and their parameters. Know why each model exists, what data characteristics it addresses, and when to deploy it over alternatives. You should be able to identify appropriate models given specific data features, interpret parameter meanings, and explain the theoretical foundations that make each approach valid.

---

Foundational Univariate Models

These models capture temporal dependence in a single variable using either past values, past errors, or both. The key distinction lies in whether the model looks backward at the variable itself or at the mistakes made in predicting it.

Autoregressive (AR) Models

An AR model predicts the current value of a series using a linear combination of its own past values. The general form of an AR($p$) model is:

$X_t = c + \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \epsilon_t$

• Past values drive predictions. The model uses $p$ lagged observations of the variable itself to forecast future values.
• Coefficients measure persistence. Each $\phi_i$ quantifies how strongly the value at time $t-i$ influences the current observation. A $\phi_1$ close to 1 means shocks persist for a long time; a $\phi_1$ close to 0 means they die out quickly.
• Stationarity is required. For an AR(1), this means $|\phi_1| < 1$. More generally, the roots of the characteristic polynomial must lie outside the unit circle. Without stationarity, forecasts can diverge.

Moving Average (MA) Models

Instead of regressing on its own past, an MA model regresses on past forecast errors (also called innovations or shocks). The general MA($q$) model is:

$X_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \cdots + \theta_q \epsilon_{t-q}$

• Past forecast errors drive predictions. The model uses $q$ lagged error terms rather than the variable's own history.
• Captures short-term dynamics. MA models excel at modeling temporary fluctuations that dissipate after $q$ periods. The autocorrelation function (ACF) of an MA($q$) process cuts off sharply at lag $q$.
• Invertibility condition. For valid estimation and a unique representation, the MA coefficients must satisfy constraints ensuring the process can be re-expressed in AR form. For MA(1), this requires $|\theta_1| < 1$.

Autoregressive Moving Average (ARMA) Models

ARMA combines both components into a single framework. An ARMA($p, q$) model is:

$X_t = c + \sum_{i=1}^{p} \phi_i X_{t-i} + \epsilon_t + \sum_{j=1}^{q} \theta_j \epsilon_{t-j}$

• Combines AR and MA components for more flexible modeling of complex autocorrelation structures.
• Parsimony advantage. An ARMA(1,1) can often achieve a better fit with just two parameters than a pure AR(5) or MA(5) would, because it captures both persistent and transient dynamics simultaneously.
• Stationary data only. ARMA assumes no trends, no unit roots, and no seasonal patterns in the series.

---

Handling Non-Stationarity

Real-world data rarely stays put. These models address trends, unit roots, and deterministic drift that violate the assumptions of basic ARMA frameworks.

Autoregressive Integrated Moving Average (ARIMA) Models

When a series has a unit root (a stochastic trend), you can't fit ARMA directly. ARIMA solves this by differencing the data first.

• Differencing creates stationarity. The $d$ parameter specifies how many times to difference the series before applying ARMA. First differencing ($d = 1$) transforms $X_t$ into $\Delta X_t = X_t - X_{t-1}$, which removes a linear stochastic trend.
• Notation: $ARIMA(p, d, q)$ combines autoregressive order, integration (differencing) order, and moving average order in one framework.
• Most common specification. $ARIMA(1,1,1)$ handles many economic series well. Use unit root tests like the Augmented Dickey-Fuller (ADF) test to determine whether differencing is needed and how many times.

Seasonal ARIMA (SARIMA) Models

Many series have patterns that repeat at regular intervals (quarters, months, weeks). SARIMA extends ARIMA to capture these.

• Adds seasonal differencing and lags. Parameters $(P, D, Q)_s$ capture patterns repeating every $s$ periods. For monthly data with annual seasonality, $s = 12$.
• Full notation: $SARIMA(p, d, q)(P, D, Q)_s$ separates non-seasonal and seasonal components explicitly. For example, $SARIMA(1,1,1)(1,1,1)_{12}$ applies first differencing and seasonal differencing at lag 12, with AR and MA terms at both levels.
• Essential for periodic data. Quarterly GDP, monthly retail sales, and daily temperature all exhibit predictable seasonal swings that pure ARIMA would miss.

---

Modeling Volatility Dynamics

Financial returns often exhibit volatility clustering: periods of high variance followed by more high variance. Standard ARMA-type models assume constant variance (homoskedasticity), so they can't capture this. These models treat variance itself as a time-varying process.

Generalized Autoregressive Conditional Heteroskedasticity (GARCH) Models

The standard $GARCH(1,1)$ model specifies the conditional variance as:

$\sigma_t^2 = \omega + \alpha \epsilon_{t-1}^2 + \beta \sigma_{t-1}^2$

where $\omega > 0$, $\alpha \geq 0$, $\beta \geq 0$, and $\alpha + \beta < 1$ for stationarity of the variance process.

• Variance depends on past shocks and past variance. The $\alpha$ term captures the impact of recent surprises on current volatility, while $\beta$ captures the persistence of volatility itself.
• Captures volatility clustering. A large $\epsilon_{t-1}^2$ raises $\sigma_t^2$, which in turn makes large values of $\epsilon_t$ more likely. This is exactly the clustering pattern seen in financial data.
• Foundation for risk management. Value-at-Risk calculations and option pricing rely heavily on GARCH-based volatility forecasts. Note that standard GARCH treats positive and negative shocks symmetrically; extensions like EGARCH and GJR-GARCH allow for asymmetric effects (the "leverage effect" where negative returns increase volatility more than positive returns).

Exponential Smoothing Models

Exponential smoothing takes a different approach: it produces forecasts as weighted averages of past observations, with weights that decay exponentially.

• Simple Exponential Smoothing (SES) uses a single smoothing parameter $\alpha \in (0,1)$. The forecast is $\hat{X}_{t+1} = \alpha X_t + (1 - \alpha) \hat{X}_t$. Higher $\alpha$ means more weight on recent data.
• Holt-Winters extends to trends and seasons. The additive version adds a trend component (smoothed by parameter $\beta$) and a seasonal component (smoothed by parameter $\gamma$). A multiplicative version handles seasons whose amplitude grows with the level of the series.
• Computationally simple and surprisingly effective. These models often serve as strong benchmarks in forecasting competitions despite their simplicity. They also have a formal connection to certain state space models (the ETS framework), which provides a proper likelihood-based foundation.

---

Multivariate and Regime-Based Models

When multiple variables interact or when the underlying data-generating process shifts over time, univariate frameworks are no longer sufficient.

Vector Autoregression (VAR) Models

A VAR($p$) model generalizes the univariate AR model to a system of $k$ variables. Each variable is regressed on $p$ lags of itself and $p$ lags of every other variable:

$\mathbf{Y}_t = \mathbf{c} + \mathbf{A}_1 \mathbf{Y}_{t-1} + \cdots + \mathbf{A}_p \mathbf{Y}_{t-p} + \mathbf{u}_t$

• All variables are treated as endogenous. There's no need to specify which variables are "dependent" and which are "independent" in advance.
• Granger causality testing. VAR enables formal tests of whether one variable's past helps predict another, beyond what the other variable's own past already explains.
• Impulse response functions (IRFs). These trace how a one-unit shock to one variable propagates through the entire system over time. IRFs are central to policy analysis in macroeconomics.

State Space Models

State space models separate what you observe from the unobserved process generating the data. They consist of two equations:

• Measurement equation: links the observed data $\mathbf{Y}_t$ to a vector of latent (hidden) states $\boldsymbol{\alpha}_t$.
• State equation: governs how the latent states evolve over time, typically as a first-order Markov process.

The Kalman filter is the recursive algorithm used to estimate the latent states. It updates state estimates each time a new observation arrives, making it naturally suited for real-time applications.

State space models handle missing data, irregular time spacing, and time-varying parameters gracefully. They also unify many other models: ARIMA, exponential smoothing (ETS), and structural time series models can all be written in state space form.

Markov Switching Models

Sometimes the data-generating process itself changes. A recession behaves differently from an expansion; a calm market behaves differently from a crisis.

• Regime-dependent parameters. Model coefficients (means, variances, or both) change based on an unobserved discrete state variable $S_t \in \{1, 2, \ldots, M\}$.
• Transition probabilities govern switching. The probability of moving from regime $i$ to regime $j$ is $p_{ij}$, and these probabilities follow a Markov chain (the next state depends only on the current state, not the full history).
• Captures structural breaks endogenously. Rather than imposing known break dates, the model infers when shifts occurred and estimates the probability of being in each regime at every point in time.

---

Quick Reference Table

---

Self-Check Questions

1. Which two models both use differencing to achieve stationarity, and what distinguishes the type of non-stationarity each addresses?

2. You observe that large forecast errors in a financial return series tend to cluster together. Which model family is designed for this phenomenon, and what is the key equation governing variance dynamics?

3. Compare and contrast VAR and State Space models: under what data conditions would you choose one over the other?

4. An FRQ presents quarterly retail sales data with both upward trend and repeating December spikes. Write the general notation for the appropriate model and explain what each parameter group captures.

5. A researcher suspects that interest rate dynamics fundamentally changed after a financial crisis but doesn't know exactly when. Which model allows the data to identify regime shifts endogenously, and what probabilistic structure governs transitions between states?