Hamilton Regime-Switching Model Guide (2026)

What Is the Hamilton Regime-Switching Model?

In March 1989, James D. Hamilton published A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle in Econometrica, volume 57(2), pages 357 to 384. His application used quarterly US real GNP growth from 1951 to 1984 and estimated recurring expansion and contraction states.

Hamilton did not invent every switching regression. His contribution was to make an unobserved Markov state practical for dependent time series, specifically an autoregression, through a recursive likelihood filter. The original model was an AR(4) with a regime-dependent mean and common innovation variance, not merely a two-Gaussian mixture.

A Markov regime-switching model lets selected parameters depend on a discrete latent state $s_t$. That can describe abrupt changes in mean, variance or autoregressive dynamics. It can also carve a continuous, heavy-tailed process into attractive-looking coloured bands that have little forecasting value. The filter is useful; the regime story must earn its keep out of sample. Compare the approach with continuous conditional variance in our GARCH guide.

---

Model, Probabilities and Durations

For a simple two-state switching mean and variance,

$$y_t=\mu_{s_t}+\sigma_{s_t}\varepsilon_t, \qquad \varepsilon_t\sim\mathcal N(0,1), \qquad s_t\in\{1,2\}.$$

The latent state follows a first-order Markov chain with row-stochastic transition matrix

$$P= \begin{bmatrix} p_{11}&1-p_{11}\\ 1-p_{22}&p_{22} \end{bmatrix}, \qquad p_{ij}=\Pr(s_t=j\mid s_{t-1}=i).$$

If $p_{ii}<1$, state duration is geometrically distributed and its expected length is

$$\mathbb E[D_i]=\frac{1}{1-p_{ii}}.$$

For $p_{11}=0.98$ and $p_{22}=0.92$, expected durations are 50 and 12.5 observations. The stationary state probabilities are 80% and 20%. These are model-implied averages, not promises that the next episode will last exactly that long. A standard Markov chain is memoryless: conditional on remaining in a state until today, its exit probability tomorrow does not depend on the episode's age.

Hamilton's GNP specification adds dependence:

$$y_t=\mu_{s_t} +\sum_{j=1}^{4}\phi_j\bigl(y_{t-j}-\mu_{s_{t-j}}\bigr) +\varepsilon_t.$$

Only the mean switches in this version. Because lagged deviations depend on past states, the filter tracks combinations of current and lagged regimes. A simple mixture likelihood is not enough.

---

Hamilton Filtering and Kim Smoothing

Let $\xi_{t-1|t-1}(i)=\Pr(s_{t-1}=i\mid y_{1:t-1})$. The one-step state prediction is

$$\xi_{t|t-1}(j)=\sum_i p_{ij}\xi_{t-1|t-1}(i).$$

Given state-conditional density $f_j(y_t)$, the observation's predictive density is

$$f(y_t\mid y_{1:t-1})= \sum_j f_j(y_t)\xi_{t|t-1}(j),$$

and Bayes' rule updates

$$\xi_{t|t}(j)= \frac{f_j(y_t)\xi_{t|t-1}(j)} {\sum_k f_k(y_t)\xi_{t|t-1}(k)}.$$

Summing the log predictive densities gives the log likelihood for numerical maximum likelihood. EM is one possible estimation method for latent-state models; it is not the definition of Hamilton estimation and direct optimisation is common.

Filtered probabilities condition only on data through $t$, so they can be used in a real-time rule. Smoothed probabilities condition on the full sample:

$$\xi_{t|T}(i)=\xi_{t|t}(i) \sum_j \frac{p_{ij}\xi_{t+1|T}(j)} {\xi_{t+1|t}(j)}.$$

This backwards recursion is associated with Kim's smoother. It is valuable for historical state classification, but a strategy that uses $\xi_{t|T}$ at time $t$ looks ahead. The statsmodels switching-model example exposes both probability series and demonstrates their different uses.

---

Python and a real-time decision rule

For daily returns, a two-state model with switching mean and variance is a transparent starting point. The code below fits several random initialisations because the likelihood can have local optima.

import pandas as pd
import statsmodels.api as sm


def fit_return_regimes(returns: pd.Series):
    y = returns.dropna().astype(float)
    model = sm.tsa.MarkovRegression(
        y, k_regimes=2, trend="c", switching_variance=True
    )
    result = model.fit(search_reps=20, disp=False)
    return {
        "result": result,
        "filtered": result.filtered_marginal_probabilities,
        "smoothed": result.smoothed_marginal_probabilities,
        "expected_durations": result.expected_durations,
    }

Do not assume state zero means "calm". State labels are arbitrary and can switch between fits. Order them after estimation by a parameter such as $\sigma_i$, then apply the same mapping to probabilities and transition rows.

A causal allocation might reduce risk when yesterday's filtered probability of the high-volatility state exceeds a threshold:

fit = fit_return_regimes(daily_returns)
high_state = 1  # determine from fitted variances, never assume
high_vol_prob = fit["filtered"][high_state]
target_risk = (1.0 - 0.5 * high_vol_prob).shift(1)

The shift matters. A probability estimated with today's close cannot receive today's close as an execution price. Threshold, exposure reduction and state count must all be selected on training data. Better still, compare continuous probability-weighted exposure with a hard switch; throwing away probability information at 0.5 can create needless turnover.

To reproduce Hamilton's AR(4), use sm.tsa.MarkovAutoregression with order=4 and switching_ar=False. Statsmodels provides a worked replication rather than forcing an independent-mixture model onto the original paper.

Hamilton reported that a typical recession regime in his specification was associated with a permanent GNP-level loss of about 3%. That is a result for a particular post-war US macro sample. It is not a calibration for equity regimes.

Forecasts and diagnostics

A fitted transition matrix gives tomorrow's state prior before tomorrow's return arrives. If today's filtered high-volatility probability is 70%, $p_{low,high}=0.02$ and $p_{high,high}=0.92$, tomorrow's predicted high-state probability is $0.30(0.02)+0.70(0.92)=0.65$. That 65% forecast, not today's smoothed classification, is the quantity available for advance risk planning.

Evaluate probability forecasts with proper scores. The one-step log predictive density rewards a model that assigns high density to realised returns and penalises unjustified certainty. Brier scores can assess a state proxy only when an external outcome is defined without using the fitted states themselves. Calling whichever state produced a large negative return "crisis" and then claiming accurate crisis classification is circular.

Check the probability integral transform of one-step predictive distributions, residual autocorrelation and tail coverage. For a switching autoregression, inspect residuals conditional on inferred state as well as in aggregate. Persistent structure means that more states are not necessarily the answer; lag dynamics, heavy-tailed innovations or exogenous variables may be missing.

Transition estimates need uncertainty. A state observed in only 20 periods cannot support a precise $p_{ii}=0.95$, however neat the implied 20-period duration looks. Report state occupancy, expected transition counts and confidence intervals. Boundary estimates near zero or one deserve particular suspicion.

For live monitoring, store the vintage of every macro input. Revised GDP makes a historical recession filter look better than one fed the releases available in real time. Market prices have their own timing problems: a regime probability calculated after the close cannot determine a position filled at that close.

Use expanding or rolling refits with a fixed schedule. Daily refitting can let state labels and parameter order jump; never map "state 1" to high volatility by index alone. Match states to the previous fit by parameter distance, then apply a documented fallback when the match is ambiguous.

---

Where Regime Switching Goes Wrong

In-sample colouring. Smoothed probabilities use the future and will make historical regimes look cleaner than the signals available at the time. Plot filtered one-step probabilities and evaluate predictive log likelihood on unseen data. A beautiful smoothed recession chart is not a trading backtest.

State proliferation. A third state almost always raises the maximised likelihood. AIC and BIC penalise complexity but do not settle the question, especially near unidentified boundaries. Require stable parameters, occupied states and out-of-sample improvement. A state containing three crash observations is often an outlier bucket.

Invalid likelihood-ratio tests. Under a one-regime null, transition probabilities and regime-specific differences are unidentified. The usual chi-squared likelihood-ratio reference distribution does not apply. Hansen (1996), Inference When a Nuisance Parameter Is Not Identified Under the Null Hypothesis, derives non-standard theory for this problem. Use a method designed for switching models or a carefully specified simulation test.

Local optima and label switching. Fit many defensible starting values, inspect convergence and compare log likelihoods. Sort states only after fitting. Standard errors near transition probabilities of zero or one can be unreliable.

Geometric durations. Markets may have duration dependence: a crisis that has lasted six months may not have the same exit probability as one that began yesterday. Hidden semi-Markov models or time-varying transition probabilities relax the memoryless assumption, at the cost of more parameters.

Distributional misspecification. Gaussian switching variance can imitate fat tails, while a Student-t single-regime model may explain them more parsimoniously. Compare both. Regime-switching GARCH is flexible but can be weakly identified and computationally delicate.

Economic instability. Means are especially hard to estimate at daily frequency. A "bull" state selected from noisy mean differences may disappear after costs. Volatility and correlation states often have clearer signal-to-noise, but still need an executable risk rule.

Engel and Hamilton (1990) applied switching trends to exchange rates in Long Swings in the Dollar. That paper's result was specific: their model rejected a random walk and forecast better in its setting, while uncovered interest parity was still rejected. Source-specific claims are more useful than saying regime models generally predict crises.

Use the time-series analysis guide for residual and stability checks, the Kalman filter guide for continuous latent states and our risk-management guide for turning probabilities into constrained exposure.

---

Frequently Asked Questions

What is a Hamilton regime-switching model?

It is a time-series model whose parameters depend on an unobserved Markov state, with probabilities inferred recursively from the observed data.

What is the difference between filtered and smoothed probabilities?

Filtered probabilities use observations through the current date and can support a live rule. Smoothed probabilities use the full sample, including future observations, and are for retrospective inference.

How long does a regime last?

With a constant transition probability, expected duration is $1/(1-p_{ii})$. Actual duration is geometric and memoryless.

Should a regime strategy use a 50% probability threshold?

Not automatically. A hard threshold discards uncertainty and can cause turnover. Select the mapping from probability to exposure out of sample and delay execution until the probability was observable.

How many states should be fitted?

Start with the smallest economically interpretable model. Compare predictive likelihood, parameter stability and state occupancy; do not rely on in-sample fit or ordinary chi-squared likelihood-ratio tests.

What should be cited?

Hamilton (1989), A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle, Econometrica 57(2), 357 to 384.