Kalman Filter for Finance: Hedge Ratios (2026)

What Is a Kalman Filter in Finance?

On 1 March 1960, Rudolf E. Kalman published A New Approach to Linear Filtering and Prediction Problems in the Journal of Basic Engineering, volume 82, pages 35 to 45. The paper recast filtering around a state vector and a recursive covariance update.

In finance, a Kalman filter estimates an unobserved quantity, such as a changing hedge ratio or fair value, as each new observation arrives. It needs only the previous state estimate and covariance, so one update does not revisit the whole history. Under a correctly specified linear-Gaussian model, the filtered state is the conditional mean and minimum mean-square-error estimate.

The qualification does real work. A filter cannot establish that two prices are cointegrated, and a time-varying coefficient can absorb almost any divergence if its process noise is generous enough. The useful question is not "does Kalman beat rolling OLS?" but "does this state-space model produce stable, causal forecasts after costs?" Our pairs trading guide and cointegration guide cover the tests that come before execution.

---

The Kalman Recursion

Write a linear state-space model as

$$x_t=F_tx_{t-1}+w_t,\qquad w_t\sim\mathcal N(0,Q_t),$$ $$y_t=H_tx_t+v_t,\qquad v_t\sim\mathcal N(0,R_t),$$

with independent state and observation noise. The filter first predicts from information through $t-1$:

$$\widehat x_{t|t-1}=F_t\widehat x_{t-1|t-1}, \qquad P_{t|t-1}=F_tP_{t-1|t-1}F_t^{\mathsf T}+Q_t.$$

The new observation creates an innovation

$$e_t=y_t-H_t\widehat x_{t|t-1}, \qquad S_t=H_tP_{t|t-1}H_t^{\mathsf T}+R_t.$$

Then

$$K_t=P_{t|t-1}H_t^{\mathsf T}S_t^{-1}, \qquad \widehat x_{t|t}=\widehat x_{t|t-1}+K_te_t.$$

The innovation is the genuine one-step forecast error. A residual computed after updating the state with $y_t$ is smaller by construction and is a poor substitute for a trading surprise.

The compact covariance update $(I-K_tH_t)P_{t|t-1}$ is algebraically correct for the optimal gain, but finite-precision arithmetic can lose symmetry or positive semidefiniteness. The Joseph form is safer:

$$P_{t|t}=(I-K_tH_t)P_{t|t-1}(I-K_tH_t)^{\mathsf T} +K_tR_tK_t^{\mathsf T}.$$

The ratio of $Q_t$ to $R_t$ controls adaptation. More state noise lets coefficients move rapidly; more observation noise makes the filter trust its prior. Neither matrix is a generic smoothing knob. Their units depend on the state and observation definitions.

---

Dynamic Hedge Ratios

For two aligned log-price series $p_t$ and $q_t$, let the latent state be intercept and hedge coefficient:

$$x_t= \begin{bmatrix}\alpha_t\\\beta_t\end{bmatrix}, \qquad x_t=x_{t-1}+w_t,$$ $$p_t= \begin{bmatrix}1&q_t\end{bmatrix}x_t+v_t.$$

Here $F=I$ and $H_t=[1,q_t]$. The random-walk state says that alpha and beta may drift without a fixed long-run mean. That is convenient, but not free: if $Q$ is too large, beta chases each price move and the resulting posterior spread looks calm because the model has fitted it away.

A one-step example makes the update concrete. Suppose the prior state is $[0,1]^{\mathsf T}$, prior covariance is $\operatorname{diag}(0.01,0.04)$, $q_t=2$, $p_t=1.98$ and $R=0.01$. Ignoring extra process noise for this step,

$$e_t=1.98-[1,2][0,1]^{\mathsf T}=-0.02,$$ $$S_t=0.01+4(0.04)+0.01=0.18,\qquad K_t=[0.0556,0.4444]^{\mathsf T}.$$

The posterior becomes approximately $[-0.0011,0.9911]^{\mathsf T}$. The 0.02 miss changes beta by -0.0089 because the prior and observation variances limit how much one point can move the estimate.

For trading, standardise the prior innovation by $\sqrt{S_t}$, size with coefficients known at the decision time and execute no earlier than the next tradable price. The online pairs-trading treatment by Palomar also uses rolling signal normalisation rather than full-sample moments.

---

Stable Python without look-ahead

The function below aligns observations, preserves the one-step innovation and uses the Joseph covariance update. Separate process variances are supplied for alpha and beta because those states have different units.

import numpy as np
import pandas as pd


def kalman_pairs(p: pd.Series, q: pd.Series, obs_var: float,
                 alpha_var: float, beta_var: float) -> pd.DataFrame:
    data = pd.concat({"p": p, "q": q}, axis=1).dropna()
    state = np.array([0.0, 1.0])
    covariance = np.eye(2)
    process_cov = np.diag([alpha_var, beta_var])
    identity = np.eye(2)
    rows = []

    for time, row in data.iterrows():
        prior_state = state
        prior_cov = covariance + process_cov
        design = np.array([[1.0, float(row["q"])]])
        innovation = float(row["p"] - (design @ prior_state).item())
        innovation_var = float(
            (design @ prior_cov @ design.T).item() + obs_var
        )
        gain = prior_cov @ design.T / innovation_var

        state = prior_state + gain[:, 0] * innovation
        ikh = identity - gain @ design
        covariance = ikh @ prior_cov @ ikh.T + obs_var * (gain @ gain.T)
        rows.append((time, state[0], state[1],
                     innovation, innovation / np.sqrt(innovation_var)))

    return pd.DataFrame(
        rows, columns=["time", "alpha", "beta", "innovation", "z"]
    ).set_index("time")

Call this on a strictly historical stream. Estimate $Q$, $R$ and initial conditions on an earlier training sample, then hold them fixed or re-estimate them through a documented walk-forward procedure. Maximum likelihood often uses a smoother internally; running that estimation on the full backtest leaks future observations even though the final filter itself runs forwards.

The code stores filtered alpha and beta after seeing the current observation but stores the prior innovation separately. A signal based on today's close must trade on a later executable price. Shift positions by one bar in a close-to-close backtest.

Other finance models use the same recursion. A time-varying market beta replaces log prices with stock and market returns. A latent mean-reverting state uses $F=\phi$, $|\phi|<1$, rather than a random walk. In each case, derive $F,H,Q,R$ from the economic model before tuning them.

Initialisation and monitoring

An initial beta of one is a convenience, not prior knowledge. Estimate an OLS state on a training window or set a diffuse prior covariance so the data can move the state quickly. The first observations are then a burn-in period, not an honest trading history. Report where scoring begins and verify that changing the prior does not alter mature results.

Missing observations require an explicit update rule. If $p_t$ or $q_t$ is unavailable, perform the time prediction but skip the measurement update; dropping the timestamp entirely also skips the accumulation of process noise and changes the model clock. For asynchronous markets, stale prices are observations with misleadingly low movement, not harmless missing values.

Monitor the innovations rather than admiring a smooth beta. Under the fitted model, standardised innovations should be roughly mean zero, unit variance and serially uncorrelated. Plot rolling mean, variance, autocorrelation and tail quantiles. Persistent positive innovations suggest biased state dynamics; autocorrelation suggests omitted lags; too many large values suggest an understated $R$, heavy tails or a broken pair.

Forecast calibration is measurable. Of observations assigned a 95% Gaussian innovation interval, roughly 95% should land inside over a stable out-of-sample sample. Check by period and volatility regime, not only in aggregate. A filter can have a low average squared error while failing exactly during the episodes that create losses.

For hedge execution, retain the covariance of alpha and beta and propagate it into spread uncertainty. A point estimate of beta near one can hide a wide interval. Scale down or stop when state uncertainty rises beyond a pre-specified limit. This is more faithful to the model than trading every z-score as equally certain.

Finally, compare against hard baselines: static OLS, rolling OLS and an exponentially weighted regression using the same execution lag. The Kalman filter earns its complexity only if its causal innovations or hedge errors improve after turnover and parameter-selection costs.

---

Where Kalman Strategies Fail

A flexible hedge ratio manufactures a spread. Dynamic regression does not imply cointegration. Compare one-step innovations with posterior residuals, monitor beta drift and test whether the out-of-sample innovation process has the assumed structure. If beta doubles merely to hide a broken relation, stop trading.

Smoothed states leak. Filtering conditions on observations through $t$. Smoothing conditions on the full sample through $T$. Smoothed paths are excellent for historical diagnosis and EM estimation on a training set, but cannot generate historical live signals. This distinction is treated formally by Durbin and Koopman's Time Series Analysis by State Space Methods.

Scaling changes the model. Running the same numerical $Q$ and $R$ on dollar prices, log prices and rebased prices gives different behaviour. Centre or scale predictors using training-only transformations, document units and inspect condition numbers.

Gaussian tails are too polite. With non-Gaussian noise, the linear filter is still a linear estimator under suitable second moments, but Gaussian likelihood intervals and optimality claims no longer automatically hold. Outliers can jerk the state. Student-t observation models, innovation clipping specified ex ante or particle filters may be more suitable, but each changes the statistical model.

Parameter likelihoods are flat. Different $Q/R$ combinations can fit a short series similarly while producing very different trading turnover. Use multiple starting values, profile the likelihood and assess economic sensitivity, not just optimiser convergence.

Costs erase responsiveness. A high process variance creates rapidly changing hedge notionals. Include both spread trades and hedge rebalancing in turnover. Add bid-ask spread, borrow fees and execution delay before selecting $Q/R$.

The filter should also pass innovation diagnostics: mean near zero, no material residual autocorrelation and forecast variance consistent with realised errors. Our time-series analysis guide covers those checks, while the statistical arbitrage guide addresses portfolio-level exposure.

---

Frequently Asked Questions

What does a Kalman filter estimate?

It recursively estimates a latent state and its covariance from a linear transition model and noisy observations.

Why use it for pairs trading?

It can update an intercept and hedge ratio without an arbitrary rolling-window cut-off. It does not establish cointegration or profitability; those need separate evidence.

Which residual should drive a signal?

Use the one-step innovation $e_t=y_t-H_t\widehat x_{t|t-1}$, usually scaled by $\sqrt{S_t}$. The posterior residual is mechanically reduced by the same observation.

Does the filter itself use future data?

No. A forward filter at $t$ uses observations through $t$. A smoother uses later data, and full-sample parameter estimation can therefore introduce look-ahead into a backtest.

How should Q and R be selected?

Estimate them on training data by likelihood or a Bayesian method, check identification and run sensitivity tests. Re-estimate only with information available at that point in a walk-forward test.

What is the canonical paper?

Kalman (1960), A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering 82, 35 to 45.