7.6 Cramer-Rao lower bound (CRLB)

The Cramer-Rao Lower Bound (CRLB) sets the minimum variance that any unbiased estimator can achieve for a given parameter estimation problem. In signal processing, it serves as the fundamental benchmark: if your estimator's variance is close to the CRLB, you know you're extracting nearly all the information the data has to offer.

The bound is derived from the Fisher information, which quantifies how much information observed data carries about unknown parameters. This guide covers the CRLB's definition, calculation, attainability conditions, extension to vector parameters, and practical considerations.

Definition of CRLB

The Cramer-Rao Lower Bound provides a hard floor on the variance of any unbiased estimator. No matter how clever your estimation algorithm is, it cannot produce a variance below this bound. The result was independently derived by Harald Cramér and C.R. Rao in the 1940s and remains one of the most widely used tools in estimation theory.

Relationship to parameter estimation

Parameter estimation is the task of inferring unknown quantities from observed data. This shows up everywhere in signal processing: system identification, spectral analysis, DOA estimation, image reconstruction, and more.

The CRLB tells you the best-case performance for any unbiased estimator applied to a given problem. This serves two purposes:

• Evaluating estimators: If your estimator's variance is near the CRLB, there's little room for improvement. If there's a large gap, a better algorithm may exist.
• Guiding system design: Before building a system, you can compute the CRLB to determine whether the desired estimation accuracy is even theoretically achievable given the noise and signal conditions.

Derivation from likelihood function

The CRLB emerges from the likelihood function $f(\mathbf{x};\theta)$, which gives the probability of observing data $\mathbf{x}$ given parameter $\theta$. The derivation proceeds in three steps:

1. Form the log-likelihood: Take $\log f(\mathbf{x};\theta)$.
2. Compute the Fisher information: This measures the expected curvature of the log-likelihood with respect to $\theta$. Formally: $I(\theta) = -\mathbb{E}\left[\frac{\partial^2 \log f(\mathbf{x};\theta)}{\partial \theta^2}\right]$ A sharply peaked log-likelihood (high curvature) means the data is very informative about $\theta$, yielding large Fisher information.
3. Invert the Fisher information: The CRLB is $1/I(\theta)$. High information means low minimum variance, and vice versa.

The key regularity conditions required are that the log-likelihood is twice differentiable with respect to $\theta$ and that differentiation and integration (expectation) can be exchanged.

Properties of CRLB

Lower bound on variance

For any unbiased estimator $\hat{\theta}$ of $\theta$:

$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$

This holds regardless of the estimation method. The bound is a property of the statistical model itself, not of any particular estimator.

Dependence on Fisher information

The CRLB is inversely proportional to the Fisher information. Factors that increase Fisher information (and thus tighten the bound) include:

• Higher SNR: Cleaner data carries more information about the parameter.
• More observations: Each independent observation contributes additional Fisher information.
• Sharper likelihood: A likelihood function that changes rapidly with $\theta$ is more informative.

The Fisher information depends on both the statistical model and the true parameter value, so the CRLB can vary across the parameter space.

Asymptotic behavior for large samples

For $N$ independent, identically distributed observations, the Fisher information typically scales linearly with $N$, so:

$\text{CRLB}(\theta) \propto \frac{1}{N}$

This means the minimum achievable variance decreases as $1/N$. Under regularity conditions, the CRLB converges to zero as $N \to \infty$, implying that arbitrarily precise estimation is possible with enough data. The rate of decrease depends on the specific problem, but the $1/N$ scaling is the most common case.

Calculation of CRLB

General formula

For a scalar parameter $\theta$ and observed data $\mathbf{x}$:

$\text{CRLB}(\theta) = \frac{1}{I(\theta)}$

where the Fisher information is:

$I(\theta) = -\mathbb{E}\left[\frac{\partial^2 \log f(\mathbf{x};\theta)}{\partial \theta^2}\right]$

An equivalent form uses the score function $s(\theta) = \frac{\partial \log f(\mathbf{x};\theta)}{\partial \theta}$:

$I(\theta) = \mathbb{E}\left[s(\theta)^2\right]$

These two forms are equal under the regularity conditions. The second form is sometimes easier to compute.

Simplifications for common distributions

For standard distributions, the CRLB has clean closed-form expressions:

• Gaussian with unknown mean (known variance $\sigma^2$, $N$ observations): $\text{CRLB}(\mu) = \frac{\sigma^2}{N}$ The sample mean achieves this bound exactly, so it is an efficient estimator.
• Gaussian with unknown variance (known mean, $N$ observations): $\text{CRLB}(\sigma^2) = \frac{2\sigma^4}{N}$
• Poisson with parameter $\lambda$ ($N$ observations): $\text{CRLB}(\lambda) = \frac{\lambda}{N}$
• Bernoulli with parameter $p$ ($N$ observations): $\text{CRLB}(p) = \frac{p(1-p)}{N}$

Examples in parameter estimation problems

Frequency estimation of a sinusoid in AWGN: Consider estimating the frequency $\omega_0$ of a sinusoidal signal $x[n] = A\sin(\omega_0 n + \phi) + w[n]$ observed over $N$ samples, where $w[n]$ is white Gaussian noise with variance $\sigma^2$. The CRLB for frequency estimation is approximately:

$\text{CRLB}(\omega_0) \approx \frac{12}{A^2 N^3 / \sigma^2}$

Notice the $N^3$ in the denominator: frequency estimation accuracy improves much faster than $1/N$ because the time aperture provides leverage. This bound is commonly used to benchmark algorithms like MUSIC, ESPRIT, and periodogram-based methods.

Linear regression: For the model $\mathbf{y} = \mathbf{H}\boldsymbol{\theta} + \mathbf{w}$ with $\mathbf{w} \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I})$, the CRLB for the parameter vector is $\sigma^2(\mathbf{H}^T\mathbf{H})^{-1}$. The ordinary least squares estimator achieves this bound exactly.

Attainability of CRLB

The CRLB is a lower bound, but not every estimation problem has an estimator that actually reaches it. Understanding when the bound is achievable is critical.

Conditions for equality

An unbiased estimator $\hat{\theta}$ achieves the CRLB if and only if the score function can be written as:

$\frac{\partial \log f(\mathbf{x};\theta)}{\partial \theta} = I(\theta)(\hat{\theta}(\mathbf{x}) - \theta)$

This means the score function must be a linear function of the estimator. This is a strong requirement. It holds for exponential family distributions (Gaussian, Poisson, Bernoulli, exponential, etc.) but not for all models.

Efficient estimators

An estimator that achieves the CRLB is called efficient. Efficient estimators have the smallest variance among all unbiased estimators.

• For exponential family distributions with sufficient statistics, efficient estimators typically exist and can be found via the sufficient statistic.
• For many other models, no efficient estimator exists at finite sample sizes. The CRLB still serves as a benchmark, but you should expect a gap between your estimator's variance and the bound.

Maximum likelihood estimator (MLE)

The MLE maximizes $f(\mathbf{x};\theta)$ with respect to $\theta$. Its key properties relative to the CRLB:

• Asymptotically efficient: As $N \to \infty$, the MLE's variance converges to the CRLB under regularity conditions. Specifically, $\hat{\theta}_{\text{MLE}} \sim \mathcal{N}(\theta, I(\theta)^{-1})$ asymptotically.
• Not necessarily unbiased at finite $N$: The MLE can be biased for small sample sizes, so comparing its MSE directly to the CRLB requires caution.
• Not necessarily efficient at finite $N$: The MLE may have variance exceeding the CRLB when data is limited.

Despite these caveats, the MLE is often the go-to estimator because of its strong asymptotic properties and its general applicability.

CRLB for multiple parameters

Vector parameter case

When estimating a vector $\boldsymbol{\theta} = [\theta_1, \theta_2, \ldots, \theta_K]^T$, the CRLB generalizes to a matrix inequality. For any unbiased estimator $\hat{\boldsymbol{\theta}}$:

$\text{Cov}(\hat{\boldsymbol{\theta}}) \geq \mathbf{I}(\boldsymbol{\theta})^{-1}$

where the inequality means that $\text{Cov}(\hat{\boldsymbol{\theta}}) - \mathbf{I}(\boldsymbol{\theta})^{-1}$ is positive semidefinite.

Fisher information matrix

The Fisher information matrix (FIM) is a $K \times K$ matrix with elements:

$[\mathbf{I}(\boldsymbol{\theta})]_{ij} = -\mathbb{E}\left[\frac{\partial^2 \log f(\mathbf{x};\boldsymbol{\theta})}{\partial \theta_i \partial \theta_j}\right]$

The diagonal entries $[\mathbf{I}]_{ii}$ measure the information about each individual parameter. The off-diagonal entries $[\mathbf{I}]_{ij}$ capture the coupling between parameters: nonzero off-diagonal terms mean that estimating one parameter is entangled with estimating another.

Inverse of Fisher information matrix

The CRLB matrix is:

$\mathbf{CRLB}(\boldsymbol{\theta}) = \mathbf{I}(\boldsymbol{\theta})^{-1}$

How to interpret it:

• Diagonal elements $[\mathbf{I}^{-1}]_{ii}$: Lower bound on $\text{Var}(\hat{\theta}_i)$. Note that this is generally larger than $1/[\mathbf{I}]_{ii}$ because of parameter coupling. The difference reflects the cost of jointly estimating correlated parameters.
• Off-diagonal elements $[\mathbf{I}^{-1}]_{ij}$: Lower bound on the covariance between $\hat{\theta}_i$ and $\hat{\theta}_j$.

If you only care about one parameter $\theta_i$, the relevant bound is the $i$-th diagonal element of $\mathbf{I}^{-1}$, not simply $1/[\mathbf{I}]_{ii}$.

Applications of CRLB

Performance benchmarking of estimators

The most common use of the CRLB is comparing an estimator's variance (or MSE) against the theoretical minimum. In practice, you would:

1. Derive or numerically compute the CRLB for your problem.
2. Run your estimator (analytically or via Monte Carlo simulation) to obtain its variance.
3. Plot both as a function of SNR or sample size.

If the estimator's variance curve tracks the CRLB closely, the estimator is near-optimal. A persistent gap suggests room for improvement or indicates that the estimator is biased.

System design and optimization

Before building a system, the CRLB can answer design questions such as:

• Minimum SNR: What SNR is needed to achieve a target estimation accuracy?
• Required number of samples: How many observations are necessary?
• Sensor placement: In array processing, the FIM depends on sensor geometry. Optimizing sensor positions to maximize the FIM (or minimize the CRLB) leads to better estimation performance.
• Waveform design: In radar and communications, the transmitted waveform affects the FIM. The CRLB can guide waveform optimization for parameter estimation tasks like range and velocity estimation.

Fundamental limits in signal processing

The CRLB reveals inherent trade-offs. For example, in spectral estimation, the CRLB shows how frequency resolution depends on observation time, SNR, and the number of sinusoidal components. These limits hold regardless of the algorithm, so they define what is physically achievable.

Relationship to other bounds

The CRLB is the most widely used variance bound, but it is not always the tightest. Several alternative bounds exist.

Bhattacharyya bound

The Bhattacharyya bound uses higher-order derivatives of the log-likelihood (not just the second derivative). It is generally tighter than the CRLB, especially in low-SNR regimes or when the likelihood function has significant higher-order structure. The trade-off is increased computational complexity.

Hammersley-Chapman-Robbins bound

The Hammersley-Chapman-Robbins (HCR) bound does not require the likelihood to be differentiable, making it applicable to a broader class of problems. It is also tighter than the CRLB in general. The HCR bound applies to both biased and unbiased estimators, but it involves an optimization over a test point, which can make computation more involved.

Comparison of tightness and applicability

The CRLB's popularity comes from its simplicity. For most well-behaved problems at moderate-to-high SNR, the CRLB is a good approximation of the actual performance limit. At low SNR or near threshold effects, tighter bounds become necessary.

Extensions and generalizations

Bayesian CRLB

In the Bayesian framework, the parameter $\theta$ is treated as a random variable with a known prior $p(\theta)$. The Bayesian CRLB (also called the Van Trees inequality or posterior CRLB) bounds the MSE of any estimator:

$\mathbb{E}[(\hat{\theta} - \theta)^2] \geq \frac{1}{I_D(\theta) + I_P(\theta)}$

where $I_D$ is the data Fisher information and $I_P$ is the prior Fisher information (from the prior distribution). The prior information tightens the bound, reflecting the benefit of incorporating prior knowledge.

Constrained CRLB

When parameters are known to satisfy constraints (e.g., non-negativity, unit norm, or linear relationships), the unconstrained CRLB is overly pessimistic. The constrained CRLB accounts for these constraints and provides a tighter bound.

For equality constraints $\mathbf{g}(\boldsymbol{\theta}) = \mathbf{0}$, the constrained CRLB is computed by projecting the FIM inverse onto the constraint manifold. This is particularly relevant in array processing where, for example, source directions may be known to lie on a plane.

CRLB for biased estimators

If an estimator has bias $b(\theta) = \mathbb{E}[\hat{\theta}] - \theta$, the standard CRLB does not directly apply. The modified bound becomes:

$\text{Var}(\hat{\theta}) \geq \frac{[1 + b'(\theta)]^2}{I(\theta)}$

where $b'(\theta) = \frac{\partial b(\theta)}{\partial \theta}$. The MSE (which includes both variance and squared bias) is then bounded by:

$\text{MSE}(\hat{\theta}) \geq \frac{[1 + b'(\theta)]^2}{I(\theta)} + b(\theta)^2$

This extension is useful for evaluating regularized or shrinkage estimators, which intentionally introduce bias to reduce variance.

Practical considerations

Numerical computation of CRLB

When closed-form expressions are unavailable, the CRLB must be computed numerically. Common approaches:

1. Finite differences: Approximate the second derivative of the log-likelihood using $\frac{\partial^2}{\partial \theta^2} \log f \approx \frac{\log f(\mathbf{x};\theta+h) - 2\log f(\mathbf{x};\theta) + \log f(\mathbf{x};\theta-h)}{h^2}$ then average over many data realizations.

2. Monte Carlo: Generate many data realizations from the model, compute the score function for each, and estimate the Fisher information as the sample variance of the score.

3. Automatic differentiation: Modern computational tools can compute exact derivatives of complex likelihood functions, which is especially useful for high-dimensional FIM computation.

For vector parameters, the FIM is a matrix that must be inverted. Ill-conditioning of the FIM (which occurs when parameters are weakly identifiable) can cause numerical instability. Regularization or careful parameterization may be needed.

Finite sample performance vs asymptotic bounds

The CRLB is exact as a bound for all sample sizes, but the MLE only achieves it asymptotically. In practice:

• At high SNR or large $N$, estimators like the MLE typically operate near the CRLB.
• At low SNR or small $N$, threshold effects can cause estimator variance to dramatically exceed the CRLB. This is common in frequency estimation, where ambiguity-driven outliers cause the MSE to spike well above the bound.
• The region where estimator performance transitions from near-CRLB to far above it is called the threshold region. Identifying this region is important for practical system design.

Always validate CRLB predictions with Monte Carlo simulations at the operating conditions of interest.

Robustness to model misspecification

The CRLB assumes the statistical model is correct. If the true data distribution differs from the assumed model:

• The Fisher information computed under the wrong model (the misspecified Fisher information) may not correspond to the actual achievable variance.
• Estimators designed for the assumed model may be biased, and the CRLB may understate the true estimation error.
• The sandwich covariance (or Huber-White estimator) provides a more realistic variance estimate under misspecification for the MLE.

To guard against model mismatch, consider robust estimation techniques, goodness-of-fit testing, and sensitivity analysis. Computing the CRLB under multiple plausible models can give a range of expected performance rather than a single optimistic number.