10.1 Loss functions

Definition of loss functions

A loss function measures how "bad" a particular estimate or decision is by assigning a numerical cost to the gap between your estimate and the true value. In Bayesian decision theory, loss functions turn vague notions of "good" and "bad" decisions into precise math, which then lets you pick the best action given your posterior beliefs.

Role in decision theory

Loss functions formalize what it costs you to be wrong. Once you define a loss function, you can:

• Calculate the expected loss of any candidate action, weighted by your posterior distribution over unknown parameters
• Compare different decision strategies on a common scale
• Select the action that minimizes expected loss, which is the Bayes-optimal decision

The key idea is that "optimal" has no meaning until you specify what you're trying to minimize. The loss function is that specification.

Relationship to utility functions

Loss and utility are two sides of the same coin. A utility function assigns higher values to outcomes you prefer, while a loss function assigns higher values to outcomes you want to avoid.

• Minimizing loss is equivalent to maximizing utility
• You can convert between them: $L = -U$ (or any affine transformation that flips the sign)
• In practice, Bayesian decision theory uses loss functions more often, but the underlying logic is identical

Types of loss functions

Squared error loss

$L(\theta, \hat{\theta}) = (\theta - \hat{\theta})^2$

This is the most common loss function in estimation. It squares the difference between the true parameter $\theta$ and your estimate $\hat{\theta}$, so large errors get penalized much more than small ones. A miss of 4 units costs 16, while a miss of 2 units costs only 4.

• The estimator that minimizes posterior expected squared error loss is the posterior mean
• Widely used in regression and standard estimation problems
• Downside: sensitivity to outliers, since squaring amplifies extreme errors

Absolute error loss

$L(\theta, \hat{\theta}) = |\theta - \hat{\theta}|$

This loss grows linearly with the size of the error rather than quadratically. That makes it more forgiving of large deviations.

• The estimator that minimizes posterior expected absolute error loss is the posterior median
• More robust to outliers than squared error loss
• Common in financial modeling and robust estimation, where occasional extreme values shouldn't dominate your estimate

0-1 loss function

$L(\theta, \hat{\theta}) = I(\theta \neq \hat{\theta})$

Here $I$ is the indicator function: you pay a cost of 1 if your estimate is wrong and 0 if it's exactly right. There's no notion of "how far off" you are.

• The estimator that minimizes posterior expected 0-1 loss is the posterior mode (the MAP estimate)
• Natural for classification problems, where predictions are discrete categories
• Not useful for continuous parameters in practice, since the probability of guessing $\theta$ exactly is zero; it's mainly applied to discrete parameter spaces or hypothesis selection

Properties of loss functions

Symmetry vs. asymmetry

A symmetric loss function penalizes overestimation and underestimation equally. Squared error and absolute error are both symmetric.

Asymmetric loss functions assign different costs depending on the direction of the error. This matters whenever one type of mistake is worse than the other:

• In risk management, underestimating a financial loss is typically far more costly than overestimating it
• In medical screening, missing a disease (false negative) can be more dangerous than a false alarm (false positive)

You can build asymmetric loss by using different weights on each side. For example, a weighted absolute error: $L(\theta, \hat{\theta}) = w_1 |\theta - \hat{\theta}|$ if $\hat{\theta} > \theta$ and $w_2 |\theta - \hat{\theta}|$ if $\hat{\theta} < \theta$, with $w_1 \neq w_2$.

Convexity and continuity

• A convex loss function has a single global minimum, which guarantees that optimization won't get stuck in local minima
• Continuity allows smooth optimization, and differentiability enables gradient-based methods
• Squared error, absolute error, and logistic loss are all convex. Absolute error is convex and continuous but not differentiable at zero (a minor practical issue)

Robustness to outliers

Some loss functions are designed to limit the influence of extreme observations:

• Absolute error loss is more robust than squared error because it doesn't square large deviations
• Huber loss is a hybrid: it behaves like squared error for small errors (smooth, differentiable near zero) and like absolute error for large errors (limiting outlier influence). You set a threshold $\delta$ that controls the transition point
• Tukey's biweight loss goes further by completely ignoring observations beyond a certain threshold, effectively giving zero weight to extreme outliers

Bayesian decision theory

Posterior expected loss

The central calculation in Bayesian decision theory is the posterior expected loss:

$\mathbb{E}[L(\theta, \hat{\theta}) | x] = \int L(\theta, \hat{\theta}) \, p(\theta | x) \, d\theta$

Here's what this does step by step:

1. Start with your posterior distribution $p(\theta | x)$, which encodes everything you know about $\theta$ after seeing data $x$
2. For a candidate estimate $\hat{\theta}$, evaluate the loss $L(\theta, \hat{\theta})$ at every possible value of $\theta$
3. Weight each loss by how plausible that $\theta$ value is (according to the posterior)
4. Integrate (sum up) to get the expected loss for that particular $\hat{\theta}$
5. Choose the $\hat{\theta}$ that makes this expected loss as small as possible

This is the Bayes estimator under your chosen loss function. Different loss functions yield different Bayes estimators (mean, median, or mode, as discussed above).

Bayes risk

The Bayes risk is the minimum achievable expected loss, averaged over both the data and the prior:

$r(\pi) = \inf_{\delta} R(\pi, \delta)$

where $\pi$ is the prior and $\delta$ is a decision rule. Think of it as the best-case performance you can hope for given your prior beliefs and loss function. It serves as a benchmark: if a decision rule achieves the Bayes risk, no other rule can do better (on average, under that prior).

Minimax decision rule

When you don't trust any particular prior, the minimax approach offers a worst-case guarantee:

$\delta^* = \arg\min_{\delta} \sup_{\theta} R(\theta, \delta)$

This rule picks the decision strategy that minimizes the maximum possible risk across all parameter values. It's conservative by design: you're optimizing for the worst-case scenario rather than the average case. Minimax rules are most useful when:

• Prior information is weak or unreliable
• The consequences of worst-case errors are severe
• You're in an adversarial setting

There's an elegant connection: the minimax rule often coincides with the Bayes rule under the "least favorable prior," the prior that makes the problem hardest.

Loss functions in estimation

Point estimation

For point estimation, the choice of loss function directly determines which summary of the posterior you should report:

Interval estimation

Loss functions also guide the construction of credible intervals. The Highest Posterior Density (HPD) interval is the shortest interval that contains a given probability (say, 95%) of the posterior mass. It minimizes expected interval length for a given coverage probability.

When using asymmetric loss functions, the resulting credible intervals can themselves be asymmetric around the point estimate, which is appropriate when overestimation and underestimation carry different costs.

Prediction

For predicting future observations $\tilde{x}$, you work with the posterior predictive distribution rather than the posterior over parameters. Predictive loss functions evaluate how well your entire predictive distribution matches reality:

• Log predictive density loss: $-\log p(\tilde{x} | x)$. Rewards sharp, well-calibrated predictions.
• Continuous Ranked Probability Score (CRPS): Compares the full predictive CDF to the observed value. More robust than log loss to outlying observations.

These predictive loss functions are central to Bayesian model comparison, since they measure out-of-sample performance while naturally accounting for uncertainty in both parameters and future data.

Loss functions in hypothesis testing

Type I vs. Type II errors

In hypothesis testing, loss functions formalize the costs of two kinds of mistakes:

• Type I error (false positive): rejecting a true null hypothesis
• Type II error (false negative): failing to reject a false null hypothesis

A loss function for testing assigns separate penalties to each error type. The Bayes-optimal test then balances these penalties against the posterior probabilities of each hypothesis. In frequentist testing, this trade-off shows up as the choice of significance level $\alpha$ (controlling Type I error) versus test power (controlling Type II error).

False discovery rate

When testing many hypotheses simultaneously (e.g., thousands of genes in a genomics study), the false discovery rate (FDR) becomes the relevant error measure. FDR is the expected proportion of false positives among all rejected hypotheses.

• The Benjamini-Hochberg procedure controls FDR using a linear step-up method on p-values
• The q-value approach provides a Bayesian-flavored FDR control, estimating the minimum FDR at which each hypothesis would be rejected
• FDR control is less conservative than family-wise error rate control (like Bonferroni), making it more practical for large-scale testing

Choosing appropriate loss functions

Context-dependent selection

There's no universally "correct" loss function. The right choice depends on what errors actually cost in your specific problem:

• Finance: Underestimating portfolio risk can lead to catastrophic losses, so asymmetric loss that penalizes underestimation more heavily is standard
• Medical diagnosis: Missing a cancer diagnosis (false negative) is typically far worse than ordering an unnecessary follow-up test (false positive), so the loss function should weight false negatives more
• Weather forecasting: Predicting no rain when it rains might be worse than predicting rain when it doesn't, depending on the downstream decision (e.g., flood preparation vs. carrying an umbrella)

The process: identify the real-world consequences of each type of error, then pick (or design) a loss function that reflects those consequences.

Sensitivity analysis

Because the choice of loss function affects your conclusions, it's good practice to check whether your results are robust:

• Run your analysis under several plausible loss functions
• If the optimal decision stays the same across different loss functions, you can be more confident in it
• If results change substantially, report the sensitivity and let the decision-maker choose which loss function best reflects their priorities

Limitations and considerations

Model misspecification

Loss functions operate within the assumed model. If the model itself is wrong (e.g., assuming normality when the data are heavy-tailed), then even the Bayes-optimal decision under your loss function may perform poorly. Robust loss functions like Huber loss can help, but they don't fully solve the problem. Model checking and validation remain essential.

Computational complexity

Computing posterior expected loss requires integrating over the posterior distribution, which can be expensive:

• For simple conjugate models, closed-form solutions exist
• For complex models, you'll typically use Monte Carlo integration: draw samples from the posterior and average the loss over those samples
• Variational inference offers a faster but approximate alternative
• In high-dimensional problems, the computational cost of evaluating certain loss functions can become a bottleneck, so there's often a practical trade-off between using an ideal loss function and one that's computationally tractable

Applications in machine learning

Loss functions for classification

Classification algorithms are trained by minimizing a loss function over the training data. Common choices:

• Cross-entropy (log) loss: The standard for probabilistic classifiers (logistic regression, neural networks). Penalizes confident wrong predictions heavily.
• Hinge loss: Used in support vector machines. Only penalizes predictions that are on the wrong side of the margin or within it.
• Exponential loss: The basis of AdaBoost. Gives exponentially increasing weight to misclassified examples.

For imbalanced classes (e.g., 95% negative, 5% positive), you can use weighted loss functions that assign higher cost to misclassifying the minority class.

Loss functions for regression

Regression loss functions optimize predictions of continuous outcomes:

• Mean squared error (MSE): The default. Corresponds to squared error loss averaged over data points.
• Mean absolute error (MAE): More robust to outliers. Corresponds to absolute error loss.
• Huber loss: Combines MSE's smoothness near zero with MAE's robustness to outliers.
• Quantile loss: Enables quantile regression, where you estimate specific percentiles of the response distribution rather than just the mean.

Advanced topics

Hierarchical loss functions

In problems with multi-level structure (e.g., students nested within schools, or tasks grouped by category), you can define loss functions at multiple levels. For instance, in hierarchical classification with a taxonomy (animal → mammal → dog), misclassifying a dog as a cat might incur less loss than misclassifying it as a plant, because the error is "closer" in the hierarchy.

Multi-task learning similarly uses a combination of shared and task-specific loss components to balance learning across related tasks.

Multi-objective loss functions

Many real problems involve competing objectives. For example, you might want a model that's both accurate and fair, or both precise and interpretable. Approaches include:

• Weighted sum: Combine individual losses as $L = w_1 L_1 + w_2 L_2$. Simple but requires choosing weights.
• Pareto optimization: Find the set of solutions where you can't improve one objective without worsening another.
• Constrained optimization: Minimize one loss subject to a constraint on another (e.g., minimize prediction error subject to a fairness constraint).

These techniques enable explicit trade-off analysis, forcing you to confront the tensions between different goals rather than ignoring them.