7.3 Confidence and Prediction Intervals in Multiple Regression

Confidence and Prediction Intervals in Multiple Regression

Confidence and prediction intervals quantify the uncertainty in our regression estimates. Rather than relying on single point estimates, these intervals give us a range of plausible values for both model coefficients and future observations.

Understanding these intervals is essential for drawing meaningful conclusions from regression output. They let you assess whether a predictor's effect is practically meaningful and how much trust to place in a prediction for a new data point.

Confidence Intervals for Coefficients

Definition and Interpretation

A confidence interval for a regression coefficient gives you a range of values that is likely to contain the true population coefficient at a specified confidence level (typically 95%). Think of it as the set of plausible values for the true effect of a predictor on the response, given your data.

For example, suppose a 95% confidence interval for the coefficient of $X_1$ is (0.5, 1.2). This means you can be 95% confident that a one-unit increase in $X_1$ is associated with an increase in the response somewhere between 0.5 and 1.2 units, holding all other predictors constant.

Calculation and Properties

The interval is built from three ingredients: the point estimate $\hat{\beta}$, its standard error $SE(\hat{\beta})$, and a critical value from the t-distribution.

The formula:

$\hat{\beta} \pm t_{\alpha/2,\; n-p-1} \times SE(\hat{\beta})$

where:

• $n$ is the sample size
• $p$ is the number of predictors
• $\alpha$ is the significance level (e.g., 0.05 for a 95% interval)
• $n - p - 1$ gives the degrees of freedom (subtracting one for the intercept and one for each predictor)

A few properties to remember:

• Narrow interval → more precise estimate of the coefficient. Wide interval → greater uncertainty.
• If the interval does not contain zero, the coefficient is statistically significant at that confidence level. If it does contain zero, you cannot rule out the possibility that the predictor has no effect.

Prediction Intervals for New Observations

Definition and Interpretation

A prediction interval gives you a range of values likely to contain a single future response $Y$ for a given set of predictor values $X_1, X_2, \ldots, X_p$. This is different from estimating a coefficient; here you're forecasting an actual observation.

For example, a 95% prediction interval for a new observation with $X_1 = 10$ and $X_2 = 5$ might be (75, 95). That means you expect the actual response for this new case to fall between 75 and 95 with 95% confidence.

Calculation and Properties

The prediction interval uses the fitted value $\hat{Y}$, the standard error of prediction, and the same t-distribution critical value:

$\hat{Y} \pm t_{\alpha/2,\; n-p-1} \times SE(\text{pred})$

where:

$SE(\text{pred}) = \sqrt{MSE \times (1 + h)}$

• $MSE$ is the mean squared error of the regression (an estimate of $\sigma^2$)
• $h$ is the leverage of the new observation, which measures how far its predictor values are from the center of the existing data

The "1" inside $\sqrt{MSE \times (1 + h)}$ is what makes prediction intervals fundamentally wider than confidence intervals for the mean response. That "1" represents the irreducible variability of individual observations around the regression line. Even if you estimated the regression function perfectly, individual responses would still scatter around it.

The width of the prediction interval grows when:

• The confidence level increases
• The sample size is small
• The data are highly variable (large $MSE$)
• The new observation's predictor values are far from the center of the data (large $h$)

Confidence vs. Prediction Intervals

Key Differences

This distinction trips up a lot of students, so it's worth being precise:

• A confidence interval for a coefficient targets a population parameter (the true $\beta$). It reflects uncertainty about where the regression line sits.
• A prediction interval targets a single future observation. It reflects uncertainty about the regression line plus the natural scatter of individual data points around that line.

Because prediction intervals account for both sources of uncertainty, they are always wider than the corresponding confidence interval for the mean response at the same predictor values.

Applications

• Confidence intervals for coefficients help you assess significance and effect size. If the 95% CI for a coefficient includes zero, that predictor is not statistically significant at the 0.05 level.
• Prediction intervals are used when you need to forecast individual outcomes. For instance, a manufacturer might use a prediction interval to estimate the range of product quality scores for a new batch, given specific production settings. The wider interval honestly communicates the uncertainty inherent in predicting any single outcome.

Factors Affecting Interval Width

Data and Sample Characteristics

• Sample size: Larger samples produce narrower intervals because standard errors shrink as you gather more data.
• Variability of the data: Higher variance in $Y$ (larger $MSE$) directly inflates both confidence and prediction intervals.
• Distance from the data center: Prediction intervals widen for observations with high leverage. If you're predicting at predictor values far from the sample means, there's less nearby data to anchor the estimate, so uncertainty increases.

Model and Interval Specifications

• Confidence level: Moving from 95% to 99% confidence requires a larger $t$-critical value, which widens the interval. You're trading precision for greater certainty.
• Number of predictors: Adding more predictors ($p$) reduces the degrees of freedom $n - p - 1$, which inflates the $t$-critical value. This effect is most noticeable when $n$ is small relative to $p$.
• Collinearity: When predictors are highly correlated with each other, the standard errors of the affected coefficients become inflated. The model struggles to separate each predictor's individual contribution, resulting in wider confidence intervals for those coefficients. The prediction interval for $Y$ may be less affected if the collinear predictors collectively still explain the response well.