Influential Linear Regression Outliers

Why This Matters

In linear regression, not all data points are created equal—some observations wield disproportionate power over your model's coefficients, predictions, and overall fit. Understanding influential outliers is essential because a single problematic point can completely distort your regression line, leading to misleading conclusions and poor predictions. You're being tested on your ability to distinguish between points that are merely unusual and points that actually change your model's story.

The key concepts here involve leverage, influence, and residual behavior—three related but distinct ideas that exam questions love to conflate. A point can have high leverage without being influential, or it can be an outlier in $Y$ without affecting coefficients much. Mastering these diagnostics means understanding what each metric actually measures and when to use which tool. Don't just memorize formulas and thresholds—know what concept each diagnostic illustrates and how they work together to protect your model's integrity.

---

Leverage-Based Diagnostics

These metrics identify observations with unusual predictor values—points that sit far from the center of your $X$-space. High leverage means the observation has the potential to influence the regression, though it may not actually do so if its $Y$-value falls in line.

Leverage (Hat Values)

• Measures distance in predictor space—specifically, how far an observation's $X$-values are from the mean of all predictors
• High leverage threshold is typically $2(p+1)/n$ where $p$ is the number of predictors and $n$ is sample size
• Leverage alone doesn't guarantee influence—a point needs both high leverage AND a large residual to actually distort your model

Mahalanobis Distance

• Accounts for correlation structure—measures multivariate distance from the centroid while adjusting for how variables covary
• Superior to Euclidean distance when predictors are correlated, since it considers the shape of the data cloud
• Threshold based on chi-squared distribution with $p$ degrees of freedom, often using critical values at $\alpha = 0.01$

---

Influence Metrics

These diagnostics measure how much the regression results actually change when you remove a specific observation. Influence combines leverage with residual size—a point must be unusual in both $X$ and $Y$ to truly distort your model.

Cook's Distance

• Combines leverage and residual size into a single summary of overall influence on all fitted values simultaneously
• Threshold of $4/n$ is commonly used, though some texts suggest flagging values above 1
• Best single metric for overall influence—if you can only compute one diagnostic, this is the one to choose

DFFITS (Difference in Fits)

• Measures change in predicted value for observation $i$ when that observation is deleted from the model
• Threshold of $\pm 2\sqrt{(p+1)/n}$ identifies points that substantially shift their own fitted values
• Scaled version of Cook's Distance—they identify similar points but DFFITS focuses on the observation's own prediction

DFBETAS (Difference in Beta Estimates)

• Coefficient-specific influence—measures how much each $\beta_j$ changes when observation $i$ is removed
• Threshold of $\pm 2/\sqrt{n}$ flags observations that substantially shift individual coefficient estimates
• Produces a matrix of values—one DFBETA for each observation-coefficient combination, revealing which coefficients are vulnerable

---

Residual-Based Diagnostics

These metrics focus on how far observations fall from the regression line in the $Y$-direction. Large residuals indicate poor fit, but a point can be an outlier in $Y$ without having much influence if it lacks leverage.

Studentized Residuals

• Standardized using leave-one-out variance—divides residual by an estimate of standard error that excludes that observation
• Follows a t-distribution under normality assumptions, enabling formal hypothesis testing for outliers
• Threshold of $|t| > 2$ or $3$ commonly used, with Bonferroni correction for multiple testing when screening many points

Outlier Detection Methods (Z-score, IQR)

• Z-score method flags observations more than 2-3 standard deviations from the mean, assuming approximate normality
• IQR method uses $Q_1 - 1.5 \times IQR$ and $Q_3 + 1.5 \times IQR$ as fences, robust to non-normality
• Univariate approaches—useful for initial screening but don't account for regression structure or multivariate relationships

---

Visual Diagnostic Tools

Plots provide intuitive ways to identify problematic observations by combining multiple diagnostic dimensions. Visualization often reveals patterns that numerical summaries miss.

Influence Plots

• Combines leverage (x-axis) and studentized residuals (y-axis) with point size proportional to Cook's Distance
• Upper-right and lower-right corners contain the most dangerous points—high leverage plus large residuals
• Quick visual screening tool—immediately identifies observations warranting further investigation

Partial Regression Plots (Added Variable Plots)

• Shows relationship between $X_j$ and $Y$ after removing effects of other predictors—both variables are residualized first
• Slope equals the coefficient $\beta_j$ in the full multiple regression model
• Reveals masked outliers—points that appear normal marginally but are influential in the partial relationship

---

Quick Reference Table

---

Self-Check Questions

1. A point has high leverage but a small studentized residual. Is it influential? Explain why Cook's Distance would be low in this case.

2. Which two diagnostics would you use together to determine both whether a point is influential and which coefficients it affects most?

3. Compare and contrast Mahalanobis Distance and simple leverage—when would you prefer one over the other, and what assumption does Mahalanobis make that leverage doesn't?

4. An FRQ presents a regression with one observation having Cook's Distance of 0.8 and asks whether it should be removed. What threshold would you cite, and what additional diagnostics might inform your decision?

5. You're examining a partial regression plot and notice one point far from the regression line. Explain why this point might not have appeared unusual in a simple scatterplot of $Y$ versus that predictor.