2.4 Model Diagnostics and Residual Analysis

Model diagnostics and residual analysis are crucial for validating regression models. They help ensure assumptions are met and identify potential issues that could affect results. These techniques involve examining residuals, checking for influential observations, and assessing model fit.

By using tools like residual plots, Q-Q plots, and influence measures, we can spot problems like non-linearity, heteroscedasticity, or outliers. This helps us refine our models and make more reliable predictions in regression analysis.

Residual Diagnostics

Assessing Model Assumptions

Top images from around the web for Assessing Model Assumptions

• 

  regression - Linear mixed effects models: what to do when the residual QQ-plot looks non-normal ... View original

  Is this image relevant?

• 

  regression - Heteroskedasticity - residual plot interpretation - Cross Validated View original

  Is this image relevant?

• 

  data visualization - Help interpreting Residuals vs Fitted Plots - Cross Validated View original

  Is this image relevant?

• 

  regression - Linear mixed effects models: what to do when the residual QQ-plot looks non-normal ... View original

  Is this image relevant?

• 

  regression - Heteroskedasticity - residual plot interpretation - Cross Validated View original

  Is this image relevant?

1 of 3

Top images from around the web for Assessing Model Assumptions

• 

  regression - Linear mixed effects models: what to do when the residual QQ-plot looks non-normal ... View original

  Is this image relevant?

• 

  regression - Heteroskedasticity - residual plot interpretation - Cross Validated View original

  Is this image relevant?

• 

  data visualization - Help interpreting Residuals vs Fitted Plots - Cross Validated View original

  Is this image relevant?

• 

  regression - Linear mixed effects models: what to do when the residual QQ-plot looks non-normal ... View original

  Is this image relevant?

• 

  regression - Heteroskedasticity - residual plot interpretation - Cross Validated View original

  Is this image relevant?

1 of 3

• Residual plots visualize the difference between observed and predicted values to assess model assumptions
  • Plot residuals against predicted values to check for patterns or trends
  • Residuals should be randomly scattered around zero with no discernible pattern
  • Non-random patterns (curves, funnels, etc.) indicate violation of linearity or homoscedasticity assumptions
• Q-Q plots (Quantile-Quantile plots) compare the distribution of residuals to a theoretical normal distribution
  • Plot observed residual quantiles against expected quantiles from a normal distribution
  • Points should fall close to a straight diagonal line if residuals are normally distributed
  • Deviations from the line indicate non-normality of residuals (heavy tails, skewness, etc.)
• Heteroscedasticity tests assess whether residual variance is constant across the range of predicted values
  • Breusch-Pagan test regresses squared residuals on predictor variables
    • Significant p-value indicates presence of heteroscedasticity
  • White's test includes squared terms and interactions of predictors in the regression
    • Robust to non-normality of residuals
• Durbin-Watson test checks for autocorrelation (correlation between residuals) in time series or ordered data
  • Calculates test statistic $d$ based on residuals: $d = \frac{\sum_{t=2}^T (e_t - e_{t-1})^2}{\sum_{t=1}^T e_t^2}$
  • $d$ ranges from 0 to 4, with values around 2 indicating no autocorrelation
  • Values significantly below 2 suggest positive autocorrelation, while values significantly above 2 suggest negative autocorrelation

Addressing Violations of Assumptions

• Transform variables (log, square root, etc.) to improve linearity or stabilize variance
• Use weighted least squares regression to account for non-constant variance
• Consider alternative models (generalized linear models, robust regression) for non-normal residuals
• Include lagged terms or differencing in time series models to address autocorrelation

Influential Observations

Measures of Influence

• Leverage measures the potential influence of an observation on the predicted values
  • High leverage points have unusual combinations of predictor values
  • Leverage for observation $i$ is the $i$-th diagonal element of the hat matrix $\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T$
  • Observations with leverage greater than $2p/n$ (where $p$ is the number of predictors and $n$ is the sample size) are considered high leverage points
• Cook's distance combines leverage and residual size to assess the overall influence of an observation on the regression coefficients
  • Measures the change in coefficients when an observation is omitted from the model
  • Cook's distance for observation $i$ is calculated as: $D_i = \frac{e_i^2}{ps^2} \cdot \frac{h_{ii}}{(1-h_{ii})^2}$
  • Observations with Cook's distance greater than $4/(n-p-1)$ are considered influential
• Outliers are observations with unusually large residuals (more than 2 or 3 standard deviations from zero)
  • Can be identified using standardized residuals or studentized residuals
  • Outliers may have high leverage and influence, but not always
• Influential points are observations that substantially change the regression results when included or excluded
  • Can be outliers, high leverage points, or both
  • Identified using Cook's distance, DFFITS, or DFBETAS measures

Handling Influential Observations

• Carefully examine influential observations for data entry errors, measurement issues, or other problems
• Consider removing influential observations if they are clearly erroneous or unrepresentative of the population
• Use robust regression methods (least absolute deviations, M-estimation, etc.) that are less sensitive to outliers
• Include indicator variables for influential observations to model their effects separately
• Report results with and without influential observations to assess their impact on conclusions