7.3 Handling Missing Data and Outliers

Data preparation is crucial for accurate analysis. Missing data can skew results, so understanding its types and causes is essential. Imputation techniques like mean/median/mode and multiple imputation help fill gaps, but it's important to consider the underlying mechanisms.

Outliers can significantly impact analysis too. Statistical methods like z-scores and IQR help identify them, while machine learning approaches offer more advanced detection. Treatment options include winsorization and trimming, but careful consideration is needed to avoid introducing bias or losing valuable information.

Missing Data Handling

Types and Causes of Missing Data

Top images from around the web for Types and Causes of Missing Data

• 

  Frontiers | Computing within-study covariances, data visualization, and missing data solutions ... View original

  Is this image relevant?

• 

  Frontiers | A Deep Learning Approach for Missing Data Imputation of Rating Scales Assessing ... View original

  Is this image relevant?

• 

  Frontiers | Comparison of Different LGM-Based Methods with MAR and MNAR Dropout Data View original

  Is this image relevant?

• 

  Frontiers | Computing within-study covariances, data visualization, and missing data solutions ... View original

  Is this image relevant?

• 

  Frontiers | A Deep Learning Approach for Missing Data Imputation of Rating Scales Assessing ... View original

  Is this image relevant?

1 of 3

Top images from around the web for Types and Causes of Missing Data

• 

  Frontiers | Computing within-study covariances, data visualization, and missing data solutions ... View original

  Is this image relevant?

• 

  Frontiers | A Deep Learning Approach for Missing Data Imputation of Rating Scales Assessing ... View original

  Is this image relevant?

• 

  Frontiers | Comparison of Different LGM-Based Methods with MAR and MNAR Dropout Data View original

  Is this image relevant?

• 

  Frontiers | Computing within-study covariances, data visualization, and missing data solutions ... View original

  Is this image relevant?

• 

  Frontiers | A Deep Learning Approach for Missing Data Imputation of Rating Scales Assessing ... View original

  Is this image relevant?

1 of 3

• Missing completely at random (MCAR): Data is missing independently of both observed and unobserved data (survey participant accidentally skips a question)
• Missing at random (MAR): Missing data depends on observed data but not on unobserved data (older participants less likely to report income)
• Missing not at random (MNAR): Missing data depends on unobserved data (participants with higher income less likely to report it)
• Causes of missing data include data entry errors, equipment malfunctions, and participant non-response

Imputation Techniques for Handling Missing Data

• Single imputation: Replaces each missing value with a single estimated value
  • Mean/median/mode imputation: Replaces missing values with the mean, median, or mode of the observed values for that variable
  • Regression imputation: Uses regression model to predict missing values based on other variables
• Multiple imputation: Creates multiple complete datasets by imputing missing values multiple times, analyzing each dataset separately, and combining results
  • Accounts for uncertainty in imputed values by introducing random variation in the imputation process
  • Produces unbiased estimates and standard errors, assuming data is MAR or MCAR

Considerations and Best Practices for Imputation

• Imputation assumes data is MAR or MCAR; if data is MNAR, imputation can introduce bias
• Multiple imputation generally preferred over single imputation, as it accounts for uncertainty in imputed values
• Include variables in imputation model that are predictive of missingness and of the variable being imputed
• Assess sensitivity of results to different imputation methods and assumptions about missing data mechanism

Outlier Detection Methods

Statistical Approaches to Outlier Detection

• Z-score method: Calculates z-scores (number of standard deviations from mean) for each observation; observations with z-scores above a threshold (e.g., 3) considered outliers
  • Assumes data follows normal distribution; sensitive to extreme values and small sample sizes
• Interquartile range (IQR) method: Identifies outliers as observations below $Q1 - 1.5 \times IQR$ or above $Q3 + 1.5 \times IQR$, where $Q1$ and $Q3$ are the first and third quartiles
  • More robust to non-normal distributions and extreme values than z-score method
• Mahalanobis distance: Measures distance of each observation from the center of the multivariate distribution, taking into account correlations between variables
  • Can detect multivariate outliers that may not be identified by univariate methods

Machine Learning Approaches to Anomaly Detection

• Unsupervised learning algorithms (e.g., isolation forest, local outlier factor) can identify observations that are distinct from the majority of the data
  • Do not require labeled data or assumptions about the distribution of the data
  • Can detect complex patterns and adapt to the structure of the data
• Supervised learning algorithms (e.g., support vector machines, neural networks) can be trained on labeled data to classify observations as outliers or non-outliers
  • Require labeled training data, which may be difficult or expensive to obtain
  • Can leverage information from multiple variables and capture complex relationships

Outlier Treatment Techniques

Winsorization: Capping Extreme Values

• Replaces extreme values with a specified percentile of the data (e.g., 5th and 95th percentiles)
  • Reduces the impact of outliers while retaining their information
  • Can be applied to one or both tails of the distribution
• Winsorization is a compromise between excluding outliers and retaining them unchanged
  • Mitigates the influence of outliers on statistical analyses and models
  • Preserves the overall distribution and sample size of the data

Data Trimming: Removing Extreme Observations

• Excludes observations below a lower threshold or above an upper threshold (e.g., below 1st percentile or above 99th percentile)
  • Eliminates the impact of outliers on analyses and models
  • Can be applied to one or both tails of the distribution
• Data trimming results in a loss of information and a reduction in sample size
  • May be appropriate when outliers are believed to be erroneous or irrelevant to the analysis
  • Should be used with caution, as it can introduce bias if the excluded observations are informative

Considerations for Outlier Treatment

• Choice of treatment method depends on the nature and cause of the outliers, as well as the goals of the analysis
  • Winsorization may be preferred when outliers are believed to be genuine but extreme observations
  • Data trimming may be preferred when outliers are believed to be erroneous or irrelevant
• Sensitivity analyses should be conducted to assess the impact of different treatment methods on the results
  • Report the chosen treatment method and provide justification for the choice
  • Consider presenting results with and without outlier treatment to ensure transparency