Statistical Prediction Unit 14 ReviewEvaluating ML Models: Metrics & Methods

Key Concepts

• Machine learning model evaluation assesses how well a trained model performs on unseen data
• Performance metrics quantify the effectiveness of a model's predictions compared to the actual outcomes
• Confusion matrix provides a tabular summary of a model's classification performance, including true positives, true negatives, false positives, and false negatives
• Overfitting occurs when a model learns the noise in the training data, leading to poor generalization on new data
• Underfitting happens when a model is too simple to capture the underlying patterns in the data, resulting in high bias and low performance
• Cross-validation techniques (k-fold, stratified k-fold) help assess a model's performance by partitioning data into subsets for training and validation
• Receiver Operating Characteristic (ROC) curve plots the true positive rate against the false positive rate at various classification thresholds
• Area Under the ROC Curve (AUC) summarizes the ROC curve into a single value, with higher values indicating better classification performance
• Bias-variance tradeoff refers to the balance between a model's ability to fit the training data (bias) and its sensitivity to small fluctuations in the data (variance)

Performance Metrics

• Accuracy measures the proportion of correct predictions out of the total number of predictions made
  • Calculated as $(TP + TN) / (TP + TN + FP + FN)$
• Precision quantifies the proportion of true positive predictions among all positive predictions
  • Calculated as $TP / (TP + FP)$
  • Useful when the cost of false positives is high (spam email classification)
• Recall (sensitivity) measures the proportion of actual positive instances that are correctly identified by the model
  • Calculated as $TP / (TP + FN)$
  • Important when the cost of false negatives is high (cancer diagnosis)
• F1 score is the harmonic mean of precision and recall, providing a balanced measure of a model's performance
  • Calculated as $2 * (precision * recall) / (precision + recall)$
• Specificity measures the proportion of actual negative instances that are correctly identified by the model
  • Calculated as $TN / (TN + FP)$
• Log loss (cross-entropy loss) quantifies the dissimilarity between predicted probabilities and actual labels, penalizing confident misclassifications more heavily

Confusion Matrix Breakdown

• Confusion matrix is a table that summarizes the performance of a classification model by comparing predicted labels to actual labels
• True Positives (TP) represent the number of instances correctly classified as positive by the model
• True Negatives (TN) represent the number of instances correctly classified as negative by the model
• False Positives (FP) represent the number of instances incorrectly classified as positive by the model (Type I error)
• False Negatives (FN) represent the number of instances incorrectly classified as negative by the model (Type II error)
  • FN are particularly concerning in medical diagnosis, where missing a disease can have severe consequences
• The main diagonal of the confusion matrix (TP and TN) represents correct classifications, while the off-diagonal elements (FP and FN) represent misclassifications
• Confusion matrix allows for the calculation of various performance metrics (accuracy, precision, recall) and helps identify areas for model improvement

Overfitting vs. Underfitting

• Overfitting occurs when a model learns the noise in the training data, leading to poor generalization on unseen data
  • Overfitted models have high variance and low bias, capturing random fluctuations in the training data
  • Symptoms of overfitting include high training accuracy but low validation accuracy, and complex decision boundaries
• Underfitting happens when a model is too simple to capture the underlying patterns in the data, resulting in high bias and low performance
  • Underfitted models have low variance and high bias, failing to capture the true relationship between features and targets
  • Symptoms of underfitting include low training and validation accuracy, and oversimplified decision boundaries
• Regularization techniques (L1/L2 regularization) can help mitigate overfitting by adding a penalty term to the loss function, discouraging complex models
• Increasing model complexity (adding more features, layers, or neurons) can help address underfitting, allowing the model to capture more intricate patterns
• The goal is to find the right balance between model complexity and generalization performance, avoiding both overfitting and underfitting

Cross-Validation Techniques

• Cross-validation is a technique for assessing a model's performance by partitioning the data into subsets for training and validation
• K-fold cross-validation divides the data into K equally-sized folds, using K-1 folds for training and the remaining fold for validation
  • The process is repeated K times, with each fold serving as the validation set once
  • The model's performance is averaged across all K iterations to obtain a more robust estimate
• Stratified k-fold cross-validation ensures that each fold has a representative distribution of the target variable, particularly useful for imbalanced datasets
• Leave-one-out cross-validation (LOOCV) is a special case of k-fold cross-validation where K equals the number of instances in the dataset
  • LOOCV is computationally expensive and may lead to high variance in the performance estimates
• Repeated k-fold cross-validation involves performing k-fold cross-validation multiple times with different random partitions, further reducing the variability of the performance estimates
• Cross-validation helps assess a model's generalization performance, reduces overfitting risk, and aids in model selection and hyperparameter tuning

ROC and AUC Analysis

• Receiver Operating Characteristic (ROC) curve is a graphical representation of a binary classifier's performance at various classification thresholds
• ROC curve plots the true positive rate (sensitivity) against the false positive rate (1-specificity) as the classification threshold varies
  • True positive rate is the proportion of actual positive instances correctly classified by the model
  • False positive rate is the proportion of actual negative instances incorrectly classified as positive by the model
• Area Under the ROC Curve (AUC) summarizes the ROC curve into a single value, representing the probability that a randomly chosen positive instance will be ranked higher than a randomly chosen negative instance
  • AUC ranges from 0 to 1, with 0.5 indicating a random classifier and 1 indicating a perfect classifier
  • Higher AUC values indicate better classification performance across all possible thresholds
• ROC and AUC are particularly useful for evaluating models in imbalanced classification problems, where accuracy alone may be misleading
• Comparing ROC curves and AUC values of different models helps in model selection, choosing the model with the best trade-off between true positive rate and false positive rate

Bias-Variance Tradeoff

• Bias refers to the error introduced by approximating a real-world problem with a simplified model
  • High bias models (underfitted) have a limited ability to capture the true underlying patterns in the data
  • Symptoms of high bias include consistent underperformance on both training and validation data
• Variance refers to the model's sensitivity to small fluctuations in the training data
  • High variance models (overfitted) learn the noise in the training data, leading to poor generalization on unseen data
  • Symptoms of high variance include large differences between training and validation performance
• The bias-variance tradeoff is the balance between a model's ability to fit the training data (bias) and its sensitivity to small fluctuations in the data (variance)
  • Increasing model complexity reduces bias but increases variance, while decreasing complexity has the opposite effect
• The goal is to find the sweet spot that minimizes both bias and variance, achieving good generalization performance
  • Regularization techniques, cross-validation, and ensemble methods can help manage the bias-variance tradeoff
• Understanding the bias-variance tradeoff is crucial for selecting appropriate model complexity and optimizing performance

Advanced Evaluation Methods

• Stratified sampling ensures that the distribution of the target variable in the sample is representative of the population, reducing bias in performance estimates
• Bootstrapping involves repeatedly sampling the data with replacement to create multiple datasets for model training and evaluation
  • Bootstrapping helps estimate the variability and confidence intervals of performance metrics
• Permutation feature importance measures the importance of each feature by randomly shuffling its values and observing the impact on model performance
  • Features whose permutation leads to a significant drop in performance are considered more important
• Partial dependence plots (PDPs) visualize the marginal effect of a feature on the model's predictions, holding other features constant
  • PDPs help interpret the relationship between a feature and the target variable, and identify non-linear dependencies
• SHAP (SHapley Additive exPlanations) values quantify the contribution of each feature to the model's predictions for individual instances
  • SHAP values provide local interpretability, helping explain why a model made a specific prediction
• Learning curves plot the model's performance on training and validation sets as a function of the training set size
  • Learning curves help diagnose overfitting, underfitting, and determine if collecting more data would improve performance
• These advanced evaluation methods provide deeper insights into model performance, feature importance, interpretability, and guide model improvement efforts