Statistical Prediction Unit 12 ReviewModel & Feature Selection Techniques

Key Concepts

• Model selection involves choosing the best model from a set of candidate models based on their performance on a given dataset
• Feature selection identifies the most relevant features (variables) in a dataset to improve model performance and reduce complexity
• Bias-variance tradeoff balances the model's ability to fit the training data (low bias) and generalize to new data (low variance)
  • High bias models underfit the data and have high training and test error
  • High variance models overfit the data and have low training error but high test error
• Regularization techniques add a penalty term to the loss function to prevent overfitting by controlling the model's complexity
• Cross-validation assesses the model's performance on unseen data by splitting the dataset into multiple subsets for training and validation
• Evaluation metrics measure the model's performance, such as accuracy, precision, recall, F1-score, and mean squared error (MSE)
• Hyperparameter tuning optimizes the model's performance by selecting the best combination of hyperparameters (learning rate, regularization strength)

Types of Models

• Linear regression models the relationship between input features and a continuous output variable using a linear equation ($y = mx + b$)
• Logistic regression predicts the probability of a binary outcome (0 or 1) using a sigmoid function to map the linear combination of input features to a probability
• Decision trees recursively split the data based on feature values to create a tree-like model for classification or regression
  • Random forests combine multiple decision trees to improve performance and reduce overfitting
  • Gradient boosting builds an ensemble of weak learners (decision trees) sequentially, with each tree correcting the errors of the previous ones
• Support vector machines (SVM) find the hyperplane that maximally separates classes in high-dimensional space
• Neural networks consist of interconnected nodes (neurons) organized in layers, learning complex patterns through backpropagation and weight adjustments
• K-nearest neighbors (KNN) classifies a data point based on the majority class of its k nearest neighbors in the feature space

Feature Selection Methods

• Filter methods rank features based on their statistical properties (correlation, mutual information) independently of the model
  • Pearson correlation measures the linear relationship between two continuous variables
  • Chi-squared test assesses the dependence between a categorical feature and the target variable
• Wrapper methods evaluate subsets of features using a specific model and search strategy (forward, backward, or recursive feature elimination)
  • Forward selection starts with an empty set and iteratively adds the most informative features
  • Backward elimination starts with all features and iteratively removes the least informative ones
• Embedded methods perform feature selection during the model training process, such as L1 (Lasso) and L2 (Ridge) regularization in linear models
• Domain knowledge can guide feature selection based on expert understanding of the problem and relevant variables
• Principal component analysis (PCA) reduces the dimensionality of the feature space by creating new uncorrelated features (principal components) that capture the most variance in the data

Model Evaluation Metrics

• Accuracy measures the proportion of correctly classified instances out of the total instances
• Precision (positive predictive value) calculates the proportion of true positive predictions among all positive predictions
• Recall (sensitivity) computes the proportion of true positive predictions among all actual positive instances
• F1-score is the harmonic mean of precision and recall, providing a balanced measure of a model's performance
• Confusion matrix summarizes the model's performance in a table, showing true positives, true negatives, false positives, and false negatives
• Area under the receiver operating characteristic curve (AUC-ROC) evaluates the model's ability to discriminate between classes at various threshold settings
• Mean squared error (MSE) and root mean squared error (RMSE) measure the average squared difference between the predicted and actual values in regression problems
• R-squared (coefficient of determination) indicates the proportion of variance in the target variable explained by the model

Overfitting and Underfitting

• Overfitting occurs when a model learns the noise in the training data, resulting in poor generalization to new data
  • Overfitted models have low bias (fit the training data well) but high variance (perform poorly on unseen data)
  • Symptoms of overfitting include high accuracy on training data but low accuracy on test data, and a large difference between training and test performance
• Underfitting happens when a model is too simple to capture the underlying patterns in the data, resulting in high bias and poor performance on both training and test data
  • Underfitted models have high bias (do not fit the training data well) and low variance (consistently perform poorly on new data)
  • Symptoms of underfitting include low accuracy on both training and test data, and a small difference between training and test performance
• Complexity of the model should be balanced to avoid overfitting and underfitting
  • Increasing model complexity (more features, deeper trees) can reduce bias but increase variance
  • Decreasing model complexity (fewer features, shallower trees) can reduce variance but increase bias

Cross-Validation Techniques

• K-fold cross-validation divides the data into K equally sized subsets (folds), using K-1 folds for training and the remaining fold for validation, repeating the process K times
  • Typically, K is set to 5 or 10, providing a balance between bias and variance in the performance estimate
  • The final performance is the average of the K validation scores
• Stratified K-fold cross-validation ensures that each fold has a similar distribution of the target variable, especially useful for imbalanced datasets
• Leave-one-out cross-validation (LOOCV) is a special case of K-fold cross-validation where K equals the number of instances, using each instance as a validation set once
• Repeated K-fold cross-validation performs K-fold cross-validation multiple times with different random splits, reducing the variance of the performance estimate
• Time series cross-validation accounts for the temporal structure of the data by using past data for training and future data for validation, avoiding information leakage

Regularization Approaches

• L1 regularization (Lasso) adds the absolute values of the coefficients to the loss function, encouraging sparse solutions and performing feature selection
• L2 regularization (Ridge) adds the squared values of the coefficients to the loss function, shrinking the coefficients towards zero and handling multicollinearity
• Elastic Net combines L1 and L2 regularization, balancing between feature selection and coefficient shrinkage
  • The mixing parameter (alpha) controls the balance between L1 and L2 regularization
  • Alpha = 0 corresponds to pure Ridge, alpha = 1 corresponds to pure Lasso
• Early stopping monitors the model's performance on a validation set during training and stops the training process when the performance starts to degrade, preventing overfitting
• Dropout randomly drops out (sets to zero) a fraction of the neurons during training in neural networks, reducing overfitting by preventing complex co-adaptations

Practical Applications

• Credit risk assessment uses logistic regression or decision trees to predict the probability of default based on a borrower's characteristics (credit score, income)
• Sentiment analysis applies text classification models (Naive Bayes, SVM) to determine the sentiment (positive, negative, neutral) of customer reviews or social media posts
• Recommender systems employ collaborative filtering (matrix factorization) or content-based filtering (KNN) to suggest products or services based on user preferences and item similarities
• Fraud detection utilizes anomaly detection techniques (isolation forest, autoencoders) to identify unusual patterns or transactions indicative of fraudulent activities
• Image classification uses convolutional neural networks (CNN) to classify images into predefined categories (objects, scenes) based on learned visual features
• Time series forecasting applies recurrent neural networks (RNN) or autoregressive models (ARIMA) to predict future values based on historical patterns and trends